Related
I have tried to reproduce Hugh Woods experiment (https://medium.com/asos-techblog/sending-messages-from-azure-data-factory-to-service-bus-86d4be6dd357) in my azure environment to sent messages from Azure DataFactory to Azure Service Bus.
I have followed the instructions, assigning the "Azure Service Bus Data Sender" role to my data factory’s managed identity.
But when I tried to sent a message to Service Bus I got this error:
Error code : 2108
Failure type : User configuration issue
Details : Invoking Web Activity failed with HttpStatusCode -
'401 : Unauthorized', message - ''
Source : Pipeline Service Bus REST API
What am I doing wrong in this case? I have this configuration in my pipeline:
{
"name": "Service Bus REST API",
"properties": {
"activities": [
{
"name": "Service Bus REST API",
"description": "Teste",
"type": "WebActivity",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"url": "https://sb-namespace-dv.servicebus.windows.net/sbt-azure-adf-ntt-data-test/messages",
"method": "POST",
"headers": {
"CustomHeader-Version": "\"1.0\"",
"Content-Type": "application/json",
"BrokerProperties": {
"value": "#concat('{\"CorrelationId\": \"',pipeline().RunId,'\"}')",
"type": "Expression"
}
},
"body": {
"value": "{\"message\":{\"value\":\"#variables('OutputDetails')\",\"type\":\"Expression\"}}",
"type": "Expression"
},
"httpRequestTimeout": "00:10:00",
"authentication": {
"type": "MSI",
"resource": "https://servicebus.azure.net"
}
}
}
],
"folder": {
"name": "999_Others/9910_DevTest/TesteServiceBusADF"
},
"annotations": []
}
}
I have followed all steps of Hugh Woods article. I have expecting to get the same results of his experiment (below), but I did not have success.
{
message : "Snapshot Avaliable"
}
The message you are getting is HTTP 401 Unauthorized. So the call to write data to the service bus is being refused.
There are 2 possibilities:
The authentication is not set up correctly
You tested it before the role assignments had propagated (can take 5 mins)
See: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-managed-service-identity
I'm using postman to make rest requests to the azure API to run a pipeline that is in synapse, in terms of permissions and the token I already get them and it works, the problem is that the pipeline receives 3 parameters but I don't know how to pass them, so I have this request, example:
https://hvdhgsad.dev.azuresynapse.net/pipelines/pipeName/createRun?api-version=2020-12-01
and the parameters I added them in the body:
{
"parameters": {
"p_dir": {
"type": "string",
"defaultValue": "val1"
},
"container": {
"type": "string",
"defaultValue": "val"
},
"p_folder": {
"type": "string",
"defaultValue": "val3"
}
}
}
but when I validate the run that was launched with the request I get this.
{
"id": "xxxxxxxxxxxxxxx",
"runId": "xxxxxxxxxxxxxxxxxxxxx",
"debugRunId": null,
"runGroupId": "xxxxxxxxxxxxxxxxxxxx",
"pipelineName": "xxxxxxxxxxxxxxxxx",
"parameters": {
"p_dir": "",
"p_folder": "",
"container": ""
},
"invokedBy": {
"id": "xxxxxxxxxxxxxxxxx",
"name": "Manual",
"invokedByType": "Manual"
},
"runStart": "2021-07-20T05:56:04.2468861Z",
"runEnd": "2021-07-20T05:59:10.1734654Z",
"durationInMs": 185926,
"status": "Failed",
"message": "Operation on target Data flow1 failed: {\"StatusCode\":\"DF-Executor-SourceInvalidPayload\",\"Message\":\"Job failed due to reason: Data preview, debug, and pipeline data flow execution failed because container does not exist\",\"Details\":\"\"}",
"lastUpdated": "2021-07-20T05:59:10.1734654Z",
"annotations": [],
"runDimension": {},
"isLatest": true
}
the params are empty, so I don't know what's wrong or missing.
what is the correct way to pass them???
ref: https://learn.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/create-pipeline-run#examples
Just created an account to answer this as i've had the same issue.
I resolved this by just having the name of the variable and its subsequent value in the body JSON.
e.g.
{"variable": "value", "variable": "value"}
Found this by following the documentation you had posted, under request body it passes the name of the variable and the value directly into the JSON body.
{
"OutputBlobNameList": [
"exampleoutput.csv"
]
}
This particular example is a list/array so it confused me by adding the brackets [] if you are passing string parameters this is unneeded.
I followed the tutorial found at
https://learn.microsoft.com/en-us/azure/devops/extend/develop/service-endpoints?view=azure-devops
The api call is right because if I don't authenticate it shows the correct path in the error message. Entering the url that it shows in the message in a browser gives the correct json.
The picklist object will act like it is trying to load but will end up showing no results found.
Can you use a personal company api to load a picklist using server endpoint?
How do you debug this. All I see in the network tab on debugging tools is the 200 from azure making the post.
This is the JSON
{
"meta": {
"record_count": 5,
"source": "mongodb",
"searchParams": {
"versions.start_date": {
"$gte": "2019-08-30T00:11:55.329Z"
}
},
"paging": {
"num": 10,
"start": 0
}
},
"screenshots": [
{
"screenshot_test_id": 3946619,
"url": "https://www.google.com",
"test_name": null,
"created_date": "2019-10-27T21:26:32.0 ```
This is the datasources from the vss.extension
"dataSources": [
{
"name": "CBTAPIReturn",
"endpointUrl": "{{endpoint.url}}",
"resultSelector": "jsonpath:$[*].screenshots"
}
]
inputs and bindings from the task
"inputs": [
{
"name": "CBTService",
"type": "connectedService:CBTServiceEndpoint",
"label": "CBT service/server end point",
"defaultValue": "",
"required": true,
"helpMarkDown": "Select the CBT end point to use. If needed,selecton 'manage', and add a new service endpoint of type 'CBT server connection'"
},
{
"name": "CBT API Response",
"type": "pickList",
"label": "CBT API Response",
"required": true,
"helpMarkDown": "Select the name combination that you want to use.",
"properties": {
"EditableOptions": "True"
}
}
],
"dataSourceBindings": [
{
"target": "CBT API Response",
"endpointId": "$(CBTService)",
"dataSourceName": "CBTAPIReturn"
```
I would like to create azure container with resource group and cluster of specific Orchestrator Kubernetes cluster.
I know by using CLI its possible but I want to do this using Azure Rest API's for Container service from link given here
learn.microsoft.com/en-us/rest/api/container-service/containerservices/createorupdate
Registered my app in AAD and gave required permissions.
Got access token and making request to below api as per link
PUT management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.ContainerService/containerServices/{containerServiceName}?api-version=2017-01-31
but I am getting error as
{
"error": {
"code": "InvalidResource"
"message": "The resource definition is invalid."
}
}
I understand we get this error when there is nothing in request body.
So my real question here is if I want to create container service with resource group and cluster using API request.
Request Body
{
"id": "myCluster",
"name": "myCluster",
"type": "Microsoft.ContainerService/containerServices",
"location": "southindia",
"tags": {
"tag": "test"
},
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes"
},
"servicePrincipalProfile": {
"clientId": "<clientid>,
"secret": "<secret>"
},
"masterProfile": {
"count": 1,
"dnsPrefix": "testabc"
},
"agentPoolProfiles": {
"name": "agentPool1234",
"count": 2,
"vmSize": "Standard_A1",
"dnsPrefix": "testabcagents"
},
"linuxProfile": {
"adminUsername": "kubeadmin",
"ssh": {
"publicKeys": [
{
"keyData": "sshkey"
}
]
}
}
}
}
Response getting
{
"code": "BadRequest",
"message": "An error has occurred in subscription <subscriptionid>, resourceGroup: tobeDeletedResourceGroup request: OrchestratorType has unknown orchestrator: ."
}
Please help me out on this
Two things are missing from Azure REST API documentation.
1) It requires version of orchestratorRelease with orchestratorType like this. "orchestratorProfile": {
"orchestratorType": "Kubernetes",
"orchestratorRelease": "1.7"
}
2) Next error i got about missing vmSize in Properties.MasterProfile.VMSize.
So I have added following updates to json
"masterProfile": {
"count": 1,
"dnsPrefix": "testabc",
"vmSize": "Standard_D2_v2"
}
Its very surprising and annoying that documentation is missing these 2 important json parameters.
agentPoolProfiles should be an array of json objects. I pulled this example from azure-cli's mock unit tests to help give you a frame of reference.
https://gist.github.com/bacongobbler/470b8d139536144edf91174916ec4036
Today Google announced a new version of the Email Migration API. The API appears to follow Google's normal REST format but I've been unable to determine the service name and version information for the API in order to use it with the google-api-python-client.
Is there a discovery document for the new API and if so what is the service name and version?
The discovery document was slightly delayed from the actual release, but it's now available through the discovery service or directly here.
For now, I've been able to write my own non-standard discovery document. It's probably buggy/incomplete but it's working for me for test messages and the Python API Client.
{
"kind": "discovery#restDescription",
"etag": "\"BgGz67p-6wsAbOn4St98QhtBGbA/blODPd9XLaVQKZp7pexaIpilnaw\"",
"discoveryVersion": "v1",
"id": "emailmigration:v2",
"name": "emailmigration",
"version": "v2",
"revision": "20130717",
"title": "Email Migration API",
"description": "API to migrate messages to Google Apps Gmail Accounts",
"ownerDomain": "google.com",
"ownerName": "Google",
"documentationLink": "https://developers.google.com/admin-sdk/email-migration/",
"protocol": "rest",
"baseUrl": "https://www.googleapis.com/email/v2/",
"basePath": "/email/v2/users/",
"rootUrl": "https://www.googleapis.com/",
"servicePath": "email/v2/users/",
"batchPath": "batch",
"parameters": {
"alt": {
"type": "string",
"description": "Data format for the response.",
"default": "json",
"enum": [
"json"
],
"enumDescriptions": [
"Responses with Content-Type of application/json"
],
"location": "query"
},
"fields": {
"type": "string",
"description": "Selector specifying which fields to include in a partial response.",
"location": "query"
},
"key": {
"type": "string",
"description": "API key. Your API key identifies your project and provides you with API access, quota, and reports. Required unless you provide an OAuth 2.0 token.",
"location": "query"
},
"oauth_token": {
"type": "string",
"description": "OAuth 2.0 token for the current user.",
"location": "query"
},
"prettyPrint": {
"type": "boolean",
"description": "Returns response with indentations and line breaks.",
"default": "true",
"location": "query"
},
"quotaUser": {
"type": "string",
"description": "Available to use for quota purposes for server-side applications. Can be any arbitrary string assigned to a user, but should not exceed 40 characters. Overrides userIp if both are provided.",
"location": "query"
},
"userIp": {
"type": "string",
"description": "IP address of the site where the request originates. Use this if you want to enforce per-user limits.",
"location": "query"
}
},
"auth": {
"oauth2": {
"scopes": {
"https://www.googleapis.com/auth/email.migration": {
"description": "Write access to Email Migration API v2."
}
}
}
},
"schemas": {
"Mail": {
"id": "Mail",
"type": "object",
"description": "The metadata for a mail message.",
"properties": {
"isDraft": {
"type": "boolean",
"description": "The message is marked as a draft when migrated to the Gmail mailbox."
},
"isInbox": {
"type": "boolean",
"description": "The message should appear in the Inbox, regardless of its labels. By default, a migrated mail message will appear in the Inbox only if it has no labels."
},
"isSent": {
"type": "boolean",
"description": "The message is marked as \"Sent Mail\" when migrated to the Gmail mailbox."
},
"isStarred": {
"type": "boolean",
"description": "The message is starred when migrated."
},
"isTrash": {
"type": "boolean",
"description": "The message is marked as \"Trash\" when migrated."
},
"isUnread": {
"type": "boolean",
"description": "The message is marked as unread when migrated. Without this property, a migrated mail message is marked as read."
},
"isDeleted": {
"type": "boolean",
"description": "The message is migrated directly into Google Vault. Once archived in Google Vault, the message is only visible to a Vault administrator."
},
"labels": {
"type": "array",
"description": "The labels applied to the message when migrated.",
"items": {
"type": "string"
}
}
}
}
},
"resources": {
"mail": {
"methods": {
"insert": {
"id": "emailmigration.mail.insert",
"path": "{userKey}/mail",
"httpMethod": "POST",
"description": "Insert a Mail resource into a user's Gmail account.",
"parameters": {
"userKey": {
"type": "string",
"description": "The email address or email alias of the user's mailbox to migrate the message to.",
"required": true,
"location": "path"
}
},
"request": {
"$ref": "Mail"
},
"scopes": [
"https://www.googleapis.com/auth/email.migration"
],
"supportsMediaUpload": true,
"mediaUpload": {
"accept": [
"message/rfc822"
],
"maxSize": "32MB",
"protocols": {
"simple": {
"multipart": true,
"path": "/upload/email/v2/users/{userKey}/mail"
},
"resumable": {
"multipart": true,
"path": "/upload/email/v2/users/{userKey}/mail"
}
}
}
}
}
}
}
}