I was looking for an API which gives me the details of a Azure Datafactory Pipeline
including start time,end time, duration etc. This is required to generate a report without manually looking into the ADF monitoring window
You can get a list of pipeline runs using Pipeline Runs - Query By Factory and then for each result run [Pipeline Runs - Query By Factory]
POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/queryPipelineRuns?api-version=2018-06-01
Checkout the sample request here
You can pick the necessary fields from the response body.
{
"value": [
{
"runId": "2f7fdb90-5df1-4b8e-ac2f-064cfa58202b",
"pipelineName": "examplePipeline",
"parameters": {
"OutputBlobNameList": "[\"exampleoutput.csv\"]"
},
"invokedBy": {
"id": "80a01654a9d34ad18b3fcac5d5d76b67",
"name": "Manual"
},
"runStart": "2018-06-16T00:37:44.6257014Z",
"runEnd": "2018-06-16T00:38:12.7314495Z",
"durationInMs": 28105,
"status": "Succeeded",
"message": "",
"lastUpdated": "2018-06-16T00:38:12.7314495Z",
"annotations": [],
"runDimension": {
"JobId": "79c1cc52-265f-41a5-9553-be65e736fbd3"
}
},
{
"runId": "16ac5348-ff82-4f95-a80d-638c1d47b721",
"pipelineName": "examplePipeline",
"parameters": {
"OutputBlobNameList": "[\"exampleoutput.csv\"]"
},
"invokedBy": {
"id": "7c5fd7ef7e8a464b98b931cf15fcac66",
"name": "Manual"
},
"runStart": "2018-06-16T00:39:49.2745128Z",
"runEnd": null,
"durationInMs": null,
"status": "Cancelled",
"message": "",
"lastUpdated": "2018-06-16T00:39:51.216097Z",
"annotations": [],
"runDimension": {
"JobId": "84a3c493-0628-4b44-852f-ef5b3a11bdab"
}
}
]
}
And here is a simple WebActivity setup:
Further, you can find similar queries answered by me here and here
Related
I am migrating an Azure devops organization to another organization,
I ran into a problem where i can not create a pipeline which created in the classic way (NOT A YAML).
This is the JSON
{
"name": "PP_NAME",
"folder": "",
"configuration": {
"designerJson": {
"options": [
{
"enabled": false,
"definition": {
"id": "DEF_ID"
},
"inputs": {
"branchFilters": "[\"+refs/heads/*\"]",
"additionalFields": "{}"
}
},
{
"enabled": false,
"definition": {
"id": "DEF_ID"
},
"inputs": {
"workItemType": "Task",
"assignToRequestor": "true",
"additionalFields": "{}"
}
}
],
"variables": {
"system.debug": {
"value": "false",
"allowOverride": true
}
},
"tags": [],
"process": {
"phases": [
{
"name": "Agent job 1",
"refName": "Job_1",
"condition": "succeeded()",
"target": {
"executionOptions": {
"type": 0
},
"allowScriptsAuthAccessOption": false,
"type": 1
},
"jobAuthorizationScope": "project"
}
],
"target": {
"agentSpecification": {
"identifier": "windows-2019"
}
},
"type": 1
},
"quality": "definition",
"path": "\\",
"repository": {
"id": "REPOSITORY_ID",
"name": "test 1",
"type": "TfsGit"
}
},
"path": "\\",
"type": "designerJson"
}
}
The output :
{
"$id": "1",
"innerException": null,
"message": "This API does not support creating pipelines of configuration type DesignerJson.",
"typeName": "Microsoft.Azure.Pipelines.WebApi.UnsupportedConfigurationTypeException, Microsoft.Azure.Pipelines.WebApi",
"typeKey": "UnsupportedConfigurationTypeException",
"errorCode": 0,
"eventId": 3000
}
According the the AZDO documentation, it is possible to create a pipeline with the classic way.
Thanks !
To create a classic pipeline, you can use this REST API Definitions - Create．
If you are not sure about the request body, you can use REST API Definitions - Get to get the definition of a classic pipeline as a reference.
I'm using postman to make rest requests to the azure API to run a pipeline that is in synapse, in terms of permissions and the token I already get them and it works, the problem is that the pipeline receives 3 parameters but I don't know how to pass them, so I have this request, example:
https://hvdhgsad.dev.azuresynapse.net/pipelines/pipeName/createRun?api-version=2020-12-01
and the parameters I added them in the body:
{
"parameters": {
"p_dir": {
"type": "string",
"defaultValue": "val1"
},
"container": {
"type": "string",
"defaultValue": "val"
},
"p_folder": {
"type": "string",
"defaultValue": "val3"
}
}
}
but when I validate the run that was launched with the request I get this.
{
"id": "xxxxxxxxxxxxxxx",
"runId": "xxxxxxxxxxxxxxxxxxxxx",
"debugRunId": null,
"runGroupId": "xxxxxxxxxxxxxxxxxxxx",
"pipelineName": "xxxxxxxxxxxxxxxxx",
"parameters": {
"p_dir": "",
"p_folder": "",
"container": ""
},
"invokedBy": {
"id": "xxxxxxxxxxxxxxxxx",
"name": "Manual",
"invokedByType": "Manual"
},
"runStart": "2021-07-20T05:56:04.2468861Z",
"runEnd": "2021-07-20T05:59:10.1734654Z",
"durationInMs": 185926,
"status": "Failed",
"message": "Operation on target Data flow1 failed: {\"StatusCode\":\"DF-Executor-SourceInvalidPayload\",\"Message\":\"Job failed due to reason: Data preview, debug, and pipeline data flow execution failed because container does not exist\",\"Details\":\"\"}",
"lastUpdated": "2021-07-20T05:59:10.1734654Z",
"annotations": [],
"runDimension": {},
"isLatest": true
}
the params are empty, so I don't know what's wrong or missing.
what is the correct way to pass them???
ref: https://learn.microsoft.com/en-us/rest/api/synapse/data-plane/pipeline/create-pipeline-run#examples
Just created an account to answer this as i've had the same issue.
I resolved this by just having the name of the variable and its subsequent value in the body JSON.
e.g.
{"variable": "value", "variable": "value"}
Found this by following the documentation you had posted, under request body it passes the name of the variable and the value directly into the JSON body.
{
"OutputBlobNameList": [
"exampleoutput.csv"
]
}
This particular example is a list/array so it confused me by adding the brackets [] if you are passing string parameters this is unneeded.
I am able to create the "message route" in azure portal and able to route messages to servicebusqueue if the query matching, I want to create the message route using the restapi instead of using azure portal, I have seen many documents but unable to find the proper one. Whether creating the message route using restapi is possible or not? if yes,How can I achieve this and please provide the respective links to refer?
I haven't tried this through REST API, but as Roman suggested,
You can check the IotHubResource_CreateOrUpdate which will help you understand how to Create or update the metadata of an Iot hub. The usual pattern to modify a property is to retrieve the IoT hub metadata and security metadata, and then combine them with the modified values in a new body to update the IoT hub.
Sample Request:
PUT https://management.azure.com/subscriptions/91d12660-3dec-467a-be2a-213b5544ddc0/resourceGroups/myResourceGroup/providers/Microsoft.Devices/IotHubs/testHub?api-version=2018-04-01
Request Body:
{
"name": "iot-dps-cit-hub-1",
"type": "Microsoft.Devices/IotHubs",
"location": "centraluseuap",
"tags": {},
"etag": "AAAAAAFD6M4=",
"properties": {
"operationsMonitoringProperties": {
"events": {
"None": "None",
"Connections": "None",
"DeviceTelemetry": "None",
"C2DCommands": "None",
"DeviceIdentityOperations": "None",
"FileUploadOperations": "None",
"Routes": "None"
}
},
"state": "Active",
"provisioningState": "Succeeded",
"ipFilterRules": [],
"hostName": "iot-dps-cit-hub-1.azure-devices.net",
"eventHubEndpoints": {
"events": {
"retentionTimeInDays": 1,
"partitionCount": 2,
"partitionIds": [
"0",
"1"
],
"path": "iot-dps-cit-hub-1",
"endpoint": "sb://iothub-ns-iot-dps-ci-245306-76aca8e13b.servicebus.windows.net/"
},
"operationsMonitoringEvents": {
"retentionTimeInDays": 1,
"partitionCount": 2,
"partitionIds": [
"0",
"1"
],
"path": "iot-dps-cit-hub-1-operationmonitoring",
"endpoint": "sb://iothub-ns-iot-dps-ci-245306-76aca8e13b.servicebus.windows.net/"
}
},
"routing": {
"endpoints": {
"serviceBusQueues": [],
"serviceBusTopics": [],
"eventHubs": [],
"storageContainers": []
},
"routes": [],
"fallbackRoute": {
"name": "$fallback",
"source": "DeviceMessages",
"condition": "true",
"endpointNames": [
"events"
],
"isEnabled": true
}
},
"storageEndpoints": {
"$default": {
"sasTtlAsIso8601": "PT1H",
"connectionString": "",
"containerName": ""
}
},
"messagingEndpoints": {
"fileNotifications": {
"lockDurationAsIso8601": "PT1M",
"ttlAsIso8601": "PT1H",
"maxDeliveryCount": 10
}
},
"enableFileUploadNotifications": false,
"cloudToDevice": {
"maxDeliveryCount": 10,
"defaultTtlAsIso8601": "PT1H",
"feedback": {
"lockDurationAsIso8601": "PT1M",
"ttlAsIso8601": "PT1H",
"maxDeliveryCount": 10
}
},
"features": "None"
},
"sku": {
"name": "S1",
"tier": "Standard",
"capacity": 1
}
}
Acumatica cloud ERP has a screen that displays failed push notifications (screenID = SM502000). I have created a new Web Services Endpoint, and added a new top-level entity from that screenID. However, no matter what fields I try to add, I get a variation on this error:
Optimization cannot be performed.The following fields cause the error:\r\nSelected: View FailedToSend has BQL delegate\r\n
Edit. I was able to create a REST end-point to fetch the results using a generic inquiry. Here is the JSON returned:
{
"id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"rowNumber": 1,
"note": null,
"Result": [
{
"id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"rowNumber": 1,
"note": null,
"Date": {
"value": 637167021636873690
},
"DateTimeStamp": {
"value": "2020-02-07T19:56:03.687369+00:00"
},
"DestinationNamePushNotifications_hookId": {
"value": "test"
},
"DestinationNamePushNotifications_hookId_description": {
"value": "test"
},
"Error": {
"value": "Send to target test failed. Message: WebHook returns not success code: NotFound; Message: Tunnel webhook.local not found"
},
"NotificationBody": {
"value": "..."
},
"PushNotifications_selected": {},
"Selected": {
"value": false
},
"SourceName": {
"value": "SO-SalesOrder"
},
"TransactionId": {
"value": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
},
"custom": {},
"files": []
},
{
"id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"rowNumber": 2,
"note": null,
"Date": {
"value": 637167021636873690
},
"DateTimeStamp": {
"value": "2020-02-07T19:56:03.687369+00:00"
},
"DestinationNamePushNotifications_hookId": {
"value": "test"
},
"DestinationNamePushNotifications_hookId_description": {
"value": "test"
},
"Error": {
"value": "Send to target test failed. Message: WebHook returns not success code: NotFound; Message: Tunnel webhook.local not found"
},
"NotificationBody": {
"value": "..."
},
"PushNotifications_selected": {},
"Selected": {
"value": false
},
"SourceName": {
"value": "SO-SalesOrder"
},
"TransactionId": {
"value": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
},
"custom": {},
"files": []
}
]
}
Here's a good tutorial for using GI to access data via API:
https://www.acumatica.com/blog/contract-based-apis-in-generic-inquiries/
Try retrieving the records one by one with a REST filter in the URL. The issue is that the target graph you added has a data view (FailedToSend) which uses a data view delegate.
Because data view delegates can be slower than no delegates the API refuses to return more than one record as an optimization.
I followed the tutorial found at
https://learn.microsoft.com/en-us/azure/devops/extend/develop/service-endpoints?view=azure-devops
The api call is right because if I don't authenticate it shows the correct path in the error message. Entering the url that it shows in the message in a browser gives the correct json.
The picklist object will act like it is trying to load but will end up showing no results found.
Can you use a personal company api to load a picklist using server endpoint?
How do you debug this. All I see in the network tab on debugging tools is the 200 from azure making the post.
This is the JSON
{
"meta": {
"record_count": 5,
"source": "mongodb",
"searchParams": {
"versions.start_date": {
"$gte": "2019-08-30T00:11:55.329Z"
}
},
"paging": {
"num": 10,
"start": 0
}
},
"screenshots": [
{
"screenshot_test_id": 3946619,
"url": "https://www.google.com",
"test_name": null,
"created_date": "2019-10-27T21:26:32.0 ```
This is the datasources from the vss.extension
"dataSources": [
{
"name": "CBTAPIReturn",
"endpointUrl": "{{endpoint.url}}",
"resultSelector": "jsonpath:$[*].screenshots"
}
]
inputs and bindings from the task
"inputs": [
{
"name": "CBTService",
"type": "connectedService:CBTServiceEndpoint",
"label": "CBT service/server end point",
"defaultValue": "",
"required": true,
"helpMarkDown": "Select the CBT end point to use. If needed,selecton 'manage', and add a new service endpoint of type 'CBT server connection'"
},
{
"name": "CBT API Response",
"type": "pickList",
"label": "CBT API Response",
"required": true,
"helpMarkDown": "Select the name combination that you want to use.",
"properties": {
"EditableOptions": "True"
}
}
],
"dataSourceBindings": [
{
"target": "CBT API Response",
"endpointId": "$(CBTService)",
"dataSourceName": "CBTAPIReturn"
```