Azure Data Factory find which pipeline in a sequence failed - azure

Question

I have a pipeline (PL_Main) containing a set of Execute pipeline activities in sequence, as the below image depicts:
Now, I was trying to create an activity (executed after PL_Main completion) that allowed me to understand if any of the 'childs' Execute Pipeline activities (PL_1, PL_2, PL3.1, PL_3.2) failed.
However, according my understanding thee pipelines output contains only the RunId and PipelineName. Is there a way that I can find what were the pipelines in my execution chain that failed (or at least the 1st one failling) and store their RunId in as a parameter for my next activity (New Activity) ?
Thanks in advance to everybody!

Answer 1

Assuming you need to pass the pipelinerunid of the failed activity within Main pipeline, you need to deign the pipeline in below way :
JSON:
{
"name": "pipeline8",
"properties": {
"activities": [
{
"name": "Execute Pipeline1",
"type": "ExecutePipeline",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"pipeline": {
"referenceName": "pipeline7",
"type": "PipelineReference"
},
"waitOnCompletion": true
}
},
{
"name": "Execute Pipeline2",
"type": "ExecutePipeline",
"dependsOn": [
{
"activity": "Execute Pipeline1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"pipeline": {
"referenceName": "pipeline9",
"type": "PipelineReference"
},
"waitOnCompletion": true
}
},
{
"name": "Execute Pipeline3",
"type": "ExecutePipeline",
"dependsOn": [
{
"activity": "Execute Pipeline2",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"pipeline": {
"referenceName": "pipeline10",
"type": "PipelineReference"
},
"waitOnCompletion": true
}
},
{
"name": "Execute Pipeline4",
"type": "ExecutePipeline",
"dependsOn": [
{
"activity": "Execute Pipeline2",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"pipeline": {
"referenceName": "pipeline11",
"type": "PipelineReference"
},
"waitOnCompletion": true
}
},
{
"name": "Fail1",
"type": "Fail",
"dependsOn": [
{
"activity": "Execute Pipeline1",
"dependencyConditions": [
"Failed"
]
}
],
"userProperties": [],
"typeProperties": {
"message": {
"value": "#activity('Execute Pipeline1').output.pipelineRunId",
"type": "Expression"
},
"errorCode": "1"
}
},
{
"name": "Fail2",
"type": "Fail",
"dependsOn": [
{
"activity": "Execute Pipeline2",
"dependencyConditions": [
"Failed"
]
}
],
"userProperties": [],
"typeProperties": {
"message": {
"value": "#activity('Execute Pipeline2').output.pipelineRunId",
"type": "Expression"
},
"errorCode": "2"
}
},
{
"name": "Fail3",
"type": "Fail",
"dependsOn": [
{
"activity": "Execute Pipeline3",
"dependencyConditions": [
"Failed"
]
},
{
"activity": "Fail5",
"dependencyConditions": [
"Skipped"
]
}
],
"userProperties": [],
"typeProperties": {
"message": "#activity('Execute Pipeline3').output.pipelineRunId",
"errorCode": "1"
}
},
{
"name": "Fail4",
"type": "Fail",
"dependsOn": [
{
"activity": "Execute Pipeline4",
"dependencyConditions": [
"Failed"
]
},
{
"activity": "Fail5",
"dependencyConditions": [
"Skipped"
]
}
],
"userProperties": [],
"typeProperties": {
"message": "#activity('Execute Pipeline4').output.pipelineRunId",
"errorCode": "4"
}
},
{
"name": "Fail5",
"type": "Fail",
"dependsOn": [
{
"activity": "Execute Pipeline3",
"dependencyConditions": [
"Failed"
]
},
{
"activity": "Execute Pipeline4",
"dependencyConditions": [
"Failed"
]
}
],
"userProperties": [],
"typeProperties": {
"message": {
"value": "#concat(activity('Execute Pipeline3').output.pipelineRunId,';',activity('Execute Pipeline4').output.pipelineRunId)",
"type": "Expression"
},
"errorCode": "5"
}
}
],
"variables": {
"test": {
"type": "String"
},
"spt": {
"type": "Array"
},
"final": {
"type": "String"
}
},
"annotations": []
}
}
Reference blog : https://datasharkx.wordpress.com/2021/08/19/error-logging-and-the-art-of-avoiding-redundant-activities-in-azure-data-factory/
Note: the fail activities capture the pipelinerunid
Finaly to capture the ID:
#activity('Execute Pipeline1').error.message

Answer 2

You can try this approach also, to get the failed pipeline run id.
For this name the Execute pipeline activities name same as child pipeline names.
First get the Failed Execute pipeline activity name (Pipeline name) in the main pipeline using the error message.
Error message:
Use set variable to get this.
Get the pipeline name from the error message.
use the below expression for that.
#substring(variables('error'), add(indexOf(variables('error'),'target'),7), sub(indexOf(variables('error'),' failed'),add(indexOf(variables('error'),'target'),7)))
You can see that we got the pipeline name in the variable
Now get all the pipeline runs from the Pipeline Runs - Query By Factory command.
POST https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.DataFactory/factories/{factoryName}/queryPipelineRuns?api-version=2018-06-01
Use this in web activity.
This will give you all pipeline runs details with names and run ids Json which you can see in the above documentation.
Now compare our pipeline name from variable in this JSON and get that particular run id which is our required failed pipeline run id. Use ForEach activity or filter for that.
Reference:
Check this SO thread for reference by KarthikBhyresh-MT