Azure datafactory v2 Execute Pipeline with For Each - azure

Question

I am trying to use "Execute Pipeline" to invoke a Pipe which has a ForEach activity. I get an error.
Json for Execute pipe:
[
{
"name": "pipeline3",
"properties": {
"activities": [
{
"name": "Test_invoke1",
"type": "ExecutePipeline",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"pipeline": {
"referenceName": "MAIN_SA_copy1",
"type": "PipelineReference"
},
"waitOnCompletion": true
}
}
],
"annotations": []
}
}
]
Jason for Invoke pipe for each activity :
[
{
"name": "MAIN_SA_copy1",
"properties": {
"activities": [
{
"name": "Collect_SA_Data",
"type": "ForEach",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"items": {
"value": "#pipeline().parameters.TableNames",
"type": "Expression"
},
"batchCount": 15,
"activities": [
{
"name": "Sink_SAdata_toDL",
"type": "Copy",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [
{
"name": "Destination",
"value": "#{pipeline().parameters.DLFilePath}/#{item()}"
}
],
"typeProperties": {
"source": {
"type": "SqlServerSource",
"sqlReaderQuery": {
"value": "#concat('SELECT * FROM ',item())",
"type": "Expression"
}
},
"sink": {
"type": "AzureBlobFSSink"
},
"enableStaging": false,
"parallelCopies": 1,
"dataIntegrationUnits": 4
},
"inputs": [
{
"referenceName": "SrcDS_StructuringAnalytics",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "ADLS",
"type": "DatasetReference",
"parameters": {
"FilePath": "#pipeline().parameters.DLFilePath",
"FileName": {
"value": "#concat(item(),'.orc')",
"type": "Expression"
}
}
}
]
}
]
}
}
],
"parameters": {
"DLFilePath": {
"type": "string",
"defaultValue": "extracts/StructuringAnalytics"
},
"TableNames": {
"type": "array",
"defaultValue": [
"fom.FOMLineItem_manual"
]
}
},
"variables": {
"QryTableColumn": {
"type": "String"
},
"QryTable": {
"type": "String"
}
},
"folder": {
"name": "StructuringAnalytics"
},
"annotations": []
},
"type": "Microsoft.DataFactory/factories/pipelines"
}
]
I get an error:
[
{
"errorCode": "BadRequest",
"message": "Operation on target Collect_SA_Data failed: The execution of template action 'Collect_SA_Data' failed: the result of the evaluation of 'foreach' expression '#pipeline().parameters.TableNames' is of type 'String'. The result must be a valid array.",
"failureType": "UserError",
"target": "Test_invoke1",
"details": ""
}
]
Input:
"pipeline": {
"referenceName": "MAIN_SA_copy1",
"type": "PipelineReference"
},
"waitOnCompletion": true,
"parameters": {
"DLFilePath": "extracts/StructuringAnalytics",
"TableNames": "[\"fom.FOMLineItem_manual\"]"
}

Answer 1

Please try updating your dynamic expression of ForEach Items as below:
{
"value": "#array(pipeline().parameters.TableNames)",
"type": "Expression"
}
Hope this helps.

Answer 2

I guess you were using the UI to set the pipeline and its parameters and I guess you expected to put the array parameter of the called pipeline as everywhere else like this:
(It is all my guess, because I just did exactly the same, with the same result)
The trick is to define the array in the Code (["table1", "table2"]):
The input in the UI will look like this:
Now it works!
It seems, that the Datafactory is otherwise treating the whole array as one element of some array. Hence, the solution with the array() function sometimes works.
It looks like a bug, defining array parameter input..
(Had to edit the answer, I first thought omiting the colons in the UI input would be enough)