Azure ARM DataFactory InvalidTemplate the runAfter property not valid; scope must belong to same level - azure

Documenting here what I learned, in case it helps someone else. I had a ARM template implementing a DataFactory pipeline, which had a weird InvalidTemplate error. I am simplifying my template to a contrived bare bones template,
resources: [
{
"name": "blah",
"type": "Microsoft.DataFactory/factories/pipelines",
"apiVersion": "2018-06-01",
"properties": {
"activities": [
{
"name": "Foo",
"type": "SetVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "hi",
"value": {
"value": "int(1)",
"type": "Expression"
}
}
},
{
"name": "CoolIf",
"type": "IfCondition",
"typeProperties": {
"expression": {
"value": "#bool(equals(variables('hi'), 1))",
"type": "Expression"
},
"ifTrueActivities": [
{
"name": "Blarg",
"type": "SetVariable",
"dependsOn": [{"activity": "Foo"}],
"userProperties": [],
"typeProperties": {
"variableName": "okay",
"value": {
"value": "#string(1 + int(variables('hi')))",
"type": "Expression"
}
}
},
],
"ifFalseActivities": []
}
}
]
}
}
]
produced the error message
ErrorCode=InvalidTemplate, ErrorMessage=The template validation
failed: 'The 'runAfter' property of template action 'BlargScope'
is not valid: the action 'FooScope' must belong to same level as
action 'BlargScope'. Available actions on same level: ''
I could not find a good answer online.

After a day of trial and error I realized you can put a dependsOn in the IfCondition parent, so below, just moves the dependsOn from the child Blarg to the parent CoolIf. And then it started working =) .
resources: [
{
"name": "blah",
"type": "Microsoft.DataFactory/factories/pipelines",
"apiVersion": "2018-06-01",
"properties": {
"activities": [
{
"name": "Foo",
"type": "SetVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "hi",
"value": {
"value": "int(1)",
"type": "Expression"
}
}
},
{
"name": "CoolIf",
"type": "IfCondition",
"dependsOn": [{"activity": "Foo"}],
"typeProperties": {
"expression": {
"value": "#bool(equals(variables('hi'), 1))",
"type": "Expression"
},
"ifTrueActivities": [
{
"name": "Blarg",
"type": "SetVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "okay",
"value": {
"value": "#string(1 + int(variables('hi')))",
"type": "Expression"
}
}
},
],
"ifFalseActivities": []
}
}
]
}
}
]
In retrospect I feel silly how simple this was haha.

Related

Problems with rule-based mapping in Azure Dataflow

I have multiple XMLs in an Azure Blob. I want to extract certain datapoints and sort them into an Azure SQL db.
For this I use Data Flow. The name of certain elements changes sometimes [random name] so i would like to set up a rule-based mapping, that fetches the right values every time .
I want to retrieve IMPORTANT INFORMATION, which is always located in the first randomly named child of category_a.
Apart from the randomly named object the rest of the structure always stays the same.
This is about the structure:
<title>
<category_a>
<random_name_1>
<object_a>
<subobject_object_a>
<p>IMPORTANT INFORMATION</p>
</subobject_object_a>
</object_a>
</random_name_1>
<random_name_2>
<object_a>
<subobject_object_a>
<p>IRRELEVANT INFORMATION</p>
</subobject_object_a>
</object_a>
</random_name_2>
</category_a>
<category_b></category_b>
How do I need to write the rule based mapping so that I always fetch this value no matter the random name in the middle of the path?
Thanks for your help
There might be no option to find or use rule-based mapping to retrieve the important information. As a work around, I have used lookup and string manipulation activities to get the result. The following is the pipeline JSON:
{
"name": "pipeline1",
"properties": {
"activities": [
{
"name": "Lookup1",
"type": "Lookup",
"dependsOn": [],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "XmlSource",
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"recursive": true,
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "XmlReadSettings",
"validationMode": "none",
"namespaces": true
}
},
"dataset": {
"referenceName": "Xml1",
"type": "DatasetReference"
},
"firstRowOnly": false
}
},
{
"name": "Set variable1",
"type": "SetVariable",
"dependsOn": [
{
"activity": "Lookup1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "tp",
"value": {
"value": "#replace(replace(split(string(activity('Lookup1').output.value[0].title.category_a),':')[0],'\"',''),'{','')",
"type": "Expression"
}
}
},
{
"name": "Set variable2",
"type": "SetVariable",
"dependsOn": [
{
"activity": "Set variable1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "tp2",
"value": {
"value": "#replace(replace(split(string(activity('Lookup1').output.value[0].title.category_a[variables('tp')]),':')[0],'\"',''),'{','')",
"type": "Expression"
}
}
},
{
"name": "Set variable3",
"type": "SetVariable",
"dependsOn": [
{
"activity": "Set variable2",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "tp3",
"value": {
"value": "#replace(replace(split(string(activity('Lookup1').output.value[0].title.category_a[variables('tp')][variables('tp2')]),':')[0],'\"',''),'{','')",
"type": "Expression"
}
}
},
{
"name": "Set variable4",
"type": "SetVariable",
"dependsOn": [
{
"activity": "Set variable3",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "tp4",
"value": {
"value": "#replace(replace(split(string(activity('Lookup1').output.value[0].title.category_a[variables('tp')][variables('tp2')][variables('tp3')]),':')[0],'\"',''),'{','')",
"type": "Expression"
}
}
},
{
"name": "Set variable5",
"type": "SetVariable",
"dependsOn": [
{
"activity": "Set variable4",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "final",
"value": {
"value": "#activity('Lookup1').output.value[0].title.category_a[variables('tp')][variables('tp2')][variables('tp3')][variables('tp4')]",
"type": "Expression"
}
}
}
],
"variables": {
"tp": {
"type": "String"
},
"tp2": {
"type": "String"
},
"tp3": {
"type": "String"
},
"tp4": {
"type": "String"
},
"final": {
"type": "String"
}
},
"annotations": []
}
}
The lookup has the source xml file. Since we know the category_a is a child element, I have started from there to obtain the child element names (in each set variable activity).
The following is the output image for reference:

Dropping columns in Azure Data Factory based on values in columns

I'm working with a dataset where I need to drop some columns which contain only NULL values. The issue is that the column names are not consistent or similar, and can change with time. I was wondering if there is a way in ADF to drop a column if all instances are NULL without having drifted columns?
I have tried unpivoting, removing rows, then re-pivoting, however after I pivot the data back to its original format, I get the following message:
"This drifted column is not in the source schema and therefore can only be referenced with pattern matching expressions"
The drifted columns don't seem to join on subsequent join functions. I have also tried setting derived columns with regex column patters to make all the drifted columns explicit, however, the byName() function doesn't seem to work with the $$ syntax; namely:
toString(byName($$))
Any ideas of how to solve this within Azure Data Factory - Data Flows would be very much appreciated!
I have used combination of both data factory pipeline activities and dataflow to achieve the requirement.
First, I have taken dataflow to output a file. I have added a new column with all values as 1 so that I can use aggregate on all other rows using this new column to group.
I have used collect() to create an array for each of the column where group by is on above created column.
Now create another derived column to replace the array by converting array to string and calculating length. If length is 2 it indicates that column contains all nulls.
Write this dataflow output to a file. The data preview of the sink will be as follows:
Create a dataflow activity to run the above dataflow and pass the following dynamic content in execute pipeline activity to filter out and write data of only required columns.
#activity('Data flow1').output.runstatus.profile.sink1.total
In pipeline2, I have used activities to get columns that are not entirely nulls, create a dynamic schema and then use this schema as mapping and write to a file only the required columns.
First, I have read the file written at the end of dataflow without header (even though the file has header). The dataset looks as shown below:
You can directly use the following pipeline JSON to build the pipeline:
{
"name": "pipeline2",
"properties": {
"activities": [
{
"name": "Lookup1",
"type": "Lookup",
"dependsOn": [],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "DelimitedTextSource",
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"recursive": true,
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "DelimitedTextReadSettings"
}
},
"dataset": {
"referenceName": "cols",
"type": "DatasetReference"
},
"firstRowOnly": false
}
},
{
"name": "ForEach1",
"type": "ForEach",
"dependsOn": [
{
"activity": "Lookup1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "#range(0,pipeline().parameters.count_of_rows)",
"type": "Expression"
},
"isSequential": true,
"activities": [
{
"name": "Append variable1",
"type": "AppendVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "props",
"value": {
"value": "Prop_#{item()}",
"type": "Expression"
}
}
}
]
}
},
{
"name": "ForEach2",
"type": "ForEach",
"dependsOn": [
{
"activity": "ForEach1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "#variables('props')",
"type": "Expression"
},
"isSequential": true,
"activities": [
{
"name": "Append variable2",
"type": "AppendVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "req_cols",
"value": {
"value": "#if(and(not(equals(activity('Lookup1').output.value[0][item()],'tp')),not(equals(activity('Lookup1').output.value[1][item()],'2'))),activity('Lookup1').output.value[0][item()],'')",
"type": "Expression"
}
}
}
]
}
},
{
"name": "Filter1",
"type": "Filter",
"dependsOn": [
{
"activity": "ForEach2",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "#variables('req_cols')",
"type": "Expression"
},
"condition": {
"value": "#not(equals(item(),''))",
"type": "Expression"
}
}
},
{
"name": "ForEach3",
"type": "ForEach",
"dependsOn": [
{
"activity": "Filter1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "#activity('Filter1').output.Value",
"type": "Expression"
},
"isSequential": true,
"activities": [
{
"name": "Append variable3",
"type": "AppendVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "mapping",
"value": {
"value": "#json(concat('{\"source\":{\"name\":\"',item(),'\"},\"sink\":{\"name\":\"',item(),'\"}}'))",
"type": "Expression"
}
}
}
]
}
},
{
"name": "Set variable1",
"type": "SetVariable",
"dependsOn": [
{
"activity": "ForEach3",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "dynamic_schema",
"value": {
"value": "#concat('{\"type\":\"TabularTranslator\",\"mappings\":',string(variables('mapping')),'}}')",
"type": "Expression"
}
}
},
{
"name": "Copy data1",
"type": "Copy",
"dependsOn": [
{
"activity": "Set variable1",
"dependencyConditions": [
"Succeeded"
]
}
],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "DelimitedTextSource",
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"recursive": true,
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "DelimitedTextReadSettings"
}
},
"sink": {
"type": "DelimitedTextSink",
"storeSettings": {
"type": "AzureBlobFSWriteSettings"
},
"formatSettings": {
"type": "DelimitedTextWriteSettings",
"quoteAllText": true,
"fileExtension": ".txt"
}
},
"enableStaging": false,
"translator": {
"value": "#json(variables('dynamic_schema'))",
"type": "Expression"
}
},
"inputs": [
{
"referenceName": "csv1",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "req_file",
"type": "DatasetReference"
}
]
}
],
"parameters": {
"count_of_rows": {
"type": "int"
}
},
"variables": {
"props": {
"type": "Array"
},
"req_cols": {
"type": "Array"
},
"test": {
"type": "String"
},
"mapping": {
"type": "Array"
},
"dynamic_schema": {
"type": "String"
}
},
"annotations": []
}
}
NOTE: In the copy data activity, the source is the original file.
If the source column names will change, then you have to use column patterns. When you match columns based on patterns, you can project those into columns using the Select transformation. Use the rule-based mapping option in the Select transformation with true() as the matching expression and $$ as the Name As property like this:

Why does setting CosmosDB throughputSettings result in "Entity with the specified id does not exist in the system"?

While trying to deploy a CosmosDB instance with 2 collections ("MyCollection1", "MyCollection2") I keep getting the error:
NotFound: Entity with the specified id does not exist in the system
So I keep searching for "resourceId" in my custom ARM template (please see below) but cannot find the erorr cause.
I don't understand, why does not the pipeline at least print the line number for me?
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"accountName": {
"defaultValue": "my-cosmosdb",
"type": "String"
}
},
"variables": {
"resourceName": "[concat(resourceGroup().name, '-', parameters('accountName'))]",
"resourceId": "[resourceId('Microsoft.DocumentDB/databaseAccounts', variables('resourceName'))]",
"apiVersion": "[providers('Microsoft.DocumentDB', 'databaseAccounts').apiVersions[0]]"
},
"outputs": {
"CosmosDbConnectionString": {
"type": "string",
"value": "[concat('AccountEndpoint=https://', variables('resourceName'), '.documents.azure.com:443/;AccountKey=', listKeys(variables('resourceId'), variables('apiVersion')).primaryMasterKey, ';')]"
},
"DatabaseName": {
"type": "string",
"value": "MyDB"
},
"CollectionName1": {
"type": "string",
"value": "MyCollection1"
},
"CollectionName2": {
"type": "string",
"value": "MyCollection2"
}
},
"resources": [
{
"type": "Microsoft.DocumentDB/databaseAccounts",
"apiVersion": "2020-03-01",
"name": "[variables('resourceName')]",
"location": "[resourceGroup().location]",
"tags": {
"defaultExperience": "DocumentDB"
},
"kind": "GlobalDocumentDB",
"properties": {
"publicNetworkAccess": "Enabled",
"enableAutomaticFailover": false,
"enableMultipleWriteLocations": false,
"isVirtualNetworkFilterEnabled": false,
"virtualNetworkRules": [],
"disableKeyBasedMetadataWriteAccess": false,
"databaseAccountOfferType": "Standard",
"consistencyPolicy": {
"defaultConsistencyLevel": "Session",
"maxIntervalInSeconds": 5,
"maxStalenessPrefix": 100
},
"locations": [
{
"locationName": "[resourceGroup().location]",
"provisioningState": "Succeeded",
"failoverPriority": 0,
"isZoneRedundant": false
}
],
"capabilities": []
}
},
{
"type": "Microsoft.DocumentDB/databaseAccounts/sqlDatabases",
"apiVersion": "2020-03-01",
"name": "[concat(variables('resourceName'), '/MyDB')]",
"dependsOn": [
"[resourceId('Microsoft.DocumentDB/databaseAccounts', variables('resourceName'))]"
],
"properties": {
"resource": {
"id": "MyDB"
},
"options": {}
}
},
{
"type": "Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers",
"apiVersion": "2020-03-01",
"name": "[concat(variables('resourceName'), '/MyDB/MyCollection1')]",
"dependsOn": [
"[resourceId('Microsoft.DocumentDB/databaseAccounts/sqlDatabases', variables('resourceName'), 'MyDB')]",
"[resourceId('Microsoft.DocumentDB/databaseAccounts', variables('resourceName'))]"
],
"properties": {
"resource": {
"id": "MyCollection1",
"indexingPolicy": {
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/*"
}
],
"excludedPaths": [
{
"path": "/\"_etag\"/?"
}
]
},
"partitionKey": {
"paths": [
"/partitionKey"
],
"kind": "Hash"
},
"uniqueKeyPolicy": {
"uniqueKeys": []
},
"conflictResolutionPolicy": {
"mode": "LastWriterWins",
"conflictResolutionPath": "/_ts"
}
},
"options": {}
}
},
{
"type": "Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers",
"apiVersion": "2020-03-01",
"name": "[concat(variables('resourceName'), '/MyDB/MyCollection2')]",
"dependsOn": [
"[resourceId('Microsoft.DocumentDB/databaseAccounts/sqlDatabases', variables('resourceName'), 'MyDB')]",
"[resourceId('Microsoft.DocumentDB/databaseAccounts', variables('resourceName'))]"
],
"properties": {
"resource": {
"id": "MyCollection2",
"indexingPolicy": {
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/*"
}
],
"excludedPaths": [
{
"path": "/\"_etag\"/?"
}
]
},
"partitionKey": {
"paths": [
"/partitionKey"
],
"kind": "Hash"
},
"uniqueKeyPolicy": {
"uniqueKeys": []
},
"conflictResolutionPolicy": {
"mode": "LastWriterWins",
"conflictResolutionPath": "/_ts"
}
},
"options": {}
}
},
{
"type": "Microsoft.DocumentDB/databaseAccounts/sqlDatabases/throughputSettings",
"apiVersion": "2020-03-01",
"name": "[concat(variables('resourceName'), '/MyDB/default')]",
"dependsOn": [
"[resourceId('Microsoft.DocumentDB/databaseAccounts/sqlDatabases', variables('resourceName'), 'MyDB')]",
"[resourceId('Microsoft.DocumentDB/databaseAccounts', variables('resourceName'))]"
],
"properties": {
"resource": {
"throughput": 400
}
}
}
]
}
UPDATE:
I have removed the part creating collections and the error is still there.
UPDATE 2:
The following part seemingly causes the error, but why?
{
"type": "Microsoft.DocumentDB/databaseAccounts/sqlDatabases/throughputSettings",
"apiVersion": "2020-03-01",
"name": "[concat(variables('resourceName'), '/MyDB/default')]",
"dependsOn": [
"[resourceId('Microsoft.DocumentDB/databaseAccounts/sqlDatabases', variables('resourceName'), 'MyDB')]",
"[resourceId('Microsoft.DocumentDB/databaseAccounts', variables('resourceName'))]"
],
"properties": {
"resource": {
"throughput": 400
}
}
}
What is wrong with the dependsOn entry "[resourceId('Microsoft.DocumentDB/databaseAccounts/sqlDatabases', variables('resourceName'), 'MyDB')]?
UPDATE 3:
Trying to deploy the complete ARM template listed above manually results in:
Try setting your throughput in the options for your database.
UPDATE: You cannot specify throughput on a resource that did not have it when initially provisioned. Databases and containers provisioned without throughput cannot be updated later to have it. Conversely, a resource provisioned with throughput, cannot be updated to remove. You must delete and recreate the resource. This will require migrating your data.
{
"type": "Microsoft.DocumentDB/databaseAccounts/sqlDatabases",
"apiVersion": "2020-03-01",
"name": "[concat(variables('resourceName'), '/MyDB')]",
"dependsOn": [
"[resourceId('Microsoft.DocumentDB/databaseAccounts', variables('resourceName'))]"
],
"properties": {
"resource": {
"id": "MyDB"
},
"options": { "throughput": "[parameters('throughput')]" }
}
},
btw, there are lots of samples you can use to start with here Cosmos DB templates
The following has worked for me, I had to replace "sqlDatabases/throughputSettings" by "sqlDatabases/containers/throughputSettings":
{
"type": "Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers/throughputSettings",
"apiVersion": "2020-03-01",
"name": "[concat(variables('resourceName'), '/', variables('DatabaseName'), '/', variables('CollectionName1'), '/default')]",
"dependsOn": [
"[resourceId('Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers', variables('resourceName'), variables('databaseName'), variables('CollectionName1'))]",
"[resourceId('Microsoft.DocumentDB/databaseAccounts/sqlDatabases', variables('resourceName'), variables('databaseName'))]",
"[resourceId('Microsoft.DocumentDB/databaseAccounts', variables('resourceName'))]"
],
"properties": {
"resource": {
"throughput": 400
}
}
}
And then similar entry for CollectionName2

Azure datafactory v2 Execute Pipeline with For Each

I am trying to use "Execute Pipeline" to invoke a Pipe which has a ForEach activity. I get an error.
Json for Execute pipe:
[
{
"name": "pipeline3",
"properties": {
"activities": [
{
"name": "Test_invoke1",
"type": "ExecutePipeline",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"pipeline": {
"referenceName": "MAIN_SA_copy1",
"type": "PipelineReference"
},
"waitOnCompletion": true
}
}
],
"annotations": []
}
}
]
Jason for Invoke pipe for each activity :
[
{
"name": "MAIN_SA_copy1",
"properties": {
"activities": [
{
"name": "Collect_SA_Data",
"type": "ForEach",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"items": {
"value": "#pipeline().parameters.TableNames",
"type": "Expression"
},
"batchCount": 15,
"activities": [
{
"name": "Sink_SAdata_toDL",
"type": "Copy",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [
{
"name": "Destination",
"value": "#{pipeline().parameters.DLFilePath}/#{item()}"
}
],
"typeProperties": {
"source": {
"type": "SqlServerSource",
"sqlReaderQuery": {
"value": "#concat('SELECT * FROM ',item())",
"type": "Expression"
}
},
"sink": {
"type": "AzureBlobFSSink"
},
"enableStaging": false,
"parallelCopies": 1,
"dataIntegrationUnits": 4
},
"inputs": [
{
"referenceName": "SrcDS_StructuringAnalytics",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "ADLS",
"type": "DatasetReference",
"parameters": {
"FilePath": "#pipeline().parameters.DLFilePath",
"FileName": {
"value": "#concat(item(),'.orc')",
"type": "Expression"
}
}
}
]
}
]
}
}
],
"parameters": {
"DLFilePath": {
"type": "string",
"defaultValue": "extracts/StructuringAnalytics"
},
"TableNames": {
"type": "array",
"defaultValue": [
"fom.FOMLineItem_manual"
]
}
},
"variables": {
"QryTableColumn": {
"type": "String"
},
"QryTable": {
"type": "String"
}
},
"folder": {
"name": "StructuringAnalytics"
},
"annotations": []
},
"type": "Microsoft.DataFactory/factories/pipelines"
}
]
I get an error:
[
{
"errorCode": "BadRequest",
"message": "Operation on target Collect_SA_Data failed: The execution of template action 'Collect_SA_Data' failed: the result of the evaluation of 'foreach' expression '#pipeline().parameters.TableNames' is of type 'String'. The result must be a valid array.",
"failureType": "UserError",
"target": "Test_invoke1",
"details": ""
}
]
Input:
"pipeline": {
"referenceName": "MAIN_SA_copy1",
"type": "PipelineReference"
},
"waitOnCompletion": true,
"parameters": {
"DLFilePath": "extracts/StructuringAnalytics",
"TableNames": "[\"fom.FOMLineItem_manual\"]"
}
Please try updating your dynamic expression of ForEach Items as below:
{
"value": "#array(pipeline().parameters.TableNames)",
"type": "Expression"
}
Hope this helps.
I guess you were using the UI to set the pipeline and its parameters and I guess you expected to put the array parameter of the called pipeline as everywhere else like this:
(It is all my guess, because I just did exactly the same, with the same result)
The trick is to define the array in the Code (["table1", "table2"]):
The input in the UI will look like this:
Now it works!
It seems, that the Datafactory is otherwise treating the whole array as one element of some array. Hence, the solution with the array() function sometimes works.
It looks like a bug, defining array parameter input..
(Had to edit the answer, I first thought omiting the colons in the UI input would be enough)

How nested deployments are enforced

I have Azure ARM parent template which has nested deployment of Microsoft.Automation/automationAccounts which in turn has nested resource Configurations. I was able to successfully deploy entire template once, which in fact created configuration in automation Account. I manually deleted configuration inside automation account and tried running template again but this nested deployment is no longer triggered at all. There is no errors, just this nested deployment is not shown up at all in history. I would assume ARM thinks since it succeeded last time it does not need to deploy or something, not sure. What might be the problem?
Here is relevant parts of template. Neither ScaleSet no nested deployment WorkerNodeDSCConfiguration is triggered at all.
{
"type": "Microsoft.Compute/virtualMachineScaleSets",
"name": "[variables('namingInfix')]",
"location": "[resourceGroup().location]",
"apiVersion": "[variables('computeApiVersion')]",
"dependsOn": [
"[concat('Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'))]",
"[concat('Microsoft.Network/applicationGateways/', variables('appGwName'))]",
"[concat('Microsoft.Network/loadBalancers/', variables('loadBalancerName'))]",
"WorkerNodeDscConfiguration"
],
"sku": {
"name": "[parameters('vmSku')]",
"tier": "Standard",
"capacity": "[parameters('instanceCount')]"
},
"properties": {
"overprovision": "false",
"singlePlacementGroup": "true",
"upgradePolicy": {
"mode": "Automatic"
},
"virtualMachineProfile": {
"licenseType": "[parameters('LicenseType')]",
"storageProfile": {
"osDisk": {
"caching": "ReadWrite",
"createOption": "FromImage"
},
"dataDisks": [],
"imageReference": "[variables('imageReference')]"
},
"osProfile": {
"computerNamePrefix": "[variables('namingInfix')]",
"adminUsername": "[parameters('adminUsername')]",
"adminPassword": "[parameters('adminPassword')]"
},
"networkProfile": {
"networkInterfaceConfigurations": [
{
"name": "[variables('nicName')]",
"properties": {
"primary": "true",
"ipConfigurations": [
{
"name": "[variables('ipConfigName')]",
"properties": {
"subnet": {
"id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/virtualNetworks/', variables('virtualNetworkName'), '/subnets/', variables('subnetName'))]"
},
"loadBalancerBackendAddressPools": [
{
"id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/loadBalancers/', variables('loadBalancerName'), '/backendAddressPools/', variables('bePoolName'))]"
}
],
"loadBalancerInboundNatPools": [
{
"id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/loadBalancers/', variables('loadBalancerName'), '/inboundNatPools/', variables('natPoolName'))]"
}
],
"ApplicationGatewayBackendAddressPools": [
{
"id": "[concat('/subscriptions/', subscription().subscriptionId,'/resourceGroups/', resourceGroup().name, '/providers/Microsoft.Network/applicationGateways/', variables('appGwName'), '/backendAddressPools/', variables('appGwBePoolName'))]"
}
]
}
}
]
}
}
]
},
"extensionProfile": {
"extensions": [
{
"name": "Microsoft.Powershell.DSC",
"properties": {
"autoUpgradeMinorVersion": true,
"typeHandlerVersion": "2.72",
"type": "DSC",
"publisher": "Microsoft.Powershell",
//"forceUpdateTag": "[parameters('DSCExtensionTagVersion')]",
"settings": {
"configurationArguments": {
"RegistrationKey": {
"UserName": "PLACEHOLDER_DONOTUSE",
"Password": "[parameters('registrationKey')]"
},
"RegistrationUrl": "[parameters('registrationUrl')]",
"NodeConfigurationName": "swarmHost",
"RebootNodeIfNeeded": true,
"ConfigurationMode": "ApplyAndAutoCorrect"
}
}
}
}
]
}
}
}
},
{
"name": "swarmmanagerdeployment",
"type": "Microsoft.Resources/deployments",
"apiVersion": "2016-09-01",
"dependsOn": [],
"properties": {
"mode": "Incremental",
"templateLink": {
"uri": "[concat(parameters('_artifactsLocation'), '/', variables('swarmmanagerdeploymentTemplateFolder'), '/', variables('swarmmanagerdeploymentTemplateFileName'), parameters('_artifactsLocationSasToken'))]",
"contentVersion": "1.0.0.0"
},
"parameters": {
"swarmmanager1Name": { "value": "[parameters('swarmmanager1Name')]" },
"swarmmanager1VmSize": { "value": "[variables('swarmmanager1VmSize')]" },
"adminUsername": { "value": "[parameters('adminUsername')]" },
"adminPassword": { "value": "[parameters('adminPassword')]" },
"dockerswarmstorageaccountName": { "value": "[variables('dockerswarmstorageaccountName')]" },
"dockerswarmstorageaccountType": { "value": "[parameters('dockerswarmstorageaccountType')]" },
"swarmmanager1NicName": { "value": "[variables('swarmmanager1NicName')]" },
"swarmmanagerpublicIPName": { "value": "[variables('swarmmanagerpublicIPName')]" },
"swarmmanager1SubnetRef": { "value": "[variables('swarmmanager1SubnetRef')]" },
"swarmmanager1ImagePublisher": { "value": "[variables('swarmmanager1ImagePublisher')]" },
"swarmmanager1ImageOffer": { "value": "[variables('swarmmanager1ImageOffer')]" },
"windowsOSVersion": { "value": "[parameters('windowsOSVersion')]" },
"swarmmanager1StorageAccountContainerName": { "value": "[variables('swarmmanager1StorageAccountContainerName')]" },
"swarmmanager1OSDiskName": { "value": "[variables('swarmmanager1OSDiskName')]" },
"swarmmanagerpublicIPDnsName": { "value": "[variables('swarmmanagerpublicIPName')]" },
"RegistrationKey": { "value": "[parameters('registrationKey')]" },
"RegistrationUrl": { "value": "[parameters('registrationUrl')]" },
"LicenseType": { "value": "[parameters('LicenseType')]" },
"_artifactsLocationSasToken": { "value": "[parameters('_artifactsLocationSasToken')]" },
"_artifactsLocation": { "value": "[parameters('_artifactsLocation')]" },
"privateKey": { "value": "[parameters('privateKey')]" },
"serverCert": { "value": "[parameters('serverCert')]" },
"CACert": { "value": "[parameters('CACert')]" }
}
}
},
{
"name": "WorkerNodeDscConfiguration",
"type": "Microsoft.Resources/deployments",
"apiVersion": "2017-05-10",
"resourceGroup": "[parameters('automationAccountRGName')]",
"dependsOn": [],
"properties": {
"mode": "Incremental",
"template": {
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.1",
"resources": [
{
"apiversion": "2015-10-31",
"location": "[resourceGroup().location]",
"name": "[parameters('automationAccountName')]",
"type": "Microsoft.Automation/automationAccounts",
"properties": {
"sku": {
"name": "Basic"
}
},
"tags": {
},
"resources": [
{
"name": "swarmhost",
"type": "configurations",
"apiVersion": "2018-01-15",
"location": "[resourceGroup().location]",
"dependsOn": [
"[concat('Microsoft.Automation/automationAccounts/', parameters('AutomationAccountName'))]"
],
"properties": {
"state": "Published",
"overwrite": "true",
"Source": {
"type": "uri",
"value": "[parameters('WorkerNodeDSCConfigURL')]"
}
}
},
{
"name": "[guid(resourceGroup().id, deployment().name)]",
"type": "Compilationjobs",
"apiVersion": "2015-10-31",
"tags": {},
"dependsOn": [
"[concat('Microsoft.Automation/automationAccounts/', parameters('AutomationAccountName'))]",
"[concat('Microsoft.Automation/automationAccounts/', parameters('AutomationAccountName'),'/Configurations/swarmhost')]"
],
"properties": {
"configuration": {
"SwarmManagerURI": "[reference('swarmmanagerdeployment').outputs.returnedIPAddress.value]"
}
}
}
]
}
]
}
}
}
its pretty hard to tell what is going on, but you can verify if something is going on by changing some vmss property, for example, to something else and running the template. it will revert it back to what it is in the template. It might be pretty tricky with the automation account stuff, because I would think it wouldnt trigger the compilation job, because the guid is the same and the resource is there already. you would need to provide a new guid each time (ARM cant help you with this, you need to do some randomization externally). Generally I prefer to use powershell to configure automation accounts compared to ARM.
You can also verify that the nested deployment is being triggered by looking at its timestamp, it should change (and it will). The resource is always deployed, but depending on your situation its properties might stay intact.

Resources