ARM deployment multi-line string with escaped quotes - azure

I am trying to deploy this ARM template
/* example_template.json */
{
"$schema": "https://schema.management.azure.com/schemas/2019-08-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"var": {
"type": "string",
"defaultValue": "
echo ${VARIABLE}
",
"metadata": {
"description": "some description"
}
}
},
"resources": [],
"outputs": {
"ouput": {
"type": "string",
"value": "[string(parameters('var'))]"
}
}
}
which successfully outputs what I want:
"outputs": {
"ouput": {
"type": "String",
"value": "\n echo ${VARIABLE}\n "
}
}
The problem is, if I am trying to use this in a bash script, $VARIABLE may have a space in it, hence I need to output to be
"outputs": {
"ouput": {
"type": "String",
"value": "\n echo \"${USER}\"\n "
}
}
to prevent argument splitting.
So, I have tried to edit my template to include the quotes
/* example_template.json */
{
"$schema": "https://schema.management.azure.com/schemas/2019-08-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"var": {
"type": "string",
"defaultValue": "
echo \"${VARIABLE}\"
",
"metadata": {
"description": "some description"
}
}
},
"resources": [],
"outputs": {
"ouput": {
"type": "string",
"value": "[string(parameters('var'))]"
}
}
}
Which gives me a validate and create error:
> az deployment group validate -f example_template.json -g resource-group-name
Failed to parse 'example_template.json', please check whether it is a valid JSON format
This seems to only happen with the multi-line string - if I put the whole defaultValue on a single line i.e.,
"defaultValue": "echo \"${VARIABLE}\""
it is successful again.
I need to use a multiline string as this variable is for a long deployment script which would be infeasible to put on one-line.
I believe this is a bug due to the parser only failing with the multiline string, but am unsure where to report it!
Does anyone know what a possible solution to this could be?
Thanks,
Akhil

Actually there is no significance to do multi-line manually like you did.
As you can see, the type of this "var" parameter is "string", so it should be a string in "defaultValue", thus the reason you got the error message. Even it worked without error when using PowerShell command like "New-AzDeployment" ,but the result is same.
As you want to "use a multiline string as this variable is for a long deployment script", you can try do this to make the defaultValue to be a string:
"parameters": {
"var": {
"type": "string",
"defaultValue": "VARIABLE={value1}\n echo \"${VARIABLE}\"",
"metadata": {
"description": "some description"
}
}
},
The output should be:
VARIABLE={value1}
echo "${VARIABLE}"
Actually az command is based on python, Power Shell command is based on c#, which made them parse json differently.
About how Azure CLI parse json, it starts reading defaultValue string with ", and cannot find the opposite " in the same line, then the error occurs.
About how Power Shell parse json, it starts reading defaultValue string with :, and read the whole content included in that :.
Deploy ARM template with Power Shell.

Related

How to reference Keyvault Secret Tags from ARM template

I have an ARM template which syncs secret value from source Keyvault into Destination one.
I also want to sync secret tags, but ARM reference that I use for 'sourceKV.secret.tags' retrieval does not work
[reference(resourceId('subscriptionId', 'resourceGroup', 'Microsoft.KeyVault/vaults/secrets', 'SourceKV', 'Secret'), '2021-04-01-preview', 'Full').tags.tagName]
any ideas what can be the issue, or what is the correct form to retrieve tags during ARM template deployment?
These work for me:
"outputs": {
"tags": {
"type": "string",
"value": "[reference('/subscriptions/xxxx/resourceGroups/yyyy/providers/Microsoft.KeyVault/vaults/zzzz/secrets/mysecret', '2022-07-01', 'Full').tags]"
},
"tagValue": {
"type": "string",
"value": "[reference('/subscriptions/xxxx/resourceGroups/yyyy/providers/Microsoft.KeyVault/vaults/zzzz/secrets/mysecret', '2022-07-01', 'Full').tags.hello]"
},
"tagValue2": {
"type": "string",
"value": "[reference(resourceId(subscription().subscriptionId, resourceGroup().name, 'Microsoft.KeyVault/vaults/secrets', 'xxxx', 'mysecret'), '2021-04-01-preview', 'Full').tags.hello]"
}
}
Will result in:
"outputs": {
"tagValue": {
"type": "String",
"value": "world"
},
"tagValue2": {
"type": "String",
"value": "world"
},
"tags": {
"type": "Object",
"value": {
"hello": "world"
}
}
}
Also works with the API version you used. It is important that you use 'Full', otherwise you won't get the tags. Note that you can use this syntax anywhere in your template. I just used it in the outputs because it is good for testing.
As I found out it is not possible to use Reference function for setting tags property value for keyvault as valid usages state
reference func only works if it is used inside properties block or for outputs; but as tags are not part of properties instead of returning value reference fun returns just string "reference(resource...)"

Is it possible to do an iterative string replace in an Azure ARM template?

I suspect it's not possible to do what I'm looking for but it's worth a shot!
I have a pipeline for provisioning Azure log query alert rules. The individual alert rules are defined as ARM parameter files, and I use a shared ARM template file to deploy them.
Here's a stripped down version of my template file with most of the parameters omitted.
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"logQuery": {
"type": "string",
"minLength": 1,
"metadata": {
"description": "Query to execute against the AI resource"
}
}
},
"variables": {
"appInsightsResourceId": "[concat(resourceGroup().id,'/providers/','microsoft.insights/components/', parameters('appInsightsResourceName'))]",
"actionGroupId": "[concat(resourceGroup().id,'/providers/','microsoft.insights/actionGroups/', parameters('actionGroupName'))]",
"linkToAiResource" : "[concat('hidden-link:', variables('appInsightsResourceId'))]"
},
"resources":[{
"name":"[parameters('alertName')]",
"type":"Microsoft.Insights/scheduledQueryRules",
"location": "northeurope",
"apiVersion": "2018-04-16",
"tags": {
"[variables('linkToAiResource')]": "Resource"
},
"properties":{
"description": "[parameters('alertDescription')]",
"enabled": "[parameters('isEnabled')]",
"source": {
"query": "[parameters('logQuery')]",
"dataSourceId": "[variables('appInsightsResourceId')]",
"queryType":"[parameters('logQueryType')]"
},
"schedule":{
"frequencyInMinutes": "[parameters('alertSchedule').Frequency]",
"timeWindowInMinutes": "[parameters('alertSchedule').Time]"
},
"action":{
"odata.type": "Microsoft.WindowsAzure.Management.Monitoring.Alerts.Models.Microsoft.AppInsights.Nexus.DataContracts.Resources.ScheduledQueryRules.AlertingAction",
"severity": "[parameters('alertSeverity')]",
"aznsAction":{
"actionGroup":"[array(variables('actionGroupId'))]"
},
"trigger":{
"thresholdOperator":"[parameters('alertTrigger').Operator]",
"threshold":"[parameters('alertTrigger').Threshold]"
}
}
}
}
]
}
You can see how I'm providing the App Insights query as a parameter, so my parameters file could look something like:
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"logQuery": {
"value": "requests | where resultCode >= 500"
}
}
}
However, these queries can be very long and hard to understand when viewing as an unbreakable JSON string. So I want to parametize this parameter (if you know what I mean) so that the key variables are defined and supplied separately. I was thinking about changing the parameters to something like this, introducing a new parameter holding an array of placeholder replacements for the parametized query...
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"logQueryVariables": [
{ "{minCode}": "500" }
],
"logQuery": {
"value": "requests | where resultCode >= {minCode}"
}
}
}
...then finding a way to iterate over the variables array and replace the placeholders in the logQuery parameter, I thought maybe I could use an ARM function or something. But I'm afraid to admit I'm stuck with this part. Is it possible to use the copy syntax to do something like this?
depends on what the end result should look like you can do this different ways, but I'd suggest against doing this in the template and suggest you do this outside of the template and feed in the result. If you really want to achieve exactly what you describe you can do that** with nested deployments. I dont think there is any other way of iterating through an array to build a string in ARM Templates.
This is the closest I could come. We do the input looping in powershell for secretObjects.
{
"type": "Microsoft.KeyVault/vaults/secrets",
"name": "[concat(variables('KeyVaultName'), '/',
variables('secretsObject').secrets[copyIndex()].secretName)]",
"apiVersion": "2018-02-14",
"dependsOn": [
"[concat('Microsoft.KeyVault/vaults/', variables('KeyVaultName'))]"
],
"copy": {
"name": "secretsCopy",
"count": "[length(variables('secretsObject').secrets)]"
},
"properties": {
"value": "[variables('secretsObject').secrets[copyIndex()].secretValue]"
}
}

ARM nested templates: _artifactLocation parameter is not populated correctly when deploying template with nested templates

I am trying to figure out how nested templates work and I have the below templates. I am trying to deploy from VS using the VS deploy mechanism:
Right click on the project > Deploy > New
"Artifact storage account" field is prepopulated with "Automatically create a storage account" and I leave it like that
Click on Deploy button
If you take a look in HelloWorldParent.json template in variables you will see two variables "nestedTemplateUri" and "nestedTemplateUriWithBlobContainerName".
It is my understanding that "nestedTemplateUri" should contain the "blob container name" but that doesn't seem to be the case.
If I deploy with resources > properties > templateLink > "uri": "[variables('nestedTemplateUri')]"
The deployment fails with:
Error: Code=InvalidContentLink; Message=Unable to download deployment
content from
'https://********.blob.core.windows.net/NestedTemplates/HelloWorld.json?sv=2017-07-29&sr=c&sig=ZCJAoOdp08qDWxbzKbXSZzX1VBCf7%2FNSt4aIznFCTPQ%3D&se=2019-03-12T03:39:09Z&sp=r'
The Storage Account is created, the templates, parameters and PS1 script are uploaded
A new deployment is not created in resource group / deployments
If I deploy with resources > properties > templateLink > "uri": "[variables('nestedTemplateUriWithBlobContainerName')]"
The deployments succeeds.
Any idea? Any help is highly appreciated!
HelloWorldParent.json
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"_artifactsLocation": {
"type": "string",
"metadata": {
"description": "The base URI where artifacts required by this template are located including a trailing '/'"
}
},
"_artifactsLocationSasToken": {
"type": "securestring",
"metadata": {
"description": "The sasToken required to access _artifactsLocation. When the template is deployed using the accompanying scripts, a sasToken will be automatically generated. Use the defaultValue if the staging location is not secured."
},
"defaultValue": ""
}
},
"variables": {
"blobContainerName": "[concat(resourceGroup().name, '-stageartifacts/')]",
"nestedTemplateUriWithBlobContainerName": "[uri(parameters('_artifactsLocation'), concat(variables('blobContainerName'), 'NestedTemplates/HelloWorld.json', parameters('_artifactsLocationSasToken')))]",
"nestedTemplateUri": "[uri(parameters('_artifactsLocation'), concat('NestedTemplates/HelloWorld.json', parameters('_artifactsLocationSasToken')))]"
},
"resources": [
{
"apiVersion": "2017-05-10",
"name": "linkedTemplate",
"type": "Microsoft.Resources/deployments",
"properties": {
"mode": "incremental",
"templateLink": {
"uri": "[variables('nestedTemplateUri')]",
"contentVersion": "1.0.0.0"
}
}
}
],
"outputs": {
"messageFromLinkedTemplate": {
"type": "string",
"value": "[reference('linkedTemplate').outputs.greetingMessage.value]"
},
"_artifactsLocation": {
"type": "string",
"value": "[parameters('_artifactsLocation')]"
}
}
}
HelloWorldParent.parameters.json
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
}
}
NestedTemplates/HelloWorld.json
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {},
"variables": {},
"resources": [],
"outputs": {
"greetingMessage": {
"value": "Hello World (1)",
"type": "string"
}
}
}
Unfortunately VS is a bit "dated" in it's support for your scenario... the problem is that you're using the URI function and the _artifactsLocation does not have a trailing slash. So you have a few options to fix:
1) in the PS1 file in VS there is a line that looks like this:
$OptionalParameters[$ArtifactsLocationName] = $StorageAccount.Context.BlobEndPoint + $StorageContainerName
If you change it to this (add a trailing /):
$OptionalParameters[$ArtifactsLocationName] = $StorageAccount.Context.BlobEndPoint + $StorageContainerName + "/"
It should work - alternatively you can just replace the entire script with this one: https://github.com/Azure/azure-quickstart-templates/blob/master/Deploy-AzureResourceGroup.ps1
Note that if you have other templates that work without the trailing slash, this will be a breaking change.
2) use concat() to create the uri instead of the uri() function. You still have to know whether there is a trailing slash but this change can be done in the template and not the PS1 file.
"nestedTemplateUri": "[concat(parameters('_artifactsLocation'), '/NestedTemplates/HelloWorld.json', parameters('_artifactsLocationSasToken'))]"
Either should work.

ARM deployment fails when properties are passed an an object

When deploying an ARM template using resource iteration, I'd like to pass the resource properties as an object.
Doing this would allow for a different set of parameters to exist within each element the copy array. The reason for this is because some properties may need to be conditionally included or excluded depending on the values of others. For example, in the case of an API Management product, the documentation states the following with regard to the subscriptionsLimit property -
Can be present only if subscriptionRequired property is present and has a value of false.
However, when deploying the example template below the deployment hangs in Azure. Looking in to the related events, I can see that the action the deploy the resource keeps failing with an Internal Server Error (500), but there are no additional details.
If I refer to each parameter in the properties object using variables('productsJArray')[copyIndex()].whatever then the deployment succeeds. However, this is undesirable as it means that every properties object would have to contain identical parameters, which is not always permissible and may cause the deployment to fail.
Example template
Note that I've output variables('productsJArray')[copyIndex()] and it is a valid object. I've even copied the output in to the template and deployed it successfully.
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"apiManagementServiceName": {
"type": "string",
"metadata": {
"description": "The name of the API Management instance."
}
},
"productsJson": {
"type": "string",
"metadata": {
"description": "A JSON representation of the Products to add."
}
}
},
"variables": {
"productsJArray": "[json(parameters('productsJson'))]"
},
"resources": [
{
"condition": "[greater(length(variables('productsJArray')), 0)]",
"type": "Microsoft.ApiManagement/service/products",
"name": "[concat(parameters('apiManagementServiceName'), '/', variables('productsJArray')[copyIndex()].name)]",
"apiVersion": "2018-06-01-preview",
"properties": "[variables('productsJArray')[copyIndex()]]",
"copy": {
"name": "productscopy",
"count": "[if(greater(length(variables('productsJArray')), 0), length(variables('productsJArray')), 1)]"
}
}
]
}
Example parameters
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"apiManagementServiceName": {
"value": "my-api-management"
},
"productsJson": {
"value": "[{\"name\":\"my-product\",\"displayName\":\"My Product\",\"description\":\"My product is awesome.\",\"state\":\"published\",\"subscriptionRequired\":true,\"approvalRequired\":false}]"
}
}
}
Output of variable 'productsJArray[0]'
"outputs": {
"properties": {
"type": "Object",
"value": {
"approvalRequired": false,
"description": "My product is awesome.",
"displayName": "My Product",
"name": "my-product",
"state": "published",
"subscriptionRequired": true
}
}
}
The issue here was that I was passing including name parameter along with other parameters when setting resource properties. This is obviously wrong, but it would have been helpful if MS had handled the error in a more human friendly way (guess they can't think of everything).
I've updated my incoming productsJson parameter -
[{\"name\":\"cs-automation\",\"properties\":{\"displayName\":\"CS Automation Subscription\",\"state\":\"published\",\"description\":\"Allows access to the ConveyorBot v1 API.\",\"subscriptionRequired\":true,\"approvalRequired\":false}}]
And I'm now passing only the required 'properties' -
"resources": [
{
"type": "Microsoft.ApiManagement/service/products",
"name": "[concat(parameters('apiManagementServiceName'), '/', variables('productsJArray')[copyIndex()].name)]",
"apiVersion": "2018-06-01-preview",
"properties": "[variables('productsJArray')[copyIndex()].properties]"
}
]

Stream Analytics Job deployed as Azure Resource Manager (ARM) template

I am trying to setup an output EventHub for a Stream Analytics Job defined as a JSON template. Without the output bit the template is successfully deployed, however when adding the output definition it fails with:
Deployment failed. Correlation ID: <SOME_UUID>. {
"code": "BadRequest",
"message": "The JSON provided in the request body is invalid. Property 'eventHubName' value 'parameters('eh_name')' is not acceptable.",
"details": {
"code": "400",
"message": "The JSON provided in the request body is invalid. Property 'eventHubName' value 'parameters('eh_name')' is not acceptable.",
"correlationId": "<SOME_UUID>",
"requestId": "<SOME_UUID>"
}
}
I've defined the ARM template as:
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"location": {
"type": "string",
"defaultValue": "westeurope"
},
"hubName": {
"type": "string",
"defaultValue": "fooIotHub"
},
"eh_name": {
"defaultValue": "fooEhName",
"type": "String"
},
"eh_namespace": {
"defaultValue": "fooEhNamespace",
"type": "String"
},
"streamAnalyticsJobName": {
"type": "string",
"defaultValue": "fooStreamAnalyticsJobName"
}
},
"resources": [{
"type": "Microsoft.StreamAnalytics/StreamingJobs",
"apiVersion": "2016-03-01",
"name": "[parameters('streamAnalyticsJobName')]",
"location": "[resourceGroup().location]",
"properties": {
"sku": {
"name": "standard"
},
"outputErrorPolicy": "Drop",
"eventsOutOfOrderPolicy": "adjust",
"eventsOutOfOrderMaxDelayInSeconds": 0,
"eventsLateArrivalMaxDelayInSeconds": 86400,
"inputs": [{
"Name": "IoTHubInputLable",
"Properties": {
"DataSource": {
"Properties": {
"iotHubNamespace": "[parameters('hubName')]",
"sharedAccessPolicyKey": "[listkeys(resourceId('Microsoft.Devices/IotHubs/IotHubKeys',parameters('hubName'), 'iothubowner'),'2016-02-03').primaryKey]",
"sharedAccessPolicyName": "iothubowner",
"endpoint": "messages/events"
},
"Type": "Microsoft.Devices/IotHubs"
},
"Serialization": {
"Properties": {
"Encoding": "UTF8"
},
"Type": "Json"
},
"Type": "Stream"
}
}],
"transformation": {
"name": "Transformation",
"properties": {
"streamingUnits": 1,
"query": "<THE SQL-LIKE CODE FOR THE JOB QUERY>"
}
},
"outputs": [{
"name": "EventHubOutputLable",
"properties": {
"dataSource": {
"type": "Microsoft.ServiceBus/EventHub",
"properties": {
"eventHubName": "parameters('eh_name')",
"serviceBusNamespace": "parameters('eh_namespace')",
"sharedAccessPolicyName": "RootManageSharedAccessKey"
}
},
"serialization": {
"Properties": {
"Encoding": "UTF8"
}
}
}
}]
}
}]
}
Checking here https://learn.microsoft.com/en-us/azure/templates/microsoft.streamanalytics/streamingjobs
it looks like the structure of the JSON for the output is as the expected one (with the properties field along with the type).
I've figured out those "Event Hub properties" from the Chrome browser using Developer Tools and checking the details of the HTTP request "GetOutputs", otherwise I am not sure where I could see how to specify those properties? The structure looks quite similar to the one for the input IoT Hub (which is working), in that case using different lables for the properties related to the IoT Hub details.
Checking this blog post https://blogs.msdn.microsoft.com/david/2017/07/20/building-azure-stream-analytics-resources-via-arm-templates-part-2-the-template/
the output part is related to PowerBI and it looks like a different structure is used for the properties: outputPowerBISource, so I've tried to use for the Event Hub the field outputEventHubSource (from the checks using Chrome Developer Tools) instead of properties, but then I get this error:
Deployment failed. Correlation ID: <SOME_UUID>. {
"code": "BadRequest",
"message": "The JSON provided in the request body is invalid. Required property 'properties' not found in JSON. Path '', line 1, position 1419.",
"details": {
"code": "400",
"message": "The JSON provided in the request body is invalid. Required property 'properties' not found in JSON. Path '', line 1, position 1419.",
"correlationId": "<SOME_UUID>",
"requestId": "<SOME_UUID>"
}
}
The command I am using to deploy this template is the Azure CLI (from a Linux Debian machine):
az group deployment create \
--name "deployStreamAnalyticsJobs" \
--resource-group "MyRGName" \
--template-file ./templates/stream-analytics-jobs.json
How do I specify an output in an Azure Resource Manager (ARM) template for a Stream Analytics Job?
Any property that contains a parameter (or any expression that needs to be evaluated must contain square brackets, e.g.
"eventHubName": "[parameters('eh_name')]",
"serviceBusNamespace": "[parameters('eh_namespace')]",
Otherwise the literal value in the quotes is used.
That help?
I found out all parameters need to be wrapped in square brakets (as pointed out in the other answer to this question).
Also to dynamically retrieve the shared access policy key (or any other parameter for an existing resource like the Event Hub) a combination of functions like listKeys and resourceId etc must be used, see below for a full example of an Event Hub described as an output for a Stream Analytics Job.
In details:
the parameters defined for eventHubName and serviceBusNamespace must be evaluated using square brackets (see how I defined those parameters in the JSON example in the body of the question I asked above),
the shared access policy could be either an hardcoded string (or a parameter as before) like sharedAccessPolicyName or dynamically retrieved using "sharedAccessPolicyKey": "[listKeys(resourceId('Microsoft.EventHub/namespaces/eventhubs/authorizationRules', parameters('eh_namespace'), parameters('eh_name'), 'RootManageSharedAccessKey'),'2017-04-01').primaryKey]" for the sharedAccessPolicyKey (this is sensitive data and it should be protected avoiding hardcoding information as a plain string)
The following JSON configuration shows an existing Event Hub defined as an output defined for the Stream Analytics Job:
"outputs": [{
"Name": "EventHubOutputLable",
"Properties": {
"DataSource": {
"Type": "Microsoft.ServiceBus/EventHub",
"Properties": {
"eventHubName": "[parameters('eh_name')]",
"serviceBusNamespace": "[parameters('eh_namespace')]",
"sharedAccessPolicyKey": "[listKeys(resourceId('Microsoft.EventHub/namespaces/eventhubs/authorizationRules', parameters('eh_namespace'), parameters('eh_name'), 'RootManageSharedAccessKey'),'2017-04-01').primaryKey]",
"sharedAccessPolicyName": "RootManageSharedAccessKey"
}
},
"Serialization": {
"Properties": {
"Encoding": "UTF8"
},
"Type": "Json"
}
}
}]

Resources