ARM deployment fails when properties are passed an an object - azure

When deploying an ARM template using resource iteration, I'd like to pass the resource properties as an object.
Doing this would allow for a different set of parameters to exist within each element the copy array. The reason for this is because some properties may need to be conditionally included or excluded depending on the values of others. For example, in the case of an API Management product, the documentation states the following with regard to the subscriptionsLimit property -
Can be present only if subscriptionRequired property is present and has a value of false.
However, when deploying the example template below the deployment hangs in Azure. Looking in to the related events, I can see that the action the deploy the resource keeps failing with an Internal Server Error (500), but there are no additional details.
If I refer to each parameter in the properties object using variables('productsJArray')[copyIndex()].whatever then the deployment succeeds. However, this is undesirable as it means that every properties object would have to contain identical parameters, which is not always permissible and may cause the deployment to fail.
Example template
Note that I've output variables('productsJArray')[copyIndex()] and it is a valid object. I've even copied the output in to the template and deployed it successfully.
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"apiManagementServiceName": {
"type": "string",
"metadata": {
"description": "The name of the API Management instance."
}
},
"productsJson": {
"type": "string",
"metadata": {
"description": "A JSON representation of the Products to add."
}
}
},
"variables": {
"productsJArray": "[json(parameters('productsJson'))]"
},
"resources": [
{
"condition": "[greater(length(variables('productsJArray')), 0)]",
"type": "Microsoft.ApiManagement/service/products",
"name": "[concat(parameters('apiManagementServiceName'), '/', variables('productsJArray')[copyIndex()].name)]",
"apiVersion": "2018-06-01-preview",
"properties": "[variables('productsJArray')[copyIndex()]]",
"copy": {
"name": "productscopy",
"count": "[if(greater(length(variables('productsJArray')), 0), length(variables('productsJArray')), 1)]"
}
}
]
}
Example parameters
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"apiManagementServiceName": {
"value": "my-api-management"
},
"productsJson": {
"value": "[{\"name\":\"my-product\",\"displayName\":\"My Product\",\"description\":\"My product is awesome.\",\"state\":\"published\",\"subscriptionRequired\":true,\"approvalRequired\":false}]"
}
}
}
Output of variable 'productsJArray[0]'
"outputs": {
"properties": {
"type": "Object",
"value": {
"approvalRequired": false,
"description": "My product is awesome.",
"displayName": "My Product",
"name": "my-product",
"state": "published",
"subscriptionRequired": true
}
}
}

The issue here was that I was passing including name parameter along with other parameters when setting resource properties. This is obviously wrong, but it would have been helpful if MS had handled the error in a more human friendly way (guess they can't think of everything).
I've updated my incoming productsJson parameter -
[{\"name\":\"cs-automation\",\"properties\":{\"displayName\":\"CS Automation Subscription\",\"state\":\"published\",\"description\":\"Allows access to the ConveyorBot v1 API.\",\"subscriptionRequired\":true,\"approvalRequired\":false}}]
And I'm now passing only the required 'properties' -
"resources": [
{
"type": "Microsoft.ApiManagement/service/products",
"name": "[concat(parameters('apiManagementServiceName'), '/', variables('productsJArray')[copyIndex()].name)]",
"apiVersion": "2018-06-01-preview",
"properties": "[variables('productsJArray')[copyIndex()].properties]"
}
]

Related

Azure Lighthouse - Get the region for subscription level deployment

I want to onboard Azure Subscription into my tenant. For that, I created an ARM template with subscriptionDeploymentSchema. If I open this template from the portal, I get prompted to choose a region. Is there any way I get the value of that built-in field into my template and use it?
I want to display value of the "Region" field (built-in) in the "Region Name", which is my field. Is that possible?
I know for the resource group level deployment I can use resourceGroup().location, but that does not work here.
Part of my template:
{
"$schema": "https://schema.management.azure.com/schemas/2018-05-01/subscriptionDeploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"mspOfferName": {
"type": "string",
"metadata": {
"description": "Specify a unique name for your offer"
}
},
"mspOfferDescription": {
"type": "string",
"metadata": {
"description": "Name of the Managed Service Provider offering"
}
},
"regionName": {
"type": "string",
"metadata": {
"description": "Should have value of the 'Region' built-in field"
}
}
},
"resources": [
...
{
"name": "myNestedTemplate",
"type": "Microsoft.Resources/deployments",
"apiVersion": "2020-10-01",
"location": "<i-want-to-use-value-from-the-built-in-field>", // <-- I WANT TO USE REGION FROM THE BUILT-IN FIELD HERE
...
}
],
...
}
Is that even possible? Or should I create a field for user to provide a region? The second option does not make any sens, as you would be prompted to provide two locations.
You can use deployment().location to get the value of the Region field.

Resource [parameters('mgName')] Location must be an expression or 'global'

I am experimenting with Azure Management Groups Arm template.
As you can see in this link, I have this Arm template:
{
"$schema": "https://schema.management.azure.com/schemas/2019-08-01/managementGroupDeploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"mgName": {
"type": "string",
"defaultValue": "[concat('mg-', uniqueString(newGuid()))]"
}
},
"resources": [
{
"type": "Microsoft.Management/managementGroups",
"apiVersion": "2021-04-01",
"name": "[parameters('mgName')]",
"scope": "/",
"location": "eastus",
"properties": {}
}
],
"outputs": {
"output": {
"type": "string",
"value": "[parameters('mgName')]"
}
}
}
Saved as mg.json and it works fine.
Later I start experimenting with validating and testing Arm template using Test-AzTemplate (https://github.com/Azure/arm-ttk). When I run following command to test Arm Template:
Test-AzTemplate -TemplatePath .\mg.json
I get this test error:
[-] Resources Should Have Location (3 ms)
Resource [parameters('mgName')] Location must be an expression or 'global'
Now when I remove "location": "eastus", line form Arm template, the test does not fail and pass the test.
My Question:
Is this location in Management Group Arm required or not required? And why it is failing when it is part of Microsoft documentation! Any idea?
Location is not required in Management Group. As you can check this Azure Create Management Group REST API documentation, location is not needed here.
That's why in the template either you can remove the location or you can provide 'global' as the value, as the test command output specifies.

Is it possible to do an iterative string replace in an Azure ARM template?

I suspect it's not possible to do what I'm looking for but it's worth a shot!
I have a pipeline for provisioning Azure log query alert rules. The individual alert rules are defined as ARM parameter files, and I use a shared ARM template file to deploy them.
Here's a stripped down version of my template file with most of the parameters omitted.
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"logQuery": {
"type": "string",
"minLength": 1,
"metadata": {
"description": "Query to execute against the AI resource"
}
}
},
"variables": {
"appInsightsResourceId": "[concat(resourceGroup().id,'/providers/','microsoft.insights/components/', parameters('appInsightsResourceName'))]",
"actionGroupId": "[concat(resourceGroup().id,'/providers/','microsoft.insights/actionGroups/', parameters('actionGroupName'))]",
"linkToAiResource" : "[concat('hidden-link:', variables('appInsightsResourceId'))]"
},
"resources":[{
"name":"[parameters('alertName')]",
"type":"Microsoft.Insights/scheduledQueryRules",
"location": "northeurope",
"apiVersion": "2018-04-16",
"tags": {
"[variables('linkToAiResource')]": "Resource"
},
"properties":{
"description": "[parameters('alertDescription')]",
"enabled": "[parameters('isEnabled')]",
"source": {
"query": "[parameters('logQuery')]",
"dataSourceId": "[variables('appInsightsResourceId')]",
"queryType":"[parameters('logQueryType')]"
},
"schedule":{
"frequencyInMinutes": "[parameters('alertSchedule').Frequency]",
"timeWindowInMinutes": "[parameters('alertSchedule').Time]"
},
"action":{
"odata.type": "Microsoft.WindowsAzure.Management.Monitoring.Alerts.Models.Microsoft.AppInsights.Nexus.DataContracts.Resources.ScheduledQueryRules.AlertingAction",
"severity": "[parameters('alertSeverity')]",
"aznsAction":{
"actionGroup":"[array(variables('actionGroupId'))]"
},
"trigger":{
"thresholdOperator":"[parameters('alertTrigger').Operator]",
"threshold":"[parameters('alertTrigger').Threshold]"
}
}
}
}
]
}
You can see how I'm providing the App Insights query as a parameter, so my parameters file could look something like:
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"logQuery": {
"value": "requests | where resultCode >= 500"
}
}
}
However, these queries can be very long and hard to understand when viewing as an unbreakable JSON string. So I want to parametize this parameter (if you know what I mean) so that the key variables are defined and supplied separately. I was thinking about changing the parameters to something like this, introducing a new parameter holding an array of placeholder replacements for the parametized query...
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"logQueryVariables": [
{ "{minCode}": "500" }
],
"logQuery": {
"value": "requests | where resultCode >= {minCode}"
}
}
}
...then finding a way to iterate over the variables array and replace the placeholders in the logQuery parameter, I thought maybe I could use an ARM function or something. But I'm afraid to admit I'm stuck with this part. Is it possible to use the copy syntax to do something like this?
depends on what the end result should look like you can do this different ways, but I'd suggest against doing this in the template and suggest you do this outside of the template and feed in the result. If you really want to achieve exactly what you describe you can do that** with nested deployments. I dont think there is any other way of iterating through an array to build a string in ARM Templates.
This is the closest I could come. We do the input looping in powershell for secretObjects.
{
"type": "Microsoft.KeyVault/vaults/secrets",
"name": "[concat(variables('KeyVaultName'), '/',
variables('secretsObject').secrets[copyIndex()].secretName)]",
"apiVersion": "2018-02-14",
"dependsOn": [
"[concat('Microsoft.KeyVault/vaults/', variables('KeyVaultName'))]"
],
"copy": {
"name": "secretsCopy",
"count": "[length(variables('secretsObject').secrets)]"
},
"properties": {
"value": "[variables('secretsObject').secrets[copyIndex()].secretValue]"
}
}

Stream Analytics Job deployed as Azure Resource Manager (ARM) template

I am trying to setup an output EventHub for a Stream Analytics Job defined as a JSON template. Without the output bit the template is successfully deployed, however when adding the output definition it fails with:
Deployment failed. Correlation ID: <SOME_UUID>. {
"code": "BadRequest",
"message": "The JSON provided in the request body is invalid. Property 'eventHubName' value 'parameters('eh_name')' is not acceptable.",
"details": {
"code": "400",
"message": "The JSON provided in the request body is invalid. Property 'eventHubName' value 'parameters('eh_name')' is not acceptable.",
"correlationId": "<SOME_UUID>",
"requestId": "<SOME_UUID>"
}
}
I've defined the ARM template as:
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"location": {
"type": "string",
"defaultValue": "westeurope"
},
"hubName": {
"type": "string",
"defaultValue": "fooIotHub"
},
"eh_name": {
"defaultValue": "fooEhName",
"type": "String"
},
"eh_namespace": {
"defaultValue": "fooEhNamespace",
"type": "String"
},
"streamAnalyticsJobName": {
"type": "string",
"defaultValue": "fooStreamAnalyticsJobName"
}
},
"resources": [{
"type": "Microsoft.StreamAnalytics/StreamingJobs",
"apiVersion": "2016-03-01",
"name": "[parameters('streamAnalyticsJobName')]",
"location": "[resourceGroup().location]",
"properties": {
"sku": {
"name": "standard"
},
"outputErrorPolicy": "Drop",
"eventsOutOfOrderPolicy": "adjust",
"eventsOutOfOrderMaxDelayInSeconds": 0,
"eventsLateArrivalMaxDelayInSeconds": 86400,
"inputs": [{
"Name": "IoTHubInputLable",
"Properties": {
"DataSource": {
"Properties": {
"iotHubNamespace": "[parameters('hubName')]",
"sharedAccessPolicyKey": "[listkeys(resourceId('Microsoft.Devices/IotHubs/IotHubKeys',parameters('hubName'), 'iothubowner'),'2016-02-03').primaryKey]",
"sharedAccessPolicyName": "iothubowner",
"endpoint": "messages/events"
},
"Type": "Microsoft.Devices/IotHubs"
},
"Serialization": {
"Properties": {
"Encoding": "UTF8"
},
"Type": "Json"
},
"Type": "Stream"
}
}],
"transformation": {
"name": "Transformation",
"properties": {
"streamingUnits": 1,
"query": "<THE SQL-LIKE CODE FOR THE JOB QUERY>"
}
},
"outputs": [{
"name": "EventHubOutputLable",
"properties": {
"dataSource": {
"type": "Microsoft.ServiceBus/EventHub",
"properties": {
"eventHubName": "parameters('eh_name')",
"serviceBusNamespace": "parameters('eh_namespace')",
"sharedAccessPolicyName": "RootManageSharedAccessKey"
}
},
"serialization": {
"Properties": {
"Encoding": "UTF8"
}
}
}
}]
}
}]
}
Checking here https://learn.microsoft.com/en-us/azure/templates/microsoft.streamanalytics/streamingjobs
it looks like the structure of the JSON for the output is as the expected one (with the properties field along with the type).
I've figured out those "Event Hub properties" from the Chrome browser using Developer Tools and checking the details of the HTTP request "GetOutputs", otherwise I am not sure where I could see how to specify those properties? The structure looks quite similar to the one for the input IoT Hub (which is working), in that case using different lables for the properties related to the IoT Hub details.
Checking this blog post https://blogs.msdn.microsoft.com/david/2017/07/20/building-azure-stream-analytics-resources-via-arm-templates-part-2-the-template/
the output part is related to PowerBI and it looks like a different structure is used for the properties: outputPowerBISource, so I've tried to use for the Event Hub the field outputEventHubSource (from the checks using Chrome Developer Tools) instead of properties, but then I get this error:
Deployment failed. Correlation ID: <SOME_UUID>. {
"code": "BadRequest",
"message": "The JSON provided in the request body is invalid. Required property 'properties' not found in JSON. Path '', line 1, position 1419.",
"details": {
"code": "400",
"message": "The JSON provided in the request body is invalid. Required property 'properties' not found in JSON. Path '', line 1, position 1419.",
"correlationId": "<SOME_UUID>",
"requestId": "<SOME_UUID>"
}
}
The command I am using to deploy this template is the Azure CLI (from a Linux Debian machine):
az group deployment create \
--name "deployStreamAnalyticsJobs" \
--resource-group "MyRGName" \
--template-file ./templates/stream-analytics-jobs.json
How do I specify an output in an Azure Resource Manager (ARM) template for a Stream Analytics Job?
Any property that contains a parameter (or any expression that needs to be evaluated must contain square brackets, e.g.
"eventHubName": "[parameters('eh_name')]",
"serviceBusNamespace": "[parameters('eh_namespace')]",
Otherwise the literal value in the quotes is used.
That help?
I found out all parameters need to be wrapped in square brakets (as pointed out in the other answer to this question).
Also to dynamically retrieve the shared access policy key (or any other parameter for an existing resource like the Event Hub) a combination of functions like listKeys and resourceId etc must be used, see below for a full example of an Event Hub described as an output for a Stream Analytics Job.
In details:
the parameters defined for eventHubName and serviceBusNamespace must be evaluated using square brackets (see how I defined those parameters in the JSON example in the body of the question I asked above),
the shared access policy could be either an hardcoded string (or a parameter as before) like sharedAccessPolicyName or dynamically retrieved using "sharedAccessPolicyKey": "[listKeys(resourceId('Microsoft.EventHub/namespaces/eventhubs/authorizationRules', parameters('eh_namespace'), parameters('eh_name'), 'RootManageSharedAccessKey'),'2017-04-01').primaryKey]" for the sharedAccessPolicyKey (this is sensitive data and it should be protected avoiding hardcoding information as a plain string)
The following JSON configuration shows an existing Event Hub defined as an output defined for the Stream Analytics Job:
"outputs": [{
"Name": "EventHubOutputLable",
"Properties": {
"DataSource": {
"Type": "Microsoft.ServiceBus/EventHub",
"Properties": {
"eventHubName": "[parameters('eh_name')]",
"serviceBusNamespace": "[parameters('eh_namespace')]",
"sharedAccessPolicyKey": "[listKeys(resourceId('Microsoft.EventHub/namespaces/eventhubs/authorizationRules', parameters('eh_namespace'), parameters('eh_name'), 'RootManageSharedAccessKey'),'2017-04-01').primaryKey]",
"sharedAccessPolicyName": "RootManageSharedAccessKey"
}
},
"Serialization": {
"Properties": {
"Encoding": "UTF8"
},
"Type": "Json"
}
}
}]

Linked ARM templates result in Invalid Template

Let's start with what I'm trying to accomplish.
What I'd like to do is create an ARM template where I'm retrieving secrets from the Azure Key Vault, without specifying too much details on the specific Key Vault. Sounds easy enough and probably something which is implemented in every production system.
Doing a quick search I discovered the syntax for such a thing is the following.
"parameters": {
"adminPassword": {
"reference": {
"keyVault": {
"id": "[resourceId(subscription().subscriptionId, parameters('vaultResourceGroup'), 'Microsoft.KeyVault/vaults', parameters('vaultName'))]"
},
"secretName": "[parameters('secretName')]"
}
},
From what I gather you need to add this in an external template as the used methods can't be used in the 'main' method.
So I started creating a 'main' ARM template and a new template called appservice.json which contains all of the stuff necessary for my App Service, including the appSettings block which need the secrets from Key Vault.
In my main template I've done the following, as described in the documentation.
{
"apiVersion": "2017-05-10",
"name": "linkedTemplate",
"type": "Microsoft.Resources/deployments",
"properties": {
"mode": "Incremental",
"templateLink": {
"uri": "[uri(deployment().properties.templateLink.uri, 'appservice.json')]",
"contentVersion": "1.0.0.0"
},
However, when deploying I'm confronted with the following error.
"error": {
"code": "InvalidTemplate",
"message": "Unable to process template language expressions for resource '/subscriptions/ba49bae7-2b37-4504-914b-441763a2bcd3/resourceGroups/cfpexchange-jan-test/providers/Microsoft.Resources/deployments/linkedTemplate' at line '1' and column '1526'. 'The language expression property 'templateLink' doesn't exist, available properties are 'name, properties'.'"
}
I also tried the following because I noticed IntelliSense in Visual Studio told me properties doesn't exits and I should use templateLink directly.
{
"apiVersion": "2017-05-10",
"name": "linkedTemplate",
"type": "Microsoft.Resources/deployments",
"properties": {
"mode": "Incremental",
"templateLink": {
"uri": "[uri(deployment().templateLink.uri, 'appservice.json')]",
"contentVersion": "1.0.0.0"
},
This of course isn't right either.
"error": {
"code": "InvalidTemplate",
"message": "Unable to process template language expressions for resource '/subscriptions/ba49bae7-2b37-4504-914b-441763a2bcd3/resourceGroups/cfpexchange-jan-test/providers/Microsoft.Resources/deployments/linkedTemplate' at line '1' and column '1526'. 'The language expression property 'templateLink' doesn't exist, available properties are 'name, properties'.'"
}
And when using it as a variable, like in the documentation
"variables": {
"sharedTemplateUrl": "[uri(deployment().properties.templateLink.uri, 'shared-resources.json')]"
},
...
"properties": {
"mode": "Incremental",
"templateLink": {
"uri": "[variables('sharedTemplateUrl')]",
"contentVersion": "1.0.0.0"
},
I'm also getting an error.
2018-07-04T19:14:34.4204720Z ##[error]Deployment template validation failed: 'The template variable 'sharedTemplateUrl' is not valid: The language expression property 'templateLink' doesn't exist, available properties are 'template, parameters, mode, debugSetting, provisioningState'.. Please see https://aka.ms/arm-template-expressions for usage details.'.
At this time I'm a bit lost. From what I understand from the documentation it appears I'm doing everything correct. Apparently I'm not. Any ideas on how to continue with this?
For completeness, the two actual files which I'm using at this time:
The main ARM template: https://github.com/Jandev/CfpExchange/blob/b998bb0c49cf369b2f7584e20556aefb5224ace0/Deployment/CFPExchange.json
The linked template: https://github.com/Jandev/CfpExchange/blob/522f48aa19d19730e5474ce11ead57a48d330389/Deployment/appservice.json
There have been multiple iterations on it in order to try and fix it, but as mentioned noting appeared to be working so far.
First of all, this is how you are supposed to use KV in a nested template. Example with Admin password:
"adminPassword": {
"reference": {
"keyVault": {
"id": "kv_resource_id"
},
"secretName": "[concat('secret', copyindex(1))]"
}
},
This section supposed to be in the nested template parameters WHEN you invoke it (just look at the example link).
Your error seems to be in the variable. So the templateLink property is only available when you deploy your main template from the url, if you use local file to deploy main template it wont work.
Compare this to remote template execution:
New-AzureRmResourceGroupDeployment -ResourceGroupName xxx -TemplateUri 'https://paste.ee/d/XI1Rc/0'
As this is a remote url, it should show you the same output, but this time with a templateLink property.
Name Type Value
=============== ========================= ==========
test Object {
"name": "theDeploymentName",
"properties": {
"templateLink": {
"uri": "theRemoteTemplateUri"
},
"template": {
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {},
"variables": {},
"resources": [],
"outputs": {
"test": {
"type": "Object",
"value": "[deployment()]"
}
}
},
"parameters": {},
"mode": "Incremental",
"provisioningState": "Accepted"
}
}

Resources