Azure ARM templates - Stream Analytics identity.principalId as output - azure

I have a working ARM Template for a Stream Analytics job which creates it's own 'managed identity' which can be queried using AZ CLI, e.g.:
'az stream-analytics job show -g <resource_group> -n <stream_analytics_job_name> -o json --query 'identity.principalId'
(Deployment is via Terraform 'azurerm_template_deployment' module).
Added an ARM template output to return this key using 'listkeys':
"outputs": {
"principalId": {
"type": "string",
"value": "[listkeys(resourceId('Microsoft.StreamAnalytics/streamingjobs', parameters('StreamAnalyticsJobName')), parameters('ASAApiVersion')).identity.principalId]"
}
}
}
Now the ARM deployment fails as seen in the resource group deployments list with:
{
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.",
"details": [
{
"code": "NotFound",
"message": "{\r\n \"code\": \"NotFound\",\r\n \"message\": \"The webpage cannot be found.\",\r\n \"details\": {\r\n \"code\": \"404\",\r\n \"message\": \"The webpage cannot be found.\",\r\n \"correlationId\": \"<redacted>\",\r\n \"requestId\": \"<redacted>\"\r\n }\r\n}"
}
]
}
From the resource group Activity Log, under the 'Write Steam Analytics Job', there are failed operations for listkeys - none of which give any further clues.
'identity.principalId' found for this Stream Analytics job via https://resources.azure.com:
...etc...
"identity": {
"principalId": "<redacted>",
"tenantId": "<redacted>",
"type": "SystemAssigned"
},
Have also tried the following ARM template output with the same result:
"outputs": {
"principalId": {
"type": "string",
"value": "[listkeys(resourceId('Microsoft.StreamAnalytics/streamingjobs', parameters('StreamAnalyticsJobName')), parameters('ASAApiVersion')).principalId]"
}
}
Similar issues found but none for Stream Analytics:
Azure ARM templates : DocumentDB primaryMasterKey as OUTPUT
How to return Redis primaryKey via ARM template output?
https://devkimchi.com/2018/01/05/list-of-access-keys-from-output-values-after-arm-template-deployment/
Research:
https://learn.microsoft.com/en-us/azure/azure-resource-manager/templates/template-outputs?tabs=azure-powershell
https://learn.microsoft.com/en-us/azure/stream-analytics/powerbi-output-managed-identity
https://www.terraform.io/docs/providers/azurerm/r/template_deployment.html
Any help much appreciated, thanks.

Came across the answer elsewhere (How to get Principal Id in app service using Arm template?) as I was trying to obtain a principal ID not a key (my bad).
Using the following ARM Template output worked:
"outputs": {
"principalId": {
"type": "string",
"value": "[reference(resourceId('Microsoft.StreamAnalytics/streamingjobs', parameters('StreamAnalyticsJobName')), parameters('ASAApiVersion'), 'Full').identity.principalId]"
}
}

Related

ARM Template - Deployment slot keyvault permissions

Trying to assign permissions for the app service deployment slot to keyvault and having a hell of a time getting it
{
"tenantId": "[subscription().tenantId]",
"objectId": "[resourceId('Microsoft.Web/sites/slots', parameters('azureAppService').webSiteName, 'DEV').identity.principalId]",
"permissions": {
"secrets": [
"Get"
]
}
}
Not sure what I'm doing wrong here, the template validation goes through, but upon deployment but I get an error. How do I specify the resource ID for the deployment slot?
Here is the error
{
"status": "Failed",
"error": {
"code": "InvalidTemplate",
"message": "Unable to process template language expressions for resource '/subscriptions/---/resourceGroups/Test/providers/Microsoft.KeyVault/vaults/KEYVAULT-TEST' at line '447' and column '9'. 'The language expression property 'identity' can't be evaluated.'",
"additionalInfo": [
{
"type": "TemplateViolation",
"info": {
"lineNumber": 447,
"linePosition": 9,
"path": ""
}
}
]
}
}
You would need to use the reference function (see documentation):
[reference(resourceId('Microsoft.Web/sites/slots', parameters('azureAppService').webSiteName, 'DEV'), '2022-03-01', 'full').identity.principalId]
"[reference(resourceId('Microsoft.Web/sites/slots', parameters('azureAppService').webSiteName, 'DEV'), '2020-06-01', 'full').identity.principalId]",
So it seems that even tho I added this to the deployment slots
"identity": {
"type": "SystemAssigned"
},
It wasn't creating the systemassigned identity. Once I manually created it and then redeployed the ARM template, it worked.

ARM template - bad request failed when deploying a linked service

I am trying to deploy a synapse instance via an ARM template and the deployment is successful via the Azure DevOps portal, but when I try to deploy the same template with an Azure Keyvault linked service I encounter the following error:
##[error]At least one resource deployment operation failed. Please list deployment
operations for details. Please see https://aka.ms/DeployOperations for usage details.
##[error]Details:
##[error]BadRequest:
After inspecting the activity logs from the Synapse instance I found out the following:
"resourceGroupName": "platform-test-group",
"resourceProviderName": {
"value": "Microsoft.Synapse",
"localizedValue": "Microsoft.Synapse"
},
"resourceType": {
"value": "Microsoft.Synapse/workspaces/linkedservices",
"localizedValue": "Microsoft.Synapse/workspaces/linkedservices"
},
"resourceId": "/subscriptions/xxxx-xxxx-xxxx-xxxx/resourcegroups/platform-test-group/providers/Microsoft.Synapse/workspaces/synapsedataapp/linkedservices/AzureKeyVault",
"status": {
"value": "Failed",
"localizedValue": "Failed"
},
"subStatus": {
"value": "NotFound",
"localizedValue": "Not Found (HTTP Status Code: 404)"
},
"submissionTimestamp": "2022-02-01T02:30:31.1471914Z",
"subscriptionId": "xxxx-xxxx-xxxx-xxxx",
"tenantId": "0f44c5d4-xxxx-xxxx-xxxxx",
"properties": {
"statusCode": "NotFound",
"serviceRequestId": null,
"statusMessage": "{\"error\":{\"code\":\"BadRequest\",\"message\":\"\"}}",
"eventCategory": "Administrative",
"entity": "/subscriptions/xxxx-xxxx-xxxx-xxxx/resourcegroups/platform-test-group/providers/Microsoft.Synapse/workspaces/synapsedataapp/linkedservices/AzureKeyVault",
"message": "Microsoft.Synapse/workspaces/linkedservices/write",
"hierarchy": "xxxx-xxxx-xxxx-xxxx/Enterprise/Group/Group-Test/xxxx-xxxx-xxxx-xxxx"
},
"relatedEvents": []
}
As you can see, the 404 error appears when the template tries to deploy to the tenant id which is not found, however, when I deploy the keyvault via the synapse UI I encounter no error.
Below is the code snippet that I use in my ARM template to deploy the keyvault to the synapse instance:
{
"name": "[concat(variables('workspaceName'), '/AzureKeyVault')]",
"type": "Microsoft.Synapse/workspaces/linkedservices",
"apiVersion": "2021-06-01-preview",
"properties": {
"annotations": [],
"type": "AzureKeyVault",
"typeProperties": {
"baseUrl": "https://data-test-kv.vault.azure.net/"
}
},
"dependsOn": [
"[variables('workspaceName')]"
]
}
Am I missing some kind of permission or connection that I need to enable? Why am I able to deploy successfully through the UI but not through the ARM template? Any comment or suggestion is greatly valued, so please feel free to comment or improve this question.
I had to contact Microsoft support and their reply was the following:
ARM templates cannot be used to create a linked service. This is due to the fact that linked services are not ARM resources, for examples, synapse workspaces, storage account, virtual networks, etc. Instead, a linked service is classified as an artifact. To still complete the task at hand, you will need to use the Synapse REST API or PowerShell. Below is the link that provides guidance on how to use the API. https://learn.microsoft.com/en-us/powershell/module/az.synapse/set-azsynapselinkedservice?view=azps-7.1.0
This limitation in ARM is applied only to Synapse and they might fix this in the future.
Additional references:
https://feedback.azure.com/d365community/idea/05e41bf1-0925-ec11-b6e6-000d3a4f07b8
https://feedback.azure.com/d365community/idea/48f1bf78-2985-ec11-a81b-6045bd7956bb
In Synapse unlike ADF, linked-services are not part of arm-templates. They are called artifacts and it comprises: Note Books, Spark Definitions, Linked Services, Pipelines etc.
You can find the full article here: https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/how-to-use-ci-cd-integration-to-automate-the-deploy-of-a-synapse/ba-p/2248060
In short, first, deploy Synapse using arm templates. And then set up the linked services:
- task: Synapse workspace deployment#1
displayName: 'Setup:Synapse KeyVault Linked Service'
inputs:
TemplateFile: '$(Build.Repository.LocalPath)/TemplateForWorkspace.json'
ParametersFile: '$(Build.Repository.LocalPath)/TemplateParametersForWorkspace.json'
azureSubscription: '${{ parameters.environments.serviceConnectionId }}'
ResourceGroupName: '$(computeResourceGroupName)'
TargetWorkspaceName: '$(synapseWorkspaceName)'
DeleteArtifactsNotInTemplate: true
OverrideArmParameters: |
synapseLinkedServiceKV: $(synapseLinkedServiceKV)
workspaceName: $(synapseWorkspaceName)
Environment: 'prod'
TemplateForWorkspace.json:
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"workspaceName": {
"type": "string"
},
"synapseLinkedServiceKV": {
"type": "string"
}
},
"variables": {
"workspaceId": "[concat('Microsoft.Synapse/workspaces/', parameters('workspaceName'))]"
},
"resources": [
{
"name": "[concat(parameters('workspaceName'), '/' , parameters('synapseLinkedServiceKV'))]",
"type": "Microsoft.Synapse/workspaces/linkedServices",
"apiVersion": "2019-06-01-preview",
"properties": {
"type": "AzureKeyVault",
"typeProperties": {
"baseUrl": "[concat('https://', parameters('synapseLinkedServiceKV'), '.vault.azure.net/')]"
},
"annotations": [],
"description": "Linked Service to Azure KeyVault. KeyVault is used to primarily fetch secrets"
},
"dependsOn": []
}
]
}
TemplateParametersForWorkspace.json:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"workspaceName": {
"value": ""
},
"synapseLinkedServiceKV": {
"value": ""
}
}
}
It deletes the existing artifacts and deploys the one above. You would first need to install the task extension on your Azure Devops for Synapse workspace deployment#1
Note above template was auto-generated. In synapse studio, goto Git Configuration and point it to your repo. It will submit the changes to the branch workspace_publish. You can copy and build on top of the specific artifact code.

How to create unique Guid at each deployment or hide the parameter newGuid() function from user in ARM template?

I need unique identifier at each custom deployment using ARM template for assigning resource name(uniqueName) which should be globally unique. As per documentation newGuid() returns a value in the format of a globally unique identifier. This function can only be used in the default value for a parameter.
As newGuid() function can be called in only parameters section, but I don't want to give the input block to user, because user can edit the field while deploying this, so how can I hide that from user or is there any other way to create same unique guid globally at each deployment?
I have tried creating same unique guid using this in variables section, but it works only for few times of deployment. I'm not sure deployment issue but it may possible because guid function does not make unique field all the time.
"variables": {
"uniqueName":"[guid(resourceGroup().id, deployment().name)]"
}
I have this template.
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"appname": {
"defaultValue": "xyz",
"type": "String"
},
"uniqueName": {
"defaultValue": "[newGuid()]",
"type": "String"
},
"myIdentity": {
"type": "String"
}
},
"variables": {
"location": "[resourceGroup().location]",
"ResourceGroupName": "[resourceGroup().name]"
},
"resources": [
{
"type": "Microsoft.Resources/deploymentScripts",
"apiVersion": "2019-10-01-preview",
"name": "[parameters('uniqueName')]",
"location": "[variables('location')]",
"kind": "AzureCLI",
"identity": {
"type": "UserAssigned",
"userAssignedIdentities": {
"[resourceID('Microsoft.ManagedIdentity/userAssignedIdentities/', parameters('myIdentity'))]": {}
}
},
"properties": {
"AzCliVersion": "2.0.80",
"timeout": "PT10M",
"arguments": "[parameters('appname')]",
"cleanupPreference": "OnSuccess",
"retentionInterval": "P1D",
"supportingScriptUris": [
"https://some-uri/test.sh"
],
"scriptContent": "[concat('./test.sh ', string(parameters('appname')), ' > $AZ_SCRIPTS_OUTPUT_PATH')]"
}
}
],
"outputs": {
"result": {
"type": "String",
"value": "[base64(string(reference(parameters('resourceName')).outputs))]"
}
}
}
Deployment error after some successful deployments is this.
{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"Conflict","message":"{\r\n \"status\": \"canceled\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource operation completed with terminal provisioning state 'canceled'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentScriptExceededMaxAllowedTime\"\r\n }\r\n ]\r\n }\r\n}"}]}
Point 1: This issue resolved after adding this property in resources.
After adding this property, make sure some containers are not running. Delete storage accounts and containers manually which are created by previous failed deployments.
"cleanupPreference": "Always"
Always: Delete the automatically created resources once script execution gets in a terminal state. If an existing storage account is used, the script service deletes the file share created in the storage account. Because the deploymentScripts resource may still be present after the resources are cleaned up, the script service persists the script execution results, for example, stdout, outputs, return value, etc. before the resources are deleted.
Initially property was set to "cleanupPreference": "OnSuccess" which was not removing the created storage account and containers in failed deployments and making a issues while next deployment.
for more details check this azure document
Point 2. Creating unique identifier can be done using newGuid() function, but only in parameters section of templates.
Which creates unique Id globally, but if you don't want to show user input box then guid() also works.
"variables": {
"uniqueName":"[guid(resourceGroup().id, deployment().name)]"
}
refer guid for more details.

How to access the iteration values in ARM Output's?

I'm working on ARM Templates. Actually, I have to deploy two resources at a time into my Azure account. For that, I used copyindex() concept with the help of following document and am able to deploy them successfully. Now am trying to display the names of deployed resources by using Output concept in ARM. But due to looping of resources deployment, its resulting in following error.
"DeploymentOutputEvaluationFailed",
"message": "Unable to evaluate template outputs: 'alertName'. Please see error details and deployment operations. Please see https://aka.ms/arm-debug for usage details.",
"details": [
{
"code": "DeploymentOutputEvaluationFailed",
"target": "alertName",
"message": "The template output 'alertName' is not valid: The language expression property 'alertMetricType' can't be evaluated.."
}
]
Could you please suggest me to "How to fetch the values of copyindex() looping as ARM Output session values"
I would suggest to reference it like this for example because you need to assemble anarray which you need to each of your template to take the output from the previous one and concat it with its own output then print the result.
"parameters": {
"state": {
"value": []
}
}
"parameters": {
"state": {
"value": "[reference(concat('loop', copyIndex())).outputs.state.value]"
}
}
then call it in output:
"outputs": {
"state": {
"type": "array",
"value": "[concat(parameters('state'), array(stuff_out))]"
}

ARM deployment failed for event grid subscription on endpoint webhook url

Using terraform and Azure ARm template , I am trying to create an azure event grid subscription on a function.
This the ARM using for the event grid subscription:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json",
"contentVersion": "1.0.0.0",
"parameters": {
"eventGridTopicName": {
"type": "string",
"metadata": {
"description": "The name of the Event Grid custom topic."
}
},
"eventGridSubscriptionName": {
"type": "string",
"metadata": {
"description": "The name of the Event Grid custom topic's subscription."
}
},
"eventGridSubscriptionUrl": {
"type": "string",
"metadata": {
"description": "The webhook URL to send the subscription events to. This URL must be valid and must be prepared to accept the Event Grid webhook URL challenge request. (RequestBin URLs are exempt from this requirement.)"
}
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]",
"metadata": {
"description": "The location in which the Event Grid resources should be deployed."
}
}
},
"resources": [{
"name": "[parameters('eventGridTopicName')]",
"type": "Microsoft.EventGrid/topics",
"location": "[parameters('location')]",
"apiVersion": "2018-01-01"
},
{
"name": "[concat(parameters('eventGridTopicName'), '/Microsoft.EventGrid/', parameters('eventGridSubscriptionName'))]",
"type": "Microsoft.EventGrid/topics/providers/eventSubscriptions",
"location": "[parameters('location')]",
"apiVersion": "2018-01-01",
"properties": {
"destination": {
"endpointType": "WebHook",
"properties": {
"endpointUrl": "[parameters('eventGridSubscriptionUrl')]"
}
},
"filter": {
"includedEventTypes": [
"All"
]
}
},
"dependsOn": [
"[parameters('eventGridTopicName')]"
]
}
]
}
Following the documentation here in order to create the subscription, we have to recover a system key in order to create the complete webhook endpoint. So following this post here, I have used an ARM template to recover the system key called evengrid_extension.
So everything goes well except during the arm deployment of the eventgrid subscription. I have this error:
Error waiting for deployment: Code="DeploymentFailed"
Message="At least one resource deployment operation failed. Please
list deployment operations for details. Please see
https://aka.ms/arm-debug for usage details."
Details=[{"code":"Conflict","message":"{\r\n
\"status\": \"Failed\",\r\n
\"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n
\"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n
\"details\": [\r\n {\r\n
\"code\": \"Url validation\",\r\n
\"message\": \"The attempt to validate the provided endpoint https://myFunctionName.azurewebsites.net/runtime/webhooks/eventgrid
failed.
\For more details, visit https:
//aka.ms/esvalidation.\"\r\n }\r\n ]\r\n }\r\n}"}]
I check my code n terraform in order to be sure that I am using the right value for all parameters in this arm template and everything is ok. I have the right topic name, the right endpoint with all value filled in. So I don't understand what I am missing here. I was wondering too if I am using the right system key. I know that there are a system key named durabletask_extension, and another one named eventgrid_extension. But in fact I have tried with both and the same error occured.
Update
Just notice that the keys i-e durabletask_extension and eventgrid_extension are both system keys. So in my arm template to recover these works well and I recover the right system key by using only eventgrid_extension.
Here my code for terraform:
resource "azurerm_eventgrid_topic" "eventgrid_topic" {
name = "topicName"
location = var.main_location
resource_group_name = azurerm_resource_group.name
}
resource "azurerm_template_deployment" "eventgrid_subscription" {
name = "EventGridSbscription"
resource_group_name = azurerm_resource_group.environment.name
template_body = file("./arm/event-grid-subscription.json")
parameters = {
eventGridTopicName = "${azurerm_eventgrid_topic.eventgrid_topic.name}"
eventGridSubscriptionName = "eventgrid-myFunctionName"
eventGridSubscriptionUrl = "https://${azurerm_function_app.function.name}.azurewebsites.net/runtime/webhooks/eventgrid?functionName=${azurerm_function_app.function.name}&code=${lookup(azurerm_template_deployment.function_key.outputs, "systemKey")}"
location = var.main_location
}
deployment_mode = "Incremental"
depends_on = [
azurerm_template_deployment.function_key
]
}
So I do not understand why my susbription deployment failed, or what I am missing in order to automate this settings with terraform.
Following the doc here I understand too that:
If you don't have access to the application code (for example, if
you're using a third-party service that supports webhooks), you can
use the manual handshake mechanism. Make sure you're using the
2018-05-01-preview API version or later (install Event Grid Azure CLI
extension) to receive the validationUrl in the validation event. To
complete the manual validation handshake, get the value of the
validationUrl property and visit that URL in your web browser. If
validation is successful, you should see a message in your web browser
that validation is successful. You'll see that event subscription's
provisioningState is "Succeeded".
So, there is a way to make a validation using terraform or another way to automate this validation ?
The template is right, you just misunderstand something in the eventGridSubscriptionUrl. Take a look at the URL. The URL shows like this:
Version 2.x runtime
https://{functionappname}.azurewebsites.net/runtime/webhooks/eventgrid?functionName={functionname}&code={systemkey}
Version 1.x runtime
https://{functionappname}.azurewebsites.net/admin/extensions/EventGridExtensionConfig?functionName={functionname}&code={systemkey}
The functionappname is what you set as the value azurerm_function_app.function.name, but functionname is not.
You get the existing function name through the Azure REST API Web Apps - Get Function.
And in Terraform, it seems there is no function resource in the function app for you to create. But you can also use the template to create the function and output the function name. Then you can set it in the URL. You can get more details about function in the Azure Template here and the function name shows in the property.

Resources