I am referencing the below codes with command but getting error for both
az datafactory linked-service create --resource-group $resourcegroup --factory-name $factoryname --linked-service-name ls_AzureKeyVault_storage --properties #ls_AzureKeyVault_storage.json > \dev\null
{
"name": "ls_AzureKeyVault_storage",
"properties": {
"annotations": [],
"type": "AzureKeyVault",
"typeProperties": {
"baseUrl": "https://kvadfconnections.vault.azure.net/"
}
}
}
az datafactory linked-service create --resource-group $resourcegroup --factory-name $factoryname --linked-service-name AzureStorageLinkedService --properties #AzureStorageLinkedService.json > \dev\null
{
"name": "AzureStorageLinkedService",
"properties": {
"annotations": [],
"type": "AzureBlobStorage",
"typeProperties": {
"connectionString": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "ls_AzureKeyVault_storage",
"type": "LinkedServiceReference"
},
"secretName": "sec-stforadfcli-connection"
}
}
},
"type": "Microsoft.DataFactory/factories/linkedservices"
}
We have tested this in our local environment, Below statements are based on our analysis.
While creating the linked service through AzureCLI cmdlet using az data factory linked-service create ,you need to pass the json file to the --properties flag .
az datafactory linked-service create --factory-name
--linked-service-name
--properties
--resource-group
[--if-match]
If you declare the properties{} list on top of the typeproperties{} in your json file then while creating the linked service to the data factory it will fail with the error that you have shared as shown in the below
You Need pass only typeProperties in json file to create a linked service with the data factory as shown in the below.
In the below example ,we are trying to create a keyvault linked service with our existing ADF.
Here is our keyvault.json file which has type properties.
{
"type": "AzureKeyVault",
"typeProperties":{
"baseUrl": "<keyvault>"
},
"annotations":[<requiredannotations],
"description":"<requireddescription>",
"parameters": {
"test":{
"type":"String",
"defaultValue":"test"
}
}
}
Here is the sample output for reference :
You can use the above keyvault.json file as reference & make the changes as per your requirement.
You can also refer this documentation, for more information about what all the properties that we can pass to AzurekeyVaultLinkedService & there respective datatypes.
Related
I am trying to deploy a synapse instance via an ARM template and the deployment is successful via the Azure DevOps portal, but when I try to deploy the same template with an Azure Keyvault linked service I encounter the following error:
##[error]At least one resource deployment operation failed. Please list deployment
operations for details. Please see https://aka.ms/DeployOperations for usage details.
##[error]Details:
##[error]BadRequest:
After inspecting the activity logs from the Synapse instance I found out the following:
"resourceGroupName": "platform-test-group",
"resourceProviderName": {
"value": "Microsoft.Synapse",
"localizedValue": "Microsoft.Synapse"
},
"resourceType": {
"value": "Microsoft.Synapse/workspaces/linkedservices",
"localizedValue": "Microsoft.Synapse/workspaces/linkedservices"
},
"resourceId": "/subscriptions/xxxx-xxxx-xxxx-xxxx/resourcegroups/platform-test-group/providers/Microsoft.Synapse/workspaces/synapsedataapp/linkedservices/AzureKeyVault",
"status": {
"value": "Failed",
"localizedValue": "Failed"
},
"subStatus": {
"value": "NotFound",
"localizedValue": "Not Found (HTTP Status Code: 404)"
},
"submissionTimestamp": "2022-02-01T02:30:31.1471914Z",
"subscriptionId": "xxxx-xxxx-xxxx-xxxx",
"tenantId": "0f44c5d4-xxxx-xxxx-xxxxx",
"properties": {
"statusCode": "NotFound",
"serviceRequestId": null,
"statusMessage": "{\"error\":{\"code\":\"BadRequest\",\"message\":\"\"}}",
"eventCategory": "Administrative",
"entity": "/subscriptions/xxxx-xxxx-xxxx-xxxx/resourcegroups/platform-test-group/providers/Microsoft.Synapse/workspaces/synapsedataapp/linkedservices/AzureKeyVault",
"message": "Microsoft.Synapse/workspaces/linkedservices/write",
"hierarchy": "xxxx-xxxx-xxxx-xxxx/Enterprise/Group/Group-Test/xxxx-xxxx-xxxx-xxxx"
},
"relatedEvents": []
}
As you can see, the 404 error appears when the template tries to deploy to the tenant id which is not found, however, when I deploy the keyvault via the synapse UI I encounter no error.
Below is the code snippet that I use in my ARM template to deploy the keyvault to the synapse instance:
{
"name": "[concat(variables('workspaceName'), '/AzureKeyVault')]",
"type": "Microsoft.Synapse/workspaces/linkedservices",
"apiVersion": "2021-06-01-preview",
"properties": {
"annotations": [],
"type": "AzureKeyVault",
"typeProperties": {
"baseUrl": "https://data-test-kv.vault.azure.net/"
}
},
"dependsOn": [
"[variables('workspaceName')]"
]
}
Am I missing some kind of permission or connection that I need to enable? Why am I able to deploy successfully through the UI but not through the ARM template? Any comment or suggestion is greatly valued, so please feel free to comment or improve this question.
I had to contact Microsoft support and their reply was the following:
ARM templates cannot be used to create a linked service. This is due to the fact that linked services are not ARM resources, for examples, synapse workspaces, storage account, virtual networks, etc. Instead, a linked service is classified as an artifact. To still complete the task at hand, you will need to use the Synapse REST API or PowerShell. Below is the link that provides guidance on how to use the API. https://learn.microsoft.com/en-us/powershell/module/az.synapse/set-azsynapselinkedservice?view=azps-7.1.0
This limitation in ARM is applied only to Synapse and they might fix this in the future.
Additional references:
https://feedback.azure.com/d365community/idea/05e41bf1-0925-ec11-b6e6-000d3a4f07b8
https://feedback.azure.com/d365community/idea/48f1bf78-2985-ec11-a81b-6045bd7956bb
In Synapse unlike ADF, linked-services are not part of arm-templates. They are called artifacts and it comprises: Note Books, Spark Definitions, Linked Services, Pipelines etc.
You can find the full article here: https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/how-to-use-ci-cd-integration-to-automate-the-deploy-of-a-synapse/ba-p/2248060
In short, first, deploy Synapse using arm templates. And then set up the linked services:
- task: Synapse workspace deployment#1
displayName: 'Setup:Synapse KeyVault Linked Service'
inputs:
TemplateFile: '$(Build.Repository.LocalPath)/TemplateForWorkspace.json'
ParametersFile: '$(Build.Repository.LocalPath)/TemplateParametersForWorkspace.json'
azureSubscription: '${{ parameters.environments.serviceConnectionId }}'
ResourceGroupName: '$(computeResourceGroupName)'
TargetWorkspaceName: '$(synapseWorkspaceName)'
DeleteArtifactsNotInTemplate: true
OverrideArmParameters: |
synapseLinkedServiceKV: $(synapseLinkedServiceKV)
workspaceName: $(synapseWorkspaceName)
Environment: 'prod'
TemplateForWorkspace.json:
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"workspaceName": {
"type": "string"
},
"synapseLinkedServiceKV": {
"type": "string"
}
},
"variables": {
"workspaceId": "[concat('Microsoft.Synapse/workspaces/', parameters('workspaceName'))]"
},
"resources": [
{
"name": "[concat(parameters('workspaceName'), '/' , parameters('synapseLinkedServiceKV'))]",
"type": "Microsoft.Synapse/workspaces/linkedServices",
"apiVersion": "2019-06-01-preview",
"properties": {
"type": "AzureKeyVault",
"typeProperties": {
"baseUrl": "[concat('https://', parameters('synapseLinkedServiceKV'), '.vault.azure.net/')]"
},
"annotations": [],
"description": "Linked Service to Azure KeyVault. KeyVault is used to primarily fetch secrets"
},
"dependsOn": []
}
]
}
TemplateParametersForWorkspace.json:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"workspaceName": {
"value": ""
},
"synapseLinkedServiceKV": {
"value": ""
}
}
}
It deletes the existing artifacts and deploys the one above. You would first need to install the task extension on your Azure Devops for Synapse workspace deployment#1
Note above template was auto-generated. In synapse studio, goto Git Configuration and point it to your repo. It will submit the changes to the branch workspace_publish. You can copy and build on top of the specific artifact code.
I am trying to create a Resource Group using the below ARM template.
{
"$schema": "https://schema.management.azure.com/schemas/2018-05-01/subscriptionDeploymentTemplate.json#",
"contentVersion": "1.0.0.1",
"parameters": {
"rgName": {
"type": "string"
},
"rgLocation": {
"type": "string"
}
},
"variables": {},
"resources": [
{
"type": "Microsoft.Resources/resourceGroups",
"apiVersion": "2018-05-01",
"location": "[parameters('rgLocation')]",
"name": "[parameters('rgName')]",
"properties": {}
}
],
"outputs": {}
}
And the Parameter file is
{
"$schema": "https://schema.management.azure.com/schemas/2018-05-01/subscriptionDeploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"rgName": {
"value": "sriram"
},
"rgLocation": {
"value": "southcentralus"
}
}
}
Is there any mistake in the above json files. Because am getting the following error.
Unable to load schema from 'https://schema.management.azure.com/schemas/2018-05-01/subscriptionDeploymentTemplate.json'. No schema request service available(768)
I ignored this error and tried running these templates in the Azure pipeline and got the following error.
"No HTTP resource was found that matches the request URI 'https://management.azure.com/subscriptions/****-****-*****-****/resourcegroups/<Rsource Group Name>/providers/Microsoft.Resources/resourceGroups/<new RG name>?api-version=2018-05-01'
Can anyone help me out? Thank you.
Your parameters file should use this https://schema.management.azure.com/schemas/2018-05-01/subscriptionDeploymentParameters.json# as a schema reference.
The schema your parameter file are using does not has affect on the creation of resource groups. With the command and your original script
az deployment create --template-file tem.json --parameters #para.json --location southcentralus
I could create the new resource group:
The error you are facing just caused by you are trying to create a new resource group within a exists resource group. This does not allowed.
Here suggest you using Command line task to achieve what you want, by using the command I show you above.
az deployment create --template-file $(Build.SourcesDirectory)/{Template}.json --parameters #$(Build.SourcesDirectory)/{parameter}.json --location southcentralus
I have extracted the ARM template belonging to the preview version of Azure App Configuration, and am setting it into our IaC repository - so far so good.
Our next logical step is to include insertion of the AppConfiguration.PrimaryKey into our Key Vault. However I do not know the name of this property, and I can not find any information on the subject online. Also I can not see the AppConfiguration/configurationStores type listed in resources.azure.com (assuming its because its still in public preview).
Does anyone know how to reference the primary key (and possibly the read-only primary key), so i can reference them through a "outputs" variable in my arm template?
Then I can let Az Cli/Az Powershell insert the secret into our Key Vault, and we obtain full automation of our IaC
I was not able to figure this out.
However by using az cli commands in a IaC script (which anyways invokes the arm template residing in a azure blob store) I circumvented the problem:
$connStrings = az appconfig credential list -n $configName| ConvertFrom-Json
$readOnlyConnString = ($connStrings | Where {$_.name -eq "Primary Read Only"}).connectionString
$primaryConnString = ($connStrings | Where {$_.name -eq "Primary"}).connectionString
#then
az keyvault secret set --vault-name $kvName --name $keyNameRO --value $readOnlyConnString
az keyvault secret set --vault-name $kvName --name $keyNamePrimary --value $primaryConnString
For an ARM template I did the following. The listkeys function returns a full list of all the values that have to do with the keys. This was hard to figure out. I hope it helps.
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"configurationStores_instance_name": {
"defaultValue": "ac-instance",
"type": "String"
}
},
"variables": {
"apiVersionVar": "[providers('Microsoft.AppConfiguration', 'configurationStores').apiVersions[0]]",
"resourceId": "[resourceId('Microsoft.AppConfiguration/configurationStores', parameters('configurationStores_instance_name'))]",
},
"resources": [
{
"type": "Microsoft.AppConfiguration/configurationStores",
"apiVersion": "2019-10-01",
"name": "[parameters('configurationStores_instance_name')]",
"location": "northcentralus",
"sku": {
"name": "standard"
},
"properties": {}
}
],
"outputs": {
"AppConfigEndpoint": {
"type": "string",
"value": "[reference(parameters('configurationStores_instance_name')).endpoint]"
},
"AppConfigKeys": {
"type": "Array",
"value": "[listkeys(variables('resourceId'), variables('apiVersionVar')).value]"
}
}
}
hope this helps!
I've searched online and browsed the available powershell cmdlets to try and find a solution for this problem but have been unsuccessful. Essentially, I have a few Data Factory pipelines that copy/archive incoming files and will use a web http post component that will invoke a Logic App that connects to a Blob container and will delete the incoming file. The issue I'm facing is that we have several automation runbooks that will rest Blob access keys every X days. When the Blob keys get reset the Logic App will fail whenever this happens because the connection is manually created in the designer itself and I can't specify a connection string that could pull from the Keyvault, as an example. Inside of the {Logic App > API Connections > Edit API Connection} we can manually update the connection string/key but obviously for an automated process we should be able to do this programmatically.
Is there a powershell cmdlet or other method I'm not seeing that would allow me to update/edit the API Connections that get created when using and Blob component inside a Logic App?
Any insights is appreciated!
Once you've rotated your key in the storage account, you can use an ARM template to update your connection API. In this ARM template, the connection api is created referencing the storage account internally so you don't have to provide the key:
azuredeploy.json file:
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"azureBlobConnectionAPIName": {
"type": "string",
"metadata": {
"description": "The name of the connection api to access the azure blob storage."
}
},
"storageAccountName": {
"type": "string",
"metadata": {
"description": "The Storage Account Name."
}
}
},
"variables": {},
"resources": [
{
"type": "Microsoft.Web/connections",
"name": "[parameters('azureBlobConnectionAPIName')]",
"apiVersion": "2016-06-01",
"location": "[resourceGroup().location]",
"scale": null,
"properties": {
"displayName": "[parameters('azureBlobConnectionAPIName')]",
"parameterValues": {
"accountName": "[parameters('storageAccountName')]",
"accessKey": "[listKeys(resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccountName')),'2015-05-01-preview').key1]"
},
"api": {
"id": "[concat('subscriptions/', subscription().subscriptionId, '/providers/Microsoft.Web/locations/', resourceGroup().location, '/managedApis/azureblob')]"
}
},
"dependsOn": []
}
]
}
azuredeploy.parameters.json file:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"azureBlobConnectionAPIName": {
"value": "myblobConnectionApiName"
},
"storageAccountName": {
"value": "myStorageAccountName"
}
}
}
You can them execute the arm template like that:
Connect-AzureRmAccount
Select-AzureRmSubscription -SubscriptionName <yourSubscriptionName>
New-AzureRmResourceGroupDeployment -Name "ExampleDeployment" -ResourceGroupName "MyResourceGroupName" `
-TemplateFile "D:\Azure\Templates\azuredeploy.json" `
-TemplateParameterFile "D:\Azure\Templates\azuredeploy.parameters.json"
to get started with ARM template and powerhsell, you cam have a look at this article:
Deploy resources with Resource Manager templates and Azure PowerShell
I am creating simple pipeline in the data factory that should only run a custom activity. The deployment template for the pipeline looks like this:
{
"type": "pipelines",
"name": "MyCustomActivityPipeline",
"dependsOn": [
"DataFactoryName",
"AzureBatchLinkedService"
],
"apiVersion": "[variables('api-version')]",
"properties": {
"description": "Custom activity sample",
"activities": [
{
"type": "Custom",
"name": "MyCustomActivity",
"linkedServiceName": {
"referenceName": "AzureBatchLinkedService",
"type": "LinkedServiceReference"
},
"typeProperties": {
"command": "cmd /c echo hello world"
}
}
]
}
}
Additionally I have created all the resources needed- the batch account with pools and the storage account. All the resources are in the same resource group and subscription. I try to trigger the pipeline using console command
Invoke-AzureRmDataFactoryV2Pipeline -DataFactory "DataFactory" -PipelineName "PipelineName" -ResourceGroupName "ResourceGroupName"
I am getting this error:
Activity MyCustomActivity failed: Can not access user batch account, please check batch account setiings.
Has anyone ever experienced such an error from ADF execution of a pipeline? The weird part is that all the resources have access to each other and are within the same resource group and subscription.
Please check the settings for the storage linked service used by batch linked service. Make sure the connection string type is SecureString