I am trying to do an adf deployment from production instance into the development instance in order to sync up the recent changes on PROD ADF.
In the production instance we do have linked services connecting to Gen1 ADLS and Gen2 ADLS.
But in DEV we do have only Gen2 ADLS available.
Is there any way to convert/map the Linked Services with Gen1 ADLS into Gen2 at the time of deployment and vice versa.
Is there any other way to achieve the same like by replacing the linked service object in ARM template?
I've tried several tests but it seems that we can not change linked services attributes via ARM templates. You can refer this post.
The linked service of Gen1 is as follows:
{
"name": "[concat(parameters('factoryName'), '/AzureBlobStorage1')]",
"type": "Microsoft.DataFactory/factories/linkedServices",
"apiVersion": "2018-06-01",
"properties": {
"annotations": [],
"type": "AzureBlobStorage",
"typeProperties": {
"connectionString": "[parameters('AzureBlobStorage1_connectionString')]"
}
},
"dependsOn": []
}
The linked service of Gen2 is as follows:
{
"name": "[concat(parameters('factoryName'), '/AzureDataLakeStorage1')]",
"type": "Microsoft.DataFactory/factories/linkedServices",
"apiVersion": "2018-06-01",
"properties": {
"annotations": [],
"type": "AzureBlobFS",
"typeProperties": {
"url": "[parameters('AzureDataLakeStorage1_properties_typeProperties_url')]",
"accountKey": {
"type": "SecureString",
"value": "[parameters('AzureDataLakeStorage1_accountKey')]"
}
}
},
"dependsOn": []
}
Related
I'm new to Azure and newer to using ARM templates.
I've got an App Service and Service Plan supporting Windows OS that needs to be changed to Linux. From what I can tell, there is no direct modification to achieve this result, I'm going to need to delete and redeploy.
I was looking at steps for manual deletion and re-build, but I'm thinking that using ARM templates would likely be more effective. I'm researching using ARM templates but not getting much information about using them for removal/modify/replacement. I'd guess that I can download the existing ARM templates and re-deploy, but there have to be a handful of gotchas, but I don't know what to look for.
My expectation is that the ARM template would not be able to deploy the custom domain and its certificate ready to go. Also, the existing template has references to snapshots that would likely be gone after deletion, so I'd expect to have to remove those references from the template prior to re-deploy.
Any guidance I can get would be greatly appreciated!
Per
One of the workaround you can follow ;
I'm researching using ARM templates but not getting much information
about using them for removal/modify/replacement
AFAIK, There is no direct command to delete the resources through which are deployed to Azure using ARM.
Instead of that you can use Azure cli as suggested in this SO THREAD,
Because after deployment there is still you can see in the deployment logs your resource are there you can delete from the portal itself.
After remove the app service from portal you can redeploy the same with adding your modifications.
We have tried after deploy the application and then remove/delete from portal as mentioned above and then re-deploy the app service with linux environment and its work fine.
You can make it use of below template(e.g):-
template.json
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"webAppName": {
"type": "string",
"defaultValue": "AzureLinuxApp",
"metadata": {
"description": "Base name of the resource such as web app name and app service plan "
},
"minLength": 2
},
"sku": {
"type": "string",
"defaultValue": "S1",
"metadata": {
"description": "The SKU of App Service Plan "
}
},
"linuxFxVersion": {
"type": "string",
"defaultValue": "php|7.4",
"metadata": {
"description": "The Runtime stack of current web app"
}
},
"location": {
"type": "string",
"defaultValue": "[resourceGroup().location]",
"metadata": {
"description": "Location for all resources."
}
}
},
"variables": {
"webAppPortalName": "[concat(parameters('webAppName'), '-webapp')]",
"appServicePlanName": "[concat('AppServicePlan-', parameters('webAppName'))]"
},
"resources": [
{
"type": "Microsoft.Web/serverfarms",
"apiVersion": "2020-06-01",
"name": "[variables('appServicePlanName')]",
"location": "[parameters('location')]",
"sku": {
"name": "[parameters('sku')]"
},
"kind": "linux",
"properties": {
"reserved": true
}
},
{
"type": "Microsoft.Web/sites",
"apiVersion": "2020-06-01",
"name": "[variables('webAppPortalName')]",
"location": "[parameters('location')]",
"kind": "app",
"dependsOn": [
"[resourceId('Microsoft.Web/serverfarms', variables('appServicePlanName'))]"
],
"properties": {
"serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('appServicePlanName'))]",
"siteConfig": {
"linuxFxVersion": "[parameters('linuxFxVersion')]"
}
}
}
]
}
app.parameter.json
{
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"webAppName": {
"value": "mylinuxappp"
}
}
}
OUTPUT DETAILS FOR REFERENCE:-
To deploy webapp with custom domain and ssl certificate need to make sure that its already verified and also need to use existing keyvault for the SSL binding . Please find this arm template for more information.
Please refer the below links for get started with Azure App service using arm template with different scenarios(step by step guidance). It should be help more to understand .
MICROSOFT DOCUMENTATIONS| Azure Resource Manager templates for App Service & Quickstart: Create App Service app using an ARM template
I am trying to deploy a synapse instance via an ARM template and the deployment is successful via the Azure DevOps portal, but when I try to deploy the same template with an Azure Keyvault linked service I encounter the following error:
##[error]At least one resource deployment operation failed. Please list deployment
operations for details. Please see https://aka.ms/DeployOperations for usage details.
##[error]Details:
##[error]BadRequest:
After inspecting the activity logs from the Synapse instance I found out the following:
"resourceGroupName": "platform-test-group",
"resourceProviderName": {
"value": "Microsoft.Synapse",
"localizedValue": "Microsoft.Synapse"
},
"resourceType": {
"value": "Microsoft.Synapse/workspaces/linkedservices",
"localizedValue": "Microsoft.Synapse/workspaces/linkedservices"
},
"resourceId": "/subscriptions/xxxx-xxxx-xxxx-xxxx/resourcegroups/platform-test-group/providers/Microsoft.Synapse/workspaces/synapsedataapp/linkedservices/AzureKeyVault",
"status": {
"value": "Failed",
"localizedValue": "Failed"
},
"subStatus": {
"value": "NotFound",
"localizedValue": "Not Found (HTTP Status Code: 404)"
},
"submissionTimestamp": "2022-02-01T02:30:31.1471914Z",
"subscriptionId": "xxxx-xxxx-xxxx-xxxx",
"tenantId": "0f44c5d4-xxxx-xxxx-xxxxx",
"properties": {
"statusCode": "NotFound",
"serviceRequestId": null,
"statusMessage": "{\"error\":{\"code\":\"BadRequest\",\"message\":\"\"}}",
"eventCategory": "Administrative",
"entity": "/subscriptions/xxxx-xxxx-xxxx-xxxx/resourcegroups/platform-test-group/providers/Microsoft.Synapse/workspaces/synapsedataapp/linkedservices/AzureKeyVault",
"message": "Microsoft.Synapse/workspaces/linkedservices/write",
"hierarchy": "xxxx-xxxx-xxxx-xxxx/Enterprise/Group/Group-Test/xxxx-xxxx-xxxx-xxxx"
},
"relatedEvents": []
}
As you can see, the 404 error appears when the template tries to deploy to the tenant id which is not found, however, when I deploy the keyvault via the synapse UI I encounter no error.
Below is the code snippet that I use in my ARM template to deploy the keyvault to the synapse instance:
{
"name": "[concat(variables('workspaceName'), '/AzureKeyVault')]",
"type": "Microsoft.Synapse/workspaces/linkedservices",
"apiVersion": "2021-06-01-preview",
"properties": {
"annotations": [],
"type": "AzureKeyVault",
"typeProperties": {
"baseUrl": "https://data-test-kv.vault.azure.net/"
}
},
"dependsOn": [
"[variables('workspaceName')]"
]
}
Am I missing some kind of permission or connection that I need to enable? Why am I able to deploy successfully through the UI but not through the ARM template? Any comment or suggestion is greatly valued, so please feel free to comment or improve this question.
I had to contact Microsoft support and their reply was the following:
ARM templates cannot be used to create a linked service. This is due to the fact that linked services are not ARM resources, for examples, synapse workspaces, storage account, virtual networks, etc. Instead, a linked service is classified as an artifact. To still complete the task at hand, you will need to use the Synapse REST API or PowerShell. Below is the link that provides guidance on how to use the API. https://learn.microsoft.com/en-us/powershell/module/az.synapse/set-azsynapselinkedservice?view=azps-7.1.0
This limitation in ARM is applied only to Synapse and they might fix this in the future.
Additional references:
https://feedback.azure.com/d365community/idea/05e41bf1-0925-ec11-b6e6-000d3a4f07b8
https://feedback.azure.com/d365community/idea/48f1bf78-2985-ec11-a81b-6045bd7956bb
In Synapse unlike ADF, linked-services are not part of arm-templates. They are called artifacts and it comprises: Note Books, Spark Definitions, Linked Services, Pipelines etc.
You can find the full article here: https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/how-to-use-ci-cd-integration-to-automate-the-deploy-of-a-synapse/ba-p/2248060
In short, first, deploy Synapse using arm templates. And then set up the linked services:
- task: Synapse workspace deployment#1
displayName: 'Setup:Synapse KeyVault Linked Service'
inputs:
TemplateFile: '$(Build.Repository.LocalPath)/TemplateForWorkspace.json'
ParametersFile: '$(Build.Repository.LocalPath)/TemplateParametersForWorkspace.json'
azureSubscription: '${{ parameters.environments.serviceConnectionId }}'
ResourceGroupName: '$(computeResourceGroupName)'
TargetWorkspaceName: '$(synapseWorkspaceName)'
DeleteArtifactsNotInTemplate: true
OverrideArmParameters: |
synapseLinkedServiceKV: $(synapseLinkedServiceKV)
workspaceName: $(synapseWorkspaceName)
Environment: 'prod'
TemplateForWorkspace.json:
{
"$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"workspaceName": {
"type": "string"
},
"synapseLinkedServiceKV": {
"type": "string"
}
},
"variables": {
"workspaceId": "[concat('Microsoft.Synapse/workspaces/', parameters('workspaceName'))]"
},
"resources": [
{
"name": "[concat(parameters('workspaceName'), '/' , parameters('synapseLinkedServiceKV'))]",
"type": "Microsoft.Synapse/workspaces/linkedServices",
"apiVersion": "2019-06-01-preview",
"properties": {
"type": "AzureKeyVault",
"typeProperties": {
"baseUrl": "[concat('https://', parameters('synapseLinkedServiceKV'), '.vault.azure.net/')]"
},
"annotations": [],
"description": "Linked Service to Azure KeyVault. KeyVault is used to primarily fetch secrets"
},
"dependsOn": []
}
]
}
TemplateParametersForWorkspace.json:
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"workspaceName": {
"value": ""
},
"synapseLinkedServiceKV": {
"value": ""
}
}
}
It deletes the existing artifacts and deploys the one above. You would first need to install the task extension on your Azure Devops for Synapse workspace deployment#1
Note above template was auto-generated. In synapse studio, goto Git Configuration and point it to your repo. It will submit the changes to the branch workspace_publish. You can copy and build on top of the specific artifact code.
I exported arm template from Datafactory V2, when Importing the template it is asking me to manually enter SQL database connection string. To minimize the human interaction I made the following changes.
{
"name": "[concat(parameters('factoryName'), '/myFactory')]",
"type": "Microsoft.DataFactory/factories/linkedServices",
"apiVersion": "2018-06-01",
"properties": {
"type": "AzureSqlDatabase",
"typeProperties": {
"connectionString": "[concat('Server=tcp:',parameters('sqlServerName'),'.database.windows.net,1433;Initial Catalog=', parameters('sqlDatabaseName'), ';Persist Security Info=False;User ID=',parameters('sqlServerUserName'),';Password=(password)',';MultipleActiveResultSets=False;Encrypt=True;TrustServerCertificate=False;Connection Timeout=30')]",
"password": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "AzureKeyVault1",
"type": "LinkedServiceReference"
},
"secretName": "sql-password"
}
}
},
"dependsOn": [
"[concat(variables('factoryId'), '/linkedServices/AzureKeyVault1')]"
]
},
So currently when deploying to Datafactory V2 and test connection to this SQL server, I got
Cannot connect to SQL Database: 'tcp:mysqlserver.database.windows.net,1433',
Database: 'mydatabase', User: 'admin'. Check the linked service configuration
is correct, and make sure the SQL Database firewall allows the integration runtime to access.
Login failed for user 'admin'., SqlErrorNumber=18456,
If I manually input all the connections in the portal UI, I can easily connect to the database and test successfully so it is not a firewall issue.
Then I think there could be 2 issue:
1.how the password from keyvault is consumed in the connectionstring. I didn't find much information about it online.
When I open the created Sql Linked service, I notice the Fully qualified domain name is missing,If i manually add it in then the connection works.
The SQL connection UI
Throwing this as an alternative answer/approach.
Store the connection string in it's entirety in Key Vault. If doing this then the reference would look like:
{
"name": "[concat(parameters('factoryName'), '/',parameters('connectionNameAdventureWorks'))]",
"type": "Microsoft.DataFactory/factories/linkedServices",
"apiVersion": "2018-06-01",
"properties": {
"annotations": [],
"type": "AzureSqlDatabase",
"typeProperties": {
"connectionString": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "[variables('azkDataAnalyticsReferenceName')]",
"type": "LinkedServiceReference"
},
"secretName": "[variables('azkAdventureWorksSecretName')]"
}
}
},
"dependsOn": [
"[concat(variables('factoryId'), '/linkedServices/',variables('azkDataAnalyticsReferenceName'))]"
]
}
And even more secure approach would be to add Data Factory as a Managed Identity and then run a sql script to add the user If doing this then all there is no need for any credentials to be passed at all.
One downside is if the DataFactory is deleted and recreated then the managed identity permissions would need to be reassigned to the sql database.
I am going to configure created azure VM (for example, install role, initialize new hdd disk etc). I see that there is a new feature Microsoft.Resources/deploymentScripts in azure ARM. As per documantation I created Managed Identity in my subscription, give Contributor permissions to newly created Managed Identity, on Subscription level. then I developed below ARM template using Microsoft.Resources/deploymentScripts feature. code pasted below. I want to paste this code into my ARM template for VM deployment.Question is if I will be able to use this approach to perform scripts like: installing role on the OS level, like IIS or WSUS, configure HDD etc...
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"name": {
"type": "string",
"defaultValue": "'John Dole'"
},
"utcValue": {
"type": "string",
"defaultValue": "[utcNow()]"
}
},
"resources": [
{
"type": "Microsoft.Resources/deploymentScripts",
"apiVersion": "2019-10-01-preview",
"name": "runPowerShellInlineWithOutput",
"location": "westeurope",
"kind": "AzurePowerShell",
"identity": {
"type": "userAssigned",
"userAssignedIdentities": {
"/subscriptions/SubID/resourceGroups/RGname/providers/Microsoft.ManagedIdentity/userAssignedIdentities/MI-ARMdeployment": {}
}
},
"properties": {
"forceUpdateTag": "[parameters('utcValue')]",
"azPowerShellVersion": "3.0",
"scriptContent": "
$output = 'hello'
Write-Output $output",
"arguments": "",
"timeout": "PT1H",
"cleanupPreference": "OnSuccess",
"retentionInterval": "P1D"
}
}
]
}
Well, yes (with some hacks), but its not meant for that. Its meant for provisioning\configuring Azure level resources, not things inside of the VM.
You have DSC extension and script extension for that (available for both windows\linux).
Logic app connectors are closed source and the 'Automation Script' option in the Azure portal strips the authentication portions of the properties node from connectors. This is what the portal hands you when you script out the ARM template for a logic app which talks to CRM.
{
"comments": "Generalized from resource: '/subscriptions/<guid>/resourceGroups/<resource group name>/providers/Microsoft.Web/connections/dynamicsCRMconnector'.",
"type": "Microsoft.Web/connections",
"name": "[parameters('connections_dynamicsCRMconnector_name')]",
"apiVersion": "2016-06-01",
"location": "eastus",
"scale": null,
"properties": {
"displayName": "CRMConnection",
"customParameterValues": {},
"api": {
"id": "/subscriptions/<guid>/providers/Microsoft.Web/locations/eastus/managedApis/dynamicscrmonline"
}
},
"dependsOn": []
}
The other connectors (SFTP, storage account, etc.) have the missing elements node documented here and there (nothing official from MS, but blog posts and sample code) but I can't find the information for the Dynamics connectors. As an example of what I would expect to see, here is how SFTP and storage accounts can be pre-configured with authentication values in ARM:
{
"type": "Microsoft.Web/connections",
"apiVersion": "2016-06-01",
"name": "[variables('sftp_conn_friendly_name')]",
"location": "[resourceGroup().location]",
"properties": {
"displayName": "SFTP connection",
"parameterValues": {
"hostName": "[variables('sftp_host')]",
"userName": "[variables('sftp_user')]",
"password": "[variables('sftp_pass')]",
"portNumber": "[variables('sftp_port')]",
"giveUpSecurityAndAcceptAnySshHostKey": true,
"disableUploadFilesResumeCapability": false
},
"api": {
"id": "[variables('sftp_conn_managed_id')]"
}
}
},
{
"type": "Microsoft.Web/connections",
"apiVersion": "2016-06-01",
"name": "[variables('storage_conn_friendly_name')]",
"location": "[resourceGroup().location]",
"properties": {
"displayName": "Blob connection",
"parameterValues": {
"accountName": "[variables('storage_account_name')]",
"accessKey": "[listKeys(variables('storage_account_name'),'2015-05-01-preview').key1]"
},
"api": {
"id": "[variables('storage_conn_managed_id')]"
}
}
}
While not a direct answer to your question, but a more general answer giving you idea how to act in such a situation. If its not documented anywhere your only hope is reversing it (and more often than not it works).
First of all, this connecter is a resource in Azure (like the ones you've written). You can use any of the available ways to get the resource properties (https://resource.azure.com, Get-AzureRmResource, REST API, various SDKs) and see what the values are like there.
Another way of going about this - creating this connector using the portal and capturing traffic with fiddler. That way you will see the exact REST call needed to créate such a connector and would be able to replicate it using ARM Template. You might not know that ARM Templates are basically proxies for REST calls. Each resource you are creating is being converted to a REST call and performed against the appropriate resource provider.