I have an Azure Data Factory with a linked Azure Databricks service like this:
The Databrick Workspace URL is not the same in all my environments so I need to parameterize it and include the parameter in the ARM template. I added a global parameter to the Data Factory and ticked "Include in ARM template" but when that was deployed, it removed the ADF's binding to the Git repo.
I have also tried it using the ARM Parameter Configuration:
I added this section under Microsoft.DataFactory/factories/linkedServices to the ARM template instead of using a global parameter:
"AzureDatabricks": {
"properties": {
"typeProperties": {
"domain": "="
}
}
},
Again it removed the ADF's Git binding when it was deployed. Does anyone know a way to parameterize a field without removing the Git binding?
After changing the ARM template as described in the question, I deployed and it did remove my Git binding. I added the binding back in and deployed again. Now I have both the Git binding and the parameter I needed.
Related
I have a couple of questions in the following line ARM template code
"variables": {
"apiVersion": "2016-06-01",
"SharedTemplateSAS": "[concat('?',split(deployment().properties.templateLink.uri,'?')[1])]",
What is the argument '?' going to be ?
Is the properties.templateLink.uri dependent on the TemplateLink URI? (I want to use templatespec and refer to linked templates by relative path)
Finally, what is the line doing exactly?
"SharedTemplateSAS":
"[concat('?',split(deployment().properties.templateLink.uri,'?')1)]",
What is the argument '?' going to be ? Is the
properties.templateLink.uri dependent on the TemplateLink URI? (I want
to use templatespec and refer to linked templates by relative path)
Looking at the sample examples in the git Hub for linked templates, the template link uri property is used in the linked templates inorder to get the uri of the childtemplate which contains the information about the resources that are going to be deployed using those linked templates.
Based on the Azure documentation, the SharedTemplateSAS variable parameters will be below format
uri": "[concat(uri(deployment().properties.templateLink.uri, 'helloworld.json'), parameters('containerSasToken'))]",
Wherein helloworld.json is the linked template which is stored in the storage account inorder to avoid the public access to that template.
if you are working with Template spec securely,template spec stores the main template and its linked templates as a resource in your Azure subscription. You use Azure RBAC to grant access to users who need to deploy the template.
You can refer this documentation, if you want to create a template spec with link template & to deploy template spec as linked template.
I have automated the Azure ADF Pipeline Deployment process using Azure DevOps CI/CD pipelines with the help of https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment (i.e) Deploying pipelines from DEV to PROD environment ADF. I am using ARM Templates of the ADF to deploy pipelines from one environment to another. Hence I will be having a separate ARM_Parameter.json corresponding to each environment(Dev/Prod).
The Problem is each ADF pipeline may have few base parameres along with it, which is not parameterized and hence it will not be available in parameter.json. Can you guys help me to replace the Dev Values with the PROD Values in Base Parameter section under each ADF Pipelines in an automated way during this automated ADF pipeline deployment process using CI/CD Pipelines?
I see two options:
If it's only for this RUN_ENVIRONMENT parameter, you could change your parameter to variable and use the system variable #pipeline().DataFactory to determine what environment you're running in.
Otherwist, you can configure the Data Factory to generate ARM Parameters for your pipeline parameter default values, but you'll have to create a custom arm-template-parameters-definition.json file. Check the documentation here
You could use Custom parameter with ARM template.
The custom parameter for Pipeline could look like this:
"Microsoft.DataFactory/factories/pipelines": {
"properties": {
"parameters": {
"RUN_ENVIRONMENT": "=:-:string"
}
}
},
Replace the Dev Values with the PROD Values in Base Parameter section
Based on your screenshot, the RUN_ENVIRONMENT is the parameter of pipeline, which means while convert to ARM template, its format like:
"resources": [
{
....
....
"properties": {
"parameters": {
"RUN_ENVIRONMENT": {
"type": "string",
"defaultValue": "pro"
}
},...
},...
}
]
It can not be replaced by using Override template parameters in ARM deploy task. Because it will prompt The template parameters 'environment' in the parameters file are not valid; they are not present in the original template and can therefore not be provided at deployment time.
To around this error, just install one extension and add the Replace token task into the pipeline which before ARM deploy task. And this task will replace the value of content during the build runtime:
For how to apply this task in our pipeline, you could have a refer to my answer1 and answer2
There is another approach to publish ADF, from master (collaboration) branch.
You can define (replace) value for every single node (property) in JSON file (ADF object).
It will resolve your problem as you can provide a separate CSV config file per each environment (stage).
Example of CSV config file (config-stage-UAT.csv):
type,name,path,value
pipeline,PL_CopyMovies,activities[0].outputs[0].parameters.BlobContainer,UAT
Then just run such cmdlet in PowerShell:
Publish-AdfV2FromJson -RootFolder "$RootFolder" -ResourceGroupName "$ResourceGroupName" -DataFactoryName "$DataFactoryName" -Location "$Location" -Stage "stage-UAT"
Check this out:
azure.datafactory.tools (PowerShell module)
I'm trying to create a webapp in azure using VSTS as the source controls. I have this:
"properties": {
"repoUrl": "https://clt-8601add0-7378-4c18-839d-8c46ac1cdd98.visualstudio.com/ResourceManagerTest/_git/[ProjectName]",
"branch": "master",
"isManualIntegration": true
}
But I get a failed provisioning state when I try this. What URL do I have to use in the template if I want to use VSTS?
I dont think VSTS allows public repos, so you would have to authenticate to the VSTS before doing this template deployment (if this is even possible).
But since you have VSTS you are probably better off configuring a proper build\release pipeline using VSTS.
I've managed to solve my own problem. Instead of using the git url provided by the repo, use: https://[username]:[Personal Access Token]#[url] to authenticate with VSTS
so in my example, I might use
https://collin:123456780abcdefg#clt-8601add0-7378-4c18-839d-8c46ac1cdd98.visualstudio.com/ResourceManagerTest/_git/[ProjectName]
Normally when i deploy through visual studio _artifactsLocation shows when editing the parameters so what should this be in VSTS and how do I set it?
2018-02-21T08:49:46.1918199Z ##[error]Deployment template validation failed: 'The value for the template parameter '_artifactsLocation' at line '1' and column '182' is not provided. Please see https://aka.ms/arm-deploy/#parameter-file for usage details.'.
2018-02-21T08:49:46.1919769Z ##[error]Task failed while creating or updating the template deployment.
You can specify it in parameters file, then specify the file path in Template parameters input box of Azure Resource Group Deployment task if you are using.
Also, the parameters can be override by specifying in Override template parameters input box of Azure Resource Group Deployment task.
If you are calling script through Azure PowerShell task, you can specify it in the arguments: -ArtifactStagingDirectory, related issue: The value for the template parameter '_artifactsLocation' is not provided
This sounds like you are using the Azure Resource Group deployment template from VS to deploy via VSTS.
It uses MSDeploy as part of the ARM template deployment to deploy your service.
The Powershell script that is generated by the VS project template uploads a ZIP-file containing your service to Blob storage, and puts the URL and other information into _artifactsLocation and other ARM template parameters.
Instead of doing that, you can remove the artifacts related parameters and the MSDeploy resource from the ARM template. Then the template ONLY contains infrastructure related resources.
After this, add a "Deploy to App Service" step in the VSTS Release pipeline after the ARM template deployment. That can then be used to deploy your service code.
If you are using a separate parameters json file, you'll need to initialise the _artifactsLocation and _artifactsLocationSasToken there. You can give them empty strings, like:
"_artifactsLocation": {
"value": ""
},
"_artifactsLocationSasToken": {
"value": ""
},
They should automatically get their values from a PowerShell script. I'm using the AzureResourceManagerTemplateDeployment#3 task, it would probably work with AzureResourceGroupDeployment#2 as well.
I see several examples of Azure Resource Manager templates, referencing artifacts directly in Git Hub.
As in the following example, taken from this quick start template:
"modulesUrl": {
"type": "string",
"defaultValue": "https://github.com/Azure/azure-quickstart-templates/raw/master/dsc-extension-azure-automation-pullserver/UpdateLCMforAAPull.zip",
"metadata": {
"description": "URL for the DSC configuration package. NOTE: Can be a Github url(raw) to the zip file (this is the default value)"
}
As an orgnaisation, we can't use free Git Hub as code is public and as we pay for VSTS already... At the moment, we have to upload artifacts to Azure Storage Accounts using the VSTS build task Azure Resource Group Deployment task and reference them from there. It would be nice if we could remove this step.
So, is there a way to reference artifacts directly from a VSTS repository in a similar way to Git Hub? I assume the URI would require some form of authentication, such as a PAT token.
All I can find is this, but I think it is referring to packages. I don't need to create packages for ARM templates and DSC configurations.
There is a task called Azure Resource Group Deployment Task, we use this to deploy the ARM template.
According to you sample template, it's using publicly accessible http/https URLs in GitHub. Afraid this is not accessible via vsts url. In VSTS you need to follow below process (Need to use a SAS Token):
You could provide some extra parameters using the output variables defined in the Azure File Copy Task (storageURI, storageToken). This are needed because in the template we use the _artifactsLocation and _artifactsLocationSasToken parameters to build the storage URL to the files.
More details please refer this blog: Setting up VSTS with ARM Templates