Normally when i deploy through visual studio _artifactsLocation shows when editing the parameters so what should this be in VSTS and how do I set it?
2018-02-21T08:49:46.1918199Z ##[error]Deployment template validation failed: 'The value for the template parameter '_artifactsLocation' at line '1' and column '182' is not provided. Please see https://aka.ms/arm-deploy/#parameter-file for usage details.'.
2018-02-21T08:49:46.1919769Z ##[error]Task failed while creating or updating the template deployment.
You can specify it in parameters file, then specify the file path in Template parameters input box of Azure Resource Group Deployment task if you are using.
Also, the parameters can be override by specifying in Override template parameters input box of Azure Resource Group Deployment task.
If you are calling script through Azure PowerShell task, you can specify it in the arguments: -ArtifactStagingDirectory, related issue: The value for the template parameter '_artifactsLocation' is not provided
This sounds like you are using the Azure Resource Group deployment template from VS to deploy via VSTS.
It uses MSDeploy as part of the ARM template deployment to deploy your service.
The Powershell script that is generated by the VS project template uploads a ZIP-file containing your service to Blob storage, and puts the URL and other information into _artifactsLocation and other ARM template parameters.
Instead of doing that, you can remove the artifacts related parameters and the MSDeploy resource from the ARM template. Then the template ONLY contains infrastructure related resources.
After this, add a "Deploy to App Service" step in the VSTS Release pipeline after the ARM template deployment. That can then be used to deploy your service code.
If you are using a separate parameters json file, you'll need to initialise the _artifactsLocation and _artifactsLocationSasToken there. You can give them empty strings, like:
"_artifactsLocation": {
"value": ""
},
"_artifactsLocationSasToken": {
"value": ""
},
They should automatically get their values from a PowerShell script. I'm using the AzureResourceManagerTemplateDeployment#3 task, it would probably work with AzureResourceGroupDeployment#2 as well.
Related
I have automated the Azure ADF Pipeline Deployment process using Azure DevOps CI/CD pipelines with the help of https://learn.microsoft.com/en-us/azure/data-factory/continuous-integration-deployment (i.e) Deploying pipelines from DEV to PROD environment ADF. I am using ARM Templates of the ADF to deploy pipelines from one environment to another. Hence I will be having a separate ARM_Parameter.json corresponding to each environment(Dev/Prod).
The Problem is each ADF pipeline may have few base parameres along with it, which is not parameterized and hence it will not be available in parameter.json. Can you guys help me to replace the Dev Values with the PROD Values in Base Parameter section under each ADF Pipelines in an automated way during this automated ADF pipeline deployment process using CI/CD Pipelines?
I see two options:
If it's only for this RUN_ENVIRONMENT parameter, you could change your parameter to variable and use the system variable #pipeline().DataFactory to determine what environment you're running in.
Otherwist, you can configure the Data Factory to generate ARM Parameters for your pipeline parameter default values, but you'll have to create a custom arm-template-parameters-definition.json file. Check the documentation here
You could use Custom parameter with ARM template.
The custom parameter for Pipeline could look like this:
"Microsoft.DataFactory/factories/pipelines": {
"properties": {
"parameters": {
"RUN_ENVIRONMENT": "=:-:string"
}
}
},
Replace the Dev Values with the PROD Values in Base Parameter section
Based on your screenshot, the RUN_ENVIRONMENT is the parameter of pipeline, which means while convert to ARM template, its format like:
"resources": [
{
....
....
"properties": {
"parameters": {
"RUN_ENVIRONMENT": {
"type": "string",
"defaultValue": "pro"
}
},...
},...
}
]
It can not be replaced by using Override template parameters in ARM deploy task. Because it will prompt The template parameters 'environment' in the parameters file are not valid; they are not present in the original template and can therefore not be provided at deployment time.
To around this error, just install one extension and add the Replace token task into the pipeline which before ARM deploy task. And this task will replace the value of content during the build runtime:
For how to apply this task in our pipeline, you could have a refer to my answer1 and answer2
There is another approach to publish ADF, from master (collaboration) branch.
You can define (replace) value for every single node (property) in JSON file (ADF object).
It will resolve your problem as you can provide a separate CSV config file per each environment (stage).
Example of CSV config file (config-stage-UAT.csv):
type,name,path,value
pipeline,PL_CopyMovies,activities[0].outputs[0].parameters.BlobContainer,UAT
Then just run such cmdlet in PowerShell:
Publish-AdfV2FromJson -RootFolder "$RootFolder" -ResourceGroupName "$ResourceGroupName" -DataFactoryName "$DataFactoryName" -Location "$Location" -Stage "stage-UAT"
Check this out:
azure.datafactory.tools (PowerShell module)
I am testing a build/release of a very simple ADF (just one activity), the build, repository, arm template export all seem ok until when I run the release task, the error shows up in the final step, that says:
"error": {
"code": "ResourceNotFound",
"message": "The Resource 'Microsoft.DataFactory/factories/htTestDF' under resource group 'xxx-rg' was not found."
}
I watched several tutorials and microsoft web sites, and tried exporting the ARM template several times, the same error occurs. Any ideas will be greatly appreciated.
Thanks for your details clarifies in comments. Now, the error message you met caused by using a different target resource group as this ARM template deploy to.
To make this more clear, I reproduce the issue based on the details you provided. Fortunately, got the same error with you. Now, let's focus on its log, then get why it cause the Not Found error.(Please set debug=true)
As I mentioned in the above pic, it is the api that this task used at first step while the template begin to apply into the corresponding resource group and deployment. For more cleared, please refer to this REST API doc firstly: Deployments - Create Or Update.
The logic of this task is compile parameters from ARM template file, pack them and use it as request body for this PUT api call. See its api doc, you can get that for this API call, its resourceGroupName and deploymentName need to be specified firstly. In another words, if you specified another target resource group, it would not find the correct target place that can apply this template definition. Because, you can see that this ARM template is preparing to applying activities SetVariable into your Data Factory HTDF3 and the defined pipelines name is HTPipe1. But these should all not exists in your target Resource Group. Thus it caused the error like this:
"error": {
"code": "ResourceNotFound",
"message": "The Resource 'Microsoft.DataFactory/factories/Merlin-1003' under resource group 'Merlin-ARM-deploy' was not found."
}
In my sample, Merlin-ARM-deploy is my target resource group.
If you want to deploy this into your target resource group, you need to create one data factory manually, or use another ARM Template to create a new one same factory in the target resource group. If you choose the previous method, just then modify the template.json file, to let its parameters correspond to the actual target resource group. But if use the second method, do not operate anything. Just apply them with task.
The ARM template generated by ADF(publish) cannot be deployed directly to a new RG.
Solution
Create RG(optional, assuming it is IAC(infra as code))
Run a powershell script task in pipeline to create an empty ADF(do not use empty ARM instead). Since it is not ARM you would need to put an optional condition to check if it already exists)
Set-AzDataFactoryV2 -ResourceGroupName "RG" -Name "ADF" -Location "North Europe"
Now we can execute the ARM template from publish folder (the one you had given)
Errors in ADF publish system.
The ARM template need to be generated in such a way that it is
idempotent(should also create afresh if not present). But it is not
at the moment. It expects an ADF to be present already(strangely).
When an empty ARM template created in another RG is used to create a fresh empty ADF in this RG(newly created), it fails. Well, it creates empty ADF but we cannot put adf_publish(default publish folder for ADF) on top of it because we get 'resource not found error'.
But when we manually create an ADF and run the adf_publish template
then it works! But ofcourse, this is not what we want.
Why does manual & powershell work(empty + publish) but not ARM template? It could be that the ARM Template has wrong location/region mentioned in it but that was not the case.(really puzzling to me)
I have an ARM template, which I will be using it to deploy resources (Mentioned as in Azure Portal) via Azure DevOps Pipeline. These ARM templates were created using my Dev Subscription. If I need to use this same ARM template to move to production, I need to manually update the subscription id etc in the ARM template and then run the respective pipelines. Is there any way to automate this manual updation process in ARM template.
I have tried using File Transform Agent job available under Azure DevOps Release pipeline. But since the values are available inside nested loops of ARM templates, it failed. Is there any PowerShell script that will be suitable for this process of updating values.
I don't want to manually update the ids under the ARM Template. Instead, I want to update it automatically.
Set Json Property task might help to set the property with specific value.
First in the marketplace search for "set Json property" task and install it for your organization.
Then you can set the property path and the value accordingly. check here for detail usage
Instead of hard coding in ARM templates you can use parameters file. For dev and prod you can have separate files and while deploying pass ARM Template along with parameter file whichever is required.
You can use AKV (Azure Key Vault) to access your subscription details in parameter file
subscription().subscriptionid will get you the details
I am trying to deploy an Azure Resource group via Octopus Deploy (within it a website).
Hyak.Common.CloudException: InvalidTemplate: Deployment template validation failed: 'The value for the template parameter '_artifactsLocation' at line '40' and column '32' is not provided.
How would I specify this in Visual Studio so that the solution can get deployed to Azure?
Thanks
From an Octopus point-of-view, you can use variable substitution on both the Template and Parameter files to sub in whatever values you need to your templates.
If you have a look at the "Template Contained in a Package" section of the Azure Resource Groups documentation, it shows you example JSON templates with variable substitution in place.
Eg.
"databaseName": {
"value": "#{DatabaseName}"
},
So in your project, you'd setup a project variable, then use the variable substitution syntax to address that variable in your template JSON (that's contained in your package), and it will get substituted before being executed at Azure.
Hope this helps
I see several examples of Azure Resource Manager templates, referencing artifacts directly in Git Hub.
As in the following example, taken from this quick start template:
"modulesUrl": {
"type": "string",
"defaultValue": "https://github.com/Azure/azure-quickstart-templates/raw/master/dsc-extension-azure-automation-pullserver/UpdateLCMforAAPull.zip",
"metadata": {
"description": "URL for the DSC configuration package. NOTE: Can be a Github url(raw) to the zip file (this is the default value)"
}
As an orgnaisation, we can't use free Git Hub as code is public and as we pay for VSTS already... At the moment, we have to upload artifacts to Azure Storage Accounts using the VSTS build task Azure Resource Group Deployment task and reference them from there. It would be nice if we could remove this step.
So, is there a way to reference artifacts directly from a VSTS repository in a similar way to Git Hub? I assume the URI would require some form of authentication, such as a PAT token.
All I can find is this, but I think it is referring to packages. I don't need to create packages for ARM templates and DSC configurations.
There is a task called Azure Resource Group Deployment Task, we use this to deploy the ARM template.
According to you sample template, it's using publicly accessible http/https URLs in GitHub. Afraid this is not accessible via vsts url. In VSTS you need to follow below process (Need to use a SAS Token):
You could provide some extra parameters using the output variables defined in the Azure File Copy Task (storageURI, storageToken). This are needed because in the template we use the _artifactsLocation and _artifactsLocationSasToken parameters to build the storage URL to the files.
More details please refer this blog: Setting up VSTS with ARM Templates