Consider following test Mapping for Data Factory Copy activity:
"translator": {
"columnMappings": "#json('{\"from\":\"to\"}')",
"type": "TabularTranslator"
}
After deploying pipeline with the help of Set-AzureRmDataFactoryV2Pipeline PowerShell cmdlet we get normally deployed pipeline with the exact columnMappings value as specified in source code. But if you try to be more dynamic:
"translator": {
"columnMappings": "#json(pipeline().parameters.Mapping)",
"type": "TabularTranslator"
}
then after deployment you'll find that translator element is completely missing in pipeline. A workaround - set translator in Azure Portal Data Factory pipeline editing UI (either in Designer or JSON modes - both options work). But if after these manipulations you save pipeline JSON to the file and attempt to deploy it via Set-AzureRmDataFactoryV2Pipeline PowerShell cmdlet - bang, translator becomes missing. Expected result - deployment shall preserve translator element, because Portal JSON Editor preserves it.
We are doing automated deployment of pipelines (as you already figured out - with the help of Set-AzureRmDataFactoryV2Pipeline) and this bug breaks our automated deployment because it requires manual postdeployment pipeline editing on Azure Portal UI.
What may be the reason of such a buggy behavior? Can you suggest an idea how to work around this bug in automated way, or how to fix the code so it can be properly deployed with Set-AzureRmDataFactoryV2Pipeline?
You could try whether "Update-Module -Name AzureRm.DataFactoryV2" helps. It might be caused by that your powershell module is out of date.
Related
I would like to create an Azure DevOps Pipeline Environment from Powershell.
Using Azure CLI or the Azure REST API however, I can not find any information on this.
There are some notions about the environments in the release but that's not what I need.
When using the portal following URL is called "/_apis/distributedtask/environments" but can't find any information about this REST API endpoint.
Does anyone know how to automate this?
You're right, If I check the network section when I create a new environment I can see it uses this api:
https://dev.azure.com/{org}/{project}/_apis/distributedtask/environments
With this JSON body:
{
"description":"",
"name":"test"
}
I don't see it domunetd but it should work :)
I created in Visual Studio 2017 a new Azure Resource Group and selected Logic Apps. In the project there is Deploy-AzureResourceGroup.ps1.
I want to change parameters in the json file accordingly with the parameters in the LogicApp.json
If I run it, it seems it working but nothing is created in Azure. I change the parameters file
{
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
"contentVersion": "1.0.0.0",
"parameters": {
"logicAppName": {
"value": "LogicApps-Test-Deploy"
},
"logicAppLocation": {
"value": "northeurope"
}
}
}
And nothing happened. I tried to generate in the Azure portal an Automatic Script: in this case I have a script and a template for all my resources (very long and complicate file).
Basically I want to create different scripts for different environment. What is the right process for that?
Don't "create different scripts for different environments". Instead, aim to have a single script ( DeployAzureResourceGroup.ps1 ), a single template to deploy your logic app ( LogicApp.json ), and different versions of your parameters file to parameterize the template. For example,
LogicApp.parameters.dev.json
LogicApp.parameters.test.json
LogicApp.parameters.prod.json
This will enable you to deploy the same infrastructure consistently and reliably, across multiple subscriptions (ie: subscriptions for dev, test, and prod environments).
Also, use a PowerShell console window or PowerShell ISE to test your work. Some of your problem could be simply trying to execute the template in Visual Studio. I've had intermittent issues in the past (mostly token cache issues) executing ARM template deployments from Visual Studio and finally just got in the practice of testing the code from a PowerShell environment, which is what most users of the scripts and templates will be doing anyway.
If you have already created an ARM template for logic app, you can also use Template feature in Azure Portal to fix any issues with your ARM template. I normally use it while developing any new ARM template. This also helps with a detailed error report and in general I find it a better and quick way of identifying any issue.
Have a look the link below -
https://azure.microsoft.com/en-gb/updates/deploy-custom-templates-from-the-preview-portal/
i have a requirement to create an ADF pipeline using ARM template in powershell and it has to take inputs/validate few things from existing ADF piepline, for that reason i have to download the ARM tempalte for existing ADF pipeline through powershell. Can we do that for single ADF pipeline or multiple ones?
Note: existing pipeline is not created through ARM deployment, so i cant use "Save-AzureRmDeploymentTemplate" as i dont have deployment name created when pipeline is created through portal..
Any help is really appreciated.
Maybe you want to take a look at Export-AzureRmResourceGroup.
But I guess you can only export the entire resource group which may contain other things. You need put your data factory in a special resource group if you only want to export ADF.
Another way is to export arm template in ADF UI.
I am trying to delete some components of data factory in ARM template json file. For an example if I have pipeline A,B,C and I would like to only deploy pipeline A into the data factory. Is there any way to accomplish this? I found only command that delete after deploy but I would prefer to delete it before deploying.
I also consider the option of deleting some of json key and value inside "resource" key in template file. Anyone ever tried this?
You could modify the ARM template json file.
Please reference this doc about Continuous integration lifecycle.
Please take a look of this post. The other answers (not mine) provided a better answer.
CI/CD pipeline for Data factory V1 using VSTS
I see several examples of Azure Resource Manager templates, referencing artifacts directly in Git Hub.
As in the following example, taken from this quick start template:
"modulesUrl": {
"type": "string",
"defaultValue": "https://github.com/Azure/azure-quickstart-templates/raw/master/dsc-extension-azure-automation-pullserver/UpdateLCMforAAPull.zip",
"metadata": {
"description": "URL for the DSC configuration package. NOTE: Can be a Github url(raw) to the zip file (this is the default value)"
}
As an orgnaisation, we can't use free Git Hub as code is public and as we pay for VSTS already... At the moment, we have to upload artifacts to Azure Storage Accounts using the VSTS build task Azure Resource Group Deployment task and reference them from there. It would be nice if we could remove this step.
So, is there a way to reference artifacts directly from a VSTS repository in a similar way to Git Hub? I assume the URI would require some form of authentication, such as a PAT token.
All I can find is this, but I think it is referring to packages. I don't need to create packages for ARM templates and DSC configurations.
There is a task called Azure Resource Group Deployment Task, we use this to deploy the ARM template.
According to you sample template, it's using publicly accessible http/https URLs in GitHub. Afraid this is not accessible via vsts url. In VSTS you need to follow below process (Need to use a SAS Token):
You could provide some extra parameters using the output variables defined in the Azure File Copy Task (storageURI, storageToken). This are needed because in the template we use the _artifactsLocation and _artifactsLocationSasToken parameters to build the storage URL to the files.
More details please refer this blog: Setting up VSTS with ARM Templates