Below I have an Azure CI pipeline written in YAML, I already created a KV with 2 secrets (Username and Password) and their respective values (admin and password). Now, I have been trying to refer the secrets in variables into task: SqlAzureDacpacDeployment#1 but it doesnt work.
If I put $Username and $Password in SqlUsername and SqlPassword, I'd get this error Cannot validate argument on parameter 'Username'.
If I put '$(Username)'and '$(Password)' in SqlUsername and SqlPassword, I'd get this error Login failed for user '***'.
What should I put there or how do I refer them properly? Thanks!
trigger:
- master
pool:
vmImage: 'windows-latest'
variables:
$Username: ""
$Password: ""
steps:
- task: AzureKeyVault#1
displayName: 'Get credentials from Key-Vault'
inputs:
azureSubscription: 'Test-SC'
KeyVaultName: 'Test-KV'
SecretsFilter: '*'
RunAsPreJob: false
- task: SqlAzureDacpacDeployment#1
displayName: 'Reg Database DDL Script'
inputs:
SqlUsername: $Username
SqlPassword: $Password
enabled: true
Found the answer, so set RunAsPreJob: true and SqlUsername: '$(Username)' SqlPassword: '$(Password)' make sure you have access policies added and the name of the KV and azureSubscription are both correct
Related
I am able to set a variable on powershell or bash script and pass the variable to another task by using ##vso[task.setvariable variable=abc;]xyz. But couldn't find any documentation to set variable on azure tasks like azure webapp deploy task or SqlAzureDacpacDeployment task. I want to catch the error by passing the variable value. Is there any effective way to catch the azure task error log for the next task?
- task: SqlAzureDacpacDeployment#1
displayName: 'Insertion SQL Task'
inputs:
azureSubscription: 'Org (xxxxx-xxxx-xxxx-xxxx-xxxx)'
ServerName: 'tcp:abc.database.windows.net'
DatabaseName: test_db
SqlUsername: '$(user)'
SqlPassword: $(pass)
deployType: SqlTask
SqlFile: 'SQL/test.sql'
enabled: true
- task: AzureWebApp#1
displayName: 'Azure Web App Deploy: $(AppName)'
inputs:
azureSubscription: 'Org (xxxxx-xxxx-xxxx-xxxx-xxxx)'
appType: webApp
ResourceGroupName: 'Test'
appName: '$(AppName)'
package: '$(Build.ArtifactStagingDirectory)\app/*.zip'
deploymentMethod: zipDeploy
enabled: true
You will need to set a name for the task.
For Azure SQL Database Deployment task, you can use the SqlDeploymentOutputFile output variable name.
- task: SqlAzureDacpacDeployment#1
displayName: 'Insertion SQL Task'
name: sqlInsert
inputs:
...
enabled: true
- script: echo "$(sqlInsert.SqlDeploymentOutputFile)"
The Azure Web App task does not provide the same mechanism. You could always call the Pipelines - Get Logs API to get what you need:
pool:
vmImage: ubuntu-latest
steps:
- powershell: Write-Host "Hello World"
name: 'HelloWorld'
- powershell: |
Write-Host "This is a script that could use $env:SYSTEM_ACCESSTOKEN"
# Construct the REST URL to obtain Build ID
$uri = "$(System.CollectionUri)$(System.TeamProject)/_apis/pipelines/$(System.DefinitionId)/runs/$(Build.BuildId)/logs?api-version=6.0-preview.1"
Write-Host "$uri"
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f "system", $env:SYSTEM_ACCESSTOKEN)))
$logs = Invoke-RestMethod -Uri $uri -Method Get -ContentType "application/json" -Headers #{Authorization=("Basic {0}" -f $base64AuthInfo)}
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
I have two parameter list and i want to pass these parameter values in sql task and webapp deploy task on azure pipeline yml.
parameters:
- name: db
type: object
default: [db1, db2, db3.......]
- name: apps
type: object
default: [app1, app2, app3......]
steps:
- ${{ each dblist in parameters.db && each applist in parameters.apps}}:
- task: SqlAzureDacpacDeployment#1
displayName: 'Azure SQL SqlTask ${{ db }}'
inputs:
azureSubscription: '$(Parameters.connectedServiceName)'
ServerName: xxxx.xxxx.windows.net
DatabaseName: ${{ dblist }}
SqlUsername: xxxxx
SqlPassword: xxxxx
deployType: SqlTask
SqlFile: 'Table.sql'
- task: AzureWebApp#1
displayName: 'Azure Web App Deploy ${{ applist }}'
continueOnError: true
inputs:
azureSubscription: '$(Parameters.connectedServiceName)'
appType: webApp
ResourceGroupName: $(group)
appName: ${{ applist }}
package: '$(build.artifactstagingdirectory)/**/*.zip'
When i run this pipeline, i got error and sequence that i want to run might not be right.
How can i run tasks like this loop:
db1
app1
db2
app2
db3
app3...... and so on
How the issue can be solved?
So you have an array of app + db that you would like to deploy in sequence if i understand.
Array are just objects so you should be able to do something like that
parameters:
- name: configurations
type: object
default:
- app: app1
db: db1
- app: app2
db: db2
steps:
- ${{ each configuration in parameters.configurations }}:
- pwsh: Write-Host Hello ${{ configuration.db }}
displayName: Deploy database
- pwsh: Write-Host Hello ${{ configuration.app }}
displayName: Deploy webapp
I currently build all our Azure Infrastructure using Terraform via Azure DevOps Pipelines. This has been working well and we have a standard pipeline which calls two templates
Plan Stage
Deployment Job
Planning Template
Runs Every Time
Apply Stage
Deployment Job
Apply Template
Runs with manual approval check
Now this works fine, but what I want to do is only have the apply step run if there are changes to make. I have found other articles on how to get a variable set in the plan stage which I can do and it works fine.
I can call this same variable in the next step
variables:
varFromPlanStage: $[stageDependencies.Plan.planning_stage.outputs['planning_stage.terraformPlanResult.terraformChanges']]
steps:
- script: echo $(varFromPlanStage)
But the problem comes in when I try and use this same variable in a condition.
I found that the way you all it is different, needing a dependency instead of stagedependancy, but no matter what I try I can't get it to work.
The pipeline looks like this.
stages:
- stage: 'Plan'
displayName: 'Planning'
jobs:
- deployment: planning_stage
displayName: 'Planning Changes'
pool:
vmImage: 'Ubuntu-20.04'
environment: 'planning'
strategy:
runOnce:
deploy:
steps:
- template: /Pipelines/10-TEST-terraform-planning-template.yml # Run the Planning Template
parameters:
terraform_version: ${{ parameters.terraform_version }}
terraform_backend_service_arm: ${{ parameters.terraform_backend_service_arm }}
terraform_backend_resource_group: ${{ parameters.terraform_backend_resource_group }}
terraform_backend_storage_account: ${{ parameters.terraform_backend_storage_account }}
terraform_backend_storage_container: ${{ parameters.terraform_backend_storage_container }}
terraform_state_key: ${{ parameters.terraform_state_key }}
git_working_directory: ${{ parameters.git_working_directory }}
# This is the Build Stage - Only do this when on the master branch (which is via a PR)
- stage: 'Apply'
condition: and(succeeded(), eq(dependencies.Plan.planning_stage.outputs['planning_stage.terraformPlanResult.terraformChanges'], 'true'))
variables:
varFromPlanStage: $[stageDependencies.Plan.planning_stage.outputs['planning_stage.terraformPlanResult.terraformChanges']]
displayName: 'Applying Changes'
jobs:
- deployment: applying_stage
displayName: 'Lets Build'
pool:
vmImage: 'Ubuntu-20.04'
environment: 'building'
strategy:
runOnce:
deploy:
steps:
- script: echo $(varFromPlanStage) # Just a test
- template: /Pipelines/20-TEST-terraform-apply-template.yml # Run the Apply Template
parameters:
terraform_version: ${{ parameters.terraform_version }}
terraform_backend_service_arm: ${{ parameters.terraform_backend_service_arm }}
terraform_backend_resource_group: ${{ parameters.terraform_backend_resource_group }}
terraform_backend_storage_account: ${{ parameters.terraform_backend_storage_account }}
terraform_backend_storage_container: ${{ parameters.terraform_backend_storage_container }}
terraform_state_key: ${{ parameters.terraform_state_key }}
git_working_directory: ${{ parameters.git_working_directory }}
And the part of the Planning Template that exports the variable is called terraformPlanResult with the var being terraformChanges
Any idea what I am doing wrong here, and why I can't call the variable as a condition but I can as part of the steps?
Thx!
This appears to work differently depending on whether you're setting the variable in a 'deployment' job, or a 'job' job. After some trial, error and googling I managed to get working for both. Examples below :-)
Example passing variables from a 'deployment' job.
# Example passing variables from a 'deployment' job.
stages:
# Create some variables to pass to next stage.
- stage: 'A'
jobs:
- deployment: 'A1'
pool:
vmImage: 'windows-2019'
environment: 'test'
strategy:
runOnce:
deploy:
steps:
# Create a variable.
- task: PowerShell#2
name: foo
displayName: 'Create a variable.'
inputs:
targetType: 'inline'
script: |
echo "##vso[task.setvariable variable=bar;isOutput=true]apple"
# Check variable.
- task: PowerShell#2
displayName: 'Check a variable.'
inputs:
targetType: 'inline'
script: |
Write-Host "$env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(foo.bar)
# Confirm condition works & variables are available for use.
- stage: 'B'
dependsOn:
- 'A'
variables:
- name: varFromStageA
# stageDependencies.stageName.deploymentName.outputs['deploymentName.stepName.variableName']
value: $[ stageDependencies.A.A1.outputs['A1.foo.bar'] ]
# dependencies.stageName.outputs['deploymentName.deploymentName.stepName.variableName']
condition: and(succeeded(), eq(dependencies.A.outputs['A1.A1.foo.bar'], 'apple'))
jobs:
- job: 'B1'
pool:
vmImage: 'windows-2019'
steps:
# Confirm variable has been passed between stages.
- task: PowerShell#2
displayName: 'Confirm var passed between stages'
inputs:
targetType: 'inline'
script: |
Write-Host "$env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(varFromStageA)
Example passing variables from a 'job' job.
# Example passing variables from a 'job' job.
stages:
# Create some variables to pass to next stage.
- stage: 'A'
jobs:
- job: 'A1'
pool:
vmImage: 'windows-2019'
steps:
# Create a variable.
- task: PowerShell#2
name: foo
displayName: 'Create a variable.'
inputs:
targetType: 'inline'
script: |
echo "##vso[task.setvariable variable=bar;isOutput=true]apple"
# Check variable.
- task: PowerShell#2
displayName: 'Check a variable.'
inputs:
targetType: 'inline'
script: |
Write-Host "$env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(foo.bar)
# Confirm condition works & variables are available for use.
- stage: 'B'
dependsOn:
- 'A'
variables:
- name: varFromStageA
# stageDependencies.stageName.jobName.outputs['stepName.variableName']
value: $[ stageDependencies.A.A1.outputs['foo.bar'] ]
# dependencies.stageName.outputs['jobName.stepName.variableName']
condition: and(succeeded(), eq(dependencies.A.outputs['A1.foo.bar'], 'apple'))
jobs:
- job: 'B1'
pool:
vmImage: 'windows-2019'
steps:
# Confirm variable has been passed between stages.
- task: PowerShell#2
displayName: 'Confirm var passed between stages'
inputs:
targetType: 'inline'
script: |
Write-Host "$env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(varFromStageA)
What's working for me (by many trials and errors) is the format:
dependencies.stage_name.outputs['job_name.step_name.variable_name']
In your case, this would be:
dependencies.Plan.outputs['planning_stage.terraformPlanResult.terraformChanges']
If that doesn't work, the only unusual thing is that your planning_stage is a deployment. You might try and change it to a regular job.
Here's a pipeline that tests various ways of referencing outputs in conditions.
For me it looks like an issue on Azure Devops (later O will preapre minimial working example and create a bug for this). Because the same syntax works correctly for regular jobs but not for deployment. Also you can use condition like condition: in(dependencies.A.result, 'Succeeded', 'SucceededWithIssues', 'Skipped') but not outputs. And according to this
"dependencies": {
"<STAGE_NAME>" : {
"result": "Succeeded|SucceededWithIssues|Skipped|Failed|Canceled",
"outputs": {
"jobName.stepName.variableName": "value"
}
},
"...": {
// another stage
}
}
it is correct syntax.
I also checked this but using life cycle hook instead of job name doesn't helped me.
What is trange that you can use output variables in job condition if ther are in the same stage.
Here is link to the issue.
I'm trying to create a multi-stage pipeline which has Variable Groups defined for each stage of the pipeline. The goal is to pass values from the group as parameters to a sub template. It seems the value of the Group, defined at the stage, is not getting passed in to the sub template. It overrides the "DEFAULTVALUE" with an empty string.
pipeline.yml
trigger:
- none
pool:
name: 'Azure Pipelines'
vmImage: windows-latest
stages:
- stage: DEV
variables:
- group: my-group-dev
jobs:
- template: sub-template.yml
parameters:
env: 'dev'
subscriptionName: '$(SubscriptionName)' # This reference from the variable group doesn't get passed in
subscriptionId: '$(SubscriptionId)'
- stage: TEST
variables:
- group: my-group-test
jobs:
- template: sub-template.yml
parameters:
env: 'test'
subscriptionName: '$(SubscriptionName)' # This reference from the variable group doesn't get passed in
subscriptionId: '$(SubscriptionId)'
sub-template.yml
parameters:
env: 'DEFAULTVALUE'
subscriptionName: 'DEFAULTVALUE'
subscriptionId: 'DEFAULTVALUE'
jobs:
- deployment: ResourceDeployment
displayName: Deploy Resources ${{ parameters.env }}
environment: ${{ parameters.env }}
strategy:
runOnce:
deploy:
steps:
- task: AzureFileCopy#4
displayName: 'Upload ARM Templates'
inputs:
sourcePath: '$(Pipeline.Workspace)/templates'
azureSubscription: '${{ parameters.subscriptionName }}'
destination: 'azureBlob'
storage: 'my-storage-account'
containerName: 'arm'
name: AzureFileCopy
- task: AzureResourceManagerTemplateDeployment#3
inputs:
deploymentScope: 'Resource Group'
azureResourceManagerConnection: '${{ parameters.subscriptionName }}'
subscriptionId: '${{ parameters.subscriptionId }}'
action: 'Create Or Update Resource Group'
resourceGroupName: 'my-resource-group'
location: 'eastus2'
templateLocation: 'URL of the file'
csmFileLink: '$(AzureFileCopy.StorageContainerUri)${{ parameters.env }}/templates/main.json$(AzureFileCopy.StorageContainerSasToken)'
I have also tried adding the variable group within the sub-template but that also doesn't parse correctly...
Does anyone know if this is possible?
It is a known issue that the service connection endpoint cannot be referenced in variable groups defined under stage level.
You can fix this issue by below workarounds:
1, Define the variable groups in the global level instead of stage level See below:
trigger:
- none
pool:
name: 'Azure Pipelines'
vmImage: windows-latest
# define the variable group under global level.
variables:
- group: my-group-dev
- group: my-group-test
stages:
- stage: DEV
jobs:
- template: sub-template.yml
parameters:
env: 'dev'
subscriptionName: '$(SubscriptionName)' # This reference from the variable group doesn't get passed in
subscriptionId: '$(SubscriptionId)'
- stage: TEST
jobs:
- template: sub-template.yml
parameters:
env: 'test'
subscriptionName: '$(SubscriptionName)' # This reference from the variable group doesn't get passed in
subscriptionId: '$(SubscriptionId)'
2, Link the variable groups on the UI page.
On the yaml pipeline edit page--> Click the 3dots-->Triggers-->Variables Tab-->Link Variable group
Please see below threads for more information.
Using a variable for the service connection
Azure subscription endpoint ID cannot be provided through a variable in build definition YAML file
You need to pass variable group name to template and on template level apply it on variables scope like this:
parameters:
env: 'DEFAULTVALUE'
subscriptionName: 'DEFAULTVALUE'
subscriptionId: 'DEFAULTVALUE'
variableGroupName: 'DEFAULTVALUE`
jobs:
- deployment: ResourceDeployment
displayName: Deploy Resources ${{ parameters.env }}
environment: ${{ parameters.env }}
variables:
- group: {{ paramaters.variableGroupName }}
strategy:
runOnce:
deploy:
steps:
- task: AzureFileCopy#4
displayName: 'Upload ARM Templates'
inputs:
sourcePath: '$(Pipeline.Workspace)/templates'
azureSubscription: '${{ parameters.subscriptionName }}'
destination: 'azureBlob'
storage: 'my-storage-account'
containerName: 'arm'
name: AzureFileCopy
- task: AzureResourceManagerTemplateDeployment#3
inputs:
deploymentScope: 'Resource Group'
azureResourceManagerConnection: '${{ parameters.subscriptionName }}'
subscriptionId: '${{ parameters.subscriptionId }}'
action: 'Create Or Update Resource Group'
resourceGroupName: 'my-resource-group'
location: 'eastus2'
templateLocation: 'URL of the file'
csmFileLink: '$(AzureFileCopy.StorageContainerUri)${{ parameters.env }}/templates/main.json$(AzureFileCopy.StorageContainerSasToken)'
This is similar to this topic:
For security reasons, we only allow you to pass information into templated code via explicit parameters.
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
The means the author of the pipeline using your template needs to commit changes where they explicitly pass the needed info into your template code.
There are some exceptions to this, where the variable is statically defined in the same file or at pipeline compile time, but generally speaking, it’s probably better to use parameters for everything that does not involve system-defined read-only dynamic variable and custom-defined dynamic output variables.
I have to variables groups
QA
PROD
both has isProd variable
and then my template is
parameters:
variableGroupName: 'QA'
jobs:
- job: ResourceDeployment
variables:
- group: ${{ parameters.variableGroupName }}
steps:
- script: echo '$(isProd)'
and main file is
trigger: none
pr: none
stages:
- stage: QA
jobs:
- template: sub-template.yml
parameters:
variableGroupName: 'QA'
- stage: PROD
jobs:
- template: sub-template.yml
parameters:
variableGroupName: 'PROD'
and I got:
and
I have a Azure DevOps YAML Pipeline to execute a Terraform deployment using the Terraform by MS DevLabs extension and an Azure Resource Manager service connection.
The last working state was using a pipeline template yaml file however I had to configure a parameter within the template and call the variable using the template expression syntax.
...
...
stages:
- stage: Plan
displayName: Terrafom Plan
jobs:
- job: DEV PLAN
displayName: Plan (DEV)
pool:
vmImage: "ubuntu-latest"
variables:
az_service_connection: "MyServiceConnection"
tf_environment: "DEV"
tf_state_rg: "DEV"
tz_state_location: "canadacentral"
tf_state_stgacct_name: "mystorageaccuontname1231231"
tf_state_container_name: "tfstate"
steps:
- template: templates/terraform-plan.yml
parameters:
az_service_connection: ${{ variables.az_service_connection }}
...
...
steps:
- task: terraformInstaller#0
displayName: "Install Terraform $(tf_version)"
inputs:
terraformVersion: $(tf_version)
- task: TerraformTaskV1#0
displayName: "Run > terraform init"
inputs:
command: "init"
commandOptions: "-input=false"
backendServiceArm: ${{ parameters.az_service_connection }}
...
...
I believe the reason why this works is because the template expression syntax ${{ variables.varname}} evaluates at compile time vs. runtime. If I didn't do it this way, i'd either get $(az_service_connection) passed into the backendServiceArm input or an empty value.
With the introduction of variable groups, i'm now facing similar behavior. I expect that the variable group evaluates after the template expression variable which causes ${{ variables.az_service_connection }} to have an empty value. I am unsure how to get this working.
How can I use variable groups with a pipeline template that uses a service connection?
I used $() syntax to pass arm connection to template:
Template file:
parameters:
- name: 'instances'
type: object
default: {}
- name: 'server'
type: string
default: ''
- name: 'armConnection'
type: string
default: ''
steps:
- task: TerraformTaskV1#0
inputs:
provider: 'azurerm'
command: 'init'
backendServiceArm: '${{ parameters.armConnection }}'
backendAzureRmResourceGroupName: 'TheCodeManual'
backendAzureRmStorageAccountName: 'thecodemanual'
backendAzureRmContainerName: 'infra'
backendAzureRmKey: 'some-terrform'
- ${{ each instance in parameters.instances }}:
- script: echo ${{ parameters.server }}:${{ instance }}
Main file:
trigger:
branches:
include:
- master
paths:
include:
- stackoverflow/09-array-parameter-for-template/*
# no PR triggers
pr: none
pool:
vmImage: 'ubuntu-latest'
variables:
- group: my-variable-group
- name: my-passed-variable
value: $[variables.myhello] # uses runtime expression
steps:
- template: template.yaml
parameters:
instances:
- test1
- test2
server: $(myhello)
armConnection: $(armConnection)
Note: Group my-variable-group contains armConnection variable