Publish file content to service bus from CI pipeline - azure

In my CI pipeline I am trying to publish message to service bus and its working when its just some hardcoded text or variables, here Using "PublishToAzureServiceBus" task .
But problem is when trying to a read file from repository and then publish that to service bus.
I have tried using read file using scripting language and put to variable but its not able to work as variable is not storing big json file.
Is there any way to read file directly when publishing message to service bus.
Below is sample code snippet for debugging
trigger:
- none
pool:
vmImage: ubuntu-latest
parameters:
- name: ProjectName
displayName: Project Name
type: string
default: DevOpsDemo
- name: repoName
displayName: repo Name
type: string
default: ProjectCode
- name: branchRef
displayName: Branch Name
type: string
default: main
variables:
- name: jobStatus
value: "Failed"
- name: projectFile
value: ""
stages:
- stage: Stage1
displayName: Stage 1
jobs:
- job: CheckOutRepo
displayName: CheckOut-Repo Display
steps:
- script: |
echo "Checkout for " ${{ parameters.ProjectName}} : ${{ parameters.repoName}} : ${{ parameters.branchRef}}
name: PrintMessage
- checkout: git://${{ parameters.ProjectName}}/${{ parameters.repoName}}#refs/heads/${{ parameters.branchRef}}
name: Checkout
- task: PythonScript#0
inputs:
scriptSource: 'inline'
script: |
import json
import requests
f = open('project-release.json')
projectFile = json.load(f)
print(projectFile)
f.close()
print("Afterclosing")
print(projectFile)
- script: |
echo "Project release file" $(cat project-release.json)
name: TestPrint
- task: CopyFiles#2
inputs:
SourceFolder: 'services'
Contents: '**'
TargetFolder: $(Build.ArtifactStagingDirectory)
name: CopyFiles
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: $(Build.ArtifactStagingDirectory)
ArtifactName: 'drop'
publishLocation: 'Container'
name: PublishArtifacts
- bash: |
echo "##vso[task.setvariable variable=jobStatus]Success"
name: setVar
- bash: |
echo "##vso[task.setvariable variable=jobStatus;isOutput=true]$(jobStatus)"
echo "##vso[task.setvariable variable=projectFile;isOutput=true]$(cat project-release.json)"
name: SetStatus
condition: always()
- stage: Stage2
displayName: Stage 2
condition: always()
jobs:
- job: Publish
pool: server
variables:
jobStatus: $[ stageDependencies.Stage1.CheckOutRepo.outputs['SetStatus.jobStatus'] ]
projectFile: $[ stageDependencies.Stage1.CheckOutRepo.outputs['SetStatus.projectFile'] ]
steps:
- task: PublishToAzureServiceBus#1
inputs:
azureSubscription: 'SBConnection'
messageBody: |
{
"Status": "$(jobStatus)",
"BuildID": "$(build.buildid)",
"BuildNumber":"$(build.buildnumber)",
"projectFile":$(cat project-release.json)
}
signPayload: false
waitForCompletion: false
condition: always()

I am able to solve this by using setvariable in bash script as below
pool:
vmImage: ubuntu-latest
stages:
- stage: Stage1
displayName: Stage 1
jobs:
- job: CheckOutRepo
displayName: CheckOut-Repo Display
steps:
- checkout: git://${{ parameters.ProjectName}}/${{ parameters.repoName}}#refs/heads/${{ parameters.branchRef}}
name: Checkout
- bash: |
data=$(cat project-release.json)
echo "##vso[task.setvariable variable=jobStatus;isOutput=true]$(jobStatus)"
echo "##vso[task.setvariable variable=data;isOutput=true]"$data
name: SetStatus
condition: always()
- stage: Stage2
displayName: Stage 2
condition: always()
jobs:
- job: Publish
pool: server
variables:
jobStatus: $[ stageDependencies.Stage1.CheckOutRepo.outputs['SetStatus.jobStatus'] ]
projectFile: $[ stageDependencies.Stage1.CheckOutRepo.outputs['SetStatus.data'] ]
steps:
- task: PublishToAzureServiceBus#1
inputs:
azureSubscription: 'SBConnection'
messageBody: |
{
"Status": "$(jobStatus)",
"BuildID": "$(build.buildid)",
"BuildNumber":"$(build.buildnumber)",
"projectFile":$(projectFile)
}
signPayload: false
waitForCompletion: false
condition: always()

Related

azurepipelines template task condition not evaluating as expected

When I remove the task condition from the deploy_data_factory_steps.yaml template the parameter displays with the expected value:
parameter is DEV2
But when I run with the condition it fails to evaluate as True:
Evaluating: and(succeeded(), eq('$(env)', 'DEV2'))
Expanded: and(True, eq('$(env)', 'DEV2'))
Result: False
azurepipelines.yaml:
- stage: DataFactoryDEV2
condition: and(succeeded(), eq('${{ parameters.runEnv }}', 'dev2'))
dependsOn: DeployDEV2
variables:
- group: xxx_DEV2
pool:
name: 'yyyDev'
jobs:
- deployment: deploy
environment: xxx-dev2
strategy:
runOnce:
deploy:
steps:
- template: templates/deploy_data_factory_steps.yml
parameters:
environ: $(env)
artifact: 'publish-factory'
azureSubscription: 'yyy - xxx'
deploy_data_factory_steps.yaml
steps:
...
- task: AzureCLI#2
condition: and(succeeded(), eq('${{ parameters.environ }}', 'DEV2'))
displayName: 'Blah blah blah'
inputs:
azureSubscription: ${{ parameters.azureSubscription }}
scriptLocation: inlineScript
scriptType: bash
inlineScript: |
echo parameter is ${{ parameters.environ }}

Pipeline Artifact not Downloading with Stages

I have recently upgraded my YAML pipeline to include Stages, since this has happened the build artifacts are not downloading in a later stage task. For the life of me I cant figure out why.
Please see my YAML Code Bellow with an explanation on how my pipeline works.
First I have the main pipeline that calls the template yaml files.
mainpipeline.yml
pool:
vmImage: 'ubuntu-latest'
resources:
repositories:
- repository: Terraform
name: VALUE/Terraform
path:
- include: /Terraform
type: git
ref: main #branch name
- repository: Website
name: VALUE/Website
path:
- include: /Website
type: git
ref: newartifactpipeline #branch name
- repository: AuthenticationServer
name: VALUE/AuthenticationServer
path:
- include: /AuthenticationServer
type: git
ref: VALUE #branch name
trigger:
branches:
include:
- master
variables:
buildConfiguration: 'Release'
stages:
- stage: build_website_files
displayName: Building Main Website Files
jobs:
- job: build_main_website
steps:
- checkout: Website
- template: buildartifact.yml#website
parameters:
#vmImage: 'windows-latest'
buildConFiguration: $(buildConfiguration)
project: Website.csproj
artifactName: Website
- stage: build_authenticationserver_files
displayName: Building AuthenticationServer Website Files
jobs:
- job: build_authenticationserver_website
steps:
- checkout: AuthenticationServer
- template: buildartifact.yml#AuthenticationServer
parameters:
# vmImage: 'windows-latest'
buildConFiguration: $(buildConfiguration)
project: AuthenticationServer.csproj
artifactName: AuthenticationServer
- stage: run_terraform_pre_build
displayName: Building Terraform Applications and Deploying Web Apps
jobs:
- job: building_terraform_applications
steps:
- checkout: Terraform
- template: /VALUE/runterraform.yml#Terraform
parameters:
terraformWorkingDirectory: '$(System.DefaultWorkingDirectory)/VALUE'
serviceConnection: 'VALUE'
azureSubscription: 'VALUE'
appconnectionname: 'VALUE'
backendresourcegroupname: 'VALUE'
backendstorageaccountname: 'VALUE'
backendcontainername: 'VALUE'
RG: 'RG_Example'
azureLocation: 'UK South'
terraformVersion: '1.0.4'
artifactName: 'Website'
- stage: run_terraform_post_build
displayName: Apply Post Build Settings
jobs:
- job: apply_post_build_settings
steps:
- checkout: Terraform
- template: /Terraform/PostBuild/runterraformpostbuild.yml#Terraform
parameters:
terraformWorkingDirectory: '$(System.DefaultWorkingDirectory)/VALUE/PostBuild'
serviceConnection: 'VALUE'
azureSubscription: 'VALUE'
appconnectionname: 'VALUE'
backendresourcegroupname: ''VALUE''
backendstorageaccountname: 'VALUE'
backendcontainername: 'VALUE'
RG: 'RG_Example'
azureLocation: 'UK South'
terraformVersion: '1.0.4'
artifactName: 'Website'
The first stage builds and calls this build artifact template yaml file, that does successfully publish the artifact file, yaml code below:
Please see screenshot bellow as proof:
buildartifact.yml
parameters:
- name: buildConfiguration
type: string
default: 'Release'
- name: project
type: string
default: 'Website.csproj'
- name: artifactName
type: string
default: 'Website'
- name: vmImage
type: string
default: 'windows-latest'
jobs:
- job: build_website
pool:
vmImage: ${{ parameters.vmImage }}
steps:
- checkout: Website
- task: CmdLine#2
inputs:
script: |
echo '$(System.DefaultWorkingDirectory)'
dir
- task: DotNetCoreCLI#2
displayName: dotnet restore
inputs:
command: restore
projects: '**/${{ parameters.project }}'
# Node.js tool installer
# Finds or downloads and caches the specified version spec of Node.js and adds it to the PATH
- task: NodeTool#0
displayName: 'Install Node .js'
inputs:
versionSpec: '14.17.3'
force32bit: false # Optional
checkLatest: false # Optional
- script: |
npm install -g #angular/cli#12.1.3
npm install
displayName: 'npm install'
- task: Npm#1
displayName: 'npm run build'
inputs:
command: 'custom'
workingDir: ClientApp
customCommand: 'build'
- task: DotNetCoreCLI#2
displayName: 'Build'
inputs:
command: 'build'
projects: '**/${{ parameters.project }}'
arguments: '--configuration ${{ parameters.buildConfiguration }}'
- task: DotNetCoreCLI#2
displayName: dotnet restore unit tests
inputs:
command: restore
projects: 'UnitTests/UnitTests.csproj'
- task: DotNetCoreCLI#2
displayName: dotnet Test
inputs:
command: test
projects: 'UnitTests/UnitTests.csproj'
arguments: '--configuration Release'
- task: DotNetCoreCLI#2
displayName: 'Publish Application'
inputs:
command: 'publish'
publishWebProjects: false
projects: '**/${{ parameters.project }}'
arguments: '--configuration ${{ parameters.buildConfiguration }} --output $(Pipeline.Workspace)/website/'
- task: PublishPipelineArtifact#1
displayName: 'Publish Artifacts'
inputs:
targetPath: '$(Pipeline.Workspace)/website/'
artifact: ${{ parameters.artifactName }}
publishLocation: 'pipeline'
With this yaml for the PublishPipelineArtifact task I have tried the following env variables for pipleines: pipeline.workspace and System.DefaultWorkingDirectory
Yet both have not worked in the later stage where the final yaml file tries to download the pipeline artifact see yaml code bellow:
runterraformanddownloadartifact.yml
parameters:
- name: terraformWorkingDirectory
type: string
default: $(System.DefaultWorkingDirectory)/Terraform
- name: serviceConnection
type: string
default: value
- name: azureSubscription
type: string
default: value
- name: appconnectionname
type: string
default: value
- name: backendresourcegroupname
type: string
default: DevOpsTerraform
- name: backendstorageaccountname
type: string
default: value
- name: backendcontainername
type: string
default: value
- name: RG
type: string
default: RG_Example
- name: azureLocation
type: string
default: UK South
- name: terraformVersion
type: string
default: 1.0.4
- name: artifactName
type: string
default: Website
jobs:
- job: Run_Terraform
displayName: Installing and Running Terraform
steps:
- checkout: Terraform
- task: TerraformInstaller#0
displayName: install
inputs:
terraformVersion: '${{ parameters.terraformVersion }}'
- task: CmdLine#2
inputs:
script: |
echo '$(System.DefaultWorkingDirectory)'
dir
- task: TerraformTaskV2#2
displayName: init
inputs:
provider: azurerm
command: init
backendServiceArm: '${{ parameters.serviceConnection }}'
backendAzureRmResourceGroupName: '${{ parameters.backendresourcegroupname }}'
backendAzureRmStorageAccountName: '${{ parameters.backendstorageaccountname }}'
backendAzureRmContainerName: '${{ parameters.backendcontainername }}'
backendAzureRmKey: terraform.tfstate
workingDirectory: '${{ parameters.terraformWorkingDirectory }}'
- task: TerraformTaskV1#0
displayName: plan
inputs:
provider: azurerm
command: plan
commandOptions: '-input=false'
environmentServiceNameAzureRM: '${{ parameters.serviceConnection }}'
workingDirectory: '${{ parameters.terraformWorkingDirectory }}'
- task: TerraformTaskV1#0
displayName: apply
inputs:
provider: azurerm
command: apply
commandOptions: '-input=false -auto-approve'
environmentServiceNameAzureRM: '${{ parameters.serviceConnection }}'
workingDirectory: '${{ parameters.terraformWorkingDirectory }}'
- job: Put_artifacts_into_place
displayName: Putting_artifacts_into_place
dependsOn: Run_Terraform
steps:
- checkout: Website
- checkout: AuthenticationServer
- task: DownloadPipelineArtifact#2
displayName: Download Build Artifacts
inputs:
artifact: '${{ parameters.artifactName }}'
patterns: /website/**/*.zip
path: $(Pipeline.Workspace)/website/
- task: AzureWebApp#1
displayName: 'Azure Web App Deploy: VALUE'
inputs:
package: $(Pipeline.Workspace)/website/**/*.zip
azureSubscription: '${{ parameters.azureSubscription }}'
ConnectedServiceName: '${{ parameters.appconnectionname}}'
appName: VALUE
ResourceGroupName: '${{ parameters.RG}}'
- task: DownloadPipelineArtifact#2
displayName: Download Build Artifacts
inputs:
artifact: '${{ parameters.artifactName}}'
patterns: /authsrv/**/*.zip
path: $(Pipeline.Workspace)/authsrv/
- task: AzureWebApp#1
displayName: 'Azure Web App Deploy: VALUE'
inputs:
package: $(Pipeline.Workspace)/authsrv/**/*.zip
azureSubscription: '${{ parameters.azureSubscription }}'
ConnectedServiceName: '${{ parameters.appconnectionname}}'
appName: VALUE
ResourceGroupName: '${{ parameters.RG}}'
Essentially the first pipeline calls these two pipelines which are wrapped in Stages. Before they were not wrapped in stages and this pipeline worked well. Since moving it to stages I have had the issue where the task: DownloadPipelineArtifact#2 completes but downloads nothing. Please see screen shot bellow:
The error I am getting at the end of the pipeline is:
##[error]Error: No package found with specified pattern: /home/vsts/work/1/website/**/*.zip<br/>Check if the package mentioned in the task is published as an artifact in the build or a previous stage and downloaded in the current job.
I have tried the following solutions without success:
File pattern for Publish Pipeline Artifact in Azure DevOps
how to use PublishPipelineArtifact#1 with build script
And consulted MS Doc: https://learn.microsoft.com/en-us/azure/devops/pipelines/artifacts/pipeline-artifacts?view=azure-devops&tabs=yaml

Azure DevOps Use a Variable as a Condition between Deployment Stages

I currently build all our Azure Infrastructure using Terraform via Azure DevOps Pipelines. This has been working well and we have a standard pipeline which calls two templates
Plan Stage
Deployment Job
Planning Template
Runs Every Time
Apply Stage
Deployment Job
Apply Template
Runs with manual approval check
Now this works fine, but what I want to do is only have the apply step run if there are changes to make. I have found other articles on how to get a variable set in the plan stage which I can do and it works fine.
I can call this same variable in the next step
variables:
varFromPlanStage: $[stageDependencies.Plan.planning_stage.outputs['planning_stage.terraformPlanResult.terraformChanges']]
steps:
- script: echo $(varFromPlanStage)
But the problem comes in when I try and use this same variable in a condition.
I found that the way you all it is different, needing a dependency instead of stagedependancy, but no matter what I try I can't get it to work.
The pipeline looks like this.
stages:
- stage: 'Plan'
displayName: 'Planning'
jobs:
- deployment: planning_stage
displayName: 'Planning Changes'
pool:
vmImage: 'Ubuntu-20.04'
environment: 'planning'
strategy:
runOnce:
deploy:
steps:
- template: /Pipelines/10-TEST-terraform-planning-template.yml # Run the Planning Template
parameters:
terraform_version: ${{ parameters.terraform_version }}
terraform_backend_service_arm: ${{ parameters.terraform_backend_service_arm }}
terraform_backend_resource_group: ${{ parameters.terraform_backend_resource_group }}
terraform_backend_storage_account: ${{ parameters.terraform_backend_storage_account }}
terraform_backend_storage_container: ${{ parameters.terraform_backend_storage_container }}
terraform_state_key: ${{ parameters.terraform_state_key }}
git_working_directory: ${{ parameters.git_working_directory }}
# This is the Build Stage - Only do this when on the master branch (which is via a PR)
- stage: 'Apply'
condition: and(succeeded(), eq(dependencies.Plan.planning_stage.outputs['planning_stage.terraformPlanResult.terraformChanges'], 'true'))
variables:
varFromPlanStage: $[stageDependencies.Plan.planning_stage.outputs['planning_stage.terraformPlanResult.terraformChanges']]
displayName: 'Applying Changes'
jobs:
- deployment: applying_stage
displayName: 'Lets Build'
pool:
vmImage: 'Ubuntu-20.04'
environment: 'building'
strategy:
runOnce:
deploy:
steps:
- script: echo $(varFromPlanStage) # Just a test
- template: /Pipelines/20-TEST-terraform-apply-template.yml # Run the Apply Template
parameters:
terraform_version: ${{ parameters.terraform_version }}
terraform_backend_service_arm: ${{ parameters.terraform_backend_service_arm }}
terraform_backend_resource_group: ${{ parameters.terraform_backend_resource_group }}
terraform_backend_storage_account: ${{ parameters.terraform_backend_storage_account }}
terraform_backend_storage_container: ${{ parameters.terraform_backend_storage_container }}
terraform_state_key: ${{ parameters.terraform_state_key }}
git_working_directory: ${{ parameters.git_working_directory }}
And the part of the Planning Template that exports the variable is called terraformPlanResult with the var being terraformChanges
Any idea what I am doing wrong here, and why I can't call the variable as a condition but I can as part of the steps?
Thx!
This appears to work differently depending on whether you're setting the variable in a 'deployment' job, or a 'job' job. After some trial, error and googling I managed to get working for both. Examples below :-)
Example passing variables from a 'deployment' job.
# Example passing variables from a 'deployment' job.
stages:
# Create some variables to pass to next stage.
- stage: 'A'
jobs:
- deployment: 'A1'
pool:
vmImage: 'windows-2019'
environment: 'test'
strategy:
runOnce:
deploy:
steps:
# Create a variable.
- task: PowerShell#2
name: foo
displayName: 'Create a variable.'
inputs:
targetType: 'inline'
script: |
echo "##vso[task.setvariable variable=bar;isOutput=true]apple"
# Check variable.
- task: PowerShell#2
displayName: 'Check a variable.'
inputs:
targetType: 'inline'
script: |
Write-Host "$env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(foo.bar)
# Confirm condition works & variables are available for use.
- stage: 'B'
dependsOn:
- 'A'
variables:
- name: varFromStageA
# stageDependencies.stageName.deploymentName.outputs['deploymentName.stepName.variableName']
value: $[ stageDependencies.A.A1.outputs['A1.foo.bar'] ]
# dependencies.stageName.outputs['deploymentName.deploymentName.stepName.variableName']
condition: and(succeeded(), eq(dependencies.A.outputs['A1.A1.foo.bar'], 'apple'))
jobs:
- job: 'B1'
pool:
vmImage: 'windows-2019'
steps:
# Confirm variable has been passed between stages.
- task: PowerShell#2
displayName: 'Confirm var passed between stages'
inputs:
targetType: 'inline'
script: |
Write-Host "$env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(varFromStageA)
Example passing variables from a 'job' job.
# Example passing variables from a 'job' job.
stages:
# Create some variables to pass to next stage.
- stage: 'A'
jobs:
- job: 'A1'
pool:
vmImage: 'windows-2019'
steps:
# Create a variable.
- task: PowerShell#2
name: foo
displayName: 'Create a variable.'
inputs:
targetType: 'inline'
script: |
echo "##vso[task.setvariable variable=bar;isOutput=true]apple"
# Check variable.
- task: PowerShell#2
displayName: 'Check a variable.'
inputs:
targetType: 'inline'
script: |
Write-Host "$env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(foo.bar)
# Confirm condition works & variables are available for use.
- stage: 'B'
dependsOn:
- 'A'
variables:
- name: varFromStageA
# stageDependencies.stageName.jobName.outputs['stepName.variableName']
value: $[ stageDependencies.A.A1.outputs['foo.bar'] ]
# dependencies.stageName.outputs['jobName.stepName.variableName']
condition: and(succeeded(), eq(dependencies.A.outputs['A1.foo.bar'], 'apple'))
jobs:
- job: 'B1'
pool:
vmImage: 'windows-2019'
steps:
# Confirm variable has been passed between stages.
- task: PowerShell#2
displayName: 'Confirm var passed between stages'
inputs:
targetType: 'inline'
script: |
Write-Host "$env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(varFromStageA)
What's working for me (by many trials and errors) is the format:
dependencies.stage_name.outputs['job_name.step_name.variable_name']
In your case, this would be:
dependencies.Plan.outputs['planning_stage.terraformPlanResult.terraformChanges']
If that doesn't work, the only unusual thing is that your planning_stage is a deployment. You might try and change it to a regular job.
Here's a pipeline that tests various ways of referencing outputs in conditions.
For me it looks like an issue on Azure Devops (later O will preapre minimial working example and create a bug for this). Because the same syntax works correctly for regular jobs but not for deployment. Also you can use condition like condition: in(dependencies.A.result, 'Succeeded', 'SucceededWithIssues', 'Skipped') but not outputs. And according to this
"dependencies": {
"<STAGE_NAME>" : {
"result": "Succeeded|SucceededWithIssues|Skipped|Failed|Canceled",
"outputs": {
"jobName.stepName.variableName": "value"
}
},
"...": {
// another stage
}
}
it is correct syntax.
I also checked this but using life cycle hook instead of job name doesn't helped me.
What is trange that you can use output variables in job condition if ther are in the same stage.
Here is link to the issue.

Conditional Stage Execution in Azure DevOps Pipelines

I want a stage in an Azure DevOps pipeline to be executed depending on the content of a variable set in a previous stage.
Here is my pipeline:
stages:
- stage: plan_dev
jobs:
- job: terraform_plan_dev
steps:
- bash: echo '##vso[task.setvariable variable=terraform_plan_exitcode;isOutput=true]2'
name: terraform_plan
- stage: apply_dev
dependsOn: plan_dev
condition: eq(stageDependencies.plan_dev.terraform_plan_dev.outputs['terraform_plan.terraform_plan_exitcode'], '2')
jobs:
- deployment: "apply_dev"
...
The idea is to skip the apply_dev stage, if the plan_dev stage shows no changes. Background is that we have manual approval for the deployment in the plan_dev stage that we want to skip if there are no changes to be approved.
Unfortunately this doesn't seem to work. No matter whether the variable terraform_plan_exitcode is set with the expected value (2) or not, the apply_dev stage is skipped.
For the syntax, I followed the documentation here that says:
stageDependencies.StageName.JobName.outputs['StepName.VariableName']
I have seen this same issue. You need to use the dependencies variable instead of the stageDependencies:
stages:
- stage: plan_dev
jobs:
- job: terraform_plan_dev
steps:
- bash: echo '##vso[task.setvariable variable=terraform_plan_exitcode;isOutput=true]2'
name: terraform_plan
- stage: apply_dev
dependsOn: plan_dev
condition: eq(dependencies.plan_dev.outputs['terraform_plan_dev.terraform_plan.terraform_plan_exitcode'], '2')
jobs:
- deployment: "apply_dev"
The following is a more complete example of something I have working with Terraform Plan + conditional Apply:
stages:
- stage: Build_zip_plan
displayName: Build portal, zip files and terraform plan
jobs:
- job: Build_portal_zip_files_terraform_plan
pool:
vmImage: 'ubuntu-latest'
steps:
- task: Cache#2
displayName: 'Register TF cache'
inputs:
key: terraform | $(Agent.OS) | $(Build.BuildNumber) | $(Build.BuildId) | $(Build.SourceVersion) | $(prefix)
path: ${{ parameters.tfExecutionDir }}
- task: TerraformInstaller#0
displayName: 'Install Terraform'
inputs:
terraformVersion: ${{ parameters.tfVersion }}
- task: TerraformTaskV1#0
displayName: 'Terraform Init'
inputs:
provider: 'azurerm'
command: 'init'
workingDirectory: ${{ parameters.tfExecutionDir }}
backendServiceArm: ${{ parameters.tfStateServiceConnection }}
backendAzureRmResourceGroupName: ${{ parameters.tfStateResourceGroup }}
backendAzureRmStorageAccountName: ${{ parameters.tfStateStorageAccount }}
backendAzureRmContainerName: ${{ parameters.tfStateStorageContainer }}
backendAzureRmKey: '$(prefix)-$(environment).tfstate'
- task: TerraformTaskV1#0
displayName: 'Terraform Plan'
inputs:
provider: 'azurerm'
command: 'plan'
commandOptions: '-input=false -out=deployment.tfplan -var="environment=$(environment)" -var="prefix=$(prefix)" -var="tenant=$(tenant)" -var="servicenow={username=\"$(servicenowusername)\",instance=\"$(servicenowinstance)\",password=\"$(servicenowpassword)\",assignmentgroup=\"$(servicenowassignmentgroup)\",company=\"$(servicenowcompany)\"}" -var="clientid=$(clientid)" -var="username=$(username)" -var="password=$(password)" -var="clientsecret=$(clientsecret)" -var="mcasapitoken=$(mcasapitoken)" -var="portaltenantid=$(portaltenantid)" -var="portalclientid=$(portalclientid)" -var="customerdisplayname=$(customerdisplayname)" -var="reportonlymode=$(reportonlymode)"'
workingDirectory: ${{ parameters.tfExecutionDir }}
environmentServiceNameAzureRM: ${{ parameters.tfServiceConnection }}
- task: PowerShell#2
displayName: 'Check Terraform plan'
name: "Check_Terraform_Plan"
inputs:
filePath: '$(Build.SourcesDirectory)/Pipelines/Invoke-CheckTerraformPlan.ps1'
arguments: '-TfPlan ''${{ parameters.tfExecutionDir }}/deployment.tfplan'''
pwsh: true
- stage:
dependsOn: Build_zip_plan
displayName: Terraform apply
condition: eq(dependencies.Build_zip_plan.outputs['Build_portal_zip_files_terraform_plan.Check_Terraform_Plan.TFChangesPending'], 'yes')
jobs:
- deployment: DeployHub
displayName: Apply
pool:
vmImage: 'ubuntu-latest'
environment: '$(prefix)'
strategy:
runOnce:
deploy:
steps:
- checkout: self
- task: Cache#2
displayName: 'Get Cache for TF Artifact'
inputs:
key: terraform | $(Agent.OS) | $(Build.BuildNumber) | $(Build.BuildId) | $(Build.SourceVersion) | $(prefix)
path: ${{ parameters.tfExecutionDir }}
- task: TerraformInstaller#0
displayName: 'Install Terraform'
inputs:
terraformVersion: ${{ parameters.tfVersion }}
- task: TerraformTaskV1#0
displayName: 'Terraform Apply'
inputs:
provider: 'azurerm'
command: 'apply'
commandOptions: 'deployment.tfplan'
workingDirectory: ${{ parameters.tfExecutionDir }}
environmentServiceNameAzureRM: ${{ parameters.tfServiceConnection }}
#Marius is correct. So this works
stages:
- stage: plan_dev
jobs:
- job: terraform_plan_dev
steps:
- bash: echo '##vso[task.setvariable variable=terraform_plan_exitcode;isOutput=true]2'
name: terraform_plan
- stage: apply_dev
dependsOn: plan_dev
variables:
varFromA: $[ stageDependencies.plan_dev.terraform_plan_dev.outputs['terraform_plan.terraform_plan_exitcode'] ]
condition: eq(dependencies.plan_dev.outputs['terraform_plan_dev.terraform_plan.terraform_plan_exitcode'], 2)
jobs:
- job: apply_dev
steps:
- bash: echo 'apply $(varFromA)'
name: terraform_apply
When you refer stage to stage dependencies you have different syntax
"dependencies": {
"<STAGE_NAME>" : {
"result": "Succeeded|SucceededWithIssues|Skipped|Failed|Canceled",
"outputs": {
"jobName.stepName.variableName": "value"
}
},
"...": {
// another stage
}
}
And when you refer to job to job across stage you have different syntax
"stageDependencies": {
"<STAGE_NAME>" : {
"<JOB_NAME>": {
"result": "Succeeded|SucceededWithIssues|Skipped|Failed|Canceled",
"outputs": {
"stepName.variableName": "value"
}
},
"...": {
// another job
}
},
"...": {
// another stage
}
}
What is funny when you have job to job in one stage we use dependecies syntax again
"dependencies": {
"<JOB_NAME>": {
"result": "Succeeded|SucceededWithIssues|Skipped|Failed|Canceled",
"outputs": {
"stepName.variableName": "value1"
}
},
"...": {
// another job
}
}
This is a bit confusing and consider this in this as
when you are on some level stage, job and refer to the same level from job to job or from stage to stage you have dependencies syntax
when you want to refer from deeper level like from job to stage you should use stageDependencies
What is funny, in above example I used this on stage level:
variables:
varFromA: $[ stageDependencies.plan_dev.terraform_plan_dev.outputs['terraform_plan.terraform_plan_exitcode'] ]
but this is evaluated at runtime and is evaluated from the job, so it is correct and is evaluated correctly.
I hope it added a value to previous answer.
TerraformTaskV2 has changesPresent output variable now, which can be used to skip apply stage.
add name: to the plan task
stages:
- stage: terraform_plan_STAGE
jobs:
- job: plan_JOB
...
steps:
...
- task: TerraformTaskV2#2
name: 'plan_TASK' # <===========
displayName: 'plan'
inputs:
provider: 'azurerm'
command: 'plan'
...
add condition: to apply stage and check if changesPresent is true
- stage: terraform_apply
dependsOn: [terraform_plan]
condition: eq(dependencies.terraform_plan_STAGE.outputs['plan_JOB.plan_TASK.changesPresent'], 'true')
reference:
https://github.com/microsoft/azure-pipelines-terraform/tree/main/Tasks/TerraformTask/TerraformTaskV2#output-variables
https://learn.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#use-outputs-in-a-different-stage

Why my multi stage full yaml azure devops pipeline breaks when moved to template?

I have a multi stage full yaml pipeline in azure devops services. The different stages consist of a build stage, a Dev stage (Deploys azure resources to an azure resource group) and a UAT stage (Deploys azure resources to a different resource group). Everything works great. However I needed to use templates because I want to reuse the steps. After moving the steps to a template, the dev stage keeps working but the UAT stage gives authorization error. It is worth mentioning, dev and uat stages use different service connections targeting different subscriptions. But why will it work when I dont use templates and simply moving the steps to a template file all of a sudden it doesn't work. Is this a known bug?
This is the error I get when using the template approach:
| The client '9e5cc21a-bb38-46b9-a16a-289fbbf9c8b9' with object
| id '9e5cc21a-bb38-46b9-a16a-289fbbf9c8b9' does not have
| authorization to perform action
| 'Microsoft.Resources/subscriptions/resourceGroups/resources/read' over scope '/subscriptions/7041f5ba-1040-4989-8e48-497b3b826d01/resourceGroups/Resource-Group-Test-A' or the scope is invalid. If access was recently granted, please refresh your credentials. StatusCode: 403 ReasonPhrase: Forbidden OperationID : 464f577e-6617-4bed-9a14-1f7487b5f209
This is the pipeline without using templates (works perfect!).
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
trigger:
- master
variables:
- group: 'Dev ARMT SFTP Connection'
resources:
repositories:
- repository: templates
type: git
name: CommonTasks
stages:
- stage: Build
jobs:
- job: Build
pool:
vmImage: 'ubuntu-latest'
steps:
- task: printAllVariables#1
- task: CopyFiles#2
inputs:
SourceFolder: '$(System.DefaultWorkingDirectory)'
Contents: 'ARM-Templates/**'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'drop'
publishLocation: 'Container'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "Build Completed..."
$workingdir = "$(Build.ArtifactStagingDirectory)"
Write-Host $workingdir
$fcontent = Get-ChildItem -Path $workingdir
Write-Host $fcontent
- stage: Dev
jobs:
- job: Dev
pool:
vmImage: 'ubuntu-latest'
steps:
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'specific'
project: '530cfd4b-51b8-4237-b2fa-f296a4cba29d'
pipeline: '30'
buildVersionToDownload: 'latest'
downloadType: 'single'
artifactName: 'drop'
downloadPath: '$(System.ArtifactsDirectory)/CommonTasks'
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'current'
downloadType: 'single'
artifactName: 'drop'
downloadPath: '$(System.ArtifactsDirectory)/SftpConnection'
- task: FileTransform#1
displayName: 'ARMT SFTP Connection - Parameters File Transform '
inputs:
folderPath: '$(System.ArtifactsDirectory)/SftpConnection/drop'
fileType: json
targetFiles: 'ARM-Templates/parameters.json'
- task: AzurePowerShell#4
displayName: 'ARMT SFTP Connection - Check Resource Existance'
inputs:
azureSubscription: 'Starwood-DT-DEV-ServiceConnection'
ScriptPath: '$(System.ArtifactsDirectory)/CommonTasks/drop/AzurePowerShell/ResourceExistance.ps1'
ScriptArguments: '-resourceGroupName Starwood-DT-DEV -resourceName $(parameters.sftp_name.value)'
azurePowerShellVersion: LatestVersion
- task: AzureResourceGroupDeployment#2
displayName: 'ARMT SFTP Connection - Deploy'
inputs:
azureSubscription: 'Starwood-DT-DEV-ServiceConnection'
resourceGroupName: 'Starwood-DT-DEV'
location: 'East US'
csmFile: '$(System.ArtifactsDirectory)/SftpConnection/drop/ARM-Templates/template.json'
csmParametersFile: '$(System.ArtifactsDirectory)/SftpConnection/drop/ARM-Templates/parameters.json'
condition: eq(variables['deployresource'],'true')
- stage: GroupATest
jobs:
- job: GroupATest
pool:
vmImage: 'ubuntu-latest'
steps:
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'specific'
project: '530cfd4b-51b8-4237-b2fa-f296a4cba29d'
pipeline: '30'
buildVersionToDownload: 'latest'
downloadType: 'single'
artifactName: 'drop'
downloadPath: '$(System.ArtifactsDirectory)/CommonTasks'
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'current'
downloadType: 'single'
artifactName: 'drop'
downloadPath: '$(System.ArtifactsDirectory)/SftpConnection'
- task: FileTransform#1
displayName: 'ARMT SFTP Connection - Parameters File Transform '
inputs:
folderPath: '$(System.ArtifactsDirectory)/SftpConnection/drop'
fileType: json
targetFiles: 'ARM-Templates/parameters.json'
- task: AzurePowerShell#4
displayName: 'ARMT SFTP Connection - Check Resource Existance'
inputs:
azureSubscription: 'Resource-Group-Test-A'
ScriptPath: '$(System.ArtifactsDirectory)/CommonTasks/drop/AzurePowerShell/ResourceExistance.ps1'
ScriptArguments: '-resourceGroupName Resouce-Group-Test-A -resourceName $(parameters.sftp_name.value)'
azurePowerShellVersion: LatestVersion
- task: AzureResourceGroupDeployment#2
displayName: 'ARMT SFTP Connection - Deploy'
inputs:
azureSubscription: 'Resource-Group-Test-A'
resourceGroupName: 'Resouce-Group-Test-A'
location: 'East US'
csmFile: '$(System.ArtifactsDirectory)/SftpConnection/drop/ARM-Templates/template.json'
csmParametersFile: '$(System.ArtifactsDirectory)/SftpConnection/drop/ARM-Templates/parameters.json'
condition: eq(variables['deployresource'],'true')
When moving to steps template (then i get the error):
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
trigger:
- master
resources:
repositories:
- repository: templates
type: git
name: CommonTasks
stages:
- stage: Build
jobs:
- job: Build
pool:
vmImage: 'ubuntu-latest'
steps:
- task: printAllVariables#1
- task: CopyFiles#2
inputs:
SourceFolder: '$(System.DefaultWorkingDirectory)'
Contents: 'ARM-Templates/**'
TargetFolder: '$(Build.ArtifactStagingDirectory)'
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'drop'
publishLocation: 'Container'
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
# Write your PowerShell commands here.
Write-Host "Build Completed..."
$workingdir = "$(Build.ArtifactStagingDirectory)"
Write-Host $workingdir
$fcontent = Get-ChildItem -Path $workingdir
Write-Host $fcontent
- stage: Dev
variables:
- group: 'Dev ARMT SFTP Connection'
jobs:
- job: Dev
pool:
vmImage: 'ubuntu-latest'
steps:
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'specific'
project: '530cfd4b-51b8-4237-b2fa-f296a4cba29d'
pipeline: '30'
buildVersionToDownload: 'latest'
downloadType: 'single'
artifactName: 'drop'
downloadPath: '$(System.ArtifactsDirectory)/CommonTasks'
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'current'
downloadType: 'single'
artifactName: 'drop'
downloadPath: '$(System.ArtifactsDirectory)/SftpConnection'
- template: YamlTemplate/azure-resource-deploy.yml#templates
parameters:
dropLocation: '$(System.ArtifactsDirectory)/SftpConnection/drop'
transformTargetPath: 'ARM-Templates/parameters.json'
resourceName: $(parameters.sftp_name.value)
resourceGroupName: 'Starwood-DT-DEV'
azureServiceConnectionName: 'Starwood-DT-DEV-ServiceConnection'
resourceLocation: 'East US'
armtTemplateFilePath: '$(System.ArtifactsDirectory)/SftpConnection/drop/ARM-Templates/template.json'
armtParemeterFilePath: '$(System.ArtifactsDirectory)/SftpConnection/drop/ARM-Templates/parameters.json'
- stage: GroupATest
variables:
- group: 'GroupA ARMT SFTP Connection'
jobs:
- job: GroupATest
pool:
vmImage: 'ubuntu-latest'
steps:
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'specific'
project: '530cfd4b-51b8-4237-b2fa-f296a4cba29d'
pipeline: '30'
buildVersionToDownload: 'latest'
downloadType: 'single'
artifactName: 'drop'
downloadPath: '$(System.ArtifactsDirectory)/CommonTasks'
- task: DownloadBuildArtifacts#0
inputs:
buildType: 'current'
downloadType: 'single'
artifactName: 'drop'
downloadPath: '$(System.ArtifactsDirectory)/SftpConnection'
- template: YamlTemplate/azure-resource-deploy.yml#templates
parameters:
dropLocation: '$(System.ArtifactsDirectory)/SftpConnection/drop'
transformTargetPath: 'ARM-Templates/parameters.json'
resourceName: $(parameters.sftp_name.value)
resourceGroupName: 'Resource-Group-Test-A'
azureServiceConnectionName: 'Resource-Group-Test-A'
resourceLocation: 'East US'
armtTemplateFilePath: '$(System.ArtifactsDirectory)/SftpConnection/drop/ARM-Templates/template.json'
armtParemeterFilePath: '$(System.ArtifactsDirectory)/SftpConnection/drop/ARM-Templates/parameters.json'
This the template structure
Parent template - (azure-resource-deploy.yml):
parameters:
- name: dropLocation
type: string
displayName: 'Artifact path to drop location'
default: ''
- name: transformTargetPath
type: string
default: ''
- name: resourceName
type: string
default: ''
- name: resourceGroupName
type: string
default: ''
- name: azureServiceConnectionName
type: string
default: ''
- name: resourceLocation
type: string
default: ''
- name: armtTemplateFilePath
type: string
default: ''
- name: armtParemeterFilePath
type: string
default: ''
steps:
- template: armt-parameter-file-transform.yml
parameters:
transformStepDisplayName: ''
folderPath: ${{parameters.dropLocation}}
targetFile: ${{parameters.transformTargetPath}}
- template: azure-resource-check.yml
parameters:
resourceName: ${{parameters.resourceName}}
resourceGroupName: ${{parameters.resourceGroupName}}
azureServiceConnectionName: ${{parameters.azureServiceConnectionName}}
- template: armt-deploy.yml
parameters:
resourceName: ${{parameters.resourceName}}
resourceGroupName: ${{parameters.resourceGroupName}}
resourceLocation: ${{parameters.resourceLocation}}
azureServiceConnectionName: ${{parameters.azureServiceConnectionName}}
templateFilePath: ${{parameters.armtTemplateFilePath}}
parametersFilePath: ${{parameters.armtParemeterFilePath}}
Templates referenced from parent:
(armt-parameter-file-transform.yml)
parameters:
- name: transformStepDisplayName
type: string
displayName: 'Display name for this step'
default: ''
- name: folderPath
type: string
displayName: 'Path to drop location'
default: ''
- name: targetFile
type: string
displayName: 'Path to paremeter file relative to drop location.'
default: ''
steps:
- task: FileTransform#1
displayName: ${{parameters.transformStepDisplayName}}
inputs:
folderPath: ${{parameters.folderPath}}
fileType: json
targetFiles: ${{parameters.targetFile}}
(azure-resource-check.yml)
parameters:
- name: resourceName
type: string
- name: resourceGroupName
type: string
- name: azureServiceConnectionName
type: string
steps:
- script: echo Echo -resourceGroupName ${{ parameters.resourceGroupName }} -resourceName ${{ parameters.resourceName }}
- task: AzurePowerShell#4
displayName: '${{ parameters.resourceName }} - Checking Resource Existance'
inputs:
azureSubscription: '${{ parameters.azureServiceConnectionName }}'
ScriptPath: '$(System.ArtifactsDirectory)/CommonTasks/drop/AzurePowerShell/ResourceExistance.ps1'
ScriptArguments: '-resourceGroupName ${{ parameters.resourceGroupName }} -resourceName ${{ parameters.resourceName }}'
azurePowerShellVersion: LatestVersion
(armt-deploy.yml)
parameters:
- name: resourceName
type: string
- name: resourceGroupName
type: string
- name: resourceLocation
type: string
- name: azureServiceConnectionName
type: string
- name: templateFilePath
type: string
- name: parametersFilePath
type: string
steps:
- task: AzureResourceGroupDeployment#2
displayName: 'ARMT Deploy - ${{parameters.resourceName}}'
inputs:
azureSubscription: ${{parameters.azureServiceConnectionName}}
resourceGroupName: ${{parameters.resourceGroupName}}
location: ${{parameters.resourceLocation}}
csmFile: ${{parameters.templateFilePath}}
csmParametersFile: ${{parameters.parametersFilePath}}
condition: eq(variables['deployresource'],'true')
This was just caused by a typo. The actual name of the resource group is Resouce-Group-Test-A, I missed the r when naming this resource group. The error does not happen in the non-templated version because when you pick the resource group you do it from a drop down list, with the correct prepopulated names, so no chance to make a mistake. However in a full yaml pipeline you have to type it in, and thats where the typo occured. If there is anything to be learned here. Pay close attention when typing resource names, errors might not be descriptive enough. Sorry If wasted anybodys precious time.

Resources