How to get the github repo name in Azure pipeline - azure

Are there any variables in Azure which I can use to get the name of the source folder(repository name). I tried using Build.SourcesDirectory and System.DefaultWorkingDirectory , both returned /Users/runner/work/1/s. I expect to get MyProjectFolderName which is the source directory of my project in github.

If you checkout self repo(This is default situation), then just follow Daniel's suggestion is ok:
$(Build.Repository.Name)
yml file like this:
trigger:
- none
pool:
vmImage: ubuntu-latest
steps:
- task: PythonScript#0
inputs:
scriptSource: 'inline'
script: |
str = "$(Build.Repository.Name)"
str.split("/")
#get the last name
print(str.split("/")[-1])
If you only checkout one repo and not check out self, then just use below yml:
trigger:
- none
pool:
vmImage: ubuntu-latest
resources:
repositories:
- repository: 222
type: github
name: xxx/222
endpoint: xxx
- repository: 333
type: github
name: xxx/333
endpoint: xxx
variables:
- name: checkoutreporef
value: $[ resources.repositories['333'].name ]
steps:
- checkout: 333
- task: PythonScript#0
inputs:
scriptSource: 'inline'
script: |
str = "$(checkoutreporef)"
str.split("/")
#get the last name
print(str.split("/")[-1])
If you checkout two and more repo, below yml will help you get the repo names:
trigger:
- none
pool:
vmImage: ubuntu-latest
resources:
repositories:
- repository: 222
type: github
name: xxx/xxx
endpoint: xxx
- repository: 333
type: github
name: xxx/xxx
endpoint: xxx
steps:
- checkout: 222
- checkout: 333
- task: PythonScript#0
inputs:
scriptSource: 'inline'
script: |
import os
#get current sub folders name
def getfoldersname():
folders = [f for f in os.listdir('.') if os.path.isdir(f)]
return folders
#print each folder name
def printfoldersname():
for folder in getfoldersname():
print(folder)
printfoldersname()

Related

Publish file content to service bus from CI pipeline

In my CI pipeline I am trying to publish message to service bus and its working when its just some hardcoded text or variables, here Using "PublishToAzureServiceBus" task .
But problem is when trying to a read file from repository and then publish that to service bus.
I have tried using read file using scripting language and put to variable but its not able to work as variable is not storing big json file.
Is there any way to read file directly when publishing message to service bus.
Below is sample code snippet for debugging
trigger:
- none
pool:
vmImage: ubuntu-latest
parameters:
- name: ProjectName
displayName: Project Name
type: string
default: DevOpsDemo
- name: repoName
displayName: repo Name
type: string
default: ProjectCode
- name: branchRef
displayName: Branch Name
type: string
default: main
variables:
- name: jobStatus
value: "Failed"
- name: projectFile
value: ""
stages:
- stage: Stage1
displayName: Stage 1
jobs:
- job: CheckOutRepo
displayName: CheckOut-Repo Display
steps:
- script: |
echo "Checkout for " ${{ parameters.ProjectName}} : ${{ parameters.repoName}} : ${{ parameters.branchRef}}
name: PrintMessage
- checkout: git://${{ parameters.ProjectName}}/${{ parameters.repoName}}#refs/heads/${{ parameters.branchRef}}
name: Checkout
- task: PythonScript#0
inputs:
scriptSource: 'inline'
script: |
import json
import requests
f = open('project-release.json')
projectFile = json.load(f)
print(projectFile)
f.close()
print("Afterclosing")
print(projectFile)
- script: |
echo "Project release file" $(cat project-release.json)
name: TestPrint
- task: CopyFiles#2
inputs:
SourceFolder: 'services'
Contents: '**'
TargetFolder: $(Build.ArtifactStagingDirectory)
name: CopyFiles
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: $(Build.ArtifactStagingDirectory)
ArtifactName: 'drop'
publishLocation: 'Container'
name: PublishArtifacts
- bash: |
echo "##vso[task.setvariable variable=jobStatus]Success"
name: setVar
- bash: |
echo "##vso[task.setvariable variable=jobStatus;isOutput=true]$(jobStatus)"
echo "##vso[task.setvariable variable=projectFile;isOutput=true]$(cat project-release.json)"
name: SetStatus
condition: always()
- stage: Stage2
displayName: Stage 2
condition: always()
jobs:
- job: Publish
pool: server
variables:
jobStatus: $[ stageDependencies.Stage1.CheckOutRepo.outputs['SetStatus.jobStatus'] ]
projectFile: $[ stageDependencies.Stage1.CheckOutRepo.outputs['SetStatus.projectFile'] ]
steps:
- task: PublishToAzureServiceBus#1
inputs:
azureSubscription: 'SBConnection'
messageBody: |
{
"Status": "$(jobStatus)",
"BuildID": "$(build.buildid)",
"BuildNumber":"$(build.buildnumber)",
"projectFile":$(cat project-release.json)
}
signPayload: false
waitForCompletion: false
condition: always()
I am able to solve this by using setvariable in bash script as below
pool:
vmImage: ubuntu-latest
stages:
- stage: Stage1
displayName: Stage 1
jobs:
- job: CheckOutRepo
displayName: CheckOut-Repo Display
steps:
- checkout: git://${{ parameters.ProjectName}}/${{ parameters.repoName}}#refs/heads/${{ parameters.branchRef}}
name: Checkout
- bash: |
data=$(cat project-release.json)
echo "##vso[task.setvariable variable=jobStatus;isOutput=true]$(jobStatus)"
echo "##vso[task.setvariable variable=data;isOutput=true]"$data
name: SetStatus
condition: always()
- stage: Stage2
displayName: Stage 2
condition: always()
jobs:
- job: Publish
pool: server
variables:
jobStatus: $[ stageDependencies.Stage1.CheckOutRepo.outputs['SetStatus.jobStatus'] ]
projectFile: $[ stageDependencies.Stage1.CheckOutRepo.outputs['SetStatus.data'] ]
steps:
- task: PublishToAzureServiceBus#1
inputs:
azureSubscription: 'SBConnection'
messageBody: |
{
"Status": "$(jobStatus)",
"BuildID": "$(build.buildid)",
"BuildNumber":"$(build.buildnumber)",
"projectFile":$(projectFile)
}
signPayload: false
waitForCompletion: false
condition: always()

How to use condition on library variable on azure pipeline yaml

I have on my azure pipelines a libary varrible named:
deployHostMachine
and in it a variable named:
Host
I need to run a task using if with condition on Host How do i do it?
My code:
parameters:
trigger:
- none
variables:
- group: DeploymentHost
- name: localForHost
value: $(Host)
- name: testEmpty
value: ''
stages:
- stage: "ManageDeployMachines"
jobs:
- job: "ManageDeployMachines"
pool:
vmImage: 'windows-latest'
steps:
- ${{ if eq(parameters.paramHost,'rnd-3.sepio.systems') }}:
- task: ManageVariableGroupTask#0
displayName: 'Set deploy host url'
inputs:
pat: 'yxtloxscphyvuevrzafozok533orcv3pqon4a7e3kzpc4je23zha'
project: '70e3803b-2a31-4f00-83a1-9fc825f4447b'
vgname: 'DeploymentHost'
orgname: 'SepioDevOfficial'
op: 'Update'
varops: '= Host => "rnd-4.sepio.systems"'

Aure pipeline : use output variable in multi-stage parametrised

I am facing an issue I could not resolve.
regarding this documentation it seams to be possible to set a variable as output of a job in the first stage, then use it as input to set a job variable in second stage.
The pbm I'm facing, is that my stages are parametrised, so their name are not constant.
here my sample code :
name: mypipeline
parameters:
- name: Application
displayName: Application to deploy the front end using front door
type: string
- name: stage_vars
displayName: stage VARS
type: object
default:
- stage_1
- stage_2
- none
trigger: none
stages:
- stage: Build
displayName: ${{ parameters.Application }} - Build.
pool:
name: Azure Pipelines
vmImage: windows-2019
jobs:
- template: ../templates/job_BuildAndPublish.yml
parameters:
BuildId: $(Build.BuildId)
importTerratest: false
importARM: true
- ${{ each stageregion in parameters.stage_vars }}:
- ${{ if ne(stageregion, 'none') }}:
- template: ../templates/isolated_web/tf_plan_stage.yml
parameters:
BuildId: $(system.BuildId)
Predecessors: 'Build'
- template: ../templates/isolated_web/tf_apply_stage.yml
parameters:
Predecessors: '${{stageregion}}_prepare'
build template is not usefull, but the ones from tf_plan_stage and tf_apply_stage are below :
tf_plan_stage.yml :
parameters:
- name: BuildId
type: string
default: $(system.BuildId)
- name: stage
type: string
default: Deploy_prepare
- name: Predecessors
type: string
default: none
stages:
- stage: ${{ parameters.stage }}
? ${{ if and(ne(parameters.Predecessors, 'previous'), ne(parameters.Predecessors, 'none') ) }}
: dependsOn: ${{parameters.Predecessors}}
displayName: ${{ parameters.TF_VAR_STAGE }} Terraform plan & publish artifact for ${{ parameters.TF_VAR_APPLICATION }}
pool:
name: Azure Pipelines
vmImage: windows-2019
jobs:
- deployment: PreDeployTerraform
displayName: ${{ parameters.TF_VAR_STAGE }} Terraform plan & publish artifact
environment: PLAN_${{ parameters.TF_VAR_ENVIRONMENT }}
timeoutInMinutes: 480
strategy:
runOnce:
deploy:
steps:
- checkout: none
# set output planAttempt output available for later Apply job
- powershell: |
echo "jobAttempt is $(System.JobAttempt)"
echo "##vso[task.setvariable variable=planAttempt;isOutput=true;]$(System.JobAttempt)"
name: setVarJobAttempt
- powershell: |
echo "variable setVarJobAttempt.planAttempt value is : $(setVarJobAttempt.planAttempt)"
name: getplanAttemptVar
tf_apply_stage.yml :
parameters:
- name: stage
type: string
default: Deploy_prepare
- name: Predecessors
type: string
default: none
stages:
- stage: ${{ parameters.stage }}
? ${{ if and(ne(parameters.Predecessors, 'previous'), ne(parameters.Predecessors, 'none') ) }}
: dependsOn: ${{parameters.Predecessors}}
displayName: ${{ parameters.TF_VAR_STAGE }} download artifact & Terraform apply changes to ${{ parameters.TF_VAR_APPLICATION }}
pool:
name: Azure Pipelines
vmImage: windows-2019
jobs:
- deployment: DeployTerraform
displayName: ${{ parameters.TF_VAR_STAGE }} download artifact & Terraform apply changes
variables:
# this one fails in pipeline : "[not recognise"
planJobAttempt: $[ stageDependencies.[parameters.Predecessors].PreDeployTerraform.outputs['setVarJobAttempt.planAttempt'] ]
# this one runs with no result in pipeline : planJobAttempt is the string "stageDependencies['parameters.Predecessors'].PreDeployTerraform.outputs['setVarJobAttempt.planAttempt']"
# planJobAttempt: $[ stageDependencies['parameters.Predecessors'].PreDeployTerraform.outputs['setVarJobAttempt.planAttempt'] ]
environment: ${{ parameters.TF_VAR_ENVIRONMENT }}
timeoutInMinutes: 480
strategy:
runOnce:
deploy:
steps:
- checkout: none
- powershell: |
echo "jobAttempts outputs are : $[ stageDependencies.[parameters.Predecessors].PreDeployTerraform.outputs ]"
- powershell: |
echo "jobAttempt is $[ stageDependencies['parameters.Predecessors'].PreDeployTerraform.outputs['setVarJobAttempt.planAttempt'] ]"
- powershell: |
echo "plan file for terraform is : $(pipeline_artifact_folder_download)/$(planJobAttempt)_${{ parameters.Predecessors }}/$(artefact_terraform_plan)_${{ parameters.TF_VAR_STAGE }}"
I tried differents things to get my system.jobAttempts from tf_plan_stage stage to tf_apply_stage stage, but without any success, as the variable in the last stage (tf_apply) seams unable to find the value from a "parametrised" stage name.
is there a way for that ?
thank-you for any answers.
Okay,
Once again, I will answer my own question :D
I've finallt find the way to use the parameters.Predecessors value in my variable declaration.
And, second, the Microsoft documentation seams not to be updated.
The tf_apply_stage.yml should be like that below ;
parameters:
- name: stage
type: string
default: Deploy_prepare
- name: Predecessors
type: string
default: none
stages:
- stage: ${{ parameters.stage }}
? ${{ if and(ne(parameters.Predecessors, 'previous'), ne(parameters.Predecessors, 'none') ) }}
: dependsOn: ${{parameters.Predecessors}}
displayName: ${{ parameters.TF_VAR_STAGE }} download artifact & Terraform apply changes to ${{ parameters.TF_VAR_APPLICATION }}
pool:
name: Azure Pipelines
vmImage: windows-2019
jobs:
- deployment: DeployTerraform
displayName: ${{ parameters.TF_VAR_STAGE }} download artifact & Terraform apply changes
variables:
*# this one give he good value for Variable*
planJobAttempt: $[ stageDependencies.${{ parameters.Predecessors }}.PreDeployTerraform.outputs['PreDeployTerraform.setVarJobAttempt.planAttempt'] ]
environment: ${{ parameters.TF_VAR_ENVIRONMENT }}
timeoutInMinutes: 480
strategy:
runOnce:
deploy:
steps:
- checkout: none
- powershell: |
echo "plan file for terraform is : $(pipeline_artifact_folder_download)/$(planJobAttempt)_${{ parameters.Predecessors }}/$(artefact_terraform_plan)_${{ parameters.TF_VAR_STAGE }}"
I put in BOLD the main changes above :
planJobAttempt: $[ stageDependencies.${{ parameters.Predecessors
}}.PreDeployTerraform.outputs['PreDeployTerraform.setVarJobAttempt.planAttempt']
]
The Job name have to be set twice in variable declaration !
Thanks for those who read my post ;)
And many thanks to this post from #Kay07949 who gives the good answers ;)

Azure Devops Pipeline

Need a small help , can someone point me what is the issue here .
What i am trying to achieve , the pipeline has to run checkout irrespective of the branch i am in , but in the build stage if the pipeline run from master it has execute some templet if from other branch different templet , i tried many option no luck .
Any pointer much appreciated .
- name: release
displayName: ReleaseVersion
type: string
default: ' '
- name: Deployment
displayName: DeploymentVersion
type: string
default: ' '
- name: Library
displayName: Library Release
type: boolean
default: True
trigger:
- none
pool:
name: DOTAzure-Ubuntu-20.04
resources:
repositories:
- repository: azureTemplates
type: githubenterprise
name: AAAAA/azure-pipelines
endpoint: BBBBBB
# You should add below variables to work complete pipeline
variables:
- name: servicename
value: XXXXX
- name: countryCode
value: YYYYY
- name: targetPort
value: 8113
stages:
- stage: Checkout
displayName: Micro Service checkout and Secrets
jobs:
- job:
steps:
- template: templates/repocheckout.yml#azureTemplates
- template: templates/retrieving_secrets.yml#azureTemplates
- template: templates/Ingesting_secrets.yml#azureTemplates
- stage: Build
displayName: Microservice Build
jobs:
- job: Master
condition: eq(variables['Build.SourceBranchName'], 'master')
steps:
- template: templates/mavenbuild_new.yml#azureTemplates
- template: templates/push_artifact_jfrog.yml#azureTemplates
parameters:
service_name: '$(countryCode)-$(servicename)'
release_version: $(release)
- job: Develop
dependsOn: Master
condition: eq(variables['Build.SourceBranchName'], 'develop')
steps:
- template: templates/mavenbuild.yml#azureTemplates
- template: templates/push_artifact_jfrog.yml#azureTemplates
parameters:
service_name: '$(countryCode)-$(servicename)'
release_version: $(release)
This doesn't look like valid yaml.
Have a look at this: https://learn.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=azure-devops&tabs=schema%2Cparameter-schema
It looks like you are missing this at the top of the file:
stages:
i.e. your yaml should be:
stages:
- stage: Checkout
displayName: Micro Service checkout and Secrets
jobs:
etc

Azure DevOps Use a Variable as a Condition between Deployment Stages

I currently build all our Azure Infrastructure using Terraform via Azure DevOps Pipelines. This has been working well and we have a standard pipeline which calls two templates
Plan Stage
Deployment Job
Planning Template
Runs Every Time
Apply Stage
Deployment Job
Apply Template
Runs with manual approval check
Now this works fine, but what I want to do is only have the apply step run if there are changes to make. I have found other articles on how to get a variable set in the plan stage which I can do and it works fine.
I can call this same variable in the next step
variables:
varFromPlanStage: $[stageDependencies.Plan.planning_stage.outputs['planning_stage.terraformPlanResult.terraformChanges']]
steps:
- script: echo $(varFromPlanStage)
But the problem comes in when I try and use this same variable in a condition.
I found that the way you all it is different, needing a dependency instead of stagedependancy, but no matter what I try I can't get it to work.
The pipeline looks like this.
stages:
- stage: 'Plan'
displayName: 'Planning'
jobs:
- deployment: planning_stage
displayName: 'Planning Changes'
pool:
vmImage: 'Ubuntu-20.04'
environment: 'planning'
strategy:
runOnce:
deploy:
steps:
- template: /Pipelines/10-TEST-terraform-planning-template.yml # Run the Planning Template
parameters:
terraform_version: ${{ parameters.terraform_version }}
terraform_backend_service_arm: ${{ parameters.terraform_backend_service_arm }}
terraform_backend_resource_group: ${{ parameters.terraform_backend_resource_group }}
terraform_backend_storage_account: ${{ parameters.terraform_backend_storage_account }}
terraform_backend_storage_container: ${{ parameters.terraform_backend_storage_container }}
terraform_state_key: ${{ parameters.terraform_state_key }}
git_working_directory: ${{ parameters.git_working_directory }}
# This is the Build Stage - Only do this when on the master branch (which is via a PR)
- stage: 'Apply'
condition: and(succeeded(), eq(dependencies.Plan.planning_stage.outputs['planning_stage.terraformPlanResult.terraformChanges'], 'true'))
variables:
varFromPlanStage: $[stageDependencies.Plan.planning_stage.outputs['planning_stage.terraformPlanResult.terraformChanges']]
displayName: 'Applying Changes'
jobs:
- deployment: applying_stage
displayName: 'Lets Build'
pool:
vmImage: 'Ubuntu-20.04'
environment: 'building'
strategy:
runOnce:
deploy:
steps:
- script: echo $(varFromPlanStage) # Just a test
- template: /Pipelines/20-TEST-terraform-apply-template.yml # Run the Apply Template
parameters:
terraform_version: ${{ parameters.terraform_version }}
terraform_backend_service_arm: ${{ parameters.terraform_backend_service_arm }}
terraform_backend_resource_group: ${{ parameters.terraform_backend_resource_group }}
terraform_backend_storage_account: ${{ parameters.terraform_backend_storage_account }}
terraform_backend_storage_container: ${{ parameters.terraform_backend_storage_container }}
terraform_state_key: ${{ parameters.terraform_state_key }}
git_working_directory: ${{ parameters.git_working_directory }}
And the part of the Planning Template that exports the variable is called terraformPlanResult with the var being terraformChanges
Any idea what I am doing wrong here, and why I can't call the variable as a condition but I can as part of the steps?
Thx!
This appears to work differently depending on whether you're setting the variable in a 'deployment' job, or a 'job' job. After some trial, error and googling I managed to get working for both. Examples below :-)
Example passing variables from a 'deployment' job.
# Example passing variables from a 'deployment' job.
stages:
# Create some variables to pass to next stage.
- stage: 'A'
jobs:
- deployment: 'A1'
pool:
vmImage: 'windows-2019'
environment: 'test'
strategy:
runOnce:
deploy:
steps:
# Create a variable.
- task: PowerShell#2
name: foo
displayName: 'Create a variable.'
inputs:
targetType: 'inline'
script: |
echo "##vso[task.setvariable variable=bar;isOutput=true]apple"
# Check variable.
- task: PowerShell#2
displayName: 'Check a variable.'
inputs:
targetType: 'inline'
script: |
Write-Host "$env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(foo.bar)
# Confirm condition works & variables are available for use.
- stage: 'B'
dependsOn:
- 'A'
variables:
- name: varFromStageA
# stageDependencies.stageName.deploymentName.outputs['deploymentName.stepName.variableName']
value: $[ stageDependencies.A.A1.outputs['A1.foo.bar'] ]
# dependencies.stageName.outputs['deploymentName.deploymentName.stepName.variableName']
condition: and(succeeded(), eq(dependencies.A.outputs['A1.A1.foo.bar'], 'apple'))
jobs:
- job: 'B1'
pool:
vmImage: 'windows-2019'
steps:
# Confirm variable has been passed between stages.
- task: PowerShell#2
displayName: 'Confirm var passed between stages'
inputs:
targetType: 'inline'
script: |
Write-Host "$env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(varFromStageA)
Example passing variables from a 'job' job.
# Example passing variables from a 'job' job.
stages:
# Create some variables to pass to next stage.
- stage: 'A'
jobs:
- job: 'A1'
pool:
vmImage: 'windows-2019'
steps:
# Create a variable.
- task: PowerShell#2
name: foo
displayName: 'Create a variable.'
inputs:
targetType: 'inline'
script: |
echo "##vso[task.setvariable variable=bar;isOutput=true]apple"
# Check variable.
- task: PowerShell#2
displayName: 'Check a variable.'
inputs:
targetType: 'inline'
script: |
Write-Host "$env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(foo.bar)
# Confirm condition works & variables are available for use.
- stage: 'B'
dependsOn:
- 'A'
variables:
- name: varFromStageA
# stageDependencies.stageName.jobName.outputs['stepName.variableName']
value: $[ stageDependencies.A.A1.outputs['foo.bar'] ]
# dependencies.stageName.outputs['jobName.stepName.variableName']
condition: and(succeeded(), eq(dependencies.A.outputs['A1.foo.bar'], 'apple'))
jobs:
- job: 'B1'
pool:
vmImage: 'windows-2019'
steps:
# Confirm variable has been passed between stages.
- task: PowerShell#2
displayName: 'Confirm var passed between stages'
inputs:
targetType: 'inline'
script: |
Write-Host "$env:MY_MAPPED_ENV_VAR"
env:
MY_MAPPED_ENV_VAR: $(varFromStageA)
What's working for me (by many trials and errors) is the format:
dependencies.stage_name.outputs['job_name.step_name.variable_name']
In your case, this would be:
dependencies.Plan.outputs['planning_stage.terraformPlanResult.terraformChanges']
If that doesn't work, the only unusual thing is that your planning_stage is a deployment. You might try and change it to a regular job.
Here's a pipeline that tests various ways of referencing outputs in conditions.
For me it looks like an issue on Azure Devops (later O will preapre minimial working example and create a bug for this). Because the same syntax works correctly for regular jobs but not for deployment. Also you can use condition like condition: in(dependencies.A.result, 'Succeeded', 'SucceededWithIssues', 'Skipped') but not outputs. And according to this
"dependencies": {
"<STAGE_NAME>" : {
"result": "Succeeded|SucceededWithIssues|Skipped|Failed|Canceled",
"outputs": {
"jobName.stepName.variableName": "value"
}
},
"...": {
// another stage
}
}
it is correct syntax.
I also checked this but using life cycle hook instead of job name doesn't helped me.
What is trange that you can use output variables in job condition if ther are in the same stage.
Here is link to the issue.

Resources