I want to update a variable based on the number of commits in git repo. I am correctly able to fetch the count but not able to update the variable for further use. Any idea what is going wrong.
variables:
# Versioning
# Version Format - 'VersionMajor.VersionMinor.VersionRelease.Revision'
# Currently the Major, Minor and Minor are all constants
- name: VersionMajorMinorRelease
value: '1.0.1'
# Revision will be calculated during pipeline run based on total commit counts for branch
- name: VersionRevision
value: ''
Further in job I am using variables as
- task: PowerShell#2
displayName: Set the build version
condition: eq('${{ parameters.Action }}', 'BuildAndDeploy')
inputs:
targetType: inline
script: |
$commitCounts= $(git rev-list --count HEAD)
$revision= $commitCounts.ToString()
Write-Host "Revision = '$revision'"
Write-Host "##vso[task.setvariable variable=VersionRevision]$revision"
Write-Host "Version of App being built = '$(VersionMajorMinorRelease).$(VersionRevision)'"
Output printed -
Revision = '1660'
Version of App being built = '1.0.1.'
Why is my variable value not getting updated in azure pipeline?
You could use the VersionRevision in next task.
According to the document Set variables in scripts:
When you use PowerShell and Bash scripts in your pipelines, it's often
useful to be able to set variables that you can then use in future
tasks.
That is reason why you able to fetch the count but not able to update the variable for further use, please try to echo it in next powershell task.
Related
I am working on a multi stage pipeline that build and deploy some c# code from staging to production
Everything works just fine but I wanted to try and customise a bit more the pipeline so I can see the actual version that is being built and deployed as part of the name of the stage.
At the current stage, this my multi stage pipeline
trigger:
batch: true
tags:
include:
- '*'
branches:
exclude:
- main
- staging
pool:
vmImage: 'ubuntu-latest'
variables:
buildNumber: "$[variables['Build.BuildNumber']]"
DOCKER_BUILDKIT: 1
dockerRegistryServiceConnectionStaging: '<My-Connection-String>'
imageRepositoryStaging: '<My-Repo-Name>'
containerRegistryStaging: '<My-Container-Name>'
dockerRegistryServiceConnectionProd: '<My-Connection-String>'
imageRepositoryProd: 'My-Repo-Name>'
containerRegistryProd: '<My-Container-Name>'
dockerfilePath: 'pathTo/Dockerfile'
solution: 'path/To/Solution.csproj'
tag: '$(Build.BuildNumber)'
stages:
- stage: 'Build_Staging'
displayName: 'Build_Staging'
jobs:
- job: buildStaging
displayName: 'DotNet Core publish and dockerize'
steps:
- powershell: |
# Write your PowerShell commands here.
Write-Host "Update Build.BuildNumber"
cd $(System.DefaultWorkingDirectory)
$Latesttag = $(git describe --tags $(git rev-list --tags --max-count=1))
Write-Host "The latest git tag is $Latesttag "
Write-Host
"##vso[build.updatebuildNumber]$Latesttag"
- task: DotNetCoreCLI#2
displayName: 'DotNet - Restore'
inputs:
command: 'restore'
projects: $(solution)
noCache: true
versioningScheme: 'off'
vstsFeed: '<Feed>'
- task: DotNetCoreCLI#2
name: 'DotnetPublish'
displayName: 'dotnet - Publish'
inputs:
command: 'publish'
projects: $(solution)
arguments: '-o publish/solution -c release'
modifyOutputPath: false
zipAfterPublish: false
publishWebProjects: false
- task: Docker#2
name: 'dockerBuildAndPush'
displayName: 'docker - Build & Push $(tag)'
inputs:
repository: $(imageRepositoryStaging)
Dockerfile: $(dockerfilePath)
containerRegistry: ${{ variables.dockerRegistryServiceConnectionStaging }}
buildContext: ${{ variables.buildContext }}
tags: |
$(Build.BuildNumber)
latest
- stage: 'Deploy_Staging'
jobs:
- deployment: 'Deploy'
environment: 'Staging'
variables:
EnvironmentName: 'Staging'
strategy:
runOnce:
deploy:
steps:
- task: AzureRmWebAppDeployment#4
displayName: 'Deploy Azure App Service To Staging'
inputs:
azureSubscription: '<Azure-Subscription>'
appType: 'webAppContainer'
DockerNamespace: '<container-namespace>'
DockerRepository: '<Repository>'
DockerImageTag: '$(Build.BuildNumber)'
WebAppName: '<WebAppName>'
The Powershell command is to override the Build.BuildNumber with the tag I am pushing to GitHub.
When I run this pipeline, in azure DevOps, I see the stage name Build_Staging_$(Build.BuildNumber) as a string.
What I would really like to see is, if I push the tag 'v1.0.0` for example, is to see the stage name like:
Build_Staging_v1.0.0
I tried to use the displayName and the output is not the one I was looking for and if I try with name instead of displayName I get the error unexpected value name
Can please please anyone help understand what am I doing wrong and how I can achieve this?
Please if my question is not 100% clear and missing any important detail, just let me know
UPDATE:
I did update the post with my entire pipeline.
This pipeline, before it used to be a single job process, and everything was working fine. But to get my hand dirty, I wanted to add stages to split and workflow based on resources and environment.
The process is still working and this is what I am expecting.
In my GitHub, when I create a tag on the main branch, this will trigger my build stage. Which thanks to the Powershell script to update the BuildNumber with the tag, I am able to build the docker image in my container registry in the following format:
docker-image-name:v1.0.1
That version can be seen at this level also:
This updated buildNumber (now is Tag) is use in Azure pipelines App Slack to check the version that has been pushed.
So far everything is good.
But I am facing the problem with the deployment job, at that level I am not able to set any Powershell script to update that same BuildNumber with the tag. I checked the documentation and nothing is mentioned about how I can add another job or step. I tried implementing this but I get errors that the value is unexpected.
Let me just share another screenshot to fully explain the issue.
Assuming I am deploying the docker image v1.0.1, everything works perfectly, but the build number in deployment stage, is not being updated, in fact in slack channel, I see the normal build number, as follow:
Instead of having the buildNumber, I would like to have my tag.
Please any help here?
Unfortunately, you won't be able to set a stage name to a dynamic variable that is set within one of its child's steps. You'll have to set it to a pipeline-level variable or predefined variable
Variable evaluation goes top-down from stages to tasks:
stages
jobs
tasks
To help explain exactly why, let's talk about how variable evaluation works in general with regard to this structure:
VARIABLE EVALUATION: Using stages as an example, you can set a stage name using any dynamic value that's present when the stage is evaluated. This means the variable is only accessed when the stage is initially "rendered". Azure DevOps requires that the variable be present before evaluation and will not retroactively update the UI if that variable is changed within a child step.
Let's talk about each and their respective limitations on what variables you can use in their names:
STAGES: pipeline-level variables, parameters (in the case of templates), or predefined variables
JOBS: stage-level variables, pipeline-level variables, parameters (in the case of templates), or predefined variables
TASKS: job-level variables, stage-level variables, pipeline-level variables, parameters (in the case of templates), or predefined variables
I did something similar by setting my build number to a repo tag. Here is the PowerShell function that sets the Build.Buildnumber variable to the tag value. You can just call it straight out or base it off a parameter if you have other version number logic.
function getTagVersion() {
$tag = iex "git describe --long --tags --always"
$a = [regex]"\d+\.\d+\.\d+\.\d+"
$b = $a.Match($tag)
$b = $b.Captures[0].value
$b = $b -replace '-', '.'
$b = $b -replace 'v', ''
Write-Host "Version found: $b"
$newBuildNumber = '$(Build.BuildNumber)' -replace $a,$b
Write-Host "##vso[build.updatebuildnumber]$newBuildNumber"
return $b
}
I can't claim credit for this code as I found it on someone's blog. But it works and I use for my release builds. You just have to call the function and it will reset the build.buildnumber to the latest tag in your repo. Its important to note, that the tag should be in normal version number format.
Example:
Tag Name: 10.1.100.0
I have some very simple variables, which I would like to change according to the environment.
I have written the code below in very different ways (including indentation) but none was fruitful. Alternatives I see are
Use variable groups (trying to avoid to have too many of them)
Write a bash script which updates the variables (will work but I
think its not a super neat solution)
variables:
- group : secrets
- name: hello
value: world
${{ if eq(parameters.environment, 'dev') }}:
- name: RabbitMQ_replicaCount
value: 3
${{ if eq(parameters.environment, 'test') }}:
RabbitMQ_replicaCount: '1'
Any other ideas will be appriciated :)
I would rather go by a PS script/Bash script for this task. Why ? The logic part of build where manipulation is needed like setting or overriding var based on branch or env can be done in a better way in script rather than the build yaml itself. Also this part un-necessary elongates the yaml.
Step 1 : Define a var in the build pipe with default env name
and may be another var whose value you want to set based on condition
Step 2 : Add a yml file(lets name it BuildEnv.yml) in your repo which actually contains your PowerShell/Bash code:
steps:
- powershell: |
if($BuildEnv -ne "Test"){
Write-Host "##vso[task.setvariable variable=BuildEnv]Dev"
Write-Host "##vso[task.setvariable variable=RabbitMQ_replicaCount]11"
}
displayName: 'Override Build Env'
# MORE CODE HERE
Step 3: Plug your yml in the build pipe as a template-
trigger:
branches:
include:
- master
name: $(date:yyyy-MM-dd_HH.mm)_$(rev:.r)
stages:
- stage: Build_Stage
displayName: Build_Stage
jobs:
- job: Build_Job
pool:
name: ABC
steps:
- template: ..\BuildEnv.yml
#REST CODE
That's it. You are done.
Reference : Template usage in Azure DevOps build - https://learn.microsoft.com/en-us/azure/devops/pipelines/process/templates?view=azure-devops
I have come across a scenario where I want to build source code dependant upon the source directory.
I have 2 languages in the same git repository (dotnet & Python).
I wanted to build the source code using single Azure Pipelines
If the commit is done for both (dotnet & Python) all the task should get executed
& if the commit is done for a specific directory only the respective language should get executed.
Please let me know how can I achieve this using -condition or if there are any other alternatives
Below is my azure-pipelines.yml
#Trigger and pool name configuration
variables:
name: files
steps:
- script: $(files) = git diff-tree --no-commit-id --name-only -r $(Build.SourceVersion)"
displayName: 'display Last Committed Files'
## Here I am getting changed files
##Dotnet/Server.cs
##Py/Hello.py
- task: PythonScript#0 ## It should only get called when there are changes in /Py
inputs:
scriptSource: 'inline'
script: 'print(''Hello, FromPython!'')'
condition: eq('${{ variables.files }}', '/Py')
- task: DotNetCoreCLI#2 ## It should only get called when there are changes in /Dotnet
inputs:
command: 'build'
projects: '**/*.csproj'
condition: eq('${{ variables.files }}', '/Dotnet')
Any help will be appreciated
I don't think it's possible to do directly what you want. All these task conditions are evaluated at the beginning of the pipeline execution. So if you set pipeline variables in any particular task, even the first one, it's already too late.
If you really want to do this, you probably have to go scripting all the way. So you set the variables in the first script using syntax from here:
(if there are C# files) echo '##vso[task.setvariable variable=DotNet]true'
(if there are Py files) echo '##vso[task.setvariable variable=Python]true'
And then in other scripts you evaluate them like:
if $(DotNet) = 'true' then dotnet build
Something among these lines. That'll probably be quite subtle so maybe it would make sense to reconsider the flow on some higher level but without extra context it's hard to say.
OK, so we are working on the micro service architecture and trying to deploy the code on azure devOps. we actually only want to trigger one pipeline when one project is changed. we are using a monorepo architecute.
This is how i have added the condition currently for the project specific build.yaml
name: $(TeamProject)_$(Build.DefinitionName)_$(SourceBranchName)_$(Date:yyyyMMdd)$(Rev:.r)
steps:
# for PowerShell Core
- pwsh: ./build.ps1
# Restore sms-messaging micro service projects
- task: DotNetCoreCLI#1
displayName: Run dotnet restore
inputs:
command: "restore"
projects: sms-messaging/src/**/*.csproj
condition: and(succeeded(), eq(variables['SmsMessaging'], 'True'))
condition contains the variable SmsMessaging from the powershell script that is executed before.
$files=$(git diff HEAD HEAD~ --name-only)
$temp=$files -split ' '
$count=$temp.Length
echo "Total changed $count files"
For ($i=0; $i -lt $temp.Length; $i++)
{
$name=$temp[$i]
echo "this is $name file"
if ($name -like "sms-messaging/*")
{
echo "changes in sms-messaging"
Write-Host "##vso[task.setvariable variable=SmsMessaging]True"
}
}
So the actual problem is that, on pushing to the repo. all the pipelines trigger. Tasks do get skipped because of the condition that's been added. But we actually don't want to trigger all the pipelines.
For that, I have created a main pipeline that would have the powerShell script and variables. and will conditionally trigger the other pipelines.
But I am unable to add a "condition" to this yaml.
resources:
pipelines:
- pipeline: SmsMessaging
project: SmsMessaging
source: SmsMessaging
trigger:
branches:
include:
- master
(i want to add condition here maybe if thats possible ? and trigger different pipelines from here. Or if there is another approach)
is this the right approach ?Any help is appreciated. Thanks
EDIT : I Also tried to login and trigger the pipelines through Powershell.
Invoke-AzDataFactoryV2Pipeline -ResourceGroupName "RGName" -DataFactoryName "FactoryName" -PipelineName "PipelineName"
but it doesnt allow me to run the pipeline saying that Organization ID is required.
You can combine your approach with path filters. You can trigger the build using path triggers in your monorepo architecture Something like:
trigger:
branches:
include:
- master
paths:
include:
- path/to/SmsMessaging/*
If this does not help, have a look at this extension
I have some test automation code that reads some values from an environment variable stored on my local machine, like this:
Environment.GetEnvironmentVariable("SAUCE_USERNAME", EnvironmentVariableTarget.User);
I'm trying to use Azure Pipelines to create this variable during pipeline execution and then read it in my test automation code. Using a YAML file.
Im reading this variable in the VS Test step of the Azure Pipeline. So if I set the variable, it has to be for the life of the Azure Pipeline.
I've tried to use the documentation here but have been unsuccessful.
Tried this code below as well but it fails with this error:
azure-pipelines.yml (Line: 39, Col: 1, Idx: 1252) - (Line: 39, Col: 1,
Idx: 1252): While scanning a simple key, could not find expected ':'.
# Create a secret variable
- powershell: |
Write-Host '##vso[task.setvariable variable=sauce.userName;issecret=true]abc'
# Attempt to output the value in various ways
- powershell: |
# Using an input-macro:
Write-Host "This works: $(sauce.userName)"
# Using the env var directly:
Write-Host "This does not work: $env:SAUCE_USERNAME"
# Using the mapped env var:
Write-Host "This works: $env:SAUCE_USERNAME"
env:
SAUCE_USERNAME: $(sauce.userName)
The easiest method is to pass the Azure DevOps(ADO) Env Variable values into your keys like this:
- task: DotNetCoreCLI#2
displayName: 'Run tests'
env:
SAUCE_USERNAME: $(sauceUsername) #this will store the value from 'sauceUsername' into SAUCE_USERNAME
SAUCE_ACCESS_KEY: $(sauceKey)
Displaying or using the value will work if you try
- bash: echo $(SAUCE_USERNAME) # will output our username stored in SAUCE_USERNAME env variable
And if you are referencing SAUCE_USERNAME in your code, the code will pick up the value from the Azure server.
This article has a good explanation
Previously, I also used Powershell, but this method is more involved and convoluted:
Create your variables in your Azure DevOps pipeline and provide those variables the values.
Create a Powershell script that you will run in the beginning to set your Env Variables. This is what my Posh looks like.
Run this Posh in the beginning as a separate step in your CI pipeline and this will set the environment variables for the VM that's being used to run your pipeline.
This is another detailed article that could help you with this.
As per request, I'm also attaching the PowerShell code that makes this possible.
Param(
[string]$sauceUserName,
[string]$sauceAccessKey,
[string]$sauceHeadlessUserName,
[string]$sauceHeadlessAccessKey
)
Write-Output "sauce.userName that was passed in from Azure DevOps=>$sauceUserName"
Write-Output "sauce.accessKey that was passed in from Azure DevOps=>$sauceAccessKey"
Write-Output "sauce.headless.userName that was passed in from Azure DevOps=>$sauceHeadlessUserName"
Write-Output "sauce.headless.access.key that was passed in from Azure DevOps=>$sauceHeadlessAccessKey"
[Environment]::SetEnvironmentVariable("SAUCE_USERNAME", "$sauceUserName", "User")
[Environment]::SetEnvironmentVariable("SAUCE_ACCESS_KEY", "$sauceAccessKey", "User")
[Environment]::SetEnvironmentVariable("SAUCE_HEADLESS_USERNAME", "$sauceUserName", "User")
[Environment]::SetEnvironmentVariable("SAUCE_HEADLESS_ACCESS_KEY", "$sauceAccessKey", "User")
I tried using both of the following syntax as suggested in answers above, but the environment variable was always blank when trying to use it in tasks further down in the pipeline (like during the build or while running tests).
[Environment]::SetEnvironmentVariable("SAUCE_USERNAME", "$(sauceUserName)", "User")
variables:
sauceUserName: '$(sauceUserName)'
What worked for me was to use the syntax to write Azure DevOps variables in an inline PowerShell script task:
- task: PowerShell#2
displayName: Add the username as an environment variable so the tests can find it.
inputs:
targetType: 'inline'
script: |
Write-Host "Making the sauceUsername available as an environment variable."
Write-Host "##vso[task.setvariable variable=SAUCE_USERNAME;]$(sauceUserName)"
My build task was then able to find the environment variable, and I could also access it in PowerShell script tasks further down in the pipeline with code like:
- task: PowerShell#2
displayName: Display the environment variable value for debugging purposes.
inputs:
targetType: 'inline'
script: |
[string] $username= $Env:SAUCE_USERNAME
Write-Host "The SAUCE_USERNAME environment variable value is '$username'."
set up pipeline variables and then try this mapping in your yaml file:
# ASP.NET Core (.NET Framework)
# Build and test ASP.NET Core projects targeting the full .NET Framework.
# Add steps that publish symbols, save build artifacts, and more:
# https://learn.microsoft.com/azure/devops/pipelines/languages/dotnet-core
pool:
vmImage: 'VS2017-Win2016'
variables:
solution: '**/*.sln'
buildPlatform: 'Any CPU'
buildConfiguration: 'Release'
yourEnvVar: '$(yourPipelineVariable)'
yourOtherEnvVar: '$(yourOtherPipelineVariable)'
We fought with this for hours thinking it was due to not being able to set environment variables but in the end it had nothing to do with that.
Here's some of our notes from trying to figure this out for Azure DevOps:
Pipeline Variables are Environment Variables and are injected into the pipeline in the first step of the process. You can confirm this by looking at the logs for the Initialize job task which lists all the environment variables.
In order to prove that the pipeline variables are in the system you can add a powershell task and inline:
Write-Host "Pipeline Variable: $(FASTLANE_APPLE_APPLICATION_SPECIFIC_PASSWORD)"
Write-Host "Environment Variable: $Env:FASTLANE_APPLE_APPLICATION_SPECIFIC_PASSWORD"
The () around the pipeline variable are critical if you want to use the pipeline variable somewhere - e.g. $(myVar)
Unrelated to the users question but perhaps helpful to those who are trying to upload to AppStore using two-factor auth (2FA). This is the documentation we were following:
We had to set FASTLANE_APPLE_APPLICATION_SPECIFIC_PASSWORD and FASTLANE_SESSION in order to get it to bypass the 2FA. We think the session one allowed us to log into AppStoreConnect but the App specific password was used for uploading. The logs sorta hint that's what's happening although you never see the session variable used.
As defined here, Secret variables should be set explicitly in step/task env variable section
To pass variables that are defined outside of the YAML it is important that you do NOT define the variable in the YAML as the one in YAML will take precedence over the exterior one.
Once the variable is defined, it can be referenced in the inline powershell script using the syntax $(variablename) however the variablename must be capitalized
per the devops hint:
To use a variable in a script, use environment variable syntax.
Replace . and space with _, capitalize the letters, and then use your platform's syntax for referencing an environment variable.
If you need to map it using an environment variable (e.g. secrets) then this is the correct syntax:
script: |
write-host "env $env:myvariable"
env:
#not sure if case matters here
myvariable: $(myvariable)
full example:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
#log the mapped variable, case doesn't matter
write-host "env $env:myvariable"
#directly access pipeline variable, CASE MATTERS!
write-host "pipeline $(MYVARIABLE)"
#show all environment variables
get-childitem env:
env:
#not sure if case matters here
myvariable: $(myvariable)