I have my normal and working release pipeline that, by given a certain deployment group, performs some tasks:
Copies a script
Executes that powershell script (on the target machines defined in the Deployment Group)
Deletes the script
I know that YAML doesn't support deployment groups, but (lucky me!) so far my deployment group has only one machine, let's call it MyTestVM .
So what I am trying to achieve mainly is simply executing a powershell script on that vm . Normally, what happenes with the release pipeline, is that you have a tentacle/release agent installed on the VM, your deployment target (which is inside the Deployment Group) is hooked up to that, and your release pipeline (thanks to the Deployment Group specification) is able to use that release agent on the machine and do whatever it wants on the VM itself.
I need the same... but through YAML ! I know there is PowerShellOnTargetMachines command available in YAML but I don't want to use that. It uses PSSession, it requires SSL certificates and many other things. I just want to use the already existing agent on the VM !
What I have in place so far:
pool: 'Private Pool'
steps:
- task: DownloadPipelineArtifact#2
inputs:
buildType: 'specific'
project: 'blahblah'
definition: 'blah'
buildVersionToDownload: 'latest'
targetPath: '$(Pipeline.Workspace)'
- task: CopyFiles#2
displayName: 'Copy Files to: C:\TestScript'
inputs:
SourceFolder: '$(Pipeline.Workspace)/Scripts/'
Contents: '**/*.ps1'
TargetFolder: 'C:\TestScript'
CleanTargetFolder: true
OverWrite: true
The first part just downloads the Artifact containing my script. And then to be honest I am not even sure I need to copy the script in the second part.. first because I don't think it is copying the script to the VM target workspace, but it is copying it to the VM where the azure pipeline agent is installed. And second: I think I can just reference it from my artifact.. but this is not the important part.
How can I make my YAML pipeline make use of the release agent installed on the VM in the same way that a normal release pipeline does?
Reached somehow a solution. First of all worth mentioning that since deployment groups don't work with YAML pipelines the way to proceed is to create an Environment and add as resource your target VM.
So I didn't need to create my own hosted agent or anything special since the problem was the target itself and not the agent running the pipeline.
By creating an Environment and adding a resource (in my case a VM) to that environment, we create also a new release agent on the target itself. So my target VM will now have 2 release agents: the old one that can be used by normal release pipelines, and the new one, attached to the Environment resource on Azure Devops that can be used by YAML pipelines.
Now I am finally able to hit my VM:
- stage: PerformScriptInVM
jobs:
- deployment: VMDeploy
pool:
vmImage: 'windows-latest'
# watch out: this creates an environment if it doesn’t exist
environment:
name: My Environment Name
resourceType: VirtualMachine
strategy:
runOnce:
deploy:
steps:
- task: DownloadPipelineArtifact#2
inputs:
buildType: 'specific'
project: 'blahblahblah'
definition: 'blah'
buildVersionToDownload: 'latest'
targetPath: '$(Pipeline.Workspace)'
- task: PowerShell#2
displayName: 'PowerShell Script'
inputs:
targetType: filePath
filePath: '$(Pipeline.Workspace)/Scripts/TestScript.ps1'
arguments: 'whatever your script needs..'
To get the job to run on the specific release agent you want, you can do two things:
Create a pool and only put your release agent into it.
pool: 'My Pool with only one release agent'
Use an existing pool, and publish/demand a capability for your agent.
On the agent machine itself, add a system environment variable (for example, MyCustomCapability. Give it a value like 1
then your pipeline becomes:
pool:
name: 'My pool with potentially more than one agent'
demands: 'MyCustomCapability'
If only this agent has this environment variable set, then only this agent can execute the job
Related
Is there a way within a Release pipeline in Azure to pass variables created in one stage to the next stage?
I see lots of documentation about using echo "##vso[task..... - This however does not seem to work within the release pipeline.
I am mainly using bash scripts and I can reuse it within the same stage in different tasks, but not subsequent stages.
This seems like an essential feature to me, passing variables through stages...
Is there a way to do this?
If you want to pass variables from one stage to another stage in yml pipelines for release, you are supposed to use echo "##vso[task....." follow the doc
For a simple example:
stages:
- stage: BuildStage
jobs:
- job: BuildJob
steps:
- bash: echo "##vso[task.setvariable variable=TestArtifactName;isoutput=true]testValue"
name: printvar
- stage: DeployWebsiteStage
lockBehavior: sequential
dependsOn: BuildStage
condition: succeeded()
variables:
BuildStageArtifactFolderName: $[stageDependencies.BuildStage.BuildJob.outputs['printvar.TestArtifactName'] ]
jobs:
- deployment: DeployWebsite
environment:
name: webapplicationdeploy
strategy:
runOnce:
deploy:
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: Write-Host "BuildStageArtifactFolderName:" $(BuildStageArtifactFolderName)
You are supposed to get the value set from stage 'BuildStage'.
If you want to pass variables from one stage to another stage in classic release pipelines,
1.Set Release Permission Manage releases for the Project Collection Build Service as Allow.
2.Toggle on 'Allow scripts to access the OAuth token' for the first stage
3.Set a variable like 'StageVar' in release scope.
4.ADD the first powershell task(inline) in the first stage for creating a variable 'myVar' in the first stage.
5.update the Release Definition and Release Variable (StageVar)
6.Add a powershell task in the second stage to retrieve the value of myVar via the Release Variable StageVar.
You could refer the blog for more details.
It works on my side,
results in the first stage:
results in the second stage:
You can make use of the Azure DevOps Variable group to store your variables and call them in your release pipelines across multiple stages and multiple pipelines within a project. You can make use of Azure CLI to make use of the Variable group.
I have stored 2 variables of database name and password in the SharedVariables group. I can call this variable group in 2 ways: 1) Via the YAML pipeline and 2) Via Classic/Release pipelines.
Yaml:-
trigger:
- main
pool:
vmImage: ubuntu-latest
variables:
- group: SharedVariables
steps:
- script: |
echo $(databaseserverpassword)
When I ran the pipeline the database server password was encrypted like the below:-
In Release pipeline:-
You can add this variable within multiple pipelines and multiple stages in your release like below:-
But the above method will help you store static values in the variable group and not the output variable of build to release Unless you specifically assign those variables manually in the variable group. You can make use of this extension > Variable Tools for Azure DevOps Services - Visual Studio Marketplace
With this extension, you can store your variables from the build in a JSON file and load that JSON file in the next release stage by calling the task from the extension.
Save the build variable in a file:-
Load the build variable in a release:-
Another method is to store the variables in a file as a build artifact and then call the build artifact in the release pipeline with the below yaml code:-
trigger:
- dev
pool:
vmImage: windows-latest
parameters:
- name: powerenvironment
displayName: Where to deploy?
type: string
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
$variable = '${{parameters.powerenvironment}}'
$variable | Out-File $(Build.ArtifactStagingDirectory)\filewithvariable.txt
Get-Content $(Build.ArtifactStagingDirectory)\filewithvariable.txt
- task: PublishBuildArtifacts#1
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'drop'
publishLocation: 'Container'
And download the artifact and run the tasks in your release pipeline.
Reference:-
Pass parameters from build to release pipelines on Azure DevOps - GeralexGR
Another simple method is to run the PowerShell script to store the build output as JSON in the published artifact and read the content in the release pipeline like below:-
ConvertTo-Json | Out-File "file.json"
Get-Content "file.json" | themnvertFrom-Json
You can also reference the dependencies from various stages and call them in another stage within a pipeline with below yaml code :-
stages:
- stage: A
jobs:
- job: A1
steps:
- bash: echo "##vso[task.setvariable variable=shouldrun;isOutput=true]true"
# or on Windows:
# - script: echo ##vso[task.setvariable variable=shouldrun;isOutput=true]true
name: printvar
- stage: B
condition: and(succeeded(), eq(dependencies.A.outputs['A1.printvar.should run], 'true'))
dependsOn: A
jobs:
- job: B1
steps:
- script: echo hello from Stage B
Reference:-
bash - How to pass a variable from build to release in azure build to release pipeline - Stack Overflow By PatrickLu-MSFT
azure devops - How to get the variable value in TFS/AzureDevOps from Build to Release Pipeline? - Stack Overflow By jessehouwing
VSTS : Can I access the Build variables from Release definition? By Calidus
Expressions - Azure Pipelines | Microsoft Learn
I am working on a multi stage pipeline that build and deploy some c# code from staging to production
Everything works just fine but I wanted to try and customise a bit more the pipeline so I can see the actual version that is being built and deployed as part of the name of the stage.
At the current stage, this my multi stage pipeline
trigger:
batch: true
tags:
include:
- '*'
branches:
exclude:
- main
- staging
pool:
vmImage: 'ubuntu-latest'
variables:
buildNumber: "$[variables['Build.BuildNumber']]"
DOCKER_BUILDKIT: 1
dockerRegistryServiceConnectionStaging: '<My-Connection-String>'
imageRepositoryStaging: '<My-Repo-Name>'
containerRegistryStaging: '<My-Container-Name>'
dockerRegistryServiceConnectionProd: '<My-Connection-String>'
imageRepositoryProd: 'My-Repo-Name>'
containerRegistryProd: '<My-Container-Name>'
dockerfilePath: 'pathTo/Dockerfile'
solution: 'path/To/Solution.csproj'
tag: '$(Build.BuildNumber)'
stages:
- stage: 'Build_Staging'
displayName: 'Build_Staging'
jobs:
- job: buildStaging
displayName: 'DotNet Core publish and dockerize'
steps:
- powershell: |
# Write your PowerShell commands here.
Write-Host "Update Build.BuildNumber"
cd $(System.DefaultWorkingDirectory)
$Latesttag = $(git describe --tags $(git rev-list --tags --max-count=1))
Write-Host "The latest git tag is $Latesttag "
Write-Host
"##vso[build.updatebuildNumber]$Latesttag"
- task: DotNetCoreCLI#2
displayName: 'DotNet - Restore'
inputs:
command: 'restore'
projects: $(solution)
noCache: true
versioningScheme: 'off'
vstsFeed: '<Feed>'
- task: DotNetCoreCLI#2
name: 'DotnetPublish'
displayName: 'dotnet - Publish'
inputs:
command: 'publish'
projects: $(solution)
arguments: '-o publish/solution -c release'
modifyOutputPath: false
zipAfterPublish: false
publishWebProjects: false
- task: Docker#2
name: 'dockerBuildAndPush'
displayName: 'docker - Build & Push $(tag)'
inputs:
repository: $(imageRepositoryStaging)
Dockerfile: $(dockerfilePath)
containerRegistry: ${{ variables.dockerRegistryServiceConnectionStaging }}
buildContext: ${{ variables.buildContext }}
tags: |
$(Build.BuildNumber)
latest
- stage: 'Deploy_Staging'
jobs:
- deployment: 'Deploy'
environment: 'Staging'
variables:
EnvironmentName: 'Staging'
strategy:
runOnce:
deploy:
steps:
- task: AzureRmWebAppDeployment#4
displayName: 'Deploy Azure App Service To Staging'
inputs:
azureSubscription: '<Azure-Subscription>'
appType: 'webAppContainer'
DockerNamespace: '<container-namespace>'
DockerRepository: '<Repository>'
DockerImageTag: '$(Build.BuildNumber)'
WebAppName: '<WebAppName>'
The Powershell command is to override the Build.BuildNumber with the tag I am pushing to GitHub.
When I run this pipeline, in azure DevOps, I see the stage name Build_Staging_$(Build.BuildNumber) as a string.
What I would really like to see is, if I push the tag 'v1.0.0` for example, is to see the stage name like:
Build_Staging_v1.0.0
I tried to use the displayName and the output is not the one I was looking for and if I try with name instead of displayName I get the error unexpected value name
Can please please anyone help understand what am I doing wrong and how I can achieve this?
Please if my question is not 100% clear and missing any important detail, just let me know
UPDATE:
I did update the post with my entire pipeline.
This pipeline, before it used to be a single job process, and everything was working fine. But to get my hand dirty, I wanted to add stages to split and workflow based on resources and environment.
The process is still working and this is what I am expecting.
In my GitHub, when I create a tag on the main branch, this will trigger my build stage. Which thanks to the Powershell script to update the BuildNumber with the tag, I am able to build the docker image in my container registry in the following format:
docker-image-name:v1.0.1
That version can be seen at this level also:
This updated buildNumber (now is Tag) is use in Azure pipelines App Slack to check the version that has been pushed.
So far everything is good.
But I am facing the problem with the deployment job, at that level I am not able to set any Powershell script to update that same BuildNumber with the tag. I checked the documentation and nothing is mentioned about how I can add another job or step. I tried implementing this but I get errors that the value is unexpected.
Let me just share another screenshot to fully explain the issue.
Assuming I am deploying the docker image v1.0.1, everything works perfectly, but the build number in deployment stage, is not being updated, in fact in slack channel, I see the normal build number, as follow:
Instead of having the buildNumber, I would like to have my tag.
Please any help here?
Unfortunately, you won't be able to set a stage name to a dynamic variable that is set within one of its child's steps. You'll have to set it to a pipeline-level variable or predefined variable
Variable evaluation goes top-down from stages to tasks:
stages
jobs
tasks
To help explain exactly why, let's talk about how variable evaluation works in general with regard to this structure:
VARIABLE EVALUATION: Using stages as an example, you can set a stage name using any dynamic value that's present when the stage is evaluated. This means the variable is only accessed when the stage is initially "rendered". Azure DevOps requires that the variable be present before evaluation and will not retroactively update the UI if that variable is changed within a child step.
Let's talk about each and their respective limitations on what variables you can use in their names:
STAGES: pipeline-level variables, parameters (in the case of templates), or predefined variables
JOBS: stage-level variables, pipeline-level variables, parameters (in the case of templates), or predefined variables
TASKS: job-level variables, stage-level variables, pipeline-level variables, parameters (in the case of templates), or predefined variables
I did something similar by setting my build number to a repo tag. Here is the PowerShell function that sets the Build.Buildnumber variable to the tag value. You can just call it straight out or base it off a parameter if you have other version number logic.
function getTagVersion() {
$tag = iex "git describe --long --tags --always"
$a = [regex]"\d+\.\d+\.\d+\.\d+"
$b = $a.Match($tag)
$b = $b.Captures[0].value
$b = $b -replace '-', '.'
$b = $b -replace 'v', ''
Write-Host "Version found: $b"
$newBuildNumber = '$(Build.BuildNumber)' -replace $a,$b
Write-Host "##vso[build.updatebuildnumber]$newBuildNumber"
return $b
}
I can't claim credit for this code as I found it on someone's blog. But it works and I use for my release builds. You just have to call the function and it will reset the build.buildnumber to the latest tag in your repo. Its important to note, that the tag should be in normal version number format.
Example:
Tag Name: 10.1.100.0
When using Azure DevOps pipelines, is it possible to deploy to on-prem servers using a yaml pipeline?
I have deployed to on premise servers with a Release (classic) pipeline using deployment groups, and I have seen instructions on deploying to Azure infrastructure using yaml pipelines.
However I can't find any examples of how to deploy to on-prem servers using yaml pipelines - is this possible yet? If so are there any examples available of how to achieve this?
As already explained in the previous answers, you need to create a new environment and add VMs to the environment (see documentation).
Using a deployment job, you also need to specify the resourceType
- deployment: VMDeploy
displayName: Deploy to VM
environment:
name: ContosoDeploy
resourceType: VirtualMachine
...
If you have multiple VMs in this environment, the job steps will be executed on all the VMs.
To target specific VMs, you can add tags (see documentation).
jobs:
- deployment: VMDeploy
displayName: Deploy to VM
environment:
name: ContosoDeploy
resourceType: VirtualMachine
tags: windows,prod # only deploy to virtual machines with both windows and prod tags
...
Yes you can. In YAML pipelines you can use "Environments" as a replacement for deployment groups. They work in a similar way in that you install the agent on the target machine and then specify the environment in your Deployment Job
Create a new Environment with an appropriate name (e.g. Dev) then add a resource, you'll be able to add either a VM or a Kubernetes cluster. Assuming that you choose VM, then you will able to download a script which you can run an target machines to install the deployment agent. This script will install and register the agent in to the environment.
Once you have the Agent registered in to the environment add a deployment job to your YAML
- stage: DeployToDev
displayName: Deploy to Dev
jobs:
- deployment: DeployWebSite
displayName: Deploy web site to Dev
environment: Dev
strategy:
runOnce:
deploy:
steps:
- task: PowerShell#2
inputs:
targetType: 'inline'
script: |
Write-Host "Deploy my code"
i have wired behavior with the copy files from hosted agent and then downloading them back to the same agent
looks like it copies the files from agent A but the same pipeline downloading them back to Agent B
with is in another machine doing another build job that is not related
Upload from ios_docker_142_linux_slave_1
Download back to different agent ios_docker_141_linux_slave_3 , why ?
- task: CopyFiles#2
inputs:
CleanTargetFolder: 'true'
SourceFolder: '$(Agent.HomeDirectory)/../${{parameters.Folderpath}}'
Contents: '**'
TargetFolder: '$(build.artifactstagingdirectory)'
This is an expected behavior if you are using parallel jobs. According to your screenshot, there are multiple jobs self-hosted connect , mac_agent, copy_back_files_to_self..
One agent one job at a time. If the agent is running a job, it will in busy status, and other jobs will look for idle agents to run . The parallel jobs is for running multiple jobs in multiple agents at a time.
To achieve what you want, you need to specify detail agent in your YAML file. The pool name needs to add to the name field, then you could add demands. You may try the following YAML Code:
stages:
- stage: Deploy
pool:
name: AgentPoolName(e.g. alm-aws-pool)
demands:
- agent.name -equals Agentname (e.g. deploy-05-agent1)
jobs:
- job: BuildJob
steps:
- script: echo Building!
We have a multi-stage YAML pipeline that does CI/CD to an existing set of Azure Resources
The stages are
Build
Deploy to Development and Run Tests
If Previous succeeded - Deploy to Production and Run Tests
We use the AzureRmWebAppDeployment task during the deployment stages and we use the AppSettings argument to that task to specify environment-specific settings. For example
- task: AzureRmWebAppDeployment#4
displayName: 'Deploy Azure App Service'
inputs:
azureSubscription: '$(azureSubscriptionEndpoint)'
appType: '$(webAppKind)'
WebAppName: 'EXISTING__AZURE_RESOURCENAME-DEV'
Package: '$(Pipeline.Workspace)/**/*.zip'
AppSettings: >
-AzureAd:CallbackPath /signin-oidc
-AzureAd:ClientId [GUID was here]
-AzureAd:Domain [domain was here]
-AzureAd:Instance https://login.microsoftonline.com/
-AzureAd:TenantId [Id was here]
-EmailServer:SMTPPassword SECRETPASSWORD
-EmailServer:SMTPUsername SECRETUSERNAME
There are two settings in that set, EmailServer: SMTPUsername and EmailServer: SMTPPassword that I want to pull from an Azure KeyVault. I know how to reference the KV secret from Azure Portal using the syntax
#Microsoft.KeyVault(SecretUri=https://our.vault.azure.net/secrets/SendGridPassword/ReferenceGuidHere)
but how do I reference the value from the YAML pipeline so it is set in Azure?
As pointed out by Thomas in this comment, Referencing Azure Key Vault secrets from CI/CD YAML
I can explicitly set the value in the YAML file like this:
-EmailServer:SMTPPassword #Microsoft.KeyVault(SecretUri=https://our.vault.azure.net/secrets/SendGridPassword/ReferenceGuidHere)
You need to set an AzureKeyVault#1 task with RunAsPreJob to true, this will make your key vault values available as CI/CD jobs environment variables so you can use it as $(KEY-OF-SECRET-VALUE) on the rest of your stages in the job.
The following piece of yaml file is a working example.
We set for python unittest a set of env variable provided from Azure key-vault
trigger:
batch: true # disable concurrent build for pipeline
branches:
include:
- '*' # CI start for all branches
pool:
vmImage: ubuntu-16.04
stages:
- stage: Test
jobs:
- job: sample_test_stage
steps:
- task: AzureKeyVault#1
inputs:
azureSubscription: 'YOUR SUBSCRIPTION HERE'
KeyVaultName: 'THE-KEY-VAULT-NAME'
SecretsFilter: '*'
RunAsPreJob: true
- task: UsePythonVersion#0
inputs:
versionSpec: '3.7'
- script : python -m unittest discover -v -s tests
displayName: 'Execute python unittest'
env: { MY-ENV-VAL-1: $(SECRET-VALUE-1), MY-ENV-VAL-2: $(SECRET-VALUE-2)}
Note that sometimes you need to approve connection beetween AzureDevops and another Azure service like KeyVault