Hi I am using agent pool in azure pipeline, so i have multiple agents so i want to parameterize the pool name like shown below but when i do this i get an error as (Line: 11, Col: 9): A template expression is not allowed in this context
parameters:
- name: pool_name
displayName: agent pool location
type: string
default: test-ashish
trigger:
- none
pool:
name: ${{parameters.pool_name}}
demands:
- Location -equals EASTUS2
steps:
- script: |
echo "This job is only for testing"
echo $()
name: Test_Job
displayName: Test Job
This issue can be solved by using the answer in this issue: Azure Devops: Is it possible to nest yaml templates inside another yaml template?.
Copyed from link:
refer code:
parameters:
buildArtifactName: "build"
solution: ""
jobs:
- job: 'BuildSolution'
pool:
vmImage: ${{parameters.vmImage}}
continueOnError: false
variables:
artifactName: ${{ parameters.buildArtifactName}}
steps:
- task: NuGetCommand#2
displayName: 'Restore NuGet packages'
inputs:
restoreSolution: ${{ parameters.solutionDir }}/${{ parameters.solution }}
configuration: ${{parameters.buildConfiguration}}
Related
I have a multi stage Azure DevOps pipeline in yaml. Each stage includes multiple jobs. The pipeline runs properly by using Azure-hosted agents. I need to run it by our company's self-hosted agent. The self-hosted agent is a virtual machine scale set (VMSS) which is defined as an agent pool in DevOps, it has been tested and is working properly. Because the self-hosted agent does not have all the capabilities we need, our company's policy is to run the jobs inside docker containers. The docker images reside in our company's private Azure Container Registy.
My question is how can I run a whole template stage in a docker container? Please note that my question is not about executing a job but all the jobs in a stage template. This link explains how to execute jobs inside a container. It doesn't give any examples of how to run all the jobs in a stage in a container, especially when the stage is defined as a template.
My pipeline in simplified form is as follows:
## Name of self-hosted agent
pool: Platform-VMSS
resources:
containers:
- container: base
image: myazurecontainerregistryname.azurecr.io/platform/myimg:1.0.0
endpoint: 'serviceconnection-acr'
trigger:
branches:
include:
- "feature/ORGUS-4810"
exclude:
- "main"
- "release"
paths:
include:
- "usecase-poc/**"
variables:
pythonVersion: 3.8
whlPackageName: py_core
srcDirectory: usecase-poc/$(whlPackageName)
${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
BuildFrom: main
PackageName: $(whlPackageName)
versionOption: custom
deployToDevPipeline: True
deployToQAPipeline: True
deployToProdPipeline: True
${{ else }}:
BuildFrom: branch_${{ lower(variables['Build.SourceBranchName'] ) }}
PackageName: $(whlPackageName)_$(BuildFrom)
versionOption: patch
deployToDevPipeline: True
deployToQAPipeline: False
deployToProdPipeline: False
stageName: 'deployToDev'
stages:
- stage: DownloadArtifact
displayName: "Download python whl from artifactory"
jobs:
- job: DownloadArtifactJob
steps:
- checkout: self
- task: UniversalPackages#0
displayName: 'Download Artifact with Universal Packages'
inputs:
vstsFeed: 'some-value/00000000-0000-0009-0000-d24300000000'
vstsFeedPackage: '00000000-0000-0000-0000-0000000'
vstsPackageVersion: 0.8.4
downloadDirectory: $(Build.ArtifactStagingDirectory)
- task: Bash#3
name: GetWheelName_Task
inputs:
targetType: 'inline'
script: |
echo $(Build.ArtifactStagingDirectory)
find $(Build.ArtifactStagingDirectory) -name '*.whl'
ArtifactName=$(find $(Build.ArtifactStagingDirectory) -name '*.whl')
echo "Artifact name value is " $ArtifactName
echo "##vso[task.setvariable variable=ArtifactName;isOutput=true]$ArtifactName"
displayName: 'Get downloaded artifact in source directory'
- task: PublishBuildArtifacts#1
displayName: "Publish downloaded artifact to pipeline's output"
inputs:
pathToPublish: $(Build.ArtifactStagingDirectory)
artifactName: whl
- stage: ConstructSharedVariablesForAllStages
displayName: Construct Shared Variables For All Stages
dependsOn: DownloadArtifact
variables:
- group: proj-shared-vg
- name: ArtifactName
value: $[stageDependencies.DownloadArtifact.DownloadArtifactJob.outputs['GetWheelName_Task.ArtifactName']]
jobs:
- job: DownloadArtifact
container: base
steps:
- task: Bash#3
displayName: "Print variable value"
inputs:
targetType: 'inline'
script: |
echo $(ArtifactName)
- task: Bash#3
name: extractWheelName
displayName: Extract Wheel Name
inputs:
targetType: inline
script: |
echo $(ArtifactName) | awk -F"/" '{print $NF}'
WhlName="py_core-0.8.4-py3-none-any.whl"
echo "##vso[task.setvariable variable=WhlName;isOutput=true]$WhlName"
- task: DownloadPipelineArtifact#2
displayName: "Download artifact from previous stage"
inputs:
buildType: 'current'
project: 'Project Name'
buildVersionToDownload: 'latest'
artifactName: whl
targetPath: '$(System.ArtifactsDirectory)'
- pwsh: |
$whlFile = Get-ChildItem -Filter *.whl -Path "$(System.ArtifactsDirectory)" | ForEach-Object { $_.fullname } | Select-Object -First 1
Write-Host "##vso[task.setvariable variable=whlFile]$whlFile"
name: SetVars
displayName: Get wheel name
## This is the section where my question is about. How can I make sure each stage runs in the self-hosted agent pool. The stage contains multiple jobs.
- template: ../templates/azure-pipeline-stage-template.yaml
parameters:
deploy: ${{variables.deployToDevPipeline}}
variableGroup: databricks-sp-vg-dev
stageName: DeployToDev
environment: DBRKS_Dev_WHL
conditionParameter: deployToDev
dependsOnStage: ConstructSharedVariablesForAllStages
- template: ../templates/azure-pipeline-stage-template.yaml
parameters:
deploy: ${{variables.deployToQAPipeline}}
variableGroup: databricks-sp-vg-qa
stageName: DeployToQA
environment: DBRKS_QA_WHL
conditionParameter: deployToQA
dependsOnStage: DeployToDev
- template: ../templates/azure-pipeline-stage-template.yaml
parameters:
deploy: ${{variables.deployToProdPipeline}}
variableGroup: databricks-sp-vg-prod
stageName: DeployToProd
environment: DBRKS_Prod_WHL
conditionParameter: deployToProd
dependsOnStage: DeployToQA
In the code above in resources the container resides in our Azure Container Registry (ACR), the endpoint is our DevOps service connection of type container registry to pull and push images to and from ACR. In the code above I have commented where the issue is. In templates where I am refering to stage templates, I would like to run them all inside a container where I have defined as a resource at the beginning of the pipeline. The stage template has multiple jobs. Here is just a sample of stage template when running to emphasize it has multiple jobs:
The highlighted stage is the one created by template:
- template: ../templates/azure-pipeline-stage-template.yaml
parameters:
deploy: ${{variables.deployToDevPipeline}}
variableGroup: databricks-sp-vg-dev
stageName: DeployToDev
environment: DBRKS_Dev_WHL
conditionParameter: deployToDev
dependsOnStage: ConstructSharedVariablesForAllStages
Question is how to enforce all the jobs in the above template run in the docker container defined as resource in our pipeline. Thank you very much for your valuable input.
Add a container field at the job level as shown below. Then all the jobs in the template will be running on specified container.
pool:
vmImage: ubuntu-latest
resources:
containers:
- container: testcontainer
image: ubuntu
stages:
- stage: template01
displayName: tempate test
jobs:
- job: template
container: testcontainer
steps:
- template: templates/template01.yaml
- stage: template02
displayName: template test
jobs:
- job: template
container: testcontainer
steps:
- template: templates/template02.yaml
Else, add a step target field to all the required tasks in a template, as referred to in this link Build and Release Tasks - Azure Pipelines | Microsoft Learn](https://learn.microsoft.com/en-us/azure/devops/pipelines/process/tasks?view=azure-devops&tabs=yaml#step-target) "https://learn.microsoft.com/en-us/azure/devops/pipelines/process/tasks?view=azure-devops&tabs=yaml#step-target)")
resources:
containers:
- container: pycontainer
image: python:3.11
steps:
- task: AnotherTask#1
target: pycontainer
I want to customize environment attribute to choose programmatically environment for approval (dev, preprod, prod). When I try to launch pipeline, I see this error. Is there an alternative?
Job : Environment $(environment) could not be found. The environment does not exist or has not been authorized for use.
variables:
environment: dev
jobs:
- deployment: test
displayName: test
timeoutInMinutes: 0
# creates an environment if it doesn't exist
environment: $(environment)
strategy:
runOnce:
deploy:
steps:
- checkout: self
clean: true
displayName : Checkout repository
- task: NodeTool#0
inputs:
versionSpec: '16.x'
checkLatest: true
For this use case you should use parameters. This is because variables are not available during the initial parsing stage of the pipeline.
parameters:
- name: "environment"
type: string
default: "development"
And then environment: ${{ parameters.environment }}
Or if you want to get fancy, you could do something like this:
parameters:
- name: "environments"
type: object
default:
- name: development
param1: value
param2: value
- name: test
param1: value
param2: value
# This will look through the environment parameter and create a job for each environment.
- ${{ each environment in parameters.environments }} :
jobs:
- deployment: test
#read in vars from a file in variables/development.yml
variables: variables/${{ environment.name }}.yml
displayName: test
timeoutInMinutes: 0
# creates an environment if it doesn't exist
environment: ${{ environment.name }}
strategy:
runOnce:
deploy:
steps:
- checkout: self
clean: true
displayName : Checkout repository
- task: NodeTool#0
inputs:
versionSpec: '16.x'
checkLatest: true
I'm having this Release created via Azure and its working perfectly. I have create the pipeline using YAML and i want to put the release in the same file. I'm using the "View YAML" approach for this to copy the YAML template and use it. But somehow its failing with this message:
ERROR ( message:Cannot find APP object with identifier "TEST01/". )
##[error]Process 'appcmd.exe' exited with code '1168'.
I guess im missing some other parameters or i really can't figure out how to use the same approach using YAML.
This is the code:
trigger:
- develop
stages:
- stage:
jobs:
- job: Build
displayName: Agent job 1
pool:
vmImage: ubuntu-latest
steps:
- checkout: self
- task: ArchiveFiles#2
displayName: Archive $(System.DefaultWorkingDirectory)
inputs:
rootFolderOrFile: "$(System.DefaultWorkingDirectory)"
includeRootFolder: false
- task: PublishBuildArtifacts#1
displayName: 'Publish Artifact: drop'
- stage:
jobs:
- job: Deploy
displayName: Deploy to IIS Dev
pool:
vmImage: windows-2019
steps:
- task: IISWebAppManagementOnMachineGroup#0
displayName: 'IIS Web App Manage'
inputs:
IISDeploymentType: iISVirtualDirectory
ParentWebsiteNameForVD: TEST01
VirtualPathForVD: /admin/test
PhysicalPathForVD: '%SystemDrive%\inetpub\_phpapps\test-center'
- task: IISWebAppDeploymentOnMachineGroup#0
displayName: 'IIS Web App Deploy'
inputs:
WebSiteName: "TEST01"
VirtualApplication: "/admin/test"
TakeAppOfflineFlag: True
XmlVariableSubstitution: True
After some research and testing seams like there is a new approach regarding deployment/releases.
Instead of using "Pipelines > Release" and create a release, if you are using the Yaml approach (defining stages etc.) you need to use it with "Environments". Create the group or environment and use it.
Two steps here are missing, one is the package and the other is env, where should be deployed.
So the solution for the above code is the following:
- stage: 'DeployTest'
displayName: 'DeployTest'
dependsOn: 'Build'
condition: and(succeeded(), eq(variables['Build.SourceBranchName'], 'develop'))
jobs:
- deployment: Test
displayName: Test
environment:
name: 'ENV-TEST-GROUP'
resourceType: VirtualMachine
strategy:
runOnce:
deploy:
steps:
- task: IISWebAppManagementOnMachineGroup#0
displayName: 'IIS Web App Manage'
inputs:
IISDeploymentType: iISVirtualDirectory
ParentWebsiteNameForVD: TEST01
VirtualPathForVD: /admin/test
PhysicalPathForVD: '%SystemDrive%\inetpub\_phpapps\test-center'
- task: IISWebAppDeploymentOnMachineGroup#0
displayName: 'IIS Web App Deploy'
inputs:
WebSiteName: "TEST01"
VirtualApplication: "/admin/test"
Package: $(Pipeline.Workspace)\drop\build_test.zip
TakeAppOfflineFlag: True
XmlVariableSubstitution: True
My current pipeline consists of 2 files:
pr-validation.yml
resources:
- repo: self
queue:
name: NonProd
variables:
- name: 'NuGetFeedId'
value: '11111b1d-1111-1ecb-1dc1-11f111111f11'
steps:
- template: pr-validation-steps.yml
parameters:
UnitTestsProjectPaths:
- '1.Tests/1.Tests.csproj'
- '2/2.Tests/2.Tests.csproj'
- '3/3.Tests/3.Tests.csproj'
and actual steps in pr-validation-steps:
parameters:
- name: UnitTestsProjectPaths
type: object
default:
- '1.Tests/1.Tests.csproj'
- '2/2.Tests/2.Tests.csproj'
- '3/3.Tests/3.Tests.csproj'
steps:
- task: DotNetCoreCLI#2
displayName: 'NuGet restore'
inputs:
command: 'restore'
vstsFeed: '$(NuGetFeedId)'
projects: '**/*.sln'
- task: DotNetCoreCLI#2
displayName: 'Build solution'
inputs:
command: 'build'
projects: '**/*.sln'
arguments: '--no-restore'
- ${{ each UnitTestsProjectPath in parameters.UnitTestsProjectPaths }}:
- task: DotNetCoreCLI#2
displayName: 'Run unit tests in ${{ UnitTestsProjectPath }} and collect coverage'
inputs:
command: 'test'
projects: '${{ UnitTestsProjectPath }}'
arguments: '--configuration $(buildConfiguration) --collect "Code coverage" --no-restore'
As you see I'm on pre-build validation for .NET Core projects and the current fragment is about running several test projects from solution, but not all of them.
Azure DevOps is saying:
'pr-validation-steps.yml (Line: 2, Col: 1): Expected a mapping'
Basically saying that something's wrong in at the beginning of the line with a 'name' entry. I tried different syntax options but nothing worked for me.
I validated the files with Tom Austin's 'Azure Pipeline YAML Validator' and it says everything's fine.
What am I doing wrong? And in case you're using any kind of effective pipeline validator - please let me know, I really need it.
First you have this
queue:
name: NonProd
which I don't recognize and Azure DevOps also.
When you remove it you should be fine.
And speaking about validator. The only one good which I know is the one on the Azure Devops:
I tested this with following:
build.yml
trigger: none
pool:
vmImage: ubuntu-latest
variables:
- name: 'NuGetFeedId'
value: '11111b1d-1111-1ecb-1dc1-11f111111f11'
steps:
- template: pr-validation-steps.yml
parameters:
UnitTestsProjectPaths:
- '1.Tests/1.Tests.csproj'
- '2/2.Tests/2.Tests.csproj'
- '3/3.Tests/3.Tests.csproj'
pr-validation-steps.yml
parameters:
- name: UnitTestsProjectPaths
type: object
default:
- '1.Tests/1.Tests.csproj'
- '2/2.Tests/2.Tests.csproj'
- '3/3.Tests/3.Tests.csproj'
steps:
- ${{ each UnitTestsProjectPath in parameters.UnitTestsProjectPaths }}:
- bash: echo 'Run unit tests in ${{ UnitTestsProjectPath }} and collect coverage'
And it works.
I have the following pipeline
variables:
azureSubscription: ...
stages:
- stage: Deploy
displayName: Deploy stage
dependsOn: Build
condition: succeeded()
jobs:
- deployment: Deploy
displayName: Deploy
environment: 'development'
strategy:
runOnce:
deploy:
steps:
- task: AzureResourceGroupDeployment#2
inputs:
action: 'Create Or Update Resource Group'
resourceGroupName: '...'
location: '...'
templateLocation: 'Linked artifact'
csmFile: '$(Pipeline.Workspace)/azure-deploy.json'
deploymentMode: 'Incremental'
The repo has the following files (at the root directory)
azure-pipelines.yaml
azure-deploy.json
and only a master branch.
I have tried:
azure-deploy.json
**azure-deploy.json
**/*azure-deploy.json
$(Build.SourcesDirectory)/azure-deploy.json
$(Pipeline.Workspace)/azure-deploy.json
$(System.DefaultWorkingDirectory)/azure-deploy.json
Having read:
Azure Pipeline Error: Could not find any file matching the template file pattern
VSTS Pipeline Deployment of ARM Error: Could not find any file matching the template file pattern
https://github.com/microsoft/azure-pipelines-tasks/issues/11520
to no avail. Any ideas?
Update: I have added a publish pipeline as suggested by #ShaykiAbramczyk
Now I get a Template file pattern matches a directory instead of a file: /home/vsts/work/1/azure-deploy.json
- stage: Build
displayName: Build stage
jobs:
- job: Build
displayName: Build
pool:
vmImage: $(vmImageName)
steps:
- task: PublishPipelineArtifact#1
inputs:
targetPath: '$(Pipeline.Workspace)'
artifact: 'azure-deploy.json'
publishLocation: 'pipeline'
"A deployment job doesn't automatically clone the source repo. You can checkout the source repo within your job with checkout: self."
Source: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/deployment-jobs?view=azure-devops
Example from my setup, I put checkout: self as the first step and now my repository is cloned before before executing the Azure PowerShell:
strategy:
runOnce:
deploy:
steps:
- checkout: self
- task: AzurePowerShell#5
displayName: Setup Network
inputs:
It is a good strategy to go for the multi-stage pipelines for what you are doing.
Build is for composing your artifacts.
Deployment jobs is for the publishing part.
So you are on the right track.
If you need sources during the deployment jobs then use the checkout step to fetch the sources. ref. Repo Checkout docs.
Just my two cents
Because you use deployment job the sources from the master branch didn't download into the agent.
You need to publish the files in the build stage and consume them in the deployment - with pipeline artifacts.
Or, just run the AzureResourceGroupDeployment in a regular job, than the .json file will be in the agent.
My objective was to create a Blank and Starter Resource Group in Azure and nothing else. And this was repeatedly giving me the error
Template file pattern matches a directory instead of a file: /home/vsts/work/1/s
And thats what got me here.
I finally sorted that out with Stringfello's
- checkout self
step.
My full pipeline is as follows.
variables:
- name: finalBuildArtifactName
value: 'aspNetCoreDropFromYaml123'
- name: BuildParameters.RestoreBuildProjects
value: '**/*.csproj'
- name: BuildParameters.TestProjects
value: '**/*[Tt]ests/*.csproj'
- name: buildConfiguration
value: 'Release'
trigger:
- master
name: $(date:yyyyMMdd)$(rev:.r)
stages:
- stage: Build ##Basically prints out some vars and copies the template files.
jobs:
- job: buildWebApp
displayName: Build Stage for somepurpose
pool:
vmImage: ubuntu-latest
steps:
- script: |
echo build.artifactstagingdirectory and build.buildnumber are as follows.
echo $(build.artifactstagingdirectory) $(build.buildnumber)
echo $(projects)
echo $(BuildConfiguration)
echo Pipeline.Workspace is $(Pipeline.Workspace)
echo The current branch is - $(Build.SourceBranchName)!!.
echo $(finalBuildArtifactName)
echo "This is the build pipe line. This produces the necessary artifacts for subsequent release pipeline."
displayName: 'Command Line Script to write out some vars'
- powershell: |
# Write your PowerShell commands here.
Write-Host "This is from power shell command task"
Write-Host "This writes out the env vars"
get-childitem -path env:*
displayName: 'PowerShell script to write out env vars'
# The following task is needed. This copies the arm template files.
# Created these templates from Visual studio 2019 as follows.
# Right click your solution and Add -> New Project -> Azure Resource Group and gave the name Vivek_Aks_Rg
- task: CopyFiles#2
inputs:
SourceFolder: 'iac/ArmTemplates/Vivek_Aks_Rg/'
Contents: 'azuredeploy*.json'
TargetFolder: '$(build.artifactStagingDirectory)/ArmTemplates'
- task: PublishBuildArtifacts#1
displayName: Publish Artifact
condition: succeededOrFailed()
inputs:
PathtoPublish: '$(build.artifactstagingdirectory)'
ArtifactName: '$(finalBuildArtifactName)'
PublishLocation: 'Container'
- stage: DeployToDev
displayName: Deploy to Dev Env
jobs:
- deployment:
pool:
vmImage: ubuntu-latest
environment: Dev
strategy:
runOnce:
deploy:
steps:
- checkout: self
- task: AzureResourceManagerTemplateDeployment#3
displayName: 'Create Azure App Service in a Given Resource Group'
inputs:
deploymentScope: 'Subscription'
azureResourceManagerConnection: 'Pay-As-You-Go(123YourSubscriptionId)'
subscriptionId: '123YourSubscriptionId'
action: 'Create Or Update Resource Group'
resourceGroupName: 'YourResourceGroupName'
location: 'Central India'
csmFile: '$(Pipeline.Workspace)/$(finalBuildArtifactName)/ArmTemplates/azuredeploy.json'
csmParametersFile: '$(Pipeline.Workspace)/$(finalBuildArtifactName)/ArmTemplates/azuredeploy.parameters.json'
deploymentMode: 'Incremental'