I'm setting up several pipelines in Azure DevOps. To make my teams life easier, I'm using job templates.
These job templates are in a a proper repository, just for them.
For every pipeline I define the repository to get the templates from.
Some tasks in these templates run powershell code, and I want this code to be in a script file, to be reusable and stored in the same repo as the template.
When the pipelines runs, the template is embeded, it tries to locate the powershell script inside project repo actually being built/deployed.
How can i achieve this?
The workaround is to have inline code which I really don't want to have.
Any constructive answer will be very appreciated.
Thanks
After some digging I couldn't find any way to specify a script file as source to powershell task in a template.
Inside pipeline definition:
resources:
repositories:
- repository: templates
type: git
name: deploy-templates
variables:
artifactName: 'Trade Data ETL - $(Build.SourceBranchName)'
stages:
- stage: Build
displayName: Build
variables:
- group: DEV-Credential-Group
- group: COMMON-Settings-Group
jobs:
- template: ssis/pipelines/stage-build.yml#templates # Template reference
parameters:
artifactName: '$(artifactName)'
Inside template file:
- task: PowerShell#2
inputs:
filePath: ssis/pipelines/scripts/build-ssis-project.ps1
arguments: '-ProjectToBuild "tradedata-ldz-ssis/tradedata-ldz-ssis.dtproj'
pwsh: true
Update 2021
According to learn.microsoft.com, you can now also check out multiple repositories without custom scripting.
If you check out more than one repository, a separate folder containing the repository is created below $(Build.SourcesDirectory).
You can define multiple repositories like this:
resources:
repositories:
- repository: devops
type: git
name: DevOps
ref: main
- repository: infrastructure
type: git
name: Infrastructure
ref: main
And in the steps simply check them out as follows:
steps:
- checkout: self
- checkout: devops
- checkout: infrastructure
# List all available repositories
- script: ls
Original Answer
Currently the resources command only supports yml files in other repositories. However, you could simply checkout the repository in a task and then run the desired powershell script.
steps:
- task: PowerShell#2
inputs:
targetType: inline
script: |
git clone -b <your-desired-branch> https://azuredevops:$($env:token)#dev.azure.com/<your-organization>/<your-project>/_git/<your-repository> <target-folder-name>
./<target-folder-name>/foo.ps1
env:
token: $(System.AccessToken)
This script would checkout an arbitrary branch and execute a script foo.ps1 in the root of the target repository.
Call - checkout: templates inside the template file. This might only work when you insert a template but it successfully sees the repository resource and pulls it down.
You can copy the script files from source directory. Currently, you have not mentioned the root folder -
ssis/pipelines/scripts/build-ssis-project.ps1
Assuming, you are building on a repo where the powershell script resides -
Try -
- task: PowerShell#1
inputs:
scriptName: '$(ScriptsDir)/ssis/pipelines/scripts/build-ssis-project.ps1'
Pass the value of ScriptsDir where it could be the build source directory or build working directory
Related
How to authorize variable in a yaml template in another repo to be used in a different repo. IOW, how to declare variables in a template once and use in multiple repos in azure devops
I am trying to migrate from classic pipelines to yaml in azure devops. So i am trying to set up a repo to host all yaml templates so it can be referenced and reused by multiple repos for builds, etc.
I wrote this yaml pipeline to sample prototyping it:
`name: FirstPL
trigger:
- my_test_branch
pool: my-agent
resources:
repositories:
- repository: blah
type: git
name: foo/bar
ref: refs/heads/poc
variables:
- template: pipeline_vars.yml#blah
steps:
- script: echo $(variable_from_pipeline_vars)
`
However when i run this i get the follwoing error:
An error occurred while loading the YAML build pipeline. Variable group was not found or is not authorized for use. For authorization details, refer to https://aka.ms/yamlauthz.
How can i declare my variables and variables groups once in a template in a repo that is dedicated to host those templates and then use them over and again in multiple repos using the resourcs syntax above? Also, I tried to find a way authorize the variables template but couldn't find anything to enable this.
How to authorize variable in a yaml template in another repo to be
used in a different repo. IOW, how to declare variables in a template
once and use in multiple repos in azure devops. However when I run
this i get the follwoing error:
An error occurred while loading the YAML build pipeline. Variable group was not found or is not authorized for use. For authorization
details, refer to
https://aka.ms/yamlauthz.
You can directly add the variable group in your azure DevOps project in the Library tab and save all your variables from pipeline_vars.yml in the variable group like below:-
Now, You can access this variable group in your YAML pipeline of multiple repos like the below:-
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
pool:
vmImage: ubuntu-latest
workspace:
clean: all
resources:
repositories:
- repository: repo_a
type: git
name: InternalProjects/repo_a
trigger:
- main
- release
- repository: repo_b
type: git
name: InternalProjects/repo_b
trigger:
- main
variables:
- group: SharedVariables
steps:
- checkout: repo_a
- checkout: repo_b
- script: |
echo $(databaseServerName)
- task: AzureCLI#2
inputs:
azureSubscription: 'xxx subscription(xxxxxxxxx-f598-44d6-b4fd-xxxxxxxxxxxx)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: 'az resource list --location uksouth'
Output:-
It asks for approving permission for the Variable group to run in the pipeline like below:-
Console:-
Tried the same with another repo repo_b in the project and it asks to approve access for repositories and variable groups like the below:-
Output:-
If you want this variable to be accessed in multiple stages/repos/pipelines within the project without authorization prompt. You can click on Security on top and allow it:-
I created one variables template and referenced it in the YAML pipeline to use across multiple repos by checking out another repo like below:-
# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml
pool:
vmImage: ubuntu-latest
workspace:
clean: all
resources:
repositories:
- repository: repo_a
type: git
name: InternalProjects/repo_a
trigger:
- main
- release
- repository: repo_b
type: git
name: InternalProjects/repo_b
trigger:
- main
variables:
- template: pipeline_vars.yml
steps:
- checkout: repo_a
- checkout: repo_b
- script: |
echo $(environmentName)
- task: AzureCLI#2
inputs:
azureSubscription: 'xxx subscription(xxxxxxxx-f598-44d6-b4fd-e2b6e97xxxxxx)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: 'az resource list --location uksouth'
Output:-
I tried to reference the same template in another repo where it does not exist it could not read the pipeline_vars.yml file as it does not exist in the repo.
You can make use of variable groups like the above to reference the variables in this pipeline.
One of the possible reasons for this is that the project that hosts the repository with the variables does not allow access to it's repositories from yaml pipelines.
To verify, go to your project's settings -> Pipelines -> Settings -> Verify "Protect access to repositories in YAML pipelines" . This setting is enabled by default. You could set it to off or add a checkout step to your pipeline yaml. See here for more information.
We are migrating from Azure DevOps to GitHub and we have Build Validations set up where if you make a change in a specific folder, the respective CI pipeline will run when a PR is created.
I am trying to make use of the PR triggers in my YAML file, however when I open a PR it doesn't seem to work.
My pipeline is:
trigger: none
pr:
branches:
include:
- develop
- release/*
- ProductionSupport/*
paths:
include:
- cicd/pipelines/common/pre-commit-ci.yaml
- src
- cicd
pool:
vmImage: ubuntu-latest
variables:
PRE_COMMIT_HOME: $(Pipeline.Workspace)/pre-commit-cache
steps:
- bash: echo "##vso[task.setvariable variable=PY]`python -V`"
displayName: Get python version
- task: Cache#2
inputs:
key: pre-commit | .pre-commit-config.yaml | "$(PY)"
path: $(PRE_COMMIT_HOME)
- bash: |
pip install --quiet pre-commit
pre-commit run
displayName: 'Run pre-commit'
As a test to make sure my branches/paths were correct I updated the triggers section to:
trigger:
branches:
include:
- develop
- release/*
- ProductionSupport/*
paths:
include:
- cicd/pipelines/common/pre-commit-ci.yaml
- src
- cicd
Then when I made a change in one of the files in these folders, the pipeline was successfully triggered. Am I specifying my PR validation incorrectly?
Your yml definition seems correct.
Since you mentioned the CI trigger work fine and you mentioned We are migrating from Azure DevOps to GitHub.
This brings me a idea that a situation that exactly reproduces what you're experiencing and you might not expect:
PR Trigger Override
For example, if your pipeline is the same one as before(Just change the pipeline source), and you didn't delete the previous build validation(Or previous pipeline name is same as the current one), then the pr part in your github yml file will be override, only the build validation on DevOps side will work.
I suggest you investigate whether you have some build validation settings to the pipeline(If your project structure is complex, this maybe difficult to find) or you can simply create a totally new pipeline with the new YAML file.
My azure repo has multiple projects in it (it is not a single solution with multiple projects), which are placed in subfolders. How to publish artifacts separately for each of the project, which are placed in different folders.
By using paths include statements you can specify a build file and a trigger for each of the subdirectories.
An example below
trigger:
branches:
include:
- refs/heads/Development
paths:
include:
- slapi
- slapi.Common
batch: True
name: $(date:yyyyMMdd)$(rev:.r)
This allows each subdirectory to trigger and build to a separate output directory as needed.
In Azure Devops you then import this build YAML file to create your build pipeline and then when you create your release pipeline you select this build pipeline as the source for the drop files
Obviously you need to complete the above example file with the actual build configuration but since the question was about using subdirectories I included the relevant parts.
For your working directories use the variables section like this to specify the project at hand
variables:
vmImageName: "windows-2022"
workingDirectory: "projectName" <- change this to your real project name
then refer to the variables in your build task like this
- task: DotNetCoreCLI#2
displayName: Build
inputs:
command: "build"
projects: |
$(workingDirectory)/*.csproj
arguments: --output $(System.DefaultWorkingDirectory)/publish_output --configuration Release
My code has multiple checkout's and when I run the pipeline, the directory path is not recognized for one of the checkout.
jobs:
- job: job1
steps:
- checkout: rep1
path: test1
- checkout: rep2
path: test2
- task: AzurePowerShell#5
inputs:
azureSubscription:xxxxx
ScriptType: 'FilePath'
ScriptPath: '$(System.DefaultWorkingDirectory)/folder1/myPowershell.ps1'
azurePowerShellVersion: 'LatestVersion'
I see a warning which says "Module path not present as expected in hosted agent, skipping step to make module available". The Powershell script is never executed. Any help, please?
According to the official docs:
If you have multiple checkout steps in your job, your source code is checked out into directories named after the repositories as a subfolder of s in (Agent.BuildDirectory). If (Agent.BuildDirectory) is C:\agent\_work\1 and your repositories are named tools and code, your code is checked out to C:\agent\_work\1\s\tools and C:\agent\_work\1\s\code.
So, it looks like the ScriptPath property of AzurePowerShell#5 task in your pipeline should be built keeping this in mind. If the PowerShell file resides in folder1 directory of the repository rep1, then the path should look like
$(Agent.BuildDirectory)/rep1/folder1/myPowershell.ps1.
I have yaml file for the azure pipline in a repo. And I need to run powershell script from a different repo.
As far as I understood I can add side repo to resources section in yaml and then use task:ShellScript#2 with scriptPath parameter. But as I understood it works relatively for repo in which yaml is placed. And I'm not sure how can I access file from a different repo.
Yes, you have to use repository resource and checkout that repo as follows:
resources:
repositories:
- repository: devops
type: github
name: kmadof/devops-templates
endpoint: kmadof
steps:
- checkout: self
- checkout: devops
- task: ShellScript#2
inputs:
scriptPath: $(Agent.BuildDirectory)/devops/scripts/some-script.sh