Trigger another build exist in project in Azure Devops - azure

I have a repo name called A with its build pipeline as azure-pipelines.yml
Then I have another repo called B with its build pipeline as azure-pipelines.yml
Both A and B are under same project ProjectA
This is the flow for
repo A, build => release (stages ops and dev)
repo B, Build create the Artifact and store the Artifact
So, what I want to achieve is as soon as the release finished from repo A it should trigger build repo B.
My pipeline A looks like this :
name: SomethingFancy
trigger:
- none
resources:
containers:
- container: docker
image: docker:1.6
- container: python3
image: python:3
variables:
major: 2
minor: 0
So I made pipeline B looks like this:
name:
trigger:
- none
resources:
pipelines:
- pipeline: SomethingFancy
source: azure-pipelines
branch: DATA-1234
project: ProjectA
trigger:
branches:
- DATA-1234
stages:
- dev
- ops
containers:
- container: docker
image: docker:1.6
So far I'm not able to run the pipeline as it complains "Pipeline Resource SomethingFancy Input Must be Valid." as per the documentation it is something # identifier for the resource (used in pipeline resource variables).
I'm referring to [this][1] for the collection of resources.
I'm also intended to use [api][2] call to queue the build of the B, but not able to find what should be the body of the post message e.g. how to add the branch of pipeline B, or how to pass the parameters to the pipeline of B
EDIT
see attached my pipeline name
[![enter image description here][3]][3]
and build source pipeline also called azurepipelines.yml and release pipeline has one stage called Dev
Now my pipeline B looks like this:
resources:
pipelines:
- pipeline: azurepipelines
source: azurepipelines
branch: DATA-1234
project: ProjectA
trigger:
branches:
- DATA-1234
stages:
- Dev
still I don't see any auto kick off of build pipeline of B.
[1]: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/resources?view=azure-devops&tabs=example#resources-pipelines
[2]: https://learn.microsoft.com/en-us/rest/api/azure/devops/build/builds/queue?view=azure-devops-rest-5.1
[3]: https://i.stack.imgur.com/2Uk7A.png

After a lot of struggle and the help of wise people here I finally manage to tame the issue. I'm posting here so that anyone can take a referebce. This is working now refer:
ListBuild and QueueTheBuild
name="ProjectA"
curl --silent -X GET -H "Authorization:Bearer $(System.AccessToken)" -H "Content-Type:application/json" $(System.TeamFoundationCollectionUri)/$(System.TeamProject)/_apis/build/definitions?api-version=6.0 --output /tmp/response.json
#Now get the build-id of your project you are interested in
#please be aware that api-version > 6 has different json output and below command
#may not help you to give the right id
id=$(cat /tmp/response.json | jq -r --arg key ${name} '.value[] | select(.name==$key)| .id' --raw-output)
#create your body to post
generate_post_data()
{
cat <<EOF
{
"sourceBranch":"refs/heads/DATA-1234",
"definition":{"id": $id}
}
EOF
}
#Now queue your build to run
#have to still verify if this command works for API_VERSION 6
curl -X POST \
--silent \
-H "Authorization:Bearer $(System.AccessToken)" \
-H "Content-Type:application/json" \
$(System.TeamFoundationCollectionUri)$(System.TeamProject)/_apis/build/builds?api-version=6.1-preview.6 \
--output /tmp/response1.json \
-d "$(generate_post_data)"
#check the outcome
cat /tmp/response1.json

As I read from the same documentation I think you should set the source property to the name of your pipeline from repo A. source: SomethingFancy

The error message looks like it's telling you it cannot find a pipeline with the name you specified, probably because, name means refers to the build numbering format in pipeline YAML e.g.
name: $(BuildID)
As #Roderick noted, the name of the pipeline should be what you see in the UI. From the main "Azure Pipelines" screen in your project. First click the "three dots" to get a submenu and then click "Rename/move". Example screenshot:
So now you should have the project name and pipeline names you need to update your YAML in pipeline B and it should work.

I noticed the the flow for repo A is build => release (stages ops and dev). I wonder if the build is the build pipeline as azure-pipelines.yml, and release (stages ops and dev) is the classic release pipeline in azure devops Releases hub? You should know that pipeline resources triggers doesnot work for classic release pipeline.
build => release (stages ops and dev) for repo A should be in the same pipeline (ie. azure-pipelines.yml). So the pipeline resources trigger you defined in pipeline B only works when the pipeline A looks like below:
name: ..
trigger:
- none
resources:
containers:
..
variables:
..
stages:
- stage: build # build the project in build stage
jobs:
- job
..
- stage: ops #stage ops
jobs:
- job:
...
- stage: dev #stage dev
jobs:
- job:
...
The source in Pipeline B is the name of the pipeline A as julie-ng mentioned. See below example:
resources:
pipelines:
- pipeline: {Can be Any String} #identifier for the resource (used in pipeline resource variables)
source: {Name of the pipeline A what you see in the UI} #name of the pipeline that produces an artifact
Name of the pipeline A:
Resource trigger in Pipeline B:
resources:
pipelines:
- pipeline: AnyString
source: pipelineA
branch: DATA-1234
If the release pipeline for repo A is the classic release pipeline. You can add this external task Trigger Build in stage dev to trigger pipeline B in stage dev:
- task: benjhuser.tfs-extensions-build-tasks.trigger-build-task.TriggerBuild#3
displayName: 'Trigger a new build of 48'
inputs:
buildDefinition: {ID of pipeline B}
buildParameters: 'variableName: variableValue'
password: '$(System.AccessToken)'
If you want to pass some variables from Pipeline A to pipeline B. you can use the buildParameters field.
In pipelien B, Click the Variables button to define a Variable to the hold the variable value. (Note: Check this option Let users override this value when running this pipeline, so that it can be overrode from A pipeline )
You can always use the Rest api to trigger the pipeline. Please see below threads for more inforamtion
this thread
send multiple parameter to Azure-Devops pipeline job via Powershell
Can you pass a file to an azure pipeline?
Update:
You can use Builds - Queue rest api To trigger a pipeline.
POST https://dev.azure.com/{organization}/{project}/_apis/build/builds?api-version=6.1-preview.6
See below example:
curl -X POST --silent \
-H "Authorization:Bearer $(System.AccessToken)" \
-H "Content-Type:application/json" \
$(System.TeamFoundationCollectionUri)$(System.TeamProject)/_apis/build/builds?api-version=6.1-preview.6 \
-d '{
"definition":{ "id": id-of-pipelineB},
"sourceBranch":"refs/heads/DATA-1234"
}'

Related

How to check if environment/devployment exists/successfull inside a job and decide to deploy/update it

In the gitlab ce omnibus 14.10.2 instance, there is a project in which it is necessary to make a pipeline in gitlab-ci.yml that would check for the existence of dynamical environment deployments
and if they are not there, then do nothing, and if they are, it would update them or manually deploy by a trigger from the webui.
My example:
#...
check-review:
variables:
GIT_STRATEGY: none
stage: dev/review/check
only:
- /^(feature|hotfix)\/(.*)/m
dependencies: []
allow_failure: true
script:
- curl -sLf ${CI_COMMIT_REF_SLUG}.domain.io --output /dev/null
when: always
ansible/deploy:
stage: dev/review
only:
- /^(feature|hotfix)\/(.*)/m
except:
- master
- staging
environment:
name: $CI_COMMIT_REF_NAME
url: https://${CI_COMMIT_REF_SLUG}.domain.io
action: start
on_stop: ansible/undeploy
when: on_success
dependencies:
- vendor # build back
script:
- make deploy
ansible/undeploy:
stage: dev/review
variables:
GIT_STRATEGY: none
environment:
name: $CI_COMMIT_REF_NAME
url: https://${CI_COMMIT_REF_SLUG}.domain.io
action: stop
when: manual
needs:
- job: ansible/deploy
script:
- make delete_dev_stand
This condition need because dev teams workinig on many branches and not everyone need to be deployed on dev envs, on the other hand with manual update only - on every update/fix code qa need to go gitlab webui & trigger job to deploy/update deployment in environments https://${CI_SERVER_HOST}/${CI_PROJECT_PATH}/-/environments/
Maybe I`ve miss something in gitlab-ci ref, only solution that I found was to make failure pipline on check stage/job but this not happy to see so many failed pipelines.
It's been two months so you probably have a solution already, but here's something you could try if it fits your use case
Approach:
Use pipeline triggers and a script that calls the Gitlab API
1. Find out if the environment exists
a. Create a script called get-deployment-strategy.sh and add it to your project
b. Create a private token in your user preferences with API read permission and add it to your CI/CD variables (GITLAB_API_PRIVATE_READ_TOKEN=token)
#!/bin/bash
# Private Token (created in user preferences) with API READ permission added to CI/CD vars
HEADER="PRIVATE-TOKEN: $GITLAB_API_PRIVATE_READ_TOKEN"
# To get the project which the environment should be read from
API_URL="https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/environments"
# URL query parameter to filter for environment with specific name
QUERY_STRING="name=$DEPLOYMENT_TARGET/$DEPLOYMENT_NAME"
# Call Gitlab API (Environments API)
API_RESPONSE="$(curl -G --header "$HEADER" "$API_URL" --data-urlencode "$QUERY_STRING")"
# The response is an empty array (string) if there is no environment with the specific name
if [ "$API_RESPONSE" = "[]" ]; then
# Deployment environment does not exist
echo "DO_STRATEGY_A"
else
echo "DO_STRATEGY_B"
fi
2. Use trigger pipelines within your check job
a. Create a trigger token in your CI/CD settings and add it to your CI/CD variables
b. Call the trigger pipeline in your check job and pass it a "deployment strategy variable"
Something like this:
check-job:
before_script:
- export DEPLOYMENT_STRATEGY=$(./get-deployment-strategy.sh)
script:
- |
curl \
--fail \
--request POST \
--form token=$DEPLOY_TRIGGER_TOKEN \
--form ref=$CI_COMMIT_BRANCH \
--form variable[DEPLOYMENT_STRATEGY]=$DEPLOYMENT_STRATEGY \
"https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/trigger/pipeline"
3. Make the deploy jobs activate only on specific DEPLOYMENT_STRATEGY
# Only runs for DO_STRATEGY_A (environment does not exist)
deploy/strategy-A:
rules:
- if: '$CI_PIPELINE_SOURCE="trigger" && $DEPLOYMENT_STRATEGY=="DO_STRATEGY_A"'
# Only runs for DO_STRATEGY_B (environment exists)
deploy/strategy-B:
rules:
- if: '$CI_PIPELINE_SOURCE="trigger" && $DEPLOYMENT_STRATEGY=="DO_STRATEGY_B"'

fail gitlab multijob pipeline if test job fails in another repository

I have a gitlab ci pipeline in my application repo, A, which calls an end to end testing Repo T to run its tests. The repo A pipeline succesfully triggers the tests from Repo T but if the test job fails in T, the job calling the test job in T from A still passes. How do I get repo A to track the result of Repo T's test job, and pass/fail its pipeline based off of the test jobs in T?
.gitlab-ci.yml for testing Repo T:
stages:
- test
test:
stage: test
image: markhobson/maven-chrome:jdk-11
artifacts:
paths:
- target/surefire-reports
script:
- mvn test
only:
- triggers
.gitlab-ci.yml from application repo A:
job1:
stage: unit-tests ...
job2:
stage: build ...
...
trigger-e2e-repo:
stage: e2e-testing
image: markhobson/maven-chrome
script:
- "curl -X POST -F token=repo-T-token -F ref=repo-T-branch https://repo-A/api/v4/projects/repo-T-id/trigger/pipeline"
only:
- repo-A-branch
Since GitLab 11.8 you can trigger a pipeline via bridge job.
In GitLab 11.8, GitLab provides a new CI/CD configuration syntax to make this task easier, and avoid needing GitLab Runner for triggering cross-project pipelines.
With bridge jobs it is possible to mirror the status of the trigger pipeline to the calling pipeline.
You can mirror the pipeline status from the triggered pipeline to the source bridge job by using strategy: depend.
Example in your case:
trigger-e2e-repo:
stage: e2e-testing
trigger:
project: repo-T
strategy: depend
If the triggered pipeline with the test jobs fails, the calling pipeline also fails.
If you only want to execute a particular job in your repository "Repo T" when executed by a bridge job, then you should use only: pipeline (only) or rules: -if '$CI_PIPELINE_SOURCE == "pipeline"' (rules:if) instead of only: triggers.
I wasn't able to use the bridge job property of mirroring a downstream job result as the version of my gitlab is before 11.8. I did manage to get it to work by creating a trigger for repo A, and making a call from repo T to repo A with this new second trigger. The remaining jobs in repo A will only be activated by triggers and setting of variables (JOB in this case ) as laid out below:
.gitlab-ci.yml for repo T:
test:
stage: test
script:
- mvn test
- "curl -X POST -F token=repo-A-token -F ref=$BRANCH -F variables[JOB]=build https://project.com/api/v4/projects/project_id/trigger/pipeline"
.gitlab-ci.yml in A
job1:
...
except:
- triggers
job2:
...
except:
- triggers
...
trigger-e2e-repo:
stage: e2e-testing
script:
- "curl -X POST -F token=repo-B-token -F ref=repo-B-branch -F variables[BRANCH]=$CI_COMMIT_REF_NAME -F https://project-B/api/v4/projects/project-B-id/trigger/pipeline"
except:
- triggers
build_application_for_prod:
stage: build_prod
script:
- "curl -X POST -F token=repo-A-token -F ref=$CI_COMMIT_REF_NAME -F variables[JOB]=deploy -F variables[SEND]=true https://foo/api/v4/projects/proj_A_id/trigger/pipeline"
only:
variables:
- $JOB == "build"
deploy_production_environment:
stage: deploy_prod
script:
- ...
only:
variables:
- $JOB == "deploy"
Note I also had to add the except statements for the jobs before the end to end tests in repo A so that they won't rerun and loop when repo A's API trigger is called later on.

pr not triggered when opening github PR (Azure pipeline YAML)

The goal
I'm pretty new to Azure and pipelines, and I'm trying to trigger a pipeline from a pr in Azure. The repo lives in Github.
Here is the pipeline yaml: pipeline.yml
trigger: none # I turned this off for to stop push triggers (but they work fine)
pr:
branches:
include:
- "*" # This does not trigger the pipeline
stages:
- stage: EchoTriggerStage
displayName: Echoing trigger reason
jobs:
- job: A
steps:
- script: echo 'Build reason::::' $(Build.Reason)
displayName: The build reason
# ... some more stages here triggered by PullRequests....
# ... some more stages here triggered by push (CI)....
The pr on Github looks like this:
The problem
However, the pipeline is not triggered, when the push triggers work just fine.
I have read in the docs but I can't see why this does not work.
The pipeline is working perfectly fine when I am triggering it through git push. However, when I try to trigger it with PR's from Github, nothing happens. In the code above, I tried turning off push triggers, and allow for all pr's to trigger the pipeline. Still nothing.
I do not want to delete the pipeline yet and create a new one.
Update
I updated the yaml file to work as suggested underneath. Since the pipeline actually runs through a push command now, the rest of the details of the yaml file are not relevant and are left out.
Other things I have tried
Opening new PR on Github
Closing/Reopening PR on Github
Making change to existing PR on Github
-> Still no triggering of pipeline.
You have a mistake in your pipeline. It should be like this:
trigger: none # turned off for push
pr:
- feature/automated-testing
steps:
- script: echo "PIPELINE IS TRIGGERED FROM PR"
Please change this
- stage:
- script: echo "PIPELINE IS TRIGGERED FROM PR"
to
- stage:
jobs:
- job:
steps:
- script: echo "PIPELINE IS TRIGGERED FROM PR"
EDIT
I used your pipeline
trigger: none # I turned this off for to stop push triggers (but they work fine)
pr:
branches:
include:
- "*" # This does not trigger the pipeline
stages:
- stage: EchoTriggerStage
displayName: Echoing trigger reason
jobs:
- job: A
steps:
- script: echo 'Build reason::::' $(Build.Reason)
displayName: The build reason
# ... some more stages here triggered by PullRequests....
# ... some more stages here triggered by push (CI)....
and all seems to be working.
Here is PR and here build for it
I didn't do that but you can try to enforce this via branch policy. TO do that please go to repo settings and then as follow:
The solution was to go to Azure Pipelines -> Edit pipeline -> Triggers -> Enable PR validation.
You can follow below steps to troubleshooting your pipeline.
1, First you need to make sure a pipeline was created from the yaml file on azure devops Portal. See example in this tutorial.
2, Below part of your yaml file is incorrect. - script task should be under steps section.
Change:
stages:
- stage:
- script: echo "PIPELINE IS TRIGGERED FROM PR"
To:
stages:
- stage:
jobs:
- job:
step:
- script: echo "PIPELINE IS TRIGGERED FROM PR"
3, I saw you used template in your yaml file. Please make sure the template yaml files are in correct format. For example:
the dockerbuild-dashboard-client.yml template of yours is a step template. You need to make sure its contents is like below:
parameters:
...
steps:
- script: echo "task 1"
- script: echo "task 2"
And webapprelease-dashboard-dev-client.yml of yours is a job template. Its contents should be like below:
parameters:
name: ''
pool: ''
sign: false
jobs:
- job: ${{ parameters.name }}
pool: ${{ parameters.pool }}
steps:
- script: npm install
- script: npm test
- ${{ if eq(parameters.sign, 'true') }}:
- script: sign
4, After the pipeline was created on azure devops Portal. You can manually run this pipeline to make sure there is no error in the yaml file and the pipeline can be successfully executed.
5, If All above are checked, but the PR trigger still is not working. You can try deleting the pipeline(created on the first step) created on Azure devops portal and recreated a new pipeline from the yaml file.

How to trigger only specific stage of pipeline with gitlab API?

I have gitlab project with ci file:
stages:
- build
- run
build:
stage: build
script:
- (some stuff)
tags:
- my_runner_tag
except:
- triggers
when: manual
run:
stage: run
script:
- (some stuff)
tags:
- my_runner_tag
except:
- triggers
when: manual
Jobs are created on every source code change, and they can be run only manually, using gitlab interface.
Now, i want to have possibility to trigger stage run with Gitlab API. Trying:
curl -X POST \
> -F token=xxxxxxxxxxxxxxx \
> -F ref=master \
> https://gitlab.xxxxx.com/api/v4/projects/5/trigger/pipeline
Returns:
{"message":{"base":["No stages / jobs for this pipeline."]}}
Seems, like i have to define stage to trigger, but i can't find a way to pass it via api call.
you are using the wrong endpoint, to do it, you need to follow the path below
list all of your pipelines and get the newest one
GET /projects/:id/pipelines
list the jobs from this pipeline
GET /projects/:id/pipelines/:pipeline_id/jobs
After that you can trigger your job
POST /projects/:id/jobs/:job_id/play
you are telling your build to run at all times except for the time they are being triggered (api call is also considered as a trigger).
change your job definition to the following:
run:
stage: run
script:
- (some stuff)
tags:
- my_runner_tag
when: manual

GitLab: Job artifacts in multi project pipelines

I have been trying to learn multi project pipelines for a while now, and apart from GitLab documentation, I have not found any study material. If I could see an example, it would really help. I have been using the following ci config for a multi project pipeline in project A, but it's not working:
trigger_job:
stage: trigger_release
trigger:
project: https://<gitlab-site>/api/v4/projects/<project-B-id>/trigger/pipeline
branch: master
strategy: depend
This leaves the pipeline in project A in a pending state forever. I used curl in the following way to finally get the config working:
trigger_job:
stage: trigger_release
script:
- curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=master https://<gitlab-site>/api/v4/projects/<project-B-id>/trigger/pipeline
However, what I really need is to collect and use the artifacts of project B pipeline in project A pipeline after the triggered job finishes. How do I do that?
Since Gitlab 11.8. you don't need to use API to trigger a pipeline, see official documentation
Example
Let's have group mygroup with 2 repos: myrepository1 and myrepository2.
Config in repository1
trigger-job:
trigger:
project: mygroup/myrepository2
branch: master
variables:
VARIABLE_TO_PASS: $CI_COMMIT_REF_NAME
Config in repository2
job-waiting-for-trigger:
stage: deploy
variables:
script:
- echo "${VARIABLE_TO_PASS} from another project pipeline"
only:
- pipelines
I haven't tried this code, but it should be correct.

Resources