I have been trying to learn multi project pipelines for a while now, and apart from GitLab documentation, I have not found any study material. If I could see an example, it would really help. I have been using the following ci config for a multi project pipeline in project A, but it's not working:
trigger_job:
stage: trigger_release
trigger:
project: https://<gitlab-site>/api/v4/projects/<project-B-id>/trigger/pipeline
branch: master
strategy: depend
This leaves the pipeline in project A in a pending state forever. I used curl in the following way to finally get the config working:
trigger_job:
stage: trigger_release
script:
- curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=master https://<gitlab-site>/api/v4/projects/<project-B-id>/trigger/pipeline
However, what I really need is to collect and use the artifacts of project B pipeline in project A pipeline after the triggered job finishes. How do I do that?
Since Gitlab 11.8. you don't need to use API to trigger a pipeline, see official documentation
Example
Let's have group mygroup with 2 repos: myrepository1 and myrepository2.
Config in repository1
trigger-job:
trigger:
project: mygroup/myrepository2
branch: master
variables:
VARIABLE_TO_PASS: $CI_COMMIT_REF_NAME
Config in repository2
job-waiting-for-trigger:
stage: deploy
variables:
script:
- echo "${VARIABLE_TO_PASS} from another project pipeline"
only:
- pipelines
I haven't tried this code, but it should be correct.
Related
I would like to trigger a GitLab project, but instead of using a branch, I would like to use a specific tag. How can this be achieved?
The following triggers the develop branch of some-project:
bridge:
stage: bridge
trigger:
project: some-project
branch: develop
You can use the parent-child pipeline feature with the ref keyword, like this:
bridge:
stage: bridge
trigger:
include:
- project: some-project
ref: 'your-tag'
file: '.gitlab-ci.yml'
You can trigger the pipeline using the API.
Example using a curl command, which you can add to your job script:
curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=my-tag \
https://gitlab.example.com/api/v4/projects/6/trigger/pipeline
When you use the CI_JOB_TOKEN to trigger pipelines, GitLab recognises the source of the job token. The pipelines become related, so you can visualise their relationships on pipeline graphs.
I have a repo name called A with its build pipeline as azure-pipelines.yml
Then I have another repo called B with its build pipeline as azure-pipelines.yml
Both A and B are under same project ProjectA
This is the flow for
repo A, build => release (stages ops and dev)
repo B, Build create the Artifact and store the Artifact
So, what I want to achieve is as soon as the release finished from repo A it should trigger build repo B.
My pipeline A looks like this :
name: SomethingFancy
trigger:
- none
resources:
containers:
- container: docker
image: docker:1.6
- container: python3
image: python:3
variables:
major: 2
minor: 0
So I made pipeline B looks like this:
name:
trigger:
- none
resources:
pipelines:
- pipeline: SomethingFancy
source: azure-pipelines
branch: DATA-1234
project: ProjectA
trigger:
branches:
- DATA-1234
stages:
- dev
- ops
containers:
- container: docker
image: docker:1.6
So far I'm not able to run the pipeline as it complains "Pipeline Resource SomethingFancy Input Must be Valid." as per the documentation it is something # identifier for the resource (used in pipeline resource variables).
I'm referring to [this][1] for the collection of resources.
I'm also intended to use [api][2] call to queue the build of the B, but not able to find what should be the body of the post message e.g. how to add the branch of pipeline B, or how to pass the parameters to the pipeline of B
EDIT
see attached my pipeline name
[![enter image description here][3]][3]
and build source pipeline also called azurepipelines.yml and release pipeline has one stage called Dev
Now my pipeline B looks like this:
resources:
pipelines:
- pipeline: azurepipelines
source: azurepipelines
branch: DATA-1234
project: ProjectA
trigger:
branches:
- DATA-1234
stages:
- Dev
still I don't see any auto kick off of build pipeline of B.
[1]: https://learn.microsoft.com/en-us/azure/devops/pipelines/process/resources?view=azure-devops&tabs=example#resources-pipelines
[2]: https://learn.microsoft.com/en-us/rest/api/azure/devops/build/builds/queue?view=azure-devops-rest-5.1
[3]: https://i.stack.imgur.com/2Uk7A.png
After a lot of struggle and the help of wise people here I finally manage to tame the issue. I'm posting here so that anyone can take a referebce. This is working now refer:
ListBuild and QueueTheBuild
name="ProjectA"
curl --silent -X GET -H "Authorization:Bearer $(System.AccessToken)" -H "Content-Type:application/json" $(System.TeamFoundationCollectionUri)/$(System.TeamProject)/_apis/build/definitions?api-version=6.0 --output /tmp/response.json
#Now get the build-id of your project you are interested in
#please be aware that api-version > 6 has different json output and below command
#may not help you to give the right id
id=$(cat /tmp/response.json | jq -r --arg key ${name} '.value[] | select(.name==$key)| .id' --raw-output)
#create your body to post
generate_post_data()
{
cat <<EOF
{
"sourceBranch":"refs/heads/DATA-1234",
"definition":{"id": $id}
}
EOF
}
#Now queue your build to run
#have to still verify if this command works for API_VERSION 6
curl -X POST \
--silent \
-H "Authorization:Bearer $(System.AccessToken)" \
-H "Content-Type:application/json" \
$(System.TeamFoundationCollectionUri)$(System.TeamProject)/_apis/build/builds?api-version=6.1-preview.6 \
--output /tmp/response1.json \
-d "$(generate_post_data)"
#check the outcome
cat /tmp/response1.json
As I read from the same documentation I think you should set the source property to the name of your pipeline from repo A. source: SomethingFancy
The error message looks like it's telling you it cannot find a pipeline with the name you specified, probably because, name means refers to the build numbering format in pipeline YAML e.g.
name: $(BuildID)
As #Roderick noted, the name of the pipeline should be what you see in the UI. From the main "Azure Pipelines" screen in your project. First click the "three dots" to get a submenu and then click "Rename/move". Example screenshot:
So now you should have the project name and pipeline names you need to update your YAML in pipeline B and it should work.
I noticed the the flow for repo A is build => release (stages ops and dev). I wonder if the build is the build pipeline as azure-pipelines.yml, and release (stages ops and dev) is the classic release pipeline in azure devops Releases hub? You should know that pipeline resources triggers doesnot work for classic release pipeline.
build => release (stages ops and dev) for repo A should be in the same pipeline (ie. azure-pipelines.yml). So the pipeline resources trigger you defined in pipeline B only works when the pipeline A looks like below:
name: ..
trigger:
- none
resources:
containers:
..
variables:
..
stages:
- stage: build # build the project in build stage
jobs:
- job
..
- stage: ops #stage ops
jobs:
- job:
...
- stage: dev #stage dev
jobs:
- job:
...
The source in Pipeline B is the name of the pipeline A as julie-ng mentioned. See below example:
resources:
pipelines:
- pipeline: {Can be Any String} #identifier for the resource (used in pipeline resource variables)
source: {Name of the pipeline A what you see in the UI} #name of the pipeline that produces an artifact
Name of the pipeline A:
Resource trigger in Pipeline B:
resources:
pipelines:
- pipeline: AnyString
source: pipelineA
branch: DATA-1234
If the release pipeline for repo A is the classic release pipeline. You can add this external task Trigger Build in stage dev to trigger pipeline B in stage dev:
- task: benjhuser.tfs-extensions-build-tasks.trigger-build-task.TriggerBuild#3
displayName: 'Trigger a new build of 48'
inputs:
buildDefinition: {ID of pipeline B}
buildParameters: 'variableName: variableValue'
password: '$(System.AccessToken)'
If you want to pass some variables from Pipeline A to pipeline B. you can use the buildParameters field.
In pipelien B, Click the Variables button to define a Variable to the hold the variable value. (Note: Check this option Let users override this value when running this pipeline, so that it can be overrode from A pipeline )
You can always use the Rest api to trigger the pipeline. Please see below threads for more inforamtion
this thread
send multiple parameter to Azure-Devops pipeline job via Powershell
Can you pass a file to an azure pipeline?
Update:
You can use Builds - Queue rest api To trigger a pipeline.
POST https://dev.azure.com/{organization}/{project}/_apis/build/builds?api-version=6.1-preview.6
See below example:
curl -X POST --silent \
-H "Authorization:Bearer $(System.AccessToken)" \
-H "Content-Type:application/json" \
$(System.TeamFoundationCollectionUri)$(System.TeamProject)/_apis/build/builds?api-version=6.1-preview.6 \
-d '{
"definition":{ "id": id-of-pipelineB},
"sourceBranch":"refs/heads/DATA-1234"
}'
Trying to trigger a pipeline to run test in project B (downstream) from project A. Wanted to know if it is possible to it run on a particular tag in project B.
(Currently, it always gets triggered on master which is the default branch.)
Invoke this command ( from https://docs.gitlab.com/ee/ci/triggers/ )
curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=master https://gitlab.example.com/api/v4/projects/9/trigger/pipeline
change ref=master to ref=your_tag, and 9 to project B id.
to get latest tag ( from https://stackoverflow.com/a/1805463/746618)
git describe $(git rev-list --tags --max-count=1)
If you have two projects with same tag you can do it like that:
Project A:
tag-job:
stage: build
trigger:
project: project-b-name
branch: $CI_COMMIT_TAG
strategy: depend
only:
- tags
Project B:
downstream-job:
stage: build
rules:
- if: '$CI_PIPELINE_SOURCE == "pipeline"'
script:
- run tests
You can specify any other particular branch in "branch: " variable of tag-job.
I have gitlab project with ci file:
stages:
- build
- run
build:
stage: build
script:
- (some stuff)
tags:
- my_runner_tag
except:
- triggers
when: manual
run:
stage: run
script:
- (some stuff)
tags:
- my_runner_tag
except:
- triggers
when: manual
Jobs are created on every source code change, and they can be run only manually, using gitlab interface.
Now, i want to have possibility to trigger stage run with Gitlab API. Trying:
curl -X POST \
> -F token=xxxxxxxxxxxxxxx \
> -F ref=master \
> https://gitlab.xxxxx.com/api/v4/projects/5/trigger/pipeline
Returns:
{"message":{"base":["No stages / jobs for this pipeline."]}}
Seems, like i have to define stage to trigger, but i can't find a way to pass it via api call.
you are using the wrong endpoint, to do it, you need to follow the path below
list all of your pipelines and get the newest one
GET /projects/:id/pipelines
list the jobs from this pipeline
GET /projects/:id/pipelines/:pipeline_id/jobs
After that you can trigger your job
POST /projects/:id/jobs/:job_id/play
you are telling your build to run at all times except for the time they are being triggered (api call is also considered as a trigger).
change your job definition to the following:
run:
stage: run
script:
- (some stuff)
tags:
- my_runner_tag
when: manual
I'm trying to run 2 pipelines for a project in GitLab, but I can't find any way to do it.
In gitlab CI you can't create multiple pipelines for one project explicitly. There are cases where multiple pipelines will run simultaneously, such as when you have jobs that run only for merge requests and other jobs that do not run on merge requests.
That said, there are ways to obtain the effect of running multiple series of jobs independently from one another.
The hacky way, before gitlab-ce 12.2
If you want to start 2 pipelines for the same project you can use pipeline triggers. This method is limited to 2 pipelines and gitlab CI is not meant to be used this way. Usually, triggers are used to start pipelines on another project.
All in your .gitlab-ci.yml:
stages:
- start
- build
###################################################
# First pipeline #
###################################################
start_other_pipeline:
stage: start
script:
# Trigger a pipeline for current project on current branch
- curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=$CI_COMMIT_REF_NAME $CI_API_V4_URL/projects/$CI_PROJECT_ID/trigger/pipeline
except:
- pipelines
build_first_pipeline:
stage: build
script:
- echo "Building first pipeline"
except:
- pipelines
###################################################
# Second pipeline #
###################################################
# Will run independently of the first pipeline.
build_second_pipeline:
stage: build
script:
- echo "Building second pipeline"
only:
- pipelines
To clean up this mess of a .gitlab-ci.yml, you can use the include keyword:
# .gitlab-ci.yml
include:
- '/first-pipeline.yml'
- '/second-pipeline.yml'
stages:
- start
- build
# This starts the second pipeline. The first pipeline is already running.
start_other_pipeline:
stage: start
script:
# Trigger a pipeline for current project on current branch
- curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=$CI_COMMIT_REF_NAME $CI_API_V4_URL/projects/$CI_PROJECT_ID/trigger/pipeline
except:
- pipelines
# first-pipeline.yml
build_first_pipeline:
stage: build
script:
- echo "Building first pipeline"
except:
- pipelines
# second-pipeline.yml
build_second_pipeline:
stage: build
script:
- echo "Building second pipeline"
only:
- pipelines
The reason this works is the use only and except in the jobs. The jobs marked with
except:
- pipelines
do not run when the pipeline has started because of a trigger coming from another pipeline, so they don't run in the second pipeline. On the other hand,
only:
- pipelines
does the exact opposite, therefore those jobs run only when the pipeline is triggered by another pipeline, so they only run in the second pipeline.
The probably right way, depending on your needs ;)
In gitlab CE 12.2, it is possible to define Directed Acyclic Graphs to specify the order that your jobs run. This way, a job can start as soon as the job it depends on (using needs) finishes.
As of GitLab 12.7 it is also possible to use parent-child pipelines for this:
# .gitlab-ci.yml
trigger_child:
trigger:
include: child.yml
do_some_stuff:
script: echo "doing some stuff"
# child.yml
do_some_more_stuff:
script: echo "doing even more stuff"
The trigger_child job completes successfully once the child pipeline has been created.