I have a problem with gitlab ci child pipelines.
Need to trigger ci pipeline automatically after each commit in repo that have more than one app. Need to configure to detect which folder/files were modified in order to know which app pipeline to trigger
Example of structure
Main/
---- applicationsA/
-------- appA1/
-------- appA2/
-------- appA3/
---- applicationsB/
-------- appB1/
-------- appB2/
-------- appB3/
Main ".gitlab-ci.yml" is:
workflow:
rules:
- if: ‘$CI_PIPELINE_SOURE == “web”’
variables:
APPNAME: $APPNAME
stages:
- child-pipelines
appA1:
stage: child-pipelines
trigger:
include:
- local: applicationA/appA1/gitlab-ci.yml
strategy: depend
rules:
- if: $APPNAME == “appA1” && $CI_PIPELINE_SOURE == “web”
appA2:
stage: child-pipelines
trigger:
include:
- local: applicationA/appA2/gitlab-ci.yml
strategy: depend
rules:
- if: $APPNAME == “appA1” && $CI_PIPELINE_SOURE == “web”
...
appA1 ".gitlab-ci.yml" is:
stages:
- build
- test
build-appA1:
stage: build
script:
- echo "Execute appA1 build!"
publish-appA1:
stage: build
script:
- echo "Execute appA1 publish!"
appA2 ".gitlab-ci.yml" is:
stages:
- build
- test
build-appA2:
stage: build
script:
- echo "Execute appA1 build!"
publish-appA2:
stage: build
script:
- echo "Execute appA1 publish!"
The purpose of this configuration is that , for example, when i change a file inside app**, the pipeline detects the changes and build the app**.
You can use rules:changes with a glob pattern and only run a certain job if anything changes in the specific app folder:
appA1:
stage: child-pipelines
trigger:
include:
- local: applicationA/appA1/gitlab-ci.yml
strategy: depend
rules:
- if: '$APPNAME == "appA1" && $CI_PIPELINE_SOURE == "web"'
changes:
- Main/applicationsA/appA1/**/*
Related
I have a monorepo where each package should be built as a docker image.
I created a trigger job for each package that triggers a child pipeline.
In the MR, my changes rule is being ignored and all child pipelines are triggered.
.gitlab-ci.yml
---
workflow:
rules:
- if: $CI_MERGE_REQUEST_ID || $CI_COMMIT_BRANCH
trigger-package-a:
stage: build
trigger:
include: .gitlab/ci/packages/package-gitlab-ci.yml
strategy: depend
rules:
- changes:
- "packages/package-a/**/*"
variables:
PACKAGE: package-a
trigger-package-b:
stage: build
trigger:
include: .gitlab/ci/packages/package-gitlab-ci.yml
strategy: depend
rules:
- changes:
- "packages/package-b/**/*"
variables:
PACKAGE: package-b
done_job:
stage: deploy
script:
- "echo DONE"
- "cat config.json"
stages:
- build
- deploy
package-gitlab-ci.yml
workflow:
rules:
- if: $CI_MERGE_REQUEST_ID
- changes:
- "packages/${PACKAGE}/**/*"
stages:
- bootstrap
- validate
cache:
key: "${PACKAGE}_${CI_COMMIT_REF_SLUG}"
paths:
- packages/${PACKAGE}/node_modules/
policy: pull
install-package:
stage: bootstrap
script:
- echo ${PACKAGE}}
- echo '{"package":${PACKAGE}}' > config.json
- "cd packages/${PACKAGE}/"
- yarn install --frozen-lockfile
artifacts:
paths:
- config.json
cache:
key: "${PACKAGE}_${CI_COMMIT_REF_SLUG}"
paths:
- packages/${PACKAGE}/node_modules/
policy: pull-push
lint-package:
script:
- yarn lint
stage: validate
needs: [install-package]
before_script:
- "cd packages/${PACKAGE}/"
test-package:
stage: validate
needs: [lint-package]
before_script:
- "echo working on ${PACKAGE}"
- "cd packages/${PACKAGE}/"
rules:
- if: $CI_MERGE_REQUEST_ID
script:
- yarn test
It looks like your downstream pipeline is defining a workflow with 2 independent rules: if and changes. This may cause the jobs to be included if the first condition in the if is met, i.e. if it is a MR pipeline. Try removing the dash in front of changes, as in the example here, to treat this as a single rule:
workflow:
rules:
- if: $CI_MERGE_REQUEST_ID
changes:
- "packages/${PACKAGE}/**/*"
EDIT: This recent issue states rules:changes does not work as expected with trigger. So you may actually need to remove the changes from the upstream pipeline and solve this in the downstream pipeline.
Side note, not directly related to your issue: the GitLab Docs provide a workflow template to run branch or MR pipelines without creating duplicates. You can use this in your upstream pipeline if it helps:
workflow:
rules:
- if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
- if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS'
when: never
- if: '$CI_COMMIT_BRANCH'
I am trying to trigger gitlab CI/CD pipeline of ProjectA from ProjectB
gitlab-ci.yml of projectB
stages:
- deploy
staging:
stage: deploy
trigger:
project: projectA
branch: main
And my gitlab-ci.yml of projectA is
image: docker:19.03.13
stages:
- build
- staging
fromprojectB: <--- when I trigger pipeline from B, I want only this job to run in build stage
stage: build
script:
..............
fromprojectC:
stage: build
fromprojectD:
stage: build
script:
.......
deploy-to-stage:
stage: staging
script:
............
I want only the fromprojectB job to run among the build stage jobs when the pipeline is triggers from projectB
How can I do this
You can pass environment variables from projectB and then use them in rules in projectA.
Example based on your input:
projectA:
stages:
- build
fromprojectB:
stage: build
rules:
- if: '$RUN_JOB_B'
when: always
- when: never
allow_failure: true
script:
- echo "Compiling the code from project B"
- echo "Compile complete."
fromprojectC:
stage: build
rules:
- if: '$RUN_JOB_C'
when: always
- when: never
allow_failure: true
script:
- echo "Compiling the code from project C"
- echo "Compile complete."
and projectB:
stages:
- build
build-job:
stage: build
variables:
RUN_JOB_B: "true"
trigger:
project: itersive/internal/rd/projecta
branch: main
Pipeline in projectB:
Pipeline in projectA:
In this setup you cannot run pipeline in projectA - all jobs require environment variables - see rules section.
I want to run the stage publish_snapshot_rpm for any branch except master or release. The pipeline runs successfully but the stage publish_snapshot_rpm is not executed. What am I missing? The part of my gitlab-ci file:
.publish:
stage: publish
tags:
- dev
dependencies:
- build
publish_snapshot_rpm:
extends: .publish
only:
# for any branch except master or release
- (?!(master|release)\b)\b\w+
script:
- some script here
You can easily achieve this by using the power of the rules keyword.
rules:
- if: $CI_COMMIT_BRANCH == master
when: never
- if: $CI_COMMIT_BRANCH =~ /^release/
when: never
- when: on_success
A simpler way with the use of except:
.publish:
stage: publish
tags:
- dev
dependencies:
- build
publish_snapshot_rpm:
extends: .publish
except:
- master
- release
script:
- some script here
I have the following stages:
- run
- notify
when run have 3 jobs: run-prod, run-stg, run-dev
which all of them are triggered only by manual action because they require env variables.
The problem is that i want to run notify step only if run actually runs (on not manual pipelines its not run) - how can i do it?
i tried with needs and only but its not working
notify jobs:
notify on run success:
stage: notify
extends: .dv.notify.slack_channel
variables:
SLACK_MESSAGE: "\"🟩 Recovery process which triggered by $GITLAB_USER_NAME finished successfully on context: $CONNECT_CONTEXT, connector: $CONNECT_NAME, connectors: $CONNECTORS_NAMES\""
SLACK_CHANNEL: C02AX03H04F #recovery
notify on run failure:
stage: notify
extends: .dv.notify.slack_channel
variables:
SLACK_MESSAGE: "\"🟥️ Recovery process which triggered by $GITLAB_USER_NAME failed on context: $CONNECT_CONTEXT, connector: $CONNECT_NAME, connectors: $CONNECTORS_NAMES\""
SLACK_CHANNEL: C02AX03H04F #recovery
run-stg:
run-stg:
extends: run
script: ...
rules:
- if: $CONNECT_CONTEXT =~ /^cs-*/
when: manual
Minimal example config that requires manual input of the CONNECT_CONTEXT variable:
stages:
- run
- notify
variables:
CONNECT_CONTEXT:
description: Your description for this variable.
run:
stage: run
rules:
- if: $CONNECT_CONTEXT =~ /^cs-*/
script:
- echo run $CONNECT_CONTEXT
notify:
stage: notify
rules:
- if: $CONNECT_CONTEXT =~ /^cs-*/
script:
- echo notify $CONNECT_CONTEXT
i have a problem with gitlab (community edition, version 14.1.2) CI with complex pipeline on my monorepo.
My structure is client/server:
root/
---- server/
-------- lib/
----------- libA/
----------- libB/
----------- libC/
-------- applications/
----------- appA/
----------- appB/
----------- appC/
---- client/
-------- applications/
------------- appA/
------------- appB/
...
Every folder (root, server, lib, libA, libB, libC etc...) have his own ".gitlab-ci.yml"
Root ".gitlab-ci.yml" is:
stages:
- build
- test
build-server:
stage: build
trigger:
include:
- local: 'server/.gitlab-ci.yml'
rules:
- changes:
- server/**/*
build-client:
stage: build
trigger:
include:
- local: 'client/.gitlab-ci.yml'
rules:
- changes:
- client/**/*
Server ".gitlab-ci.yml" is:
stages:
- build
- test
build-lib:
stage: build
trigger:
include:
- local: 'lib/.gitlab-ci.yml'
rules:
- changes:
- lib/**/*
build-applications:
stage: build
trigger:
include:
- local: 'applications/.gitlab-ci.yml'
rules:
- changes:
- applications/**/*
lib ".gitlab-ci.yml" is:
stages:
- build
- test
build-libA:
stage: build
script:
- echo "Execute libA build!"
rules:
- changes:
- libA/**/*
build-libB:
stage: build
script:
- echo "Execute libB build!"
rules:
- changes:
- libB/**/*
If i change a file inside libA only the ".gitlab-ci.yml" of root folder is triggered, other subfolders not detect file changes and not trigger the build.
The purpose of this configuration is that , for example, when i change a file inside libA, the pipeline detects the changes and build the libA.
Somone can help me to resolve? I hope the structure and the problem is clear. Thanks.
UPDATE
I'm using gitlab 14.1.0
Thanks to DavidC for the answer but with your solution I have not solved my problem, especially with the trigger $CI_PROJECT_PATH seems not to work.
After some time I finally got a solution (which can be evolved with variables)
Root ".gitlab-ci.yml" is:
stages:
- build
- test
build-server:
stage: build
trigger:
include:
- local: '/server/.gitlab-ci.yml'
rules:
- changes:
- server/**/*
build-client:
stage: build
trigger:
include:
- local: '/client/.gitlab-ci.yml'
rules:
- changes:
- client/**/*
Server ".gitlab-ci.yml" is:
stages:
- build
- test
build-lib:
stage: build
trigger:
include:
- local: '/server/lib/.gitlab-ci.yml'
rules:
- changes:
- server/lib/**/*
build-applications:
stage: build
trigger:
include:
- local: '/server/applications/.gitlab-ci.yml'
rules:
- changes:
- server/applications/**/*
lib ".gitlab-ci.yml" is:
stages:
- build
- test
build-libA:
stage: build
script:
- echo "Execute libA build!"
rules:
- changes:
- server/lib/libA/**/*
build-libB:
stage: build
script:
- echo "Execute libB build!"
rules:
- changes:
- server/lib/libB/**/*
Pay attention to this line from the gitlab documentation: "Parent and child pipelines were introduced with a maximum depth of one child pipeline level, which was subsequently increased to two. A parent pipeline can activate many child pipelines and these child pipelines can activate their own child pipelines. It is not possible to activate another level of child pipeline. " refer to: https://docs.gitlab.com/ee/ci/pipelines/parent_child_pipelines.html#nested-child-pipelines
Thanks for help!
It seems like GitLab child-pipeline context execution path is the same as the root directory of your repository, and is not relative to the path of the child-pipeline gitlab-ci.yml file.
The only solution so far seems to either give the path to the directory you want as a variable to your child-pipeline or to directly define it in the .gitlab-ci.yml.
Example :
Root ".gitlab-ci.yml" is:
stages:
- build
- test
build-server:
stage: build
variables:
CI_ROOT_DIR: server # you don't need to provide it, if you define it in the server/.gitlab-ci.yml file
trigger:
include:
- local: '$CI_ROOT_DIR/.gitlab-ci.yml'
rules:
- changes:
- $CI_ROOT_DIR/**/*
build-client:
stage: build
variables:
CI_ROOT_DIR: client
trigger:
include:
- local: '$CI_ROOT_DIR/.gitlab-ci.yml'
rules:
- changes:
- $CI_ROOT_DIR/**/*
Server ".gitlab-ci.yml" is:
stages:
- build
- test
variables:
CI_ROOT_DIR: $CI_PROJECT_PATH/server # default
build-lib:
stage: build
variables:
CI_ROOT_DIR: $CI_ROOT_DIR/lib # you don't need to provide it, if you define it in the server/lib/.gitlab-ci.yml file
trigger:
include:
- local: '$CI_ROOT_DIR/.gitlab-ci.yml'
rules:
- changes:
- $CI_ROOT_DIR/**/*
build-applications:
stage: build
variables:
CI_ROOT_DIR: $CI_ROOT_DIR/applications # you don't need to provide it, if you define it in the server/applications/.gitlab-ci.yml file
trigger:
include:
- local: '$CI_ROOT_DIR/.gitlab-ci.yml'
rules:
- changes:
- $CI_ROOT_DIR/**/*
lib ".gitlab-ci.yml" is:
stages:
- build
- test
variables:
CI_ROOT_DIR: $CI_PROJECT_PATH/server/lib # default
build-libA:
stage: build
script:
- echo "Execute libA build!"
rules:
- changes:
- $CI_ROOT_DIR/libA/**/*
build-libB:
stage: build
script:
- echo "Execute libB build!"
rules:
- changes:
- $CI_ROOT_DIR/libB/**/*
It would be better tho if it was possible to choose the context of the execution of the pipeline when triggered by the parent-pipeline or to have a CI_CHILD_PIPELINE_DIR variable available by Gitlab predefined environment variables