Instruct gitlab ci to run only certain stages for a commit - gitlab

Just like git push -o ci.skip can be used to skip all stages, is it possible to also just as easily specify a whitelist of stages to run while pushing?
Currently I have to manually go to gitlab UI after a push and skip the stages that I don't want running. I also have to track stage dependencies manually while doing this.
Can this be done more easily?


You can set CICD variables with git push options. Therefore, you can use such variables to set whether a job runs or not using only:variables/except:variables or rules:
For example:
my_job:
rules:
- if: $MY_VARIABLE_NEVER
when: never
- if: $MY_VARIABLE_ALWAYS
when: always
- when: on_success
# ...
Then you can use the push option to set the variables in your rules:
# excludes my_job from the pipeline
git push -o ci.variable="MY_VARIABLE_NEVER=true"
This can't be applied for an entire stage directly, but you can apply the applicable only:/except: or rules: for each job in the stage you want to control.

Related

Unexpected behaviour of "rules" in GitLab CI

I have some problems with understanding how and why "rules" in GitLab CI work.
I have written a minimal code showing the issue in a GitLab project: https://gitlab.com/daniel.grabowski/gitlab-ci-rules-problems It contains two directories ("files1/" and "files2/") with some files in them and a .gitlab-ci.yml file.
My configuration
Here's the CI configuration:
stages:
- build
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "push"
.job_tpl:
image: alpine:latest
stage: build
variables:
TARGET_BRANCH: $CI_DEFAULT_BRANCH
rules:
- if: $CI_COMMIT_BRANCH == $TARGET_BRANCH
changes:
- $FILES_DIR/**/*
variables:
JOB_ENV: "prod"
- if: $CI_COMMIT_BRANCH != $TARGET_BRANCH
changes:
- $FILES_DIR/**/*
when: manual
allow_failure: true
variables:
JOB_ENV: "dev"
script:
- echo "CI_COMMIT_BRANCH=$CI_COMMIT_BRANCH"
- echo "TARGET_BRANCH=$TARGET_BRANCH"
- echo "JOB_ENV=$JOB_ENV"
files1 job:
extends: .job_tpl
variables:
FILES_DIR: files1
files2 job:
extends: .job_tpl
variables:
FILES_DIR: files2
As you can see in the above code I'm using workflow to run only "branch pipelines" and have two "twin" jobs configured to watch for changes in one of the project's directories each. The TARGET_BRANCH variable is of course unnecessary in the demo project but i need something like this in the real one and it shows one of my problems. Additionally the jobs behave differently depending on the branch for which they are run.
My expectations
What I want to achieve is:
Each of the jobs should be added to a pipeline only when I push changes to files1/ or files2/ directory respectively.
When I push changes to a branch different then "main" a manual job responsible for the changed directory shoud be added to a pipeline.
When I merge changes to the "main" branch a job responsible for the changed directory shoud be added to a pipeline and it should be automatically started.
Test scenario
I'm creating a new branch from "main", make some change in the file1/test.txt and push the branch to GitLab.
what I expect: a pipeline created with only "files1 job" runnable manually
what I get: a pipeline with both jobs (both manual). Actually I've found explanation of such behaviour here: https://docs.gitlab.com/ee/ci/jobs/job_control.html#jobs-or-pipelines-run-unexpectedly-when-using-changes - "The changes rule always evaluates to true when pushing a new branch or a new tag to GitLab."
On the same branch I make another change in the file1/test.txt and make push.
what I expect: a pipeline created with only "files1 job" runnable manually
what I get: exactly what I expect since the branch isn't a "new" one
I create a Merge Request from my branch to main and make the merge.
what I expect: a pipeline created with only "files1 job" which starts automatically
what I get: a pipeline created with only "files1 job" but a manual one
My questions/problems
Can you suggest me any way to bypass the issue with "changes" evaluating always to "true" on new branches? Actually it behaves exactly as I want it if I don't use "rules" but let's assume I need "rules".
Why the jobs run as "manual" on the main branch in spite of the "if" condition in which both CI_COMMIT_BRANCH and TARGET_BRANCH variables are (or should be) set to "main". To debug it I'm printing those vars in job's "script" and when I run it on "main" pipeline I'm getting:
$ echo "CI_COMMIT_BRANCH=$CI_COMMIT_BRANCH"
CI_COMMIT_BRANCH=main
$ echo "TARGET_BRANCH=$TARGET_BRANCH"
TARGET_BRANCH=main
$ echo "JOB_ENV=$JOB_ENV"
JOB_ENV=dev
so theoretically CI should enter into the "automatic" job path.
Generally I find the CI "rules" quite inconvenient and confusing but as I understand it GitLab prefers them to "only/except" solution so I'm trying to refactor my CI/CD to use them which will fail if I don't find solution for the above difficulties :(

Gitlab CI doesnt trigger pipelines on commit to the specified branch

I am trying to trigger a pipeline on any type of change/commit (push/merge/webIDE etc) to a specific branch called test branch. Not sure why it doesn't trigger at all even after I make a change to a file in that specific branch.
default:
tags:
- specs
stages:
- validate
Trigger job:
stage: validate
script:
- curl -X POST -F token=$ENTDV_PL_TRIGGER_TOKEN -F "ref=master" https://gitlab.com/api/v4/projects/1252972/trigger/pipeline
variables:
CI_DEBUG_TRACE: "true"
rules:
- if: '$CI_COMMIT_BRANCH == "testbranch"'

If you want Trigger job to only be run when changes are made on testbranch, you can modify your Trigger job to the following:
Trigger job:
stage: validate
script:
- curl -X POST -F token=$ENTDV_PL_TRIGGER_TOKEN -F "ref=master" https://gitlab.com/api/v4/projects/1252972/trigger/pipeline
variables:
CI_DEBUG_TRACE: "true"
only:
- testbranch
The Gitlab CI documentation is actually quite good. Here is a link to how "only" works: https://docs.gitlab.com/ee/ci/yaml/index.html#only--except
From the docs: "Use the only:refs and except:refs keywords to control when to add jobs to a pipeline based on branch names or pipeline types."
You can do some other neat things like specifying only merge_requests or using regex for triggering jobs and what not.

GitLab CI/CD yml - How to add into pipeline job if new branch created. And exclude for further pushes?

In our git process we have hothix process. The process is to create a hotfix brunch from master and make changes and then merge to master back.
The issue that I'm faсing now is how to add job to pipelne if branch just created and exclude from pipeline for any further push?
When branch is created we need to set up custom environment. Create a sandbox, push code and push some test data. This is need only when branch is created. In case of changes and pushes, environment is available and no need in building it again. Just build code and push a release tag
I'm trying something like this:
job-name:
stage: build
script:
- myScript
rules:
- if: '$CI_COMMIT_BRANCH =~ /^hotfix.*/'
changes:
- force-app/**/*
when: never
- if: '$CI_COMMIT_BRANCH =~ /^hotfix.*/'
allow_failure: false
when: manual
And for sure it's not working as changes: not part of if policy:
Rules are evaluated in order until a match is found. If a match is found, the attributes are checked to see if the job should be added to the pipeline.

Gitlab CI can trigger other project pipeline stage?

I have an A project and an E2E project. I want to deploy A project trigger E2E pipeline run test but I just want the trigger test stage. we don't need trigger E2E to build deploy ...etc
e2e_tests:
stage: test
trigger:
project: project/E2E
branch: master
strategy: depend
stage: test
I have tried to use the stage in config. but got error unknown keys: stage
have any suggestions?

In your E2E project, the one that receives the trigger, you can tell a job to only run when the pipeline source is a trigger using the rules syntax:
build-from-trigger:
stage: build
when: never
rules:
- if: "$CI_COMMIT_REF_NAME == 'master' && $CI_PIPELINE_SOURCE == 'trigger'
when: always
script:
- ./build.sh # this is just an example, you'd run whatever you normally would here
The first when statement, when: never sets the default for the job. By default, this job will never run. Then using the rule syntax, we set a condition that will allow the job to run. If the CI_COMMIT_REF_NAME variable (the branch or tag name) is master AND the CI_PIPELINE_SOURCE variable (whatever kicked off this pipeline) is a trigger, then we run this job.
You can read about the when keyword here: https://docs.gitlab.com/ee/ci/yaml/#when, and you can read the rules documentation here: https://docs.gitlab.com/ee/ci/yaml/#rules

Disable pipeline for every commit in Gitlab and only run it on open merge request

The CI pipeline runs on every commit in my Gitlab repository at work. Is there way to disable that and only run the CI pipeline on an open merge request to the master branch?

There is currently no configuration option to do that. Here are some things that can be used to "disable" a pipeline build.
Adding [ci skip] inside the commit message will not trigger a pipeline on push.
Using except and only options on all jobs inside the pipeline. To avoid duplication in this case, you can use Anchors.
Update: GitLab 11.7
When pushing to GitLab you can skip triggering a pipeline by passing ci.skip option to the push command: git push -o ci.skip

Update in 2020 because solution with only and except are candidates for deprecation : https://docs.gitlab.com/ee/ci/yaml/#onlyexcept-basic
Still in .gitlab-ci.yml you should use now rules : See https://docs.gitlab.com/ee/ci/yaml/#rules
I simply set this rule on my build job and the job is "blocked" and wait a manual trigger from the UI.
rules:
- when: manual
Note that we can create more advanced rules with conditions to trigger for exemple if we see a git tag.
https://docs.gitlab.com/ee/ci/yaml/#exclude-jobs-with-rules-from-certain-pipelines

Yes, but only if you set it on each job within the pipeline.
https://docs.gitlab.com/ee/ci/yaml/#onlyexcept-basic
job:
script: echo 'test'
only:
- merge_requests
That job will only run on commits that are part of a merge request, rather than every push of a commit to a branch. If you set every job to this setting then the pipeline will never run.

You can just add [ci skip] to the commit message and builds will be skipped
https://gitlab.com/gitlab-org/gitlab-ce/issues/14499

You could add something like this in the beginning of your .gitlab-ci.yaml
workflow:
rules:
- if: "$CI_COMMIT_BRANCH"
when: never
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"'
For more info look at Merge request pipelines

No. (Not yet !)
You can follow the expected feature development here.
https://gitlab.com/gitlab-org/gitlab-ce/issues/23902
However, you can automatically trigger pipelines using "maofr"'s code
https://gitlab.com/gitlab-org/gitlab-ce/issues/23902#note_88958643

For merge we can use this guide #onlyrefs--exceptrefs
merge_requests
For pipelines created when a merge request is created or updated. Enables merge request pipelines, merged results pipelines, and merge trains.
and for generally triggering pipeline we can use only: and variables: together:
script:
- . diff.sh $SMS_MESSAGE
only:
variables:
- $ACTION == "diff"
then triggering it is easy using curl:
curl -X POST \
-F token=$TOKEN \ # your token
-F ref=$REF_NAME \ # branch name
-F "variables[ACTION]=diff" \ # variable
https://gitlab.com/api/v4/projects/0/trigger/pipeline # your project endpoint
This part variables[ACTION]=diff will assign diff into ACTION and it will apply to .gitlab-ci.yml and only: section.

job:
stage: build
script:
- echo "Do your build here"
except:
- pushes
Use except pushes to stop create a new pipeline on every pushed commits. For reference check this

Update 2022 :
According to the documention of Gitlab on: https://docs.gitlab.com/ee/ci/yaml/#when
you can use when: manual to run the job only when triggered manually.
Example:
deploy_job:
stage: deploy
script:
- make deploy
when: manual
You don't need also rules for that.

Resources