I've been stuck on trying to run a GitLab pipeline from a branch in my repo and its driving me crazy. I have a sample branch called test-branch, but every time I try to run the pipeline manually GitLab shows an error message saying
Pipeline cannot be run
Failed to build pipeline!
I am the owner of the group hence by inherited permissions, I should be able to run a pipeline on any branch by default. I am able to manually run this same pipeline from the main branch, but for any other branch, it simply won't budge. Here is the .gitlab-ci.yaml file I am using
stages:
- build_artifacts
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "push"
when: never
- if: $CI_PIPELINE_SOURCE == "web"
Run-Pipeline:
stage: build_artifacts
script:
- |
echo "Somevalue" > file.json # Just an example script
artifacts:
paths:
- file.json
Here are some steps I have takes to try and resolve this
Change the actual pipeline steps
Delete and recreate the branch with different code
Scoured the pipeline troubleshooting docs, but I don't see this anywhere
This has to be the worst error message GitLab shows, there is no way for me to debug what is going wrong. What am I missing here?
I figured out the problem. Apparently, GitLab has a certain naming convention for the variables. I added a user defined variable in the gitlab-ci config called SAMPLE-VARIABLE. Turns out you cannot use - in the name of the variable. Renaming it to SAMPLE_VARIABLE did the trick.
However, it would be good for GitLab to give clear error messages rather than throwing such a generic message at the end user and expecting them to figure it out. I spent almost a whole entire day trying out different things to figure out what was exactly going wrong here
Check first the pipeline through the Lint CI configuration
That will test the validity of your GitLab CI/CD configuration before committing the changes.
An incorrect syntax could explain the runtime error message.
Related
I have some problems with understanding how and why "rules" in GitLab CI work.
I have written a minimal code showing the issue in a GitLab project: https://gitlab.com/daniel.grabowski/gitlab-ci-rules-problems It contains two directories ("files1/" and "files2/") with some files in them and a .gitlab-ci.yml file.
My configuration
Here's the CI configuration:
stages:
- build
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "push"
.job_tpl:
image: alpine:latest
stage: build
variables:
TARGET_BRANCH: $CI_DEFAULT_BRANCH
rules:
- if: $CI_COMMIT_BRANCH == $TARGET_BRANCH
changes:
- $FILES_DIR/**/*
variables:
JOB_ENV: "prod"
- if: $CI_COMMIT_BRANCH != $TARGET_BRANCH
changes:
- $FILES_DIR/**/*
when: manual
allow_failure: true
variables:
JOB_ENV: "dev"
script:
- echo "CI_COMMIT_BRANCH=$CI_COMMIT_BRANCH"
- echo "TARGET_BRANCH=$TARGET_BRANCH"
- echo "JOB_ENV=$JOB_ENV"
files1 job:
extends: .job_tpl
variables:
FILES_DIR: files1
files2 job:
extends: .job_tpl
variables:
FILES_DIR: files2
As you can see in the above code I'm using workflow to run only "branch pipelines" and have two "twin" jobs configured to watch for changes in one of the project's directories each. The TARGET_BRANCH variable is of course unnecessary in the demo project but i need something like this in the real one and it shows one of my problems. Additionally the jobs behave differently depending on the branch for which they are run.
My expectations
What I want to achieve is:
Each of the jobs should be added to a pipeline only when I push changes to files1/ or files2/ directory respectively.
When I push changes to a branch different then "main" a manual job responsible for the changed directory shoud be added to a pipeline.
When I merge changes to the "main" branch a job responsible for the changed directory shoud be added to a pipeline and it should be automatically started.
Test scenario
I'm creating a new branch from "main", make some change in the file1/test.txt and push the branch to GitLab.
what I expect: a pipeline created with only "files1 job" runnable manually
what I get: a pipeline with both jobs (both manual). Actually I've found explanation of such behaviour here: https://docs.gitlab.com/ee/ci/jobs/job_control.html#jobs-or-pipelines-run-unexpectedly-when-using-changes - "The changes rule always evaluates to true when pushing a new branch or a new tag to GitLab."
On the same branch I make another change in the file1/test.txt and make push.
what I expect: a pipeline created with only "files1 job" runnable manually
what I get: exactly what I expect since the branch isn't a "new" one
I create a Merge Request from my branch to main and make the merge.
what I expect: a pipeline created with only "files1 job" which starts automatically
what I get: a pipeline created with only "files1 job" but a manual one
My questions/problems
Can you suggest me any way to bypass the issue with "changes" evaluating always to "true" on new branches? Actually it behaves exactly as I want it if I don't use "rules" but let's assume I need "rules".
Why the jobs run as "manual" on the main branch in spite of the "if" condition in which both CI_COMMIT_BRANCH and TARGET_BRANCH variables are (or should be) set to "main". To debug it I'm printing those vars in job's "script" and when I run it on "main" pipeline I'm getting:
$ echo "CI_COMMIT_BRANCH=$CI_COMMIT_BRANCH"
CI_COMMIT_BRANCH=main
$ echo "TARGET_BRANCH=$TARGET_BRANCH"
TARGET_BRANCH=main
$ echo "JOB_ENV=$JOB_ENV"
JOB_ENV=dev
so theoretically CI should enter into the "automatic" job path.
Generally I find the CI "rules" quite inconvenient and confusing but as I understand it GitLab prefers them to "only/except" solution so I'm trying to refactor my CI/CD to use them which will fail if I don't find solution for the above difficulties :(
I'm trying to ensure that my Gitlab CI only runs on a particular host, regardless of the Enable/Disable CI status in the GUI. I have this at the top of my .gitlab-ci.yml file, but CI now runs nowhere. I've confirmed that the host value is as expected by disabling my header and printing it from one of the script jobs.
workflow:
rules:
- if: '$CI_SERVER_HOST != "run-here.example.com"'
when: never
In other words, I'm sure that the value of CI_SERVER_HOST is run-here.example.com, and my pipelines still don't run anywhere. Removing the above header causes them to run everywhere.
You will also have to add an always to run it in other cases.
workflow:
rules:
- if: '$CI_SERVER_HOST != "run-here.example.com"'
when: never
- when: always
Could you check with this workflow.
I am working on a multi-pipeline project, and using trigger keyword to trigger a downstream pipeline, but I'm not able to pass artifacts created in the upstream project. I am using needs to get the artifact like so:
Downstream Pipeline block to get artifacts:
needs:
- project: workspace/build
job: build
ref: master
artifacts: true
Upstream Pipeline block to trigger:
build:
stage: build
artifacts:
paths:
- ./policies
expire_in: 2h
only:
- master
script:
- echo 'Test'
allow_failure: false
triggerUpstream:
stage: deploy
only:
- master
trigger:
project: workspace/deploy
But I am getting the following error:
This job depends on other jobs with expired/erased artifacts:
I'm not sure what's wrong.
Looks like there is a problem sharing artifacts between pipelines as well as between projects. It is known bug and has been reported here:
https://gitlab.com/gitlab-org/gitlab/-/issues/228586
You can find a workaround there but since it needs to add access token to project it is not the best solution.
Your upstream pipeline job "Build" is set to only store its artifacts for 2 hours (from the expire_in: 2h line. Your downstream pipeline must have run at least 2 hours later than the artifacts were created, so the artifact expired and was erased, generating that error.
To solve it you can either update the expire_in field to however long you need them to be active (so for example if you know the downstream pipeline will run up to 5 days later, set it to 5d for 5 days), or rerun the Build job to recreate the artifacts.
You can read more about the expire_in keyword and artifacts in general from the docs
It isn't a problem with expired artifacts, the error is incorrect. In my case I am able to download the artifacts as a zip directly from the UI on the executed job. My expire_in is set to 1 week yet I am still getting this message.
I have an A project and an E2E project. I want to deploy A project trigger E2E pipeline run test but I just want the trigger test stage. we don't need trigger E2E to build deploy ...etc
e2e_tests:
stage: test
trigger:
project: project/E2E
branch: master
strategy: depend
stage: test
I have tried to use the stage in config. but got error unknown keys: stage
have any suggestions?
In your E2E project, the one that receives the trigger, you can tell a job to only run when the pipeline source is a trigger using the rules syntax:
build-from-trigger:
stage: build
when: never
rules:
- if: "$CI_COMMIT_REF_NAME == 'master' && $CI_PIPELINE_SOURCE == 'trigger'
when: always
script:
- ./build.sh # this is just an example, you'd run whatever you normally would here
The first when statement, when: never sets the default for the job. By default, this job will never run. Then using the rule syntax, we set a condition that will allow the job to run. If the CI_COMMIT_REF_NAME variable (the branch or tag name) is master AND the CI_PIPELINE_SOURCE variable (whatever kicked off this pipeline) is a trigger, then we run this job.
You can read about the when keyword here: https://docs.gitlab.com/ee/ci/yaml/#when, and you can read the rules documentation here: https://docs.gitlab.com/ee/ci/yaml/#rules
The CI pipeline runs on every commit in my Gitlab repository at work. Is there way to disable that and only run the CI pipeline on an open merge request to the master branch?
There is currently no configuration option to do that. Here are some things that can be used to "disable" a pipeline build.
Adding [ci skip] inside the commit message will not trigger a pipeline on push.
Using except and only options on all jobs inside the pipeline. To avoid duplication in this case, you can use Anchors.
Update: GitLab 11.7
When pushing to GitLab you can skip triggering a pipeline by passing ci.skip option to the push command: git push -o ci.skip
Update in 2020 because solution with only and except are candidates for deprecation : https://docs.gitlab.com/ee/ci/yaml/#onlyexcept-basic
Still in .gitlab-ci.yml you should use now rules : See https://docs.gitlab.com/ee/ci/yaml/#rules
I simply set this rule on my build job and the job is "blocked" and wait a manual trigger from the UI.
rules:
- when: manual
Note that we can create more advanced rules with conditions to trigger for exemple if we see a git tag.
https://docs.gitlab.com/ee/ci/yaml/#exclude-jobs-with-rules-from-certain-pipelines
Yes, but only if you set it on each job within the pipeline.
https://docs.gitlab.com/ee/ci/yaml/#onlyexcept-basic
job:
script: echo 'test'
only:
- merge_requests
That job will only run on commits that are part of a merge request, rather than every push of a commit to a branch. If you set every job to this setting then the pipeline will never run.
You can just add [ci skip] to the commit message and builds will be skipped
https://gitlab.com/gitlab-org/gitlab-ce/issues/14499
You could add something like this in the beginning of your .gitlab-ci.yaml
workflow:
rules:
- if: "$CI_COMMIT_BRANCH"
when: never
- if: '$CI_PIPELINE_SOURCE == "merge_request_event" && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "master"'
For more info look at Merge request pipelines
No. (Not yet !)
You can follow the expected feature development here.
https://gitlab.com/gitlab-org/gitlab-ce/issues/23902
However, you can automatically trigger pipelines using "maofr"'s code
https://gitlab.com/gitlab-org/gitlab-ce/issues/23902#note_88958643
For merge we can use this guide #onlyrefs--exceptrefs
merge_requests
For pipelines created when a merge request is created or updated. Enables merge request pipelines, merged results pipelines, and merge trains.
and for generally triggering pipeline we can use only: and variables: together:
script:
- . diff.sh $SMS_MESSAGE
only:
variables:
- $ACTION == "diff"
then triggering it is easy using curl:
curl -X POST \
-F token=$TOKEN \ # your token
-F ref=$REF_NAME \ # branch name
-F "variables[ACTION]=diff" \ # variable
https://gitlab.com/api/v4/projects/0/trigger/pipeline # your project endpoint
This part variables[ACTION]=diff will assign diff into ACTION and it will apply to .gitlab-ci.yml and only: section.
job:
stage: build
script:
- echo "Do your build here"
except:
- pushes
Use except pushes to stop create a new pipeline on every pushed commits. For reference check this
Update 2022 :
According to the documention of Gitlab on: https://docs.gitlab.com/ee/ci/yaml/#when
you can use when: manual to run the job only when triggered manually.
Example:
deploy_job:
stage: deploy
script:
- make deploy
when: manual
You don't need also rules for that.