I have a very basic Gitlab CICD yaml file that is supposed to run two jobs on completion of another job. I'd like one job to automatically set a variable, while the other job prompts the user to define one. These jobs are called auto and manual respectively. However, when I run the file below, the manual step just doesn't set anything and continues to run without prompting the user for variables. This does make sense, but how exactly do I force this job to prompt the user for variable definitions?
stages:
- A
- B
.template
script:
- echo ${MY_VARIABLE}
setup:
stage: A
script:
- echo "do nothing yet"
allow_failure: false
when: manual
auto:
stage: B
variables:
MY_VARIABLE: "I am automatically set"
when: on_success
dependencies:
- setup
manual:
stage: B
when: manual
dependencies:
- setup
Currently, the above results in:
auto output
I am automatically set
manual output
What I want:
UPDATE: It doesn't look like there is a way to force the above.
Related
I have some problems with understanding how and why "rules" in GitLab CI work.
I have written a minimal code showing the issue in a GitLab project: https://gitlab.com/daniel.grabowski/gitlab-ci-rules-problems It contains two directories ("files1/" and "files2/") with some files in them and a .gitlab-ci.yml file.
My configuration
Here's the CI configuration:
stages:
- build
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "push"
.job_tpl:
image: alpine:latest
stage: build
variables:
TARGET_BRANCH: $CI_DEFAULT_BRANCH
rules:
- if: $CI_COMMIT_BRANCH == $TARGET_BRANCH
changes:
- $FILES_DIR/**/*
variables:
JOB_ENV: "prod"
- if: $CI_COMMIT_BRANCH != $TARGET_BRANCH
changes:
- $FILES_DIR/**/*
when: manual
allow_failure: true
variables:
JOB_ENV: "dev"
script:
- echo "CI_COMMIT_BRANCH=$CI_COMMIT_BRANCH"
- echo "TARGET_BRANCH=$TARGET_BRANCH"
- echo "JOB_ENV=$JOB_ENV"
files1 job:
extends: .job_tpl
variables:
FILES_DIR: files1
files2 job:
extends: .job_tpl
variables:
FILES_DIR: files2
As you can see in the above code I'm using workflow to run only "branch pipelines" and have two "twin" jobs configured to watch for changes in one of the project's directories each. The TARGET_BRANCH variable is of course unnecessary in the demo project but i need something like this in the real one and it shows one of my problems. Additionally the jobs behave differently depending on the branch for which they are run.
My expectations
What I want to achieve is:
Each of the jobs should be added to a pipeline only when I push changes to files1/ or files2/ directory respectively.
When I push changes to a branch different then "main" a manual job responsible for the changed directory shoud be added to a pipeline.
When I merge changes to the "main" branch a job responsible for the changed directory shoud be added to a pipeline and it should be automatically started.
Test scenario
I'm creating a new branch from "main", make some change in the file1/test.txt and push the branch to GitLab.
what I expect: a pipeline created with only "files1 job" runnable manually
what I get: a pipeline with both jobs (both manual). Actually I've found explanation of such behaviour here: https://docs.gitlab.com/ee/ci/jobs/job_control.html#jobs-or-pipelines-run-unexpectedly-when-using-changes - "The changes rule always evaluates to true when pushing a new branch or a new tag to GitLab."
On the same branch I make another change in the file1/test.txt and make push.
what I expect: a pipeline created with only "files1 job" runnable manually
what I get: exactly what I expect since the branch isn't a "new" one
I create a Merge Request from my branch to main and make the merge.
what I expect: a pipeline created with only "files1 job" which starts automatically
what I get: a pipeline created with only "files1 job" but a manual one
My questions/problems
Can you suggest me any way to bypass the issue with "changes" evaluating always to "true" on new branches? Actually it behaves exactly as I want it if I don't use "rules" but let's assume I need "rules".
Why the jobs run as "manual" on the main branch in spite of the "if" condition in which both CI_COMMIT_BRANCH and TARGET_BRANCH variables are (or should be) set to "main". To debug it I'm printing those vars in job's "script" and when I run it on "main" pipeline I'm getting:
$ echo "CI_COMMIT_BRANCH=$CI_COMMIT_BRANCH"
CI_COMMIT_BRANCH=main
$ echo "TARGET_BRANCH=$TARGET_BRANCH"
TARGET_BRANCH=main
$ echo "JOB_ENV=$JOB_ENV"
JOB_ENV=dev
so theoretically CI should enter into the "automatic" job path.
Generally I find the CI "rules" quite inconvenient and confusing but as I understand it GitLab prefers them to "only/except" solution so I'm trying to refactor my CI/CD to use them which will fail if I don't find solution for the above difficulties :(
I have following jobs and stages and with this yml configuration the test stage runs on schedule and regular pipelines when I set $RUN_JOB variable to true in my schedule and in my project's CI/CD variables. But this also schedules scheduled-test-1 and scheduled-test-2 in my scheduled pipelines.
What I want to do is that the test stage should continue to run on schedule and regular pipelines but scheduled-test-1 and scheduled-test-2 should not be scheduled with test stage.
stages:
build
test
deploy
scheduled-test-1
scheduled-test-2
build:
script:
- echo $Service_Version
only:
- develop
except:
- schedules
test:
script:
- echo $Service_Version
only:
variables:
- $RUN_JOB
deploy:
script:
- echo $Service_Version
only:
- develop
except:
- schedules
scheduled-test-1:
script:
- echo $Service_Version
only:
- schedules
scheduled-test-2:
script:
- echo $Service_Version
only:
- schedules
There might be a simpler option but, using artifacts:reports as in Exporting environment variables from one stage to the next in GitLab CI might help.
The idea would be to export a RUN_TEST environment variable (a flag to signal the test job has been executed), and use it in the scheduled-test-x jobs in a rules section:
rules:
- if: $RUN_TEST != ""
While those scheduled-test-x jobs might still be scheduled, at least they would not run, but their rule would be false.
From what I have just ran into, the "needs" line in a gitlab-ci.yml file only checks to see if the job that is defined in the "needs" line is being run - not if it passes or fails.
I ran the below code in my pipeline and the "build-latest" job runs even if the "test-
tag" job fails.
I only want the "build-latest" job to run if the "test-tag" job passes.
How is this achieved?
build-latest:
stage: publish
image:
name: gcr.io/go-containerregistry/crane:debug
entrypoint: [""]
rules:
#- if: $CI_COMMIT_TAG != null
- if: $CI_COMMIT_REF_NAME == "add-latest-tagging"
when: always
needs:
- test-tag
script:
- crane auth login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
The issue lies with the fact that you added
when: always
It is true that since you specified needs, the build-latest job will need the job test-tag to execute first.
After test-tag job concluded it will evaluate if it should execute the build-latest job.
By adding the always clause to the build-latest job will force it to execute, even if the test-tag fails. Provided test-tag job has at least concluded
Long story sort, you should remove the when always clause
If you want a job to run only when one or more previous jobs pass, then you want to put it in a separate stage.
Not sure how you've broken up the jobs without more of the CI file, but assuming:
test-tag job is in stage: test
stage: publish comes after test
Then it should work the way you want simply by removing the needs: option from your build-latest job.
Here is my .gitlab-ci.yml file:
script1:
only:
refs:
- merge_requests
- master
changes:
- script1/**/*
script: echo 'script1 done'
script2:
only:
refs:
- merge_requests
- master
changes:
- script2/**/*
script: echo 'script2 done'
I want script1 to run whenever there is a change in script1 directory; likewise script2.
I tested these with a change in script1, a change in script2, change in both the directories, and no change in either of these directories.
Former 3 cases are passing as expected but 4th case, the one with no change in either directory, is failing.
In the overview, Gitlab gives the message
Could not retrieve the pipeline status. For troubleshooting steps, read thedocumentation.
In the Pipelines tab, I have an option to Run pipeline. Clicking on that gives the error
An error occurred while trying to run a new pipeline for this Merge Request.
If there is no job, I want the pipeline to succeed.
Gitlab pipelines do not have any independent validity outside of jobs. A pipeline, by definition, consists of one or more jobs. In your example 4 above no jobs are created. The simplest hack you can add to your pipeline is a job which always runs:
dummyjob:
script: exit 0
I have a pipeline that runs automatically when code is pushed to gitlab. There's a terraform apply step that I want to be able to run manually in one case (resources destroyed/recreated) and automatically in another (resources simply added or destroyed.) I almost got this with a manual step but can't see how to get the pipeline to be automatic in the safe case. The manual terraform apply step would not be the last in the pipeline.
Is it possible to say 'do step C if step A completed or step B completed'? Kind of branch the pipeline? Or could I do it with two pipelines, and failure in one triggers the other?
Current partial test code (gitlab CI yaml) here:
# stop with a warning if resources will be created and destroyed
check:
stage: check
script:
- ./terraformCheck.sh
allow_failure: true
# Apply changes manually, whether there is a warning or not
override:
stage: deploy
environment:
name: production
script:
- ./terraformApply.sh
dependencies:
- plan
when: manual
allow_failure: false
only:
- master
log:
stage: log
environment:
name: production
script:
- ./terraformLog.sh
when: always
only:
- master