I need to run 2 jobs parallel on a different OS. For this scenario I've to active runners on different servers with the required OS. Each runner has a unique tag, which I use for the jobs. But the jobs are running sequential, not parallel. Is there any keyword which I have to use, to run both jobs parallel?
my gitlab-ci.yml
stages:
- test
rhel8:
stage: test
rules:
- if: $TEST == "rhel8" || $TEST == "all"
tags:
- rhel8
script:
- echo "Test RHEL 8"
rhel7:
stage: test
rules:
- if: $TEST == "rhel7" || $TEST == "all"
tags:
- rhel7
script:
- echo "Test RHEL 7"
The parallel keyword is what you are looking for. Here is an example of using it to run the same job with different runner tags:
stages:
- test
rhel:
stage: test
rules:
- if: $TEST =~ '/^rhel.*/' || $TEST == "all"
parallel:
matrix:
- TEST: rhel7
- TEST: rhel8
tags:
- $TEST
script:
- echo "Test $TEST"
Related
My goal is to manually start the stage so that the jobs is done sequentially.
My pipeline:
stages:
- dev_docker
job-1:
stage: dev_docker
when: manual
script:
- echo "job-1 started"
- sleep 5
- echo "job-1 done"
job-2:
stage: dev_docker
needs: ["job-1"]
script:
- echo "job-2 started"
- sleep 5
- echo "job-2 done"
job-3:
stage: dev_docker
needs: ["job-2"]
script:
- echo "job-3 started"
- sleep 5
- echo "job-3 done"
job-4:
stage: dev_docker
needs: ["job-3"]
script:
- echo "job-4 started"
- sleep 5
- echo "job-4 done"
My problem is that after starting the second job, the rest of the jobs are not started.
My public pipeline: https://gitlab.com/Yatakoi/needs/-/blob/main/.gitlab-ci.yml
It may be a bug within GitLab itself, someone here has the exact same issue as you, with a WIP fix potentially on the way, as mentioned in the comment
I have the following gitlab configuration:
stages:
- test
- stage1
- stage2
test:
rules:
- if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "schedule"
when: never
stage: test
script:
- 'echo "Running Test"'
my_stage1:
stage: stage1
script:
- 'echo "Running stage 1"'
my_stage2:
stage: stage2
script:
- 'echo "Running stage 2"'
and I create a merge request. I expect that all three stages are run in that case, but the first stage is not run. Why? How to fix it? The documentation on that it very unclear!
The content of CI_PIPELINE_SOURCE is merge-request-event.
Looking at the documentation here, when you are using a when:never it needs to be followed by a specific success clause, in order for it to run in other cases e.g.:
test:
rules:
- if: $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "schedule"
when: never
- when: on_success
I have a gitlab pipeline similar to the below one
stages
- test
p1::test:
stage: test
script:
- echo " parallel 1"
p2::test:
stage: test
script:
- echo " parallel 2"
p3::test:
stage: test
script:
- echo " parallel 3"
p4::test:
stage: test
script:
- echo " parallel 4"
All these four jobs will run in parallel, how can I get to know the status of the stage test,
I want to notify Success if all four are passed, Failed if anyone of the job fails.
One easy way to tell if the prior stage (and everything before it) has passed or failed is to add another stage with two jobs that use opposite when keywords.
If a job has when: on_success (the default) it will only run if all prior jobs have succeeded (or if they failed but have allow_failure: true, or have when: manual and haven't run yet). If any job has failed, it will not.
If a job has when: on_failure it will run if any of the prior jobs has failed.
This can be useful for cleaning up build artifacts, or rolling back changes, but it can apply for your use case too. For example, you could use two jobs like this:
stages:
- test
- verify_tests
p1::test:
stage: test
script:
- echo " parallel 1"
p2::test:
stage: test
script:
- echo " parallel 2"
p3::test:
stage: test
script:
- echo " parallel 3"
p4::test:
stage: test
script:
- echo " parallel 4"
tests_passed:
stage: verify_tests
when: on_success # this is the default, so you could leave this off. I'm adding it for clarity
script:
- # do whatever you need to when the tests all pass
tests_failed:
stage: verify_tests
when: on_failure # this will only run if a job in a prior stage fails
script:
- # do whatever you need to when a job fails
You can do this for each of your stages if you need to know the status after each one programmatically.
I have a little problem with my GitLab pipeline.
I would like to run a manual job with scheduled rule or find a way to run a scheduled pipe with my jobs without rewrite the pipe.
As you see in the example I have 2 firstjob tagged job. One is manually and one is scheduled. My problem that if I run the scheduled workflow, the AC-test won't start and if I try to run the FirstJob by scheduled rule, it won't start because of when: manual section.
Here is my example:
stages:
- firstjob
- test
- build
- deploy
FirstJob:
stage: firstjob
script:
- echo "Hello Peoples!"
- sleep 1
when: manual
allow_failure: false
FirstJobSchedule:
stage: firstjob
script:
- echo "Hello Scheduled Peoples!"
- sleep 1
only:
- schedule
allow_failure: false
AC-test:
needs: [FirstJob]
stage: test
script:
- echo "AC Test is running"
- sleep 10
ProdJobBuild:
stage: build
needs: [AC-test]
script:
- echo "Building thing to prod"
ProdJobDeploy:
stage: deploy
needs: [ProdJobBuild]
script:
- echo "Deploying thing to prod"
Is there a possibility to solve this problem somehow?
Did somebody ever suffer from this problem?
There's a way to do that with only:, but I'd suggest moving to rules: as only: is going to be deprecated.
So you will not need two jobs with different conditions, you can do a branching condition:
stages:
- firstjob
- test
- build
- deploy
workflow:
rules:
- if: $CI_MERGE_REQUEST_IID
- if: $CI_COMMIT_TAG
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
FirstJob:
stage: firstjob
script:
- echo "Hello Peoples!"
- sleep 1
rules:
- if: $CI_PIPELINE_SOURCE == "schedule"
# when: always # is a default value
- when: manual
# allow_failure: false # is a default value
AC-test:
needs: [FirstJob]
stage: test
script:
- echo "AC Test is running"
- sleep 10
ProdJobBuild:
stage: build
needs: [AC-test]
script:
- echo "Building thing to prod"
With it, the pipeline checks if the job is called by a schedule, and runs.
And if not, stays manual.
*I took the freedom to pick the MR-style of workflow to avoid the double pipelines.
I have to run a pipeline based on some conditional which I want to evaluate in .gitlab-ci.yml config. file. Basically, I want to create jobs based on if a condition is true. Below is my current .gitlab-ci.yml.
# This is a test to run multiple pipeline with sing .gitlab-ci.yml file.
# identifier stage will identify which pipeline (A or B) to run and only jobs
# of that pipeline would be executed and rest would be skipped.
# variables:
# PIPE_TYPE: "$(mkdir identifier; echo 'B' > identifier/type.txt; cat identifier/type.txt)"
# PIPE_TYPE: "B"
stages:
#- identify
- build
- test
#identify:
# stage: identify
# before_script:
# - mkdir "identifier"
# - echo "B" > identifier/type.txt
# script:
# - PIPE_TYPE=$(cat identifier/type.txt)
# - echo $PIPE_TYPE
# artifacts:
# paths:
# - identifier/type.txt
before_script:
# - mkdir "identifier"
# - echo "B" > identifier/type.txt
# - export PIPE_TYPE=$(cat identifier/type.txt)
- export PIPE_TYPE="B"
build_pipeline_A:
stage: build
only:
refs:
- master
variables:
- $PIPE_TYPE == "A"
script:
- echo $PIPE_TYPE
- echo "Building using A."
- mkdir "buildA"
- touch buildA/info.txt
artifacts:
paths:
- buildA/info.txt
build_pipeline_B:
stage: build
only:
refs:
- master
variables:
- $PIPE_TYPE == "B"
script:
- echo "Building using B."
- mkdir "buildB"
- touch buildB/info.txt
artifacts:
paths:
- buildB/info.txt
test_pipeline_A:
stage: test
script:
- echo "Testing A"
- test -f "buildA/info.txt"
only:
refs:
- master
variables:
- $PIPE_TYPE == "A"
dependencies:
- build_pipeline_A
test_pipeline_B:
stage: test
script:
- echo "Testing B"
- test -f "buildB/info.txt"
only:
refs:
- master
variables:
- $PIPE_TYPE == "B"
dependencies:
- build_pipeline_B
Here, I have two pipelines A with jobs build_pipeline_A and test_pipeline_A and second pipeline as B with build_pipeline_B and test_pipeline_B jobs.
First I thought I can create a job identify which would evaluate some logic and write which pipeline to be used in a file (identifier/type.txt) job and update PIPE_TYPE variable. This variable can be used in all the jobs under only:variables testing and would create the job if PIPE_TYPE is equal to job's pipeline type, unfortunately, this didn't work.
In second try, I thought of using global variables and try to evaluate the expression there and set it to PIPE_TYPE this didn't work either.
In my last try I used a before_script which would evaluate the expression and set it in PIPE_TYPE in hopes of on:variables will able to pick PIPE_TYPE value but no luck with this approach too.
I ran out of ideas at this point and decided to post the question.
my test's .gitlab-ci.yaml file, it's a public repo. so please feel free to poke around it.