I've 3 stages:
- provision
- cpp tests
- python tests
I need to run provision before running tests. Gitlab suggests using artifacts to pass result between stages but I'm afraid it's not possible in my scenario since ansible does lots of different stuff (not just generate a few config files/binaries). Ideally, I'd like to be able to run all three stages in the same container because in my scenario stages are logical and essentially can be merged into one. I'd want to avoid it as this would make .gitlab-ci.yml harder to understand.
If you have 3 tasks which can be merged into one and what you want to achieve is only to have 3 separated functions running in the same container to make the .gitlab-ci.yml file easier to understand, I would recommend using yaml anchors (see below).
.provision_template: &provision_definition
- XXX
.cpp_tests_template: &cpp_tests_definition
- YYY
.python_tests_template: &python_tests_definition
- ZZZ
my_job:
script:
- *provision_definition
- *cpp_tests_definition
- *python_tests_definition
Related
I'm absolutely new to all this ci/cd thing and its documentation is too extensive. So i'd like to apologize in advance if there is too easy question.
I have gitlab repo with many configs for different services i.e:
project_folder
--global_conf.json
--service1
----config_folder
------....
--service2
----config_folder
------....
--service3
----config_folder
------....
I'd like to know what should I do (just general plan and some key words to search documentation more precisely already would be fine) to send to hosts [host1:port, host2:port, host3:port] different strings with commands "check {service_name} for changes" or "check {global_config} for changes" e.t.c, depending on the file that was committed.
I already have my service on each host that could perform different operations for different task's strings, so I need just send them that task.
You'll propably end up using "only: changes" as explained here.
Something like this should fit:
stages:
- trigger
service1:
stage: trigger
script:
- curl http://host1:port1
only:
changes:
- service1/*
service2:
stage: trigger
script:
- curl http://host2:port2
only:
changes:
- service2/*
Lets say I have a test block for all my microservices (15-20+). Tests take a long time since there are so many disparate modules in this monorepo.
Lets say I only want to run 1 or maybe 2 at a time if and only if specific code changes have been made underneath a path. How can I best do this? For assembling I do something like this (not sure if this is terrible or not)
Ultimately, I'm trying to only build and test relevant things if they're relevant (based on if they or a related module I can define change)
Module-specific assembles
x:
stage: build
image: gradle:6.0-jdk11
script:
- gradle :x:assemble
artifacts:
paths:
- x/build/libs
only:
changes:
- x
- x/*
- x/*/**
y_build:
stage: build
image: gradle:6.0-jdk11
script:
- gradle :y:assemble
artifacts:
paths:
- y/build/libs
only:
changes:
- y
- y/*
- y/*/**
Current block for testing
test:
stage: test
image: gradle:6.0-jdk11
services:
- name: gitlab-registry.company.com/nap/dynamodb-local:1252954
command: [ "-inMemory", "-sharedDb" ]
alias: dynamodb
script: gradle check
There are many ways that a Gitlab CI Pipeline can be triggered, but the basic way is when a commit is pushed, no matter what the change is. Currently, there isn't a way to inspect which parts of the code are changed, and only run some steps or others based on the result, but you can control which steps run based on the branch or tag name.
So for example, you could have steps that only run when the branch name starts with something like "microservice_3_", and then make sure that when editing Microservice 3, you always start the branch name with "microserver_3_", though this could get complicated the more microservices you have to support.
The easier option (in terms of the pipeline definition) is to maintain the microservices in separate repositories, each with their own pipelines and tests. This way each individual pipeline only runs against a specific component and doesn't care about changes to the others. Then if you need to, you can combine the microservices in another repository (included as git submodules) and have a pipeline that only cares about the services as a whole.
This would add extra overhead while developing the project(s), but it makes the pipelines easier to manage.
I am planning to schedule jobs on runners based on variable from pipeline web UI. I have two runners registered for a project and currently both are on same machine for now.Both runners are having the same tags so differentiation from tags is not possible too.
This is my yaml file
stages :
- common
- specific
test1:
stage: common
tags:
- runner
- linux
script:
- echo $CI_RUNNER_DESCRIPTION
- echo "This is run on all runners"
test2:
stage: specific
tags:
- runner
- linux
script: echo "This is run on runner 1"
only:
variables:
- $num == $CI_RUNNER_DESCRIPTION
test3:
stage: specific
tags:
- runner
- linux
script: echo "This is run on runner 2"
only:
variables:
- $num == $CI_RUNNER_DESCRIPTION
So variable on which the selection happens is "num". The description of the runners can be 1 or 2.
The default value for num is 0 and according to variable passed from pipeline UI, the jobs are to be selected and run.
But when I execute the test with num 1 or 2, only test1 gets executed which is common to all runners.
Is such a implementation possible or am I facing issue because the runners are on same machine ?
Using variables only is not the right approach to select runners. Right now you select one of the 2 runners and try not to execute job if runner description doesn't match. Even if this worked it wouldn't make much sense as you don't know if the job will be executed or not.
I suggest you add specific labels for runners and specify which job should run on which runner.
test2:
stage: specific
tags:
- runner_1
script: echo "This is run on runner 1"
test3:
stage: specific
tags:
- runner_2
script: echo "This is run on runner 2"
If you want to select runner for the job based on UI input, maybe you can add variable to tags section. To be honest I didn't test such solution and donno if it's gonna work. Yet still it would require to create specific tags for your runners.
tags:
- $tag_variable_selected_from_UI
You cannot do this currently with gitlab. As you noted in your comment on the accepted answer, this will be available when https://gitlab.com/gitlab-org/gitlab/-/issues/35742 is resolved.
That being said, while the "accepted" answer correctly states "only" is the incorrect way to do this, it suggests another way that also doesnt work.
Once dynamic tagging support is added into gitlab-ci, this will be possible. Currently, the solution is to provide unique jobs each with their own tags.
Depending on what exactly you are trying to accomplish, you can keep your code as DRY as possible for now by adding job templates in combination with extends to reduce duplication to a couple of lines per job.
I know that you can reuse blocks of code in a before script using yaml anchors:
.something_before: &something_before
- echo 'something before'
before_script:
- *something_before
- echo "Another script step"
but this doesn't seem to work when the .something_before is declared in a shared .yml file via the include:file. It also does not seem that extends works for before_script. Does anyone know a way of reusing some steps in a before_script from a shared .yml file?
EDIT: My use case is that I have 2 gitlab projects with almost identical before_script steps. I don't want to have to change both projects whenever there's a change, so I have a third, separate Gitlab project that has a .yml template that I am including via include:file in both projects. I want to put all the common code in that shared template, and just have like two lines before_script for the git project that has the two extra steps.
Yaml anchors don't work with included files. You need to use the extends keyword. But what you want to achieve won't work with before_script as code in your template will be overwritten in the job which uses the template if there is a before_script as well.
Do you really need a before_script in your specific job or can you achieve the same with a normal script? If yes you can do something like this:
Template File:
.something_before:
before_script:
- echo 'something before'
- echo 'something more before'
Project Pipeline:
include:
- project: 'my-group/my-project'
file: '/something_before.yml'
stages:
- something
something:
stage: something
extends: .something_before
script:
- echo 'additional stuff to do'
And your before_script section will be merged into the something job and executed before the script part.
See if GitLab 13.6 (November 2020) does make it easier:
Include multiple CI/CD configuration files as a list
Previously, when adding multiple files to your CI/CD configuration using the include:file syntax, you had to specify the project and ref for each file. In this release, you now have the ability to specify the project, ref, and provide a list of files all at once. This prevents you from having to repeat yourself and makes your pipeline configuration less verbose.
See Documentation) and Issue.
And even, with GitLab 14.9 (March 2022):
Include the same CI/CD template multiple times
Previously, trying to have standard CI/CD templates that you reuse in many places was complicated because each template could only be included in a pipeline once.
We dropped this limitation in this release, so you can include the same configuration file as many times as you like.
This makes your CI/CD configuration more flexible as you can define identical includes in multiple nested configurations, and rest assured that there will be no conflicts or duplication.
See Documentation and Issue.
You can use extends without any problem, but you will need to overwrite the entire before_script block.
If you want to change just a piece of your before_script, use a shell script to do it
Set the if condition inside of your template
before_script
- |
if [ condition ]
then
commands here
fi
AFTER EDIT: You can use variables to achieve it
Project 1: VAR = command 1
Project 2: VAR = command 2
You can set the content of env var on the gitlab-ci.yml file or on the CI/CD settings in each project!
Is there any way to extend and run only a specific job from another pipeline in my current pipeline without copy-pasting it?
For example I have two pipelines:
1. build -> code_check -> auto_test -> deploy
2. auto_test* -> report
I want to execute pipeline 2 where auto_test* executes on another runner while keeping the job's keys exactly as they are in pipeline 1 (except for "tags" which I add in the job to be able to use another runner).
I have a process restriction that I can't change anything in pipeline 1 config so I need a way to execute only a specific job.
I have tried to do that through include .gitlab-ci.yaml+extends:. It somewhat works but pipeline 2 will have all jobs from both pipelines and it is not what I would like to see.
The most straightforward way would be just to copy on each update auto_test job specification from pipeline 1 into my gitlab-ci YAML of pipeline 2 and adding tags: ["MyRunner"] but I hoped there is a built-in way to do that.