I have a Gitlab-CI use case that I can't properly figure out. The background is that I have multiple environments that need to have their own artifacts built and deployed. Let's say a simplified case looks like this:
stages:
- build
- deploy
AppOne:Build:
extends: .build
environment:
name: AppOne
AppOne:Deploy:
extends: .deploy
needs: ['AppOne:Build']
environment:
name: AppOne
AppTwo:Build:
extends: .build
environment:
name: AppTwo
AppTwo:Deploy:
extends: .deploy
needs: ['AppTwo:Build']
environment:
name: AppTwo
.build:
stage: build
script:
- build.sh
.deploy:
stage: deploy
when: manual
script:
- deploy.sh
The process above works, but as you can see it's kind of long and repetitious with two apps and super simple process. The real process has three stages and up to 20 apps. I would like to know if there's a better way to express this? The build processes are identical save for a variable (set in the environment).
I guess I mainly want to know if there's a better way to express the "needs" without repeating it for each app, i.e. the app needs to be built before it can be manually deployed. I could do needs: ['${NAME}:Build'] but that requires the $NAME:Build step to actually exist.
Please help, oh collective wisdom of the Internet.
Related
I need to pass a file path to a trigger job where the file path is found within a specified json file in a separate job. Something along the lines of this...
stages:
- run_downstream_pipeline
variables:
- FILE_NAME: default_file.json
.get_path:
stage: run_downstream_pipeline
needs: []
only:
- schedules
- triggers
- web
script:
- apt-get install jq
- FILE_PATH=$(jq '.file_path' $FILE_NAME)
run_pipeline:
extends: .get_path
variables:
PATH: $FILE_PATH
trigger:
project: my/project
branch: staging
strategy: depend
I can't seem to find any workaround to do this, as using extends won't work since Gitlab wont allow for a script section in a trigger job.
I thought about trying to use the Gitlab API trigger method, but I want the status of the downstream pipeline to actually show up in the pipeline UI and I want the upstream pipeline to depend on the status of the downstream pipeline, which from my understanding is not possible when triggering via the API.
Any advice would be appreciated. Thanks!
You can use artifacts:reports:dotenv for setting variables dynamically for subsequent jobs.
stages:
- one
- two
my_job:
stage: "one"
script:
- FILE_PATH=$(jq '.file_path' $FILE_NAME) # In script, get the environment URL.
- echo "FILE_PATH=${FILE_PATH}" >> variables.env # Add the value to a dotenv file.
artifacts:
reports:
dotenv: "variables.env"
example:
stage: two
script: "echo $FILE_PATH"
another_job:
stage: two
trigger:
project: my/project
branch: staging
strategy: depend
Variables in the dotenv file will automatically be present for jobs in subsequent stages (or that declare needs: for the job).
You can also pull artifacts into child pipelines, in general.
But be warned you probably don't want to override the PATH variable, since that's a special variable used to help you find builtin binaries.
I have a .gitlab-ci.yml pipeline with a simple job that needs to run in several environments. Something similar to the following:
test:v1.0:
stage: test
environment:
name: v1.0
tags:
- v1.0
script:
- ./run.sh $VERSION
test:v2.0:
stage: test
environment:
name: v2.0
tags:
- v2.0
script:
- ./run.sh $VERSION
test:v2.5:
stage: test
environment:
name: v2.5
tags:
- v2.5
script:
- ./run.sh $VERSION
Does GitLab have any kind of mechanism to create a job by iterating an array? Something similar to Ansible's loops. The idea is to avoid copy-pasting the same job over and over, when only environment or runner tag changes. I couldn't see anything in documentation, and all feature requests about this topic seem closed. Is there any workaround to achieve the same behaviour and accepted as best practice by the community?
I know from other questions here that one proposed solution could be:
test:all:
stage: test
script:
- Iterate here with v1.0, v2.0, v2.5, etc
The issue with this approach is that only one job is created, also you lose the ability to choose runners and other capabilities from GitLab environments feature, so I'd rather avoid this one.
With the possibility of using variables in tags which was implemented just recently and the use of a parallel matrix you can do the following:
test:
stage: test
script:
- ./run.sh $VERSION
environment:
name: $VERSION
tags:
- $VERSION
parallel:
matrix:
- VERSION: [v1.0, v2.0, v3.0]
This will create a job for the three defined versions and each job will run in parallel.
I am trying to set up a CI with minimal code duplication using .gitlab-ci.yml.
With that, I am separating the configuration in separate files and reusing parts of it that are common.
I have a separate repository with Gitlab CI settings: gitlab-ci and several projects that use it to form their own CI pipelines.
Contents of gitlab-ci repository
template_jobs.yml:
.sample:
rules:
- if: '$CI_PIPELINE_SOURCE == "push"'
when: on_success
- when: never
jobs_architectureA.yml:
include:
- local: '/template_jobs.yml'
.script_core: &script_core
- echo "Running stage"
test_archA:
extends:
- .sample
stage: test
tags:
- architectureA
script:
- *script_core
jobs_architectureB.yml:
include:
- local: '/template_jobs.yml'
.script_core: &script_core
- echo "Running stage"
test_archB:
extends:
- .sample
stage: test
tags:
- architectureB
script:
- *script_core
Project with code contents:
In the actual project (separate repositories per project, and I have a lot of them), I have the following:
.gitlab-ci.yml:
stages:
- test
include:
- project: 'gitlab-ci'
file: '/jobs_architectureA.yml'
- project: 'gitlab-ci'
file: '/jobs_architectureB.yml'
This configuration works fine and allows to include only some architectures for some modules while sharing rules between the job templates.
However, it's easy to notice one code duplication: both jobs_architectureA.yml and jobs_architectureB.yml contain a common section:
.script_core: &script_core
- echo "Running stage"
It would be ideal to move it into a separate file: template_scripts.yml and include from both jobs_architectureA.yml* and jobs_architectureB.yml. However, that results in the invalid YAML (at least from Gitlab's point of view).
With that, I make a conclusion that I can share the rules as the mechanism of their usage is via extends keyword; however, I am not able to do it with the scripts: as it uses &/* anchoring mechanic on the YAML level.
Ideally, I want something along the lines of:
Contents of the ideal (conceptually) gitlab-ci repository
template_jobs.yml:
.sample:
rules:
- if: '$CI_PIPELINE_SOURCE == "push"'
when: on_success
- when: never
template_scripts.yml:
.script_core: &script_core
- echo "Running stage"
jobs_architectureA.yml:
include:
- local: '/template_jobs.yml'
- local: '/template_scripts.yml'
test_archA:
extends:
- .sample
stage: test
tags:
- architectureA
script:
- *script_core # this becomes invalid, as script_core is in the other file, even though it is included at the top
jobs_architectureB.yml:
include:
- local: '/template_jobs.yml'
- local: '/template_scripts.yml'
test_archB:
extends:
- .sample
stage: test
tags:
- architectureB
script:
- *script_core # this becomes invalid, as script_core is in the other file, even though it is included at the top
Am I doing something wrong?
Am I hitting a limitation of the Gitlab mechanic? Is it the implementation of the include directive in this specific YML type, that limits me?
Do I have options to achieve something close to the desired behaviour?
Note, while this might not look like a big deal, in reality, I have many more pieces to the scripts, and the actual script is much larger. Thus, currently, it is duplicated code all over the place which is very prone to mistakes.
my solution is to not include template_jobs.yml and template_scripts.yml directly in jobs_architectureA.yml but only in the "final" .gitlab-ci.yml
taking you exemple, /template_jobs.yml//template_scripts.yml do not change.
jobs_architectureA.yml loses the include:
test_archA:
extends:
- .sample
stage: test
tags:
- architectureB
script:
- *script_core # this becomes invalid, as script_core is in the other file, even though it is included at the top
and .gitlab-ci.yml becomes:
stages:
- test
include:
- local: '/template_jobs.yml'
- local: '/template_scripts.yml'
- project: 'gitlab-ci'
file: '/jobs_architectureA.yml'
- project: 'gitlab-ci'
file: '/jobs_architectureB.yml'
in reality, I have many more pieces to the scripts, and the actual script is much larger
Adding to Cyril's solution, GitLab 13.12 (May 2021) can help scale those includes:
Support wildcards when including YAML CI/CD configuration files
The includes: keyword for CI/CD pipelines lets you break one long .gitlab-ci.yml file into multiple smaller files to increase readability.
It also makes it easier to reuse configuration in multiple places.
Frequently there are multiple files included into a single pipeline, and they all might be stored in the same place.
In this release, we add support to use the * wildcard with the local includes: keyword. You can now make your includes: sections more dynamic, less verbose, and easier to read, check out how we are dogfooding it in GitLab.
See Documentation and Issue.
The extends: keyword can be used with multiple jobs to extend from (on recent versions of GitLab). You can simply do the following:
include:
- local: '/template_jobs.yml'
.script_core:
- echo "Running stage"
test_archA:
extends:
- .sample
- .script_core
stage: test
tags:
- architectureA
And after that move the .script_core to an import.
I'm trying to improve the project building script, described in YML-file, the improvement itself seems quite trivial, but the idea of accidentally ruining the auto-builds scares me a bit.
Right now there are several branches, version tags and other stuff in the project.
A development branch, not built by the runners would be of use, because copying a huge project somehow between virtual machines to test the build on different platforms is not convenient at all. So, I want to exclude from builds some "prj-dev" branch.
And there we have:
stages:
- build
- linuxbuild
job:
tags:
- win2008build
stage: build
script:
# something complex
job1:
tags:
- linux
stage: linuxbuild
script:
# something else complex
I googled and found a solution like:
stages:
- build
- linuxbuild
job:
tags:
- win2008build
branches:
except:
- *dev-only
But it seems that our pipelines are quite different, the tags are not git tags, they are pipeline tags. So, I'm considering rather to use a config like:
stages:
- build
- linuxbuild
job:
tags:
- win2008build
except:
branches:
- *dev-only
...which would mean "build as usual, but not my branch". There are complications in trying it both ways, I'm pretty sure someone should know the recipe for sure.
So, if you please, -- how do I exclude my dev branch without changing the pipelines, config only? Is it possible at all?
All you need to do is use except in the gitlab-ci.yml file and add your branches directly below like this:
mybuild:
stage: test
image: somedockerimage
script:
- some script running
except:
- branch-name
This is working on my project without problems.
We have a project hosted on an internal Gitlab installation.
The Pipeline of the project has 3 stages:
Build
Tests
Deploy
The objective is to hide or disable the Deploy stage when Tests fails
The problem is that we can't use artifacts because they are lost each time our machines reboot.
My question: Is there an alternative solution to artifacts to achieve this task?
The used .gitlab-ci.yml looks like this:
stages:
- build
- tests
- deploy
build_job:
stage: build
tags:
# - ....
before_script:
# - ....
script:
# - ....
when: manual
only:
- develop
- master
all_tests:
stage: tests
tags:
# - ....
before_script:
# - ....
script:
# - ....
when: manual
only:
- develop
- master
prod:
stage: deploy
tags:
# - ....
script:
# - ....
when: manual
environment: prod
I think you might have misunderstood the purpose of the built-in CI. The goal is to have building and testing all automated on each commit or at least every push. Having all tasks set to manual execution gives you almost no advantage over external CI tools like Jenkins or Bamboo. Your only advantage to local execution of the targets right now is having visibility in a central place.
That said there is no way to conditionally show or hide CI tasks, because it's against the basic idea. If you insist on your idea, you could look up the artifacts of the previous stages and abort the manual execution in case something is wrong.
The problem is that we can't use artifacts because they are lost each time our machines reboot
AFAIK artifacts are uploaded to the master and not saved on the runners. You should be fine having your artifacts passed from stage to stage.
By the way, the default for when is on_success which means to execute build only when all builds from prior stages succeed.