I'm trying to improve the project building script, described in YML-file, the improvement itself seems quite trivial, but the idea of accidentally ruining the auto-builds scares me a bit.
Right now there are several branches, version tags and other stuff in the project.
A development branch, not built by the runners would be of use, because copying a huge project somehow between virtual machines to test the build on different platforms is not convenient at all. So, I want to exclude from builds some "prj-dev" branch.
And there we have:
stages:
- build
- linuxbuild
job:
tags:
- win2008build
stage: build
script:
# something complex
job1:
tags:
- linux
stage: linuxbuild
script:
# something else complex
I googled and found a solution like:
stages:
- build
- linuxbuild
job:
tags:
- win2008build
branches:
except:
- *dev-only
But it seems that our pipelines are quite different, the tags are not git tags, they are pipeline tags. So, I'm considering rather to use a config like:
stages:
- build
- linuxbuild
job:
tags:
- win2008build
except:
branches:
- *dev-only
...which would mean "build as usual, but not my branch". There are complications in trying it both ways, I'm pretty sure someone should know the recipe for sure.
So, if you please, -- how do I exclude my dev branch without changing the pipelines, config only? Is it possible at all?
All you need to do is use except in the gitlab-ci.yml file and add your branches directly below like this:
mybuild:
stage: test
image: somedockerimage
script:
- some script running
except:
- branch-name
This is working on my project without problems.
Related
I have a Gitlab-CI use case that I can't properly figure out. The background is that I have multiple environments that need to have their own artifacts built and deployed. Let's say a simplified case looks like this:
stages:
- build
- deploy
AppOne:Build:
extends: .build
environment:
name: AppOne
AppOne:Deploy:
extends: .deploy
needs: ['AppOne:Build']
environment:
name: AppOne
AppTwo:Build:
extends: .build
environment:
name: AppTwo
AppTwo:Deploy:
extends: .deploy
needs: ['AppTwo:Build']
environment:
name: AppTwo
.build:
stage: build
script:
- build.sh
.deploy:
stage: deploy
when: manual
script:
- deploy.sh
The process above works, but as you can see it's kind of long and repetitious with two apps and super simple process. The real process has three stages and up to 20 apps. I would like to know if there's a better way to express this? The build processes are identical save for a variable (set in the environment).
I guess I mainly want to know if there's a better way to express the "needs" without repeating it for each app, i.e. the app needs to be built before it can be manually deployed. I could do needs: ['${NAME}:Build'] but that requires the $NAME:Build step to actually exist.
Please help, oh collective wisdom of the Internet.
I am setting up Gitlab CI for the first time, and I want it to create two releases for each commit. My .gitlab-ci.yml file looks like this:
stages:
- compile
- test
- build release
compile apps:
stage: compile
script:
- scons
artifacts:
paths:
- deliverables/
check version:
stage: test
script:
- check_version.sh
build releasefile:
stage: build release
script:
- build_release.sh
artifacts:
paths:
- release/
For my second version, I want to run scons in compile apps with a flag (scons --special) and then run all next jobs as well on those deliverables. My deliverables are named the same for both versions, and if I just create jobs for both the normal and special version, my "check version" job will check the normal version twice. My options:
Create a really long pipeline that runs everything of the normal version and then everything of the special version. I don't like this solution, it looks hideous and can make errors less visible when the pipeline is expanded later.
Change my scons and shell scripts.
Create two pipelines on each commit, one with a Gitlab CI flag and one without (I don't know how to do this).
Create a "split" pipeline that only uses stuff from the job that it is based on (I don't know how to do this).
For the last case, my pipeline would look something like this:
-----+------ Compile normal ----- Check version ----- Build releasefile
|
+----- Compile special ----- Check version ----- Build releasefile
I would prefer option 3 or 4 and I've been looking at Directed Acyclic Graph Pipelines, but I can't get those to work in the way I want. Is there a way to do either of these?
You can do this by creating a DAG pipeline with needs. If you don't use needs (or the older dependencies), all artifacts from previous stages will be downloaded, which in this case is problematic due to the overlap in the artifact folders / names.
You can also use extends to avoid duplication in your job declarations. The full .gitlab-ci.yml config could be something like this:
stages:
- compile
- test
- build release
compile apps:
stage: compile
script:
- scons
artifacts:
paths:
- deliverables/
compile apps special:
extends:
- compile apps
script:
- scons --special
check version:
stage: test
script:
- check_version.sh
needs:
- compile apps
check version special:
extends:
- check version
needs:
- compile apps special
build releasefile:
stage: build release
script:
- build_release.sh
artifacts:
paths:
- release/
needs:
- compile apps
- check version
build releasefile special:
extends:
- build releasefile
needs:
- compile apps special
- check version special
extends works well in this context because it doesn't combine YAML list items, but instead overwrites them (keys with different names would get merged, on the other hand). So in this case, the whole script and needs declarations get overwritten by the inheriting job.
I am trying to add needs between jobs in the Gitlab CI yaml configuration file.
stages:
- build
- test
- package
- deploy
maven-build:
stage: build
only:
- merge_requests
- master
- branches
...
test:
stage: test
needs: [ "maven-build" ]
only:
- merge_requests
- master
...
docker-build:
stage: package
needs: [ "test" ]
only:
- master
...
deploy-stage:
stage: deploy
needs: [ "docker-build" ]
only:
- master
...
deploy-prod:
stage: deploy
needs: [ "docker-build" ]
only:
- master
when: manual
...
I have used the GitLab CI online lint tools to check my syntax, it is correct.
But when I pushed the codes, it always complains:
'test' job needs 'maven-build' job
but it was not added to the pipeline
You can also test your .gitlab-ci.yml in CI Lint
The GitLab CI did not run at all.
Update: Finally I made it. I think the needs position is sensitive, move all needs under the stage, it works. My original scripts included some other configuration between them.
CI-jobs that depend on each other need to have the same limitations!
In your case that would mean to share the same only targets:
stages:
- build
- test
maven-build:
stage: build
only:
- merge_requests
- master
- branches
test:
stage: test
needs: [ "maven-build" ]
only:
- merge_requests
- master
- branches
that should work from my experience^^
Finally I made it. I think the needs position is sensitive, move all needs under the stage, it works
Actually... that might no longer be the case with GitLab 14.2 (August 2021):
Stageless pipelines
Using the needs keyword in your pipeline configuration helps to reduce cycle times by ignoring stage ordering and running jobs without waiting for others to complete.
Previously, needs could only be used between jobs on different stages.
In this release, we’ve removed this limitation so you can define a needs relationship between any job you want.
As a result, you can now create a complete CI/CD pipeline without using stages by including needs in every job to implicitly configure the execution order.
This lets you define a less verbose pipeline that takes less time to create and can run even faster.
See Documentation and Issue.
The rule in both jobs should be that same or otherwise GitLab cannot create job dependency between the jobs when the trigger rule is different.
I don't know why, but if the jobs are in different stages (as in my case), you have to define the jobs that will be done later with "." at the start.
Another interesting thing is GitLab's own CI/CD Lint online text editor does not complain there is an error. So you have to start the pipeline to see the error.
Below, notice the "." in ".success_notification" and ".failure_notification"
stages:
- prepare
- build_and_test
- deploy
- notification
#SOME CODE
build-StandaloneWindows64:
<<: *build
image: $IMAGE:$UNITY_VERSION-windows-mono-$IMAGE_VERSION
variables:
BUILD_TARGET: StandaloneWindows64
.success_notification:
needs:
- job: "build-StandaloneWindows64"
artifacts: true
stage: notification
script:
- wget https://raw.githubusercontent.com/DiscordHooks/gitlab-ci-discord-webhook/master/send.sh
- chmod +x send.sh
- ./send.sh success $WEBHOOK_URL
when: on_success
.failure_notification:
needs:
- job: "build-StandaloneWindows64"
artifacts: true
stage: notification
script:
- wget https://raw.githubusercontent.com/DiscordHooks/gitlab-ci-discord-webhook/master/send.sh
- chmod +x send.sh
- ./send.sh failure $WEBHOOK_URL
when: on_failure
#SOME CODE
I am trying to set up a CI with minimal code duplication using .gitlab-ci.yml.
With that, I am separating the configuration in separate files and reusing parts of it that are common.
I have a separate repository with Gitlab CI settings: gitlab-ci and several projects that use it to form their own CI pipelines.
Contents of gitlab-ci repository
template_jobs.yml:
.sample:
rules:
- if: '$CI_PIPELINE_SOURCE == "push"'
when: on_success
- when: never
jobs_architectureA.yml:
include:
- local: '/template_jobs.yml'
.script_core: &script_core
- echo "Running stage"
test_archA:
extends:
- .sample
stage: test
tags:
- architectureA
script:
- *script_core
jobs_architectureB.yml:
include:
- local: '/template_jobs.yml'
.script_core: &script_core
- echo "Running stage"
test_archB:
extends:
- .sample
stage: test
tags:
- architectureB
script:
- *script_core
Project with code contents:
In the actual project (separate repositories per project, and I have a lot of them), I have the following:
.gitlab-ci.yml:
stages:
- test
include:
- project: 'gitlab-ci'
file: '/jobs_architectureA.yml'
- project: 'gitlab-ci'
file: '/jobs_architectureB.yml'
This configuration works fine and allows to include only some architectures for some modules while sharing rules between the job templates.
However, it's easy to notice one code duplication: both jobs_architectureA.yml and jobs_architectureB.yml contain a common section:
.script_core: &script_core
- echo "Running stage"
It would be ideal to move it into a separate file: template_scripts.yml and include from both jobs_architectureA.yml* and jobs_architectureB.yml. However, that results in the invalid YAML (at least from Gitlab's point of view).
With that, I make a conclusion that I can share the rules as the mechanism of their usage is via extends keyword; however, I am not able to do it with the scripts: as it uses &/* anchoring mechanic on the YAML level.
Ideally, I want something along the lines of:
Contents of the ideal (conceptually) gitlab-ci repository
template_jobs.yml:
.sample:
rules:
- if: '$CI_PIPELINE_SOURCE == "push"'
when: on_success
- when: never
template_scripts.yml:
.script_core: &script_core
- echo "Running stage"
jobs_architectureA.yml:
include:
- local: '/template_jobs.yml'
- local: '/template_scripts.yml'
test_archA:
extends:
- .sample
stage: test
tags:
- architectureA
script:
- *script_core # this becomes invalid, as script_core is in the other file, even though it is included at the top
jobs_architectureB.yml:
include:
- local: '/template_jobs.yml'
- local: '/template_scripts.yml'
test_archB:
extends:
- .sample
stage: test
tags:
- architectureB
script:
- *script_core # this becomes invalid, as script_core is in the other file, even though it is included at the top
Am I doing something wrong?
Am I hitting a limitation of the Gitlab mechanic? Is it the implementation of the include directive in this specific YML type, that limits me?
Do I have options to achieve something close to the desired behaviour?
Note, while this might not look like a big deal, in reality, I have many more pieces to the scripts, and the actual script is much larger. Thus, currently, it is duplicated code all over the place which is very prone to mistakes.
my solution is to not include template_jobs.yml and template_scripts.yml directly in jobs_architectureA.yml but only in the "final" .gitlab-ci.yml
taking you exemple, /template_jobs.yml//template_scripts.yml do not change.
jobs_architectureA.yml loses the include:
test_archA:
extends:
- .sample
stage: test
tags:
- architectureB
script:
- *script_core # this becomes invalid, as script_core is in the other file, even though it is included at the top
and .gitlab-ci.yml becomes:
stages:
- test
include:
- local: '/template_jobs.yml'
- local: '/template_scripts.yml'
- project: 'gitlab-ci'
file: '/jobs_architectureA.yml'
- project: 'gitlab-ci'
file: '/jobs_architectureB.yml'
in reality, I have many more pieces to the scripts, and the actual script is much larger
Adding to Cyril's solution, GitLab 13.12 (May 2021) can help scale those includes:
Support wildcards when including YAML CI/CD configuration files
The includes: keyword for CI/CD pipelines lets you break one long .gitlab-ci.yml file into multiple smaller files to increase readability.
It also makes it easier to reuse configuration in multiple places.
Frequently there are multiple files included into a single pipeline, and they all might be stored in the same place.
In this release, we add support to use the * wildcard with the local includes: keyword. You can now make your includes: sections more dynamic, less verbose, and easier to read, check out how we are dogfooding it in GitLab.
See Documentation and Issue.
The extends: keyword can be used with multiple jobs to extend from (on recent versions of GitLab). You can simply do the following:
include:
- local: '/template_jobs.yml'
.script_core:
- echo "Running stage"
test_archA:
extends:
- .sample
- .script_core
stage: test
tags:
- architectureA
And after that move the .script_core to an import.
We have a project hosted on an internal Gitlab installation.
The Pipeline of the project has 3 stages:
Build
Tests
Deploy
The objective is to hide or disable the Deploy stage when Tests fails
The problem is that we can't use artifacts because they are lost each time our machines reboot.
My question: Is there an alternative solution to artifacts to achieve this task?
The used .gitlab-ci.yml looks like this:
stages:
- build
- tests
- deploy
build_job:
stage: build
tags:
# - ....
before_script:
# - ....
script:
# - ....
when: manual
only:
- develop
- master
all_tests:
stage: tests
tags:
# - ....
before_script:
# - ....
script:
# - ....
when: manual
only:
- develop
- master
prod:
stage: deploy
tags:
# - ....
script:
# - ....
when: manual
environment: prod
I think you might have misunderstood the purpose of the built-in CI. The goal is to have building and testing all automated on each commit or at least every push. Having all tasks set to manual execution gives you almost no advantage over external CI tools like Jenkins or Bamboo. Your only advantage to local execution of the targets right now is having visibility in a central place.
That said there is no way to conditionally show or hide CI tasks, because it's against the basic idea. If you insist on your idea, you could look up the artifacts of the previous stages and abort the manual execution in case something is wrong.
The problem is that we can't use artifacts because they are lost each time our machines reboot
AFAIK artifacts are uploaded to the master and not saved on the runners. You should be fine having your artifacts passed from stage to stage.
By the way, the default for when is on_success which means to execute build only when all builds from prior stages succeed.