I use gitflow for my branching feature-branches, develop, master + release. On top, I have multiple environments dev and multiple tests in one account and stage and prod in another. Now I want to have one "runner" in each account and want to deploy to each of these environments with Gitlab CI.
Branches
feature/*
develop
master
release
Environments
Account test - one runner here
dev
test
Account prod - one runner here
stage
prod
I came across environments ... but do not know what benefit they bring beyond visibility, I tinker with tags and could link runners to jobs by tags ... which is quite inflexible.
Can you help me out how to do this without having one job for each environment, resulting in a lot of jobs and a loooong .gitlab-ci.yml?
Here, do we need to have one job for each environment? Since you have 2 account groups, one test (feature, develop, etc.) and the other one as prod (master, release), are you ok with having 2 jobs for each of these?
Related
My Gitlab CI jobs fail because of RAM limitations.
Page https://docs.gitlab.com/ee/user/gitlab_com/index.html says:
All your CI/CD jobs run on n1-standard-1 instances with 3.75GB of RAM, CoreOS and the latest Docker Engine installed.
Below it says:
The gitlab-shared-runners-manager-X.gitlab.com fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don’t run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times.
So, how do I enable these n1-standard-2 runners (which have 7.5 GB RAM)? I've read the docs over and over but can't seem to find any instructions.
Disclaimer: I did not check if you can use them with a project and if they picked up for your gitlab CI/CD - but that is the way, of how you can check for available Runners and their tags, and how to use them. The terminology GitLab projects as well as community forks of them reads like that this is only for Official GitLab projects and their forks - and not for random projects on GitLab.
You can check all the available runners in your projects CI/CD Settings under Runners, and you will see a list of runners there like:
As you can see there are Runners tagged with gitlab-org. Base on the description you can not run them, without using a tag. Hence that you need to adapt your .gitlab-ci.yml file with those appropriate tags.
EG:
job:
tags:
- gitlab-org
see GitLab documentation for tags
I have set up a PR Pipeline in Azure. As part of this pipeline I run a number of regression tests. These run against a regression test database - we have to clear out the database at the start of the tests so we are certain what data is in there and what should come out of it.
This is all working fine until the pipeline runs multiple times in parallel - then the regression database is being written to multiple times and the data returned from it is not what is expected.
How can I stop a pipeline running in parallel - I've tried Google but can't find exactly what I'm looking for.
If the pipeline is running, the the next build should wait (not for all pipelines - I want to set it on a single pipeline), is this possible?
Depending on your exact use case, you may be able to control this with the right trigger configuration.
In my case, I had a pipeline scheduled to kick off every time a Pull Request is merged to the main branch in Azure. The pipeline deployed the code to a server and kicked off a suite of tests. Sometimes, when two merges occurred just minutes apart, the builds would fail due to a shared resource that required synchronisation being used.
I fixed it by Batching CI Runs
I changed my basic config
trigger:
- main
to use the more verbose syntax allowing me to turn batching on
trigger:
batch: true
branches:
include:
- main
With this in place, a new build will only be triggered for main once the previous one has finished, no matter how many commits are added to the branch in the meantime.
That way, I avoid having too many builds being kicked off and I can still use multiple agents where needed.
One way to solve this is to model your test regression database as an "environment" in your pipeline, then use the "Exclusive Lock" check to prevent concurrent "deployment" to that "environment".
Unfortunately this approach comes with several disadvantages inherent to "environments" in YAML pipelines:
you must set up the check manually in the UI, it's not controlled in source code.
it will only prevent that particular deployment job from running concurrently, not an entire pipeline.
the fake "environment" you create will appear in alongside all other environments, cluttering the environment view if you happen to use environments for "real" deployments. This is made worse by this view being a big sack of all environments, there's no grouping or hierarchy.
Overall the initial YAML reimplementation of Azure Pipelines mostly ignored the concepts of releases, deployments, environments. A few piecemeal and low-effort aspects have subsequently been patched in, but without any real overarching design or apparent plan to get to parity with the old release pipelines.
You can use "Trigger Azure DevOps Pipeline" extension by Maik van der Gaag.
It needs to add to you DevOps and configure end of the main pipeline and point to your test pipeline.
Can find more details on Maik's blog.
According to your description, you could use your own self-host agent.
Simply deploy your own self-hosted agents.
Just need to make sure your self host agent environment is the same as your local development environment.
Under this situation, since your agent pool only have one available build agent. When multiple builds triggered, only one build will be running simultaneously. Others will stay in queue with a specific order for agents. Unless the prior build finished, it will not run with next build.
For other pipeline, just need to keep use the host agent pool.
I have a C++ project that is compiled and packaged for multiple OS (Linux, Windows, MacOS) as well as multiple CPU architectures (i386, x86_64, arm, Aarch64)
For this I'm using Jenkins to grab the source code and run the build script in parallel on each system. It's a simple working solution, since my build script deals with the system differences.
Now I'm looking into Gitlab CI/CD, and it has many things I find appealing ( being able to keep the build script as part of the repository, very well integrated with the git repo and ticketing system, natural use of Docker containers, etc), but I cannot find any way to run the same pipeline in multiple architectures/systems parallel to each other.
So, say that my build script is:
build:
stage: build
script:
- uname -m > arch.txt
artifacts:
paths:
- arch.txt
How can I tell Gitlab that I want to run this job in multiple runners/Docker containers/systems at the same time? All the documentation I've read so far deals with running multiple tests on one build, integrating multiple projects or deploying in different environments depending on branches. Nothing I've read so far tries to do many separate builds, test and package them individually and report on their independent results. Is this feasible on Gitlab CI/CD?
GitLab uses "runners" to execute CI jobs. Runners are installed wherever you want to run a CI job, so if you want to run on multiple architectures then you will need to install runners on systems for each architecture. Runner install documentation can be found here:
https://docs.gitlab.com/runner/install/index.html
For Linux-based jobs it is common to use Docker for job execution - this doesn't give architectural flexibility, but it does allow you to test on different flavors and with different software using containerisation. For other architectures you may need to install runners yourself, or use other peoples shared runners.
While you are installing the runner software there are some keys steps:
you have the opportunity to link each runner to your GitLab project, which means it will show up in the runners list under Project > Settings > CI/CD.
you will have the opportunity to assign "tags" to the runners. Tags can be used to help identify a runner or group of runners by an arbitrary name (e.g. you could add "Windows x86_64" as a tag, or "Windows" and "x86_64" tags). These tags can be used in jobs to select a runner.
Once you have your runners installed you can get editing your .gitlab-ci.yml file.
GitLab CI files are broken up into "stages". Jobs in each stage can run in parallel. Stage names are defined at the top of the file.
stages:
- build
- deploy
Each CI job can be attached to a stage by using the stage: entry:
build job:
stage: build
script:
- echo "I am a build stage job"
In your case you will need to create multiple jobs for each architecture you want to build for. Attaching these to the same stage will allow them to run in parallel.
To control where each job runs you have two main mechanisms:
Tags - tags allow you to pin a job to a runner tag. You can specify multiple tags using the tags: entry which forms an AND list (e.g. win tag AND x86_64 tag). When that job runs GitLab will find a runner that has all the required tags, and run the job there.
Image - When running on Docker / Kubernetes you can specify a docker image to use for the runner. To use a docker image you first need to specify a runner that can run docker images (e.g. a docker-in-docker or kubernetes runner), which might, for example, be tagged with docker or kubernetes. Then you use the image: entry to specify the docker image.
Here's an example showing both tags and images:
build win x86_64:
stage: build
tags:
- win
- x86_64
script:
- echo "I am a build stage job for win x86_64"
build win 32:
stage: build
tags:
- win
- 32-bit
script:
- echo "I am a build stage job for win 32"
build debian:
stage: build
tags:
- docker
image: debian:stretch
script:
- echo "I am a build stage job for debian, running on docker using debian:stretch image"
There is currently no support for dynamic jobs, or running one job on multiple runners / architectures, so this involves a bit of manual effort. On the positive side it makes GitLab CI files easy to read, and easy to see what will run during CI execution.
Check out the latest GitLab release 11.5 (Nov. 2018) with:
Parallel attribute for faster pipelines
The speed of pipelines is an important factor for any team, and running tests or other parallelizable tasks tends to take a big chunk of the time for any build.
Adding this new keyword gives teams the ability to simply parallelize tests, allowing everyone to accelerate their software delivery process.
To use this feature, simply set the parallel attribute to how many copies of the task you’d like to split it into, and GitLab will handle the work of automatically creating the appropriate number of jobs running your task.
As commented by March, this is not an exact fit for the OP (which is about running the same pipeline in multiple architectures/systems parallel to each other).
See documentation and issue.
parallel allows you to configure how many instances of a job to run in parallel.
This value has to be greater than or equal to two (2) and less or equal than 50.
This creates N instances of the same job that run in parallel.
They’re named sequentially from job_name 1/N to job_name N/N.
test:
script: rspec
parallel: 5
In high performance computing is crucial to have a code tested against many different architectures/compilers: from a laptop to a supercomputer.
Assuming that we have
N testing machines/workers (each one running gitlab-ci-runner);
M tests,
what shall be the correct layout of .gitlab-ci.yml to ensure that each of the N machines runs all the tests?
Looks to me that adding just more workers ends up in a round-robin like assignment of the jobs.
Thanks for your help.
You could use tags in your .gitlab-ci.yml and on the runners to distribute your tests to all mashines you want. Gitlab CI is very open to those use cases. I assume that you don't use docker for testing.
To accomplish your goal, do following steps:
Install your gitlab-ci-multi-runners on your N mashines BUT if the runner ask you for tags, tag the mashine with a specifc name e.g. "MashineWithManyCPUsWhatever". Please use the gitlab-ci-multi-runner register command to do so. You can alternatively change the tags in Gitlab on the administration view after registration.
The type of the runners should be "shared" not "specific".
When you installed and tagged every runner on your mashines, tag you jobs.
Example:
Every job will now be excecuted on the specific mashine with the specific tag.
For more Information see Gitlab CI Documentation
I'm relatively new to continuous integration servers. I've been using GitLab (v6.5) for a while to manage projects, but I'd like to begin using the GitLab CI to ensure tests pass and builds succeed.
My testing setup consists of two virtual machines: one machine for GitLab and another machine for the GitLab CI (and runners). However, in production I only have a single machine, which is running GitLab. The GitLab team posted an interesting blog post a while back that emphasized:
If you are running tests on the CI server you are doing it wrong!
It was a very informative post, but I didn't come away feeling like I understood this specific point. Does this mean one shouldn't run GitLab and GitLab CI on the same server? Does it mean one shouldn't run GitLab CI and GitLab CI runners on the same server? Or both-- Do I need three servers, one for each task?
From the same post:
Anybody who can push to a branch that is tested on a CI server can easily own that server.
This implies to me that the runners are the security risk since they can run stuff contained in a commit. If that's the case, what's the typical implementation? Put GitLab and GitLab CI on the same machine, but the runners on a separate machine? Wouldn't it still suck if the runner machine was compromised? So people are okay losing their runner machine as long as their code machine is safe?
I would really like to understand this a bit more-- definitely before I implement it in production. Is there any possible yet safe way to implement GitLab, GitLab CI, and GitLab CI runners all on the same machine?
Ideally you're fine running gitlab-ci and gitlab on the same host. Others may disagree with me but the orechestrator (the gitlab-ci node) doesn't do any of the heavy lifting. Its strictly job meta IO and warehousing the results.
With that being said, I would not put the runners on the same machine. Gitlab-CI Runners are resource intensive and will be executing at full tilt on whichever machine you place them on. Its a good idea if you're running in production to put these on spot instances to help curb some of the costs of running the often cpu/memory hungry builds - but can be impractical as your instances are not always on at that point.
I've had some success with putting my gitlab-ci runner's in digital ocean on small instances. I'm not doing HUGE builds, but the idea is to distribute the work load against several servers so your CI server:
Is responsive
Can build multiple project builds at once
Can exercise isolation (this is kind of arbitrary in this list)
and a few other things that don't come to mind right away.
Hope this helps!