I am facing an out of memory error in one of my CI/CD pipelines, so I would like to customise the configuration of my GitLab's shared runners, for example by using a config.toml file. I would also like to avoid self-hosting a GitLab Runner instance, if possible.
Is there a way of doing that ?
As far as I know, there is no way of changing the config.
However, according to this doc, I can pick from 3 machine sizes up to 16GB RAM, and add the appropriate tag at the job level in my gitlab-ci.yaml.
Note that this will impact the CI/CD minutes cost factor.
For GitLab Premium and Ultimate (os, not free), you do have GitLab 15.4 (September 2022), which comes with:
More powerful Linux machine types for GitLab SaaS runners
When you run jobs on GitLab SaaS Linux runners, you now have access to more powerful machine types: medium and large. With these two machine types, you have more choices for your GitLab SaaS CI/CD jobs. And with 100% job isolation on an ephemeral virtual machine, and security and autoscaling fully managed by GitLab, you can confidently run your critical CI/CD jobs on GitLab SaaS.
See Documentation and Issue.
Related
I'm new to CI/CD.
I installed Gitlab Runner on my VPS for my project.
The first pipeline was successfully passed after the push to the master branch.
Questions:
Gitlab Runner does not listen to any port. Instead, it connects to the Gitlab server itself and checks if there are tasks for it. Is that true?
If so, how often does Gitlab Runner connect to gitlab.com?
Is it dangerous to keep Gitlab Runner and the project on the same VPS?
Gitlab Runner does not listen to any port. Instead, it connects to the Gitlab server itself and checks if there are tasks for it. Is that true?
Yes. It is a "pull" mechanism exclusively, as far as the jobs go. Though the runner may open ports for additional functionality, such as the session server.
If so, how often does Gitlab Runner connect to gitlab.com?
By default, every 3 seconds to check for new jobs (assuming the concurrency limit has not been reached). This can be configured through the check_interval configuration
Is it dangerous to keep Gitlab Runner and the project on the same VPS?
It can be, so this should be avoided where possible. The degree of danger depends somewhat on which executor you use and how you configure your server.
The file system of your GitLab server is extremely sensitive; it contains many critical secrets and operational components that should not be exposed to anyone (including CI jobs). Besides this, jobs would also have the potential to cause performance problems -- if a job uses too much memory/cpu/IO/etc., it could cause resource contention and harm/crash your other GitLab components on the server.
You might be able to get away with a single VPS if you deploy the server and runner as separate containers with a Docker or Helm deployment of GitLab and the runner on the VPS host; you can use separate logical Docker volumes and set resource constraints via Docker... but there is still some additional risk in such a scenario and configuration mistakes (which may be easier to make than one might think) can have serious consequences.
My Gitlab CI jobs fail because of RAM limitations.
Page https://docs.gitlab.com/ee/user/gitlab_com/index.html says:
All your CI/CD jobs run on n1-standard-1 instances with 3.75GB of RAM, CoreOS and the latest Docker Engine installed.
Below it says:
The gitlab-shared-runners-manager-X.gitlab.com fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don’t run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times.
So, how do I enable these n1-standard-2 runners (which have 7.5 GB RAM)? I've read the docs over and over but can't seem to find any instructions.
Disclaimer: I did not check if you can use them with a project and if they picked up for your gitlab CI/CD - but that is the way, of how you can check for available Runners and their tags, and how to use them. The terminology GitLab projects as well as community forks of them reads like that this is only for Official GitLab projects and their forks - and not for random projects on GitLab.
You can check all the available runners in your projects CI/CD Settings under Runners, and you will see a list of runners there like:
As you can see there are Runners tagged with gitlab-org. Base on the description you can not run them, without using a tag. Hence that you need to adapt your .gitlab-ci.yml file with those appropriate tags.
EG:
job:
tags:
- gitlab-org
see GitLab documentation for tags
How can I determine what is the minimum hardware requirement for install the gitlab-runner on a host?
I want a specific runner which will run tests, build docker images, and deploy the apps.
Gitlab documentation gives some advices and put focus on memory :
Depending on how you decide to configure GitLab Runner and what tools you use to exercise your application in the CI environment, GitLab Runner can consume significant amount of available memory.
After that, it's really depending the type of application you want to test/build/deploy. Test and build job will be the largest resource consumers. You can also set limits on cpu/memory for docker runner.
I have a self-hosted gitlab enterprise server (11.2.3-ee), and run CI builds for C++ in docker images. There is a pool of runners on different physical hosts, and for the most part, any job can be scheduled on any runner. The physical hosts are not identical--some are faster than others.
The admin section provides a way to view statistics at a job level. The admin page on the server (at https://gitlab.example.com/admin/jobs) shows this, but I cannot find anything in the API. I've tried looking in the source but am not familiar enough with the organization to have found anything yet.
Is there API access for job stats across the entire instance (not just project level)? Or, am I constrained to scraping the admin screen?
I'm relatively new to continuous integration servers. I've been using GitLab (v6.5) for a while to manage projects, but I'd like to begin using the GitLab CI to ensure tests pass and builds succeed.
My testing setup consists of two virtual machines: one machine for GitLab and another machine for the GitLab CI (and runners). However, in production I only have a single machine, which is running GitLab. The GitLab team posted an interesting blog post a while back that emphasized:
If you are running tests on the CI server you are doing it wrong!
It was a very informative post, but I didn't come away feeling like I understood this specific point. Does this mean one shouldn't run GitLab and GitLab CI on the same server? Does it mean one shouldn't run GitLab CI and GitLab CI runners on the same server? Or both-- Do I need three servers, one for each task?
From the same post:
Anybody who can push to a branch that is tested on a CI server can easily own that server.
This implies to me that the runners are the security risk since they can run stuff contained in a commit. If that's the case, what's the typical implementation? Put GitLab and GitLab CI on the same machine, but the runners on a separate machine? Wouldn't it still suck if the runner machine was compromised? So people are okay losing their runner machine as long as their code machine is safe?
I would really like to understand this a bit more-- definitely before I implement it in production. Is there any possible yet safe way to implement GitLab, GitLab CI, and GitLab CI runners all on the same machine?
Ideally you're fine running gitlab-ci and gitlab on the same host. Others may disagree with me but the orechestrator (the gitlab-ci node) doesn't do any of the heavy lifting. Its strictly job meta IO and warehousing the results.
With that being said, I would not put the runners on the same machine. Gitlab-CI Runners are resource intensive and will be executing at full tilt on whichever machine you place them on. Its a good idea if you're running in production to put these on spot instances to help curb some of the costs of running the often cpu/memory hungry builds - but can be impractical as your instances are not always on at that point.
I've had some success with putting my gitlab-ci runner's in digital ocean on small instances. I'm not doing HUGE builds, but the idea is to distribute the work load against several servers so your CI server:
Is responsive
Can build multiple project builds at once
Can exercise isolation (this is kind of arbitrary in this list)
and a few other things that don't come to mind right away.
Hope this helps!