I have a question regarding Gitlab runners here. I've looked and searched quite much but haven't found any relevant information yet.
I'm going to update the Gitlab server for the company I work for, and we have many Gitlab runners.
There will be a down time for the Gitlab server update and I was wondering what will happen to the active Gitlab runners during this downtime?
Will the Gitlab runners function as normal after the update or will I need to register all runners again?
Anyone familiar with this?
Thanks :)
They should work again as normal, but it depends how exactly the runners are set up. Where are the runners deployed?
E.g. for local development purposes i have deployed some runners locally via docker and mix their usage with the shared "company" runners.
Related
I am facing an out of memory error in one of my CI/CD pipelines, so I would like to customise the configuration of my GitLab's shared runners, for example by using a config.toml file. I would also like to avoid self-hosting a GitLab Runner instance, if possible.
Is there a way of doing that ?
As far as I know, there is no way of changing the config.
However, according to this doc, I can pick from 3 machine sizes up to 16GB RAM, and add the appropriate tag at the job level in my gitlab-ci.yaml.
Note that this will impact the CI/CD minutes cost factor.
For GitLab Premium and Ultimate (os, not free), you do have GitLab 15.4 (September 2022), which comes with:
More powerful Linux machine types for GitLab SaaS runners
When you run jobs on GitLab SaaS Linux runners, you now have access to more powerful machine types: medium and large. With these two machine types, you have more choices for your GitLab SaaS CI/CD jobs. And with 100% job isolation on an ephemeral virtual machine, and security and autoscaling fully managed by GitLab, you can confidently run your critical CI/CD jobs on GitLab SaaS.
See Documentation and Issue.
I'm new to CI/CD.
I installed Gitlab Runner on my VPS for my project.
The first pipeline was successfully passed after the push to the master branch.
Questions:
Gitlab Runner does not listen to any port. Instead, it connects to the Gitlab server itself and checks if there are tasks for it. Is that true?
If so, how often does Gitlab Runner connect to gitlab.com?
Is it dangerous to keep Gitlab Runner and the project on the same VPS?
Gitlab Runner does not listen to any port. Instead, it connects to the Gitlab server itself and checks if there are tasks for it. Is that true?
Yes. It is a "pull" mechanism exclusively, as far as the jobs go. Though the runner may open ports for additional functionality, such as the session server.
If so, how often does Gitlab Runner connect to gitlab.com?
By default, every 3 seconds to check for new jobs (assuming the concurrency limit has not been reached). This can be configured through the check_interval configuration
Is it dangerous to keep Gitlab Runner and the project on the same VPS?
It can be, so this should be avoided where possible. The degree of danger depends somewhat on which executor you use and how you configure your server.
The file system of your GitLab server is extremely sensitive; it contains many critical secrets and operational components that should not be exposed to anyone (including CI jobs). Besides this, jobs would also have the potential to cause performance problems -- if a job uses too much memory/cpu/IO/etc., it could cause resource contention and harm/crash your other GitLab components on the server.
You might be able to get away with a single VPS if you deploy the server and runner as separate containers with a Docker or Helm deployment of GitLab and the runner on the VPS host; you can use separate logical Docker volumes and set resource constraints via Docker... but there is still some additional risk in such a scenario and configuration mistakes (which may be easier to make than one might think) can have serious consequences.
My Gitlab CI jobs fail because of RAM limitations.
Page https://docs.gitlab.com/ee/user/gitlab_com/index.html says:
All your CI/CD jobs run on n1-standard-1 instances with 3.75GB of RAM, CoreOS and the latest Docker Engine installed.
Below it says:
The gitlab-shared-runners-manager-X.gitlab.com fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don’t run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times.
So, how do I enable these n1-standard-2 runners (which have 7.5 GB RAM)? I've read the docs over and over but can't seem to find any instructions.
Disclaimer: I did not check if you can use them with a project and if they picked up for your gitlab CI/CD - but that is the way, of how you can check for available Runners and their tags, and how to use them. The terminology GitLab projects as well as community forks of them reads like that this is only for Official GitLab projects and their forks - and not for random projects on GitLab.
You can check all the available runners in your projects CI/CD Settings under Runners, and you will see a list of runners there like:
As you can see there are Runners tagged with gitlab-org. Base on the description you can not run them, without using a tag. Hence that you need to adapt your .gitlab-ci.yml file with those appropriate tags.
EG:
job:
tags:
- gitlab-org
see GitLab documentation for tags
I am trying to host a new project with Gitlab. It is a private Python project. I was able to test some initial tests with Gitlab CI.
I don't use cache while running tests,
While exploring the runner section in settings, there is a warning shown,
GitLab Runners do not offer secure isolation between projects that
they do builds for. You are TRUSTING all GitLab users who can push
code to project A, B or C to run shell scripts on the machine hosting
runner X.
what are the security risks in using a shared test runner? Is it safe to run private projects on a shared runner? What precautions can be taken while running tests on a shared runner?
Thank you for any insight.
GitLab CI runner offers the following executor types:
shell
docker
ssh
docker-ssh
parallels
virtualbox
The security concerns you should have are mainly from using ssh and shell runners.
shell is unsafe unless you're in a controlled environment.
This is because it's, literally, a simple shell. The user running your build will have access to everything else going on for that user, and that includes other projects.
ssh is susceptible to man-in-the-middle attacks.
If you're dealing with private crypto keys in your builds, beware that they may be stolen.
Fortunately, http://gitlab.com seems to be sharing only docker runners.
docker runners are generally safe* because every build runs in a new container, so there's nothing to worry.
You can read further about GitLab CI Runner security here.
* unless you're doing the nasty privileged mode!
Using Gitlab shared runners does not comes at no risk. As Gitlab caches the content of the repos where the runners are running, your code is still stored somewhere in the Gitlab's instances. Whoever can have access to these instances as a user capable to read the cache folder, can access to yours.
If you're already running Gitlab as SaaS, I guess it doesn't make a big difference.
I'm relatively new to continuous integration servers. I've been using GitLab (v6.5) for a while to manage projects, but I'd like to begin using the GitLab CI to ensure tests pass and builds succeed.
My testing setup consists of two virtual machines: one machine for GitLab and another machine for the GitLab CI (and runners). However, in production I only have a single machine, which is running GitLab. The GitLab team posted an interesting blog post a while back that emphasized:
If you are running tests on the CI server you are doing it wrong!
It was a very informative post, but I didn't come away feeling like I understood this specific point. Does this mean one shouldn't run GitLab and GitLab CI on the same server? Does it mean one shouldn't run GitLab CI and GitLab CI runners on the same server? Or both-- Do I need three servers, one for each task?
From the same post:
Anybody who can push to a branch that is tested on a CI server can easily own that server.
This implies to me that the runners are the security risk since they can run stuff contained in a commit. If that's the case, what's the typical implementation? Put GitLab and GitLab CI on the same machine, but the runners on a separate machine? Wouldn't it still suck if the runner machine was compromised? So people are okay losing their runner machine as long as their code machine is safe?
I would really like to understand this a bit more-- definitely before I implement it in production. Is there any possible yet safe way to implement GitLab, GitLab CI, and GitLab CI runners all on the same machine?
Ideally you're fine running gitlab-ci and gitlab on the same host. Others may disagree with me but the orechestrator (the gitlab-ci node) doesn't do any of the heavy lifting. Its strictly job meta IO and warehousing the results.
With that being said, I would not put the runners on the same machine. Gitlab-CI Runners are resource intensive and will be executing at full tilt on whichever machine you place them on. Its a good idea if you're running in production to put these on spot instances to help curb some of the costs of running the often cpu/memory hungry builds - but can be impractical as your instances are not always on at that point.
I've had some success with putting my gitlab-ci runner's in digital ocean on small instances. I'm not doing HUGE builds, but the idea is to distribute the work load against several servers so your CI server:
Is responsive
Can build multiple project builds at once
Can exercise isolation (this is kind of arbitrary in this list)
and a few other things that don't come to mind right away.
Hope this helps!