How can I determine what is the minimum hardware requirement for install the gitlab-runner on a host?
I want a specific runner which will run tests, build docker images, and deploy the apps.
Gitlab documentation gives some advices and put focus on memory :
Depending on how you decide to configure GitLab Runner and what tools you use to exercise your application in the CI environment, GitLab Runner can consume significant amount of available memory.
After that, it's really depending the type of application you want to test/build/deploy. Test and build job will be the largest resource consumers. You can also set limits on cpu/memory for docker runner.
Related
I am facing an out of memory error in one of my CI/CD pipelines, so I would like to customise the configuration of my GitLab's shared runners, for example by using a config.toml file. I would also like to avoid self-hosting a GitLab Runner instance, if possible.
Is there a way of doing that ?
As far as I know, there is no way of changing the config.
However, according to this doc, I can pick from 3 machine sizes up to 16GB RAM, and add the appropriate tag at the job level in my gitlab-ci.yaml.
Note that this will impact the CI/CD minutes cost factor.
For GitLab Premium and Ultimate (os, not free), you do have GitLab 15.4 (September 2022), which comes with:
More powerful Linux machine types for GitLab SaaS runners
When you run jobs on GitLab SaaS Linux runners, you now have access to more powerful machine types: medium and large. With these two machine types, you have more choices for your GitLab SaaS CI/CD jobs. And with 100% job isolation on an ephemeral virtual machine, and security and autoscaling fully managed by GitLab, you can confidently run your critical CI/CD jobs on GitLab SaaS.
See Documentation and Issue.
I am new to Gitlab, and bit confuse with the Gitlab Image keyword..
If we already have gitlab runners specified to run the commands they why are we separately use Image keywords in the gitlab-ci.yml (in Any stage or Jobs)enter image description here
One important concept to be aware of is that there are a few different executors that a gitlab-runner may use. On gitlab.com, a docker+machine executor is used. The docker/kubernetes executors are the only context where the image: keyword is relevant.
When jobs run in these executor types, they do not run directly in the host context of the runner. Instead, a docker container is created to run each job. For this, a Docker image must be used. The default image works for general use cases, but you may want to choose a different ready-made image for your use. This can make your environment more repeatable as well as reduce job times by avoiding time that would otherwise need to be spent installing software requirements.
For example, if I have a job that builds python software, I may choose to use image: "python:3.10.2" because it will contain the Python runtime and tools I need for building Python software.
By contrast, if you are using a shell executor, your jobs execute directly in the host -- you can simply use software available on the runner host itself and there is no image: to choose.
My Gitlab CI jobs fail because of RAM limitations.
Page https://docs.gitlab.com/ee/user/gitlab_com/index.html says:
All your CI/CD jobs run on n1-standard-1 instances with 3.75GB of RAM, CoreOS and the latest Docker Engine installed.
Below it says:
The gitlab-shared-runners-manager-X.gitlab.com fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don’t run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times.
So, how do I enable these n1-standard-2 runners (which have 7.5 GB RAM)? I've read the docs over and over but can't seem to find any instructions.
Disclaimer: I did not check if you can use them with a project and if they picked up for your gitlab CI/CD - but that is the way, of how you can check for available Runners and their tags, and how to use them. The terminology GitLab projects as well as community forks of them reads like that this is only for Official GitLab projects and their forks - and not for random projects on GitLab.
You can check all the available runners in your projects CI/CD Settings under Runners, and you will see a list of runners there like:
As you can see there are Runners tagged with gitlab-org. Base on the description you can not run them, without using a tag. Hence that you need to adapt your .gitlab-ci.yml file with those appropriate tags.
EG:
job:
tags:
- gitlab-org
see GitLab documentation for tags
We have created our own self hosted agents on an AKS cluster.
We have 3 replicas, using a StatefulSet which mounts a Persistent Volume (PV) in order to use docker's cache. This means that /var/lib/docker is a mount on each replica.
Up until everything works fine.
However, it seems like that agent-0 does most of the work while agent-1 and agent-2 are working way less. This causes a problem, as PV-0 is almost using it's entire capacity and PV-1 and PV-2 aren't:
agent-0
/dev/sdc 98.3G 83.8G 14.5G 85% /var/lib/docker
agent-1
/dev/sdd 98.3G 35.3G 63.0G 36% /var/lib/docker
agent-2
/dev/sdc 98.3G 2.1G 96.2G 2% /var/lib/docker
Is there a way to control Azure DevOps' pipelines so that the load will be spread between the agents instead of always using agent-0?
We have ended up with a different solution, moving from Docker in docker (dind) to Docker outside of docker (DooD).
The main reason that we initially wanted to use our own self hosted agents was the need for docker layer caching, as it takes too much time to build docker images using the hosted agents.
At some point, we figured that we would have to support ARM based images, and start to use docker buildx which made us use the docker dind solution.
We are aware of the risk that is included by mounting the docker.sock:
In order to use Docker from within a Docker container, you need to bind-mount the Docker socket. This has very serious security implications - namely, code inside the container can now run as root on your Docker host. If you're sure you want to do this, see the bind mount documentation on Docker.com.
https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/docker?view=azure-devops#using-docker-within-a-docker-container
But for the time being, this solves our issue as we won't have to manage PVs per pods, but rather use the same docker to build all of our non-ARM images.
A side note: as for ARM, we would still be using a different agent, which will be running docker buildx in order to build docker images.
What you could try is to configure multi-agent for your agent job and type 3 for your 3 agents, if you want to spread the load:
I also think that it will only consume the amount of agents corresponding to your parallel job count for your organization. As stated here:
Each agent can run only one job at a time. To run multiple jobs in
parallel you must configure multiple agents. You also need sufficient
parallel jobs.
EDIT
The above suggested method works only with tasks who support slicing. Otherwise it will just parallel the same execution. In case your tasks dont support slicing, but you have independent Tasks in your pipeline, you can split the Agent Job in 2 or more, without setting the Dependency on previous job, and the jobs will run in parallel if you have sufficient parallel jobs.
Is there a way to control Azure DevOps' pipelines so that the load will be spread between the agents instead of always using agent-0?
I am afraid there is no such out of box way to control the build/release workloads to be evenly distributed to all agents, if we could not use the feature multi-agent.
As workaround, we could clone the our pipeline/job into three, each with different demands. Then set custom capabilities defined for agents:
Check this thread for some more details.
Hope this helps.
I am new to gitlab CI and I am fascinated with it. I managed already to get the pipelines working even using docker containers, so I am familiar with the flow for setting jobs and artifacts. I just wish now to understand how this works. My questions are about the following:
Runners
Where is actually everything happening? I mean, which computer is running my builds and executables? I understand that Gitlab has its own shared runners that are available to the users, does this mean that if a shared runner grabs my jobs, is it going to run wherever those runners are hosted? If I register my own runner in my laptop, and use that specific runner, my builds and binaries will be run in my computer?
Artifacts
In order to run/test code, we need the binaries, which from the build stage they are grabbed as artifacts. For the build part if I use cmake, for example, in the script part of the CI.yml file I create a build directory and call cmake .. and so on. Once my job is succesful, if I want the binary i have to go in gitlab and retrieve it myself. So my question is, where is everything saved? I notice that the runner, withing my project, creates something like refs/pipeline/, but where is this actually? how could I get those files and new directories in my laptop
Working space
Pretty much, where is everything happening? the runners, the execution, the artifacts?
Thanks for your time
Everything that happens in each job/step in a pipeline happens on the runner host itself, and depends on the executor you're using (shell, docker, etc.), or on the Gitlab server directly.
If you're using gitlab.com, they have a number of shared runners that the Gitlab team maintains and you can use for your project(s), but as they are shared with everyone on gitlab.com, it can be some time before your jobs are run. However, no matter if you self host or use gitlab.com, you can create your own runners specific for your project(s).
If you're using the shell executor, while the job is running you could see the files on the filesystem somewhere, but they are cleaned up after that job finishes. It's not really intended for you to access the filesystem while the job is running. That's what the job script is for.
If you're using the docker executor, the gitlab-runner service will start a docker instance from the image you specify in .gitlab-ci.yml (or use the default that is configurable). Then the job is run inside that docker instance, and it's deleted immediately after the job finishes.
You can add your own runners anywhere -- AWS, spare machine lying around, even your laptop, and jobs would be picked up by any of them. You can also turn off shared runners and force it to be run on one of your runners if needed.
In cases where you need an artifact after a build/preparatory step, it's created on the runner as part of the job as above, but then the runner automatically uploads the artifact to the gitlab server (or another service that implements the S3 protocol like AWS S3 or Minio). Unless you're using S3/minio, it will only be accessible through the gitlab UI interface, or through the API. In the UI however, it will show up on any related MR's, and also the Pipeline page, so it's fairly accessible.