Gitlab:14.4.0
Gitlab-Runner:14.4.0 & 13.0.1
After I upgrade Gitlab from 13.6.3 to 14.4.0, all Runner could not pick up jobs
I thought it was Runner version too old，but Gitlab-Runner:14.4.0 on other Linux vm show the same problem
gitlab-runner status looks normal
But when I use gitlab-runner --debug run on each vm, Runner work for few minutes, the runner on that vm could pick up jobs, now I run this command manual every 5 minute to keep other developer working
the problem is we deploy gitlab use docker version image on hand wirte Deployment deploy on Kubernetes, the background jobs fail and stuck many times, after we transfer data to new Gitlb which deploy by helm problem solved
I had the same problem. But I figured it came from the GitLab instance iteself.
The problem was thensolved by upgrading to 14.4.2. I don't think it's the update itself, but the restart of the GitLab instance after the upgrade proved useful.
Ran into a similar issue (GitLab EE v14.2.3-ee) whereby all 3 runners where in a wait state for minutes at a time, before any pending jobs were eventually picked up. Resolved the issue by...
removing all runners,
resetting the config.toml file (which was cluttered with [[runner]] entries from former runners),
restarting GitLab, and
then creating new runners.
Specifically, I performed the following steps...
Delete all runners via "GitLab console > Admin > Overview > Runners".
Log into runner host(s), find config.toml, and clean out the entire file except for...
concurrent=5
check_interval=0
[session_server]
session_timeout = 1800
Log into GitLab EE host and restart GitLab EE.
sudo gitlab-ctl restart
Log into the runner host(s), and create new runners (follow directions from "GitLab console > Admin > Overview > Runners" )
After the above corrective action, the runners now immediately pick up pending jobs.
Related
I added a new virtualbox runner to my gitlab self hosted solution and I'm getting this warning on it:
Runner has never contacted this instance
and it nevers runs any jobs
Bouncing the runner will definitely help, else re-register the runner.
Also, you should check the status of the Runner with the below command.
gitlab-runner status
If you are using the runner in Windows Server, then go to the path where you have stored the .exe file and run the below command:
.\<.exe> status
If the runner is in stopped state, the start the runner by using the same commands but just replace status with start.
I have a Gitlab server from the company where the project and the pipeline are configured. By default, every time a commit is done, the pipeline starts to execute in the Gitlab server.
I have my personalized VM, which is completely different from Gitlab. I want that the pipeline will be executed in my personalized VM instead of the Gitlab server. What should I do so that the pipeline runs on the VM and not on the Gitlab server?
I have configured the following runner in config.toml that is located in $MYPROJECT/:
[[runners]]
name = "Project-name"
url = "https://gitlab.server/"
token = "TOKEN ID"
executor = "shell"
shell = "bash"
There are things that I don't understand.
If I want to execute the pipeline in my personalized VM, should I install Gitlab runner in the VM [1]?
Should I have the project source code in the VM so that it can read the config.toml file every time there is a commit?
If I register the runner with the token key in the Gitlab server, how the Gitlab server knows that the pipeline is to be executed in the VM and not in the server [2]?
Should I use the executor docker or shell, to execute the pipeline in the VM?
[1] https://docs.gitlab.com/runner/install/linux-manually.html
[2] https://docs.gitlab.com/runner/register/#registering-runners
For running a job on a machine you need a GitLab Runner installed on that machine, connected with the GitLab server.
The project source code is fetched automatically in front of every run
You can use a tag (e.g. "MyVM") when registering the runner. Then you can set the same tag into your job so that this job is only executed by this runner. See: https://docs.gitlab.com/ee/ci/runners/configure_runners.html#use-tags-to-control-which-jobs-a-runner-can-run
You need to use docker if you want to use docker in your VM (which needs to be installed before there). Otherwise use shell.
I use Gitlab for doing Continuous Integration and Development and all of a sudden I get this error message "There has been a runner system failure, please try again"
There's no real error message or error code.
I've tried restarting the gitlab runner, using gitlab-runner restart, I've done a reboot of the server its running on but I keep getting this error message on Gitlab whenever I push a code change.
After a couple of hours, I realized the issue is that the server that Gitlab Runner is running on has no space left.
I logged into the server in question. Looked at the Gitlab log file using the following command:
journalctl -u gitlab-runner
And it showed me the following logs:
May 21 08:20:41 gitlab-runner[18936]: Checking for jobs... received job=178911 repo_url=https://.......git runner=f540b942
May 21 08:20:41 gitlab-runner-01 gitlab-runner[18936]: WARNING: Failed to process runner builds=0 error=open /tmp/trace543210445: no space left on device executor=docker runner=f540b942
To fix this issue I ran docker conatiner prune which clears out stopped containers.
Alternatively you could use docker system prune which would remove all unused objects.
See https://linuxize.com/post/how-to-remove-docker-images-containers-volumes-and-networks/ for more information about those docker commands.
Afterwards, I no longer got the error on Gitlab when pushing changes.
When I git push I have to go to the server the gitlab-runner is configured on and enter in the terminal "sudo gitlab-runner run" or "gitlab-runner run". and then the pipeline would start. And I have to be in the runtime-platform all the time. This defeats the point of a pipeline.
It used to work so that when I git pushed, the pipeline would start automatically, I didn't have to enter the command at all. So is there a way to set that up?
There are the following two methods to achieve that.
nohup gitlab-runner run & will keep your runner running as a user process and uses a config file that defaults to /home/<user>/.gitlab-runner/config.toml (see man nohup)
sudo systemctl start gitlab-runner will start as a service, and uses a config file that defaults to /etc/gitlab-runner/config.toml (see man systemctl)
In both cases, you will be able to logout and the runner will stay active.
The gitlab documentation is pretty clear :
GitLab offers a continuous integration service. For each commit or push to trigger your CI pipeline, you must:
Add a .gitlab-ci.yml file to your repository’s root directory.
Ensure your project is configured to use a Runner.
.gitlab-ci.yml part
You need to create a file named .gitlab-ci.yml in the root directory of your repository. The script part depends of what you want to do in the job.
Runner part
You need to install and configure the runner.
The most simple is to use shared runner
I am using Gitlab-CI-Multi-Runner on Windows (64 bit), and have configured it to run with the default shell. The connectivity with the Gitlab server is fine, but when it tries to run a build, it is cloning the repository using HTTP endpoint. My repo is quite big, and the Gitlab NGINX fails with an error. SSH endpoint works perfectly.
How do I make my GitLab-CI-Multi-Runner use the SSH endpoint when cloning the repo for executing a build?
Note: The same problem existed with the older Gitlab-CI-Runner for Windows which was written using C#.
The problem is related to the timeout of the unicorn engine workers, that is too short to allow big repositories to be cloned/pushed.
in /etc/gitlab/gitlab.rb uncomment and set
unicorn['worker_timeout'] = 1200
(hopefully enough).
then issue
# gitlab-ctl reconfigure
# gitlab-ctl restart
that works for me.