Gitlab CI 9.5 service is not running - linux

I am searching a solution since 2 weeks on the web and I really need some help.
I am facing 3 problems:
Linux Gitlab-runner is not running
I have been trying to install gilab-runner with all the ways (GitLab's official repository, manualy, docker).
Everytime, when I am launching the command "gitlab-runner status" the answer is always "The server is not running." I have tried a million times to uninstall the service and re-install it but I do not want to work. I have register runners of all kind and with/without the sudo user. Without any success. This is my setup server:
Config
Ubuntu 16.04.1
Docker container gitlab 9.4.3
Port:
webservice :8088
https : 4433
ssh : 2222
gitlab-runner 9.5.0
How to reproduce
Register a shell runner http://192.168.1.10:8088/
Launch the command "sudo service gitlab-runner status"
Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since ven. 2017-08-25 15:17:45 CEST; 45s ago
Process: 13201 ExecStart=/usr/bin/gitlab-ci-multi-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --syslog --user gitlab-runner (code=exited, status=1/FAILURE)
Main PID: 13201 (code=exited, status=1/FAILURE)
systemd1: gitlab-runner.service: Unit entered failed state.
systemd1: gitlab-runner.service: Failed with result 'exit-code'.
Windows gitlab-runner Error 500
Because of my problem to install gitlab-runner in Linux, I have tried to install it on another computer on Windows 10.
It worked and finally the commande gitlab-runner status answered me "Service is running" (but this is just a temporary solution, I really need to make it work on linux).
Anyway, I have added a CI script to a test program and launch the job but it was turning in loop over and over.
When I launch the command "gitlab-runner --debug run":
...
passfile: true
extension: cmd
job=183 project=19 runner=679ccd01
Using Shell executor... job=183 project=19 runner=679ccd01
Waiting for signals... job=183 project=19 runner=679ccd01
WARNING: Job failed: exit status 128 job=183 project=19 runner=679ccd01
WARNING: Submitting job to coordinator... failed job=183 runner=679ccd01 status=500 Internal Server Error
WARNING: Submitting job to coordinator... failed job=183 runner=679ccd01 status=500 Internal Server Error
...
Gitlab.com and run command
So I have decided to add my project on gitlab.com, to test it.
git#gitlab.com:sandbox_test/test_ci.git
Once again the job was turning in infinite loop until I launch on my Windows computer the command "gitlab-runner run".
Dialing: tcp gitlab.com:443 ...
Feeding runners to channel builds=0
Checking for jobs... received job=30315630 repo_url=https://gitlab.com/sandbox_test/test_ci.git runner=d98c0af1
Failed to requeue the runner: builds=1 runner=d98c0af1
Running with gitlab-ci-multi-runner 9.5.0 (413da38)
on Windows_shell_gitlab_com (d98c0af1) job=30315630 project=3992201 runner=d98c0af1
Shell configuration: environment: []
dockercommand: []
command: cmd
arguments:
- /C
passfile: true
extension: cmd
job=30315630 project=3992201 runner=d98c0af1
Using Shell executor... job=30315630 project=3992201 runner=d98c0af1
Waiting for signals... job=30315630 project=3992201 runner=d98c0af1
Job succeeded job=30315630 project=3992201 runner=d98c0af1
Why is it necessary to launch the run command to make work my job on gitlab.com?
I expect when I run a new job it will figure out by itself without to launch manually the gitlab-runner on the CI computer...
Script .gitlab-ci.yml
Validate on CI Lint
stages:
- build
- test
- deploy
build:
stage: build
script:
- echo "building"
test:
stage: test
script:
- echo "test"
I really need answers very fast, thanks for your help.
Best Regards,Clement
UPDATE 1
I have resoved a part of my problems :
Linux Gitlab-runner is not running
Launch the command "gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --syslog --user gitlab-runner"
First Error : chdir /home/gitlab-runner: no such file or directory
Solution: sudo mkdir /home/gitlab-runner
Second Error : open /etc/gitlab-runner/config.toml: permission denied
Solution : sudo chmod 755 /etc/gitlab-runner/config.toml

I have resoved a part of my problems :
Linux Gitlab-runner is not running
Launch the command "gitlab-runner run --working-directory /home/gitlab-runner --config /etc/gitlab-runner/config.toml --service gitlab-runner --syslog --user gitlab-runner"
First Error : chdir /home/gitlab-runner: no such file or directory
Solution: sudo mkdir /home/gitlab-runner
Second Error : open /etc/gitlab-runner/config.toml: permission denied
Solution : sudo chmod 755 /etc/gitlab-runner/config.toml

Related

Gitlab runner status show "the service is not installed" but it was installed and run successfully

I installed Gitlab-runner in WSL Ubuntu 20. Firstly, I have registered runner and ran successfully. But when I check the status of Gitlab-runner it show:
namlb#Admin:/etc/init.d$ sudo gitlab-runner status
Runtime platform arch=amd64 os=linux pid=2170 revision=e0218c92
version=14.3.2
gitlab-runner: the service is not installed
And another thing that confused me is that my runner just execute job when I already run gitlab-runner run command, whilst in some guides they just need run gitlab-runner start and job will be execute when triggered.
Here is some info about my gitlab-runner:
namlb#Admin:~$ sudo gitlab-runner start
[sudo] password for namlb:
Runtime platform arch=amd64 os=linux pid=2563 revision=e0218c92 version=14.3.2
namlb#Admin:~$ gitlab-runner status
Runtime platform arch=amd64 os=linux pid=2609 revision=e0218c92 version=14.3.2
FATAL: The --user is not supported for non-root users
namlb#Admin:~$ sudo gitlab-runner status
Runtime platform arch=amd64 os=linux pid=2635 revision=e0218c92 version=14.3.2
gitlab-runner: the service is not installed
namlb#Admin:~$ gitlab-runner verify
Runtime platform arch=amd64 os=linux pid=2686 revision=e0218c92 version=14.3.2
WARNING: Running in user-mode.
WARNING: The user-mode requires you to manually start builds processing:
WARNING: $ gitlab-runner run
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...
Verifying runner... is alive runner=22WNSZ6v
Run "service gitlab-runner status" for more information.
When I got the message "gitlab-runner: the service is not installed" I realized that the file /usr/lib/gitlab-runner/gitlab-runner had incorrect permissions.
for Ubuntu
Change your /etc/init.d/gitlab-runner
...
DAEMON="/usr/bin/gitlab-runner"
DESC="GitLab Runner"
USER="gitlab-runner"
...
--exec "$DAEMON" -- "run" "--working-directory" "/var/lib/gitlab-runner" "--config" "/etc/gitlab-runner/config.toml" "--service" "gitlab-runner" "--syslog" "--user" "$USER"
...
Variable $DAEMON not defined and is used
status)
status_of_proc -p "$PIDFILE" "$DAEMON" "$DESC"
;;
*)

Failed to connect to containerd: failed to dial

Just installed Docker CE following official instructions with the repository in Ubuntu 14.04
Installation went successfully, the daemon is running
$ ps aux | grep docker
[...] /usr/bin/dockerd --raw-logs [...]
My user is in the docker group:
$ groups
[...] docker
The cli can't seem to communicate (same with sudo)
$ docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock.
Is the docker daemon running?
The socket seems to have the correct permissions:
$ ls -l /var/run/docker.sock
srw-rw---- 1 root docker 0 Feb 4 16:21 /var/run/docker.sock
The log seems to claim about some issues though
$ sudo tail -f /var/log/upstart/docker.log
Failed to connect to containerd: failed to dial "/var/run/docker/containerd/docker-containerd.sock": dial unix:///var/run/docker/containerd/docker-containerd.sock: timeout
/var/run/docker.sock is up
time="2018-02-04T16:22:21.031459040+01:00" level=info msg="libcontainerd: started new docker-containerd process" pid=17147
INFO[0000] starting containerd module=containerd revision=89623f28b87a6004d4b785663257362d1658a729 version=v1.0.0
INFO[0000] setting subreaper... module=containerd
containerd: invalid argument
time="2018-02-04T16:22:21.056685023+01:00" level=error msg="containerd did not exit successfully" error="exit status 1" module=libcontainerd
Any advice to make this work ?
Relog and Docker restart already done of course
As #bobbear suggested and is actually mentioned in the official doc one of the prerequisites is:
Version 3.10 or higher of the Linux kernel. The latest version of the kernel available for you platform is recommended.
After having checked my Kernel version:
$ uname -a
Linux [...] 3.2.[...]-generic [...]-Ubuntu [...] x86_64
I searched for candidates:
$ apt-cache search linux-image
And installed my new_kernel:
$ sudo apt-get install \
linux-image-new_kernel \
linux-headers-new_kernel \
linux-image-extra-new_kernel
Same situation happend on me. IS because your linux kernel version too low !!! check it use command "uname -r" , if the version below "3.10" (for example: debian 7 whezzy default version is 3.2 ) ,even you install docker-ce suceessfully, you will still can not start docker daemon success.That why! All most answers on the web tell you to 'restart' bla bla bla... but they did not consider this problem.

What starts this docker process on my laptop?

Every time I boot up my Lubuntu 16.04 laptop I can see I have a running docker container:
$ ps -ef | grep docker
root 1724 1 3 21:17 ? 00:01:30 /usr/bin/dockerd -H fd://
root 1774 1724 0 21:17 ? 00:00:04 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
root 4750 1774 0 21:17 ? 00:00:00 docker-containerd-shim 72541a4648b890132985daf2357d1130b8b5208cf12ede607b93ab2987629719 /var/run/docker/libcontainerd/72541a4648b890132985daf2357d1130b8b5208cf12ede607b93ab2987629719 docker-runc
stephane 10755 1793 0 22:07 pts/0 00:00:00 grep docker
It serves a Jenkins application on the port 80 and requesting localhost/ in the browser redirects to http://localhost/login?from=%2F and shows a Jenkins warning page:
Unlock Jenkins
To ensure Jenkins is securely set up by the administrator, a password has been written to the log (not sure where to find it?) and this file on the server:
A wget request shows:
$ wget localhost/
--2017-05-23 22:09:55-- http://localhost/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2017-05-23 22:09:55 ERROR 403: Forbidden.
How can I know which service is firing up this docker process ?
I looked in the /etc/init.d/ directory:
$ l /etc/init.d/
alsa-utils* checkroot-bootclean.sh* halt* mattermostd* nginxd* rc* single* uuidd*
anacron* checkroot.sh* hostname.sh* mountall-bootclean.sh* ntp* rc.local* skeleton whoopsie*
apachedsd* console-setup* httpd* mountall.sh* ondemand* rcS* ssh* x11-common*
apparmor* cron* hwclock.sh* mountdevsubfs.sh* openvpn* README tomcatd*
apport* cups* irqbalance* mountkernfs.sh* php-fpm* reboot* udev*
avahi-daemon* cups-browsed* keyboardd* mountnfs-bootclean.sh* plymouth* redis* ufw*
bluetooth* dbus* killprocs* mountnfs.sh* plymouth-log* resolvconf* umountfs*
bootmisc.sh* docker* kmod* mysqld* postfix* rsync* umountnfs.sh*
cgroupfs-mount* dropboxd* lightdm* networking* pppd-dns* rsyslog* umountroot*
checkfs.sh* grub-common* mariadbd* network-manager* procps* sendsigs* urandom*
The /etc/init.d/docker is mine and removing it from the directory, a reboot still comes up with a running docker process.
I removed the /etc/init.d/docker file, rebooted, and there is a docker process:
$ ps -ef | grep docker
root 1560 1 5 22:15 ? 00:00:06 /usr/bin/dockerd -H fd://
root 1645 1560 0 22:15 ? 00:00:00 docker-containerd -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metrics-interval=0 --start-timeout 2m --state-dir /var/run/docker/libcontainerd/containerd --shim docker-containerd-shim --runtime docker-runc
root 4644 1645 0 22:15 ? 00:00:00 docker-containerd-shim 069db46cca05d43c35f05ff50aaa836507cbf69e4e3d9443b6b859d0edb5b076 /var/run/docker/libcontainerd/069db46cca05d43c35f05ff50aaa836507cbf69e4e3d9443b6b859d0edb5b076 docker-runc
stephane 5520 1741 0 22:17 pts/0 00:00:00 grep docker
So I looked up for anything docker in all these files, but found nothing named docker:
$ cd /etc/init.d/
[stephane#stephane-ThinkPad-X301 init.d]
$ grep.sh docker
[stephane#stephane-ThinkPad-X301 init.d]
This docker process is there every time I start my laptop, even when off line.
What starts this docker process ?
Lubuntu 16.04 comes with systemd by default. At some point you must have started up a jenkins instance in docker - it's hard to tell exactly what started the process initially. However, systemd would be what is currently causing it to start. In order to stop it from running, run the following commands:
systemctl status docker <- Find out of systemctl thinks docker is running.
It'll likely show something like this:
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
Active: active (running) since Sun 2017-05-21 22:59:46 EDT; 1 day 17h ago
Docs: http://docs.docker.com
Main PID: 1314 (dockerd-current)
Tasks: 14 (limit: 8192)
CGroup: /system.slice/docker.service
└─1314 /usr/bin/dockerd-current --add-runtime oci=/usr/libexec/docker/docker-runc-current --default-runtime=oci --containerd /run/containerd.sock --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --selinux-enabled --log-driver=journald
To stop it, run systemctl stop docker and then systemctl disable docker. As a last resort if this doesn't work, you can run systemctl mask docker.
Docker is being started by systemd in your environment. You can disable the entire engine by running:
sudo systemctl disable docker
sudo systemctl stop docker
You can also stop only the container that is running (the shim and Jenkins application):
sudo docker ps # lists the running containers along with their container id
sudo docker update --restart=no $container_id
sudo docker stop $container_id
If you know that you do not need this container and want to permanently delete it, you can run this instead of the above two last commands:
sudo docker rm -f $container_id
The -f switch also stops the container if it's currently running.
Edit: from your comment, your container is running under swarm mode which is redeploying it. To stop that first find the stack or service that is running it.
sudo docker stack ls
sudo docker service ls
If you see a stack listed, you can remove that with:
sudo docker stack rm $stack_name
If there are no stacks listed, or they don't apply to this container, you can delete the service with:
sudo docker service rm $service_name

Jenkins fails to start on centOS7

I am using jenkins on my centOS7/linux server. When I start jenkins and checked the status it showed me like this.
>jenkins.service - Jenkins Service
> Loaded: loaded (/etc/systemd/system/jenkins.service; enabled; vendor preset: > disabled)
> Active: failed (Result: exit-code) since Mon 2017-02-20 22:52:19 PST; 22s > ago
> Process: 40251 ExecStart=/usr/bin/java -jar /usr/local/bin/jenkins.war
>(code=exited, status=1/FAILURE)
> Main PID: 40251 (code=exited, status=1/FAILURE)
>Feb 20 22:52:19 CentOS7 systemd[1]: Started Jenkins Service.
>Feb 20 22:52:19 CentOS7 systemd[1]: Starting Jenkins Service...
>Feb 20 22:52:19 CentOS7 java[40251]: Error: Unable to access jarfile >/usr/l...ar
>Feb 20 22:52:19 CentOS7 systemd[1]: jenkins.service: main process exited, >c...RE
>Feb 20 22:52:19 CentOS7 systemd[1]: Unit jenkins.service entered failed state.
>Feb 20 22:52:19 CentOS7 systemd[1]: jenkins.service failed.
>Hint: Some lines were ellipsized, use -l to show in full.
So I uninstalled the jenkins sudo yum remove jenkins by this command, and installed it again sudo yum install jenkins.
Now again facing the same issue.
Can anyone tell me what to do.
Thanks!!
Before you can install Jenkins, you need to setup a Java virtual machine on your system
yum install java-1.8.0-openjdk.x86_64
And set two environment variables: JAVA_HOME and JRE_HOME.
echo 'export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk' | sudo tee -a /etc/profile
echo 'export JRE_HOME=/usr/lib/jvm/jre' | sudo tee -a /etc/profile
source /etc/profile
Then install jenkins and allow inbound traffic on port 8080.
You can see more details from how to install jenkins on Centos 7.
Hope this helps.
As per Jun, Jenkins required Java 11.
Refer :
Required Java version for Jenkins
at first you cannot get clue, on run systemctl status jenkins.service
until I try to change JENKINS_USER on /etc/init.d/jenkins to root, and show me
Jenkins requires Java versions [17, 11] but you are running with Java 1.8 from /usr/lib/j
Once I upgraded Java to 11 then it started working.

Job for firebird3.0.service failed because a configured resource limit was exceeded

Error when install and after when try to start Firebird 3.0 Service.
Job for firebird3.0.service failed because a configured resource limit was exceeded. See "systemctl status firebird3.0.service" and "journalctl -xe" for details.
invoke-rc.d: initscript firebird3.0, action "start" failed.
dpkg: error processing package firebird3.0-server (--configure):
subprocess installed post-installation script returned error exit status 1
Processing triggers for libc-bin (2.23-0ubuntu3) ...
Processing triggers for systemd (229-4ubuntu7) ...
Processing triggers for ureadahead (0.100.0-19) ...
Errors were encountered while processing:
firebird3.0-server
E: Sub-process /usr/bin/dpkg returned an error code (1)
See return from "service firebird3.0 start":
Job for firebird3.0.service failed because a configured resource limit was exceeded. See "systemctl status firebird3.0.service" and "journalctl -xe" for details
See return from "journalctl -xe":
-- Unit firebird3.0.service has begun starting up.
Ago 26 15:41:22 server14 systemd[1]: firebird3.0.service: PID file /var/run/firebird/3.0default.pid not readable (yet?) after start: No such file or directory
Ago 26 15:41:22 server14 firebird[3509]: Security database error
Ago 26 15:41:22 server14 systemd[1]: firebird3.0.service: Daemon never wrote its PID file. Failing.
Ago 26 15:41:22 server14 systemd[1]: Failed to start Firebird Database Server ( SuperServer ).
-- Subject: Unit firebird3.0.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit firebird3.0.service has failed.
--
-- The result is failed.
Ago 26 15:41:22 server14 systemd[1]: firebird3.0.service: Unit entered failed state.
Ago 26 15:41:22 server14 systemd[1]: firebird3.0.service: Failed with result 'resources'.
I've tried many thing to solve but only way at moment is the manual start:
start-stop-daemon --quiet --start --exec /usr/sbin/fbguard --pidfile /var/run/firebird/3.0/firebird.pid -b -m -- -daemon -forever -pidfile /var/run/firebird/3.0/firebird.pid
And manual stop:
start-stop-daemon --stop --signal KILL --exec /usr/sbin/fbguard
start-stop-daemon --stop --signal KILL --exec /usr/sbin/firebird
Any ideas?
The directory /run/firebird/3.0 is not created on installation on debian based systems. So the systemd script does not work.
Workaround:
As user root do
create the directory:
mkdir -p /run/firebird/3.0
chown to firebird:
chown -R firebird:firebird /run/firebird
After doing this, firebird 3.0 should run as expected
As /run normally is an temporary directory in Debian, you could change the sytemd startup script to always execute directory creation before start of the service:
/lib/systemd/system/firebird3.0 should then look like this:
[Unit]
Description=Firebird Database Server ( SuperServer )
After=network.target
Conflicts=firebird3.0-classic.socket
[Service]
User=firebird
Group=firebird
Type=forking
# Run ExecStartPre with root-permissions
PermissionsStartOnly=true
ExecStartPre=-/bin/mkdir -p /run/firebird/3.0
ExecStartPre=/bin/chown -R firebird:firebird /run/firebird
PIDFile=/run/firebird/3.0/default.pid
ExecStart=/usr/sbin/fbguard -pidfile /run/firebird/3.0/default.pid -daemon -forever
RuntimeDirectory=firebird/3.0
StandardError=syslog
[Install]
WantedBy=multi-user.target
PermissionsStartOnly=true is necessary to be able to execute all statements except the service itself (ExecStart) as root. This is important to create the subdirectories in /run.
BY the way: the - (minus) in the first ExecStartPre line makes run the script without stopping on errors returned from directory creation, helps if directory exists, for example after an service restart.
Don't forget to reload systemd:
systemctl --system daemon-reload

Resources