Systemd Service for jar file gets "operation timed out" error after few minues or stay in "activating mode" - linux

the service unit is:
[Unit]
Description=test
After=syslog.target
After=network.target
[Service]
Type=forking
ExecStart=/bin/java -jar /home/ec2-user/test.jar
TimeoutSec=300
[Install]
WantedBy=multi-user.target
it starts fine for 1-4 minues. But later it fails:
tail /var/log/messages:
Feb 27 18:43:44 ip-172-31-40-48 systemd: Reloading.
Feb 27 18:44:06 ip-172-31-40-48 systemd: Starting test...
Feb 27 18:44:06 ip-172-31-40-48 java: 5.1.73
Feb 27 18:44:06 ip-172-31-40-48 java: Starting the internal [HTTP/1.1] server on port 8182
Feb 27 18:49:06 ip-172-31-40-48 systemd: test.service operation timed out.Terminating.
Feb 27 18:49:06 ip-172-31-40-48 systemd: test.service: control process exited, code=exited status=143
Feb 27 18:49:06 ip-172-31-40-48 systemd: Failed to start test.
Feb 27 18:49:06 ip-172-31-40-48 systemd: Unit test.service entered failed state.
systemctl status test.service (while restarting- stays in activating mode):
test.service - Setsnew
Loaded: loaded (/etc/systemd/system/test.service; enabled)
Active: activating (start) since Sun 2015-03-01 14:29:36 EST; 2min 30s ago
Control: 32462 (java)
CGroup: /system.slice/test.service
systemctl status test.service (after fail):
test.service - test
Loaded: loaded (/etc/systemd/system/test.service; enabled)
Active: failed (Result: exit-code) since Fri 2015-02-27 18:49:06 EST; 18min ago
Process: 27954 ExecStart=/bin/java -jar /home/ec2-user/test.jar (code=exited, status=143)
when running the jar in command line it works just fine.
tried changing the jar location because I thought it's a permissions problem
selinux is off
How can i fix this issue so I could start the jar on boot? there any alternatives? (RHEL7 do not include service command)

You made the service type forking, but this service does not fork. It just runs directly. Thus systemd waited five minutes for the program to daemonize itself, and it never did. The correct type for such a service is simple.
You also disabled SELinux, which is another problem you should resolve.

Related

Unable to run node app using systemd

This going to be my first node deployment. I have this app which runs fine on my Debian 9 server using this command:
cd /srv/myapp && NODE_ENV=production yarn start
And print out this message:
yarn run v1.6.0
$ babel-node index.js
In order to demonize the app using systemd, I created `/lib/systemd/system/myapp.service:
[Unit]
Description=Myapp
[Service]
ExecStart=/home/john/start.sh
Type=simple
User=john
Restart=on-failure
[Install]
WantedBy=multi-user.target
and in start.sh I have:
cd /srv/myapp && NODE_ENV=production yarn start
Hoewever, when I run systemctl start myapp the node app does not start to listen on port 3000, as expected (netstat -tulpn | grep :3000
returns no results)
# systemctl status myapp
● myapp.service - Myapp
Loaded: loaded (/lib/systemd/system/myapp.service; disabled; vendor preset: enabled
Active: failed (Result: exit-code) since Sun 2018-05-13 06:14:04 EDT; 5s ago
Process: 8852 ExecStart=/home/bob/start.sh (code=exited, status=203/EXEC)
Main PID: 8852 (code=exited, status=203/EXEC)
May 13 06:14:04 9606 systemd[1]: myapp.service: Unit entered failed state.
May 13 06:14:04 9606 systemd[1]: myapp.service: Failed with result 'exit-code'.
May 13 06:14:04 9606 systemd[1]: myapp.service: Service hold-off time over, scheduling
May 13 06:14:04 9606 systemd[1]: Stopped myapp.
May 13 06:14:04 9606 systemd[1]: myapp.service: Start request repeated too quickly.
May 13 06:14:04 9606 systemd[1]: Failed to start myapp.
May 13 06:14:04 9606 systemd[1]: myapp.service: Unit entered failed state.
May 13 06:14:04 9606 systemd[1]: myapp.service: Failed with result 'exit-code'.
I have other variations on myapp.service but could not manage to run node.
What could be wrong here? How can I fix it?
Process: 8852 ExecStart=/home/bob/start.sh (code=exited, status=203/EXEC)
According to systemd.exec(5), this means systemd could not execute the specified file:
203 EXIT_EXEC The actual process execution failed (specifically, the execve(2) system call). Most likely this is caused by a missing or non-accessible executable file.
You should check if /home/bob/start.sh is executable and has correct shebang specified (that is, the first line of your script must be #!/bin/bash).

systemd stops process even with incorrect ExecStop

I'm trying to manage multiple redis instances with systemd. Below is my systemd unit file
[Unit]
Description=Redis instances
After=network.target
[Service]
Type=simple
ExecStart=/home/redis/bin/redis-server /home/redis/conf/redis-%i.conf
ExecStop=/home/redis/bin/redis-cli -p %i INFO # <= INFO should not stop redis process
User=redis
Group=redis
[Install]
WantedBy=multi-user.target
I could successfully start redis-server by
$ sudo systemctl start redis#6379
$ systemctl status redis#6379
● redis#6379.service - Redis instances
Loaded: loaded (/usr/lib/systemd/system/redis#.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2017-09-05 17:48:42 JST; 2h 52min ago
Main PID: 86962 (redis-server)
CGroup: /system.slice/system-redis.slice/redis#6379.service
└─86962 /home/redis/bin/redis-server 0.0.0.0:6379 [cluster]
As a test of stopping redis-server, I intentionally used INFO instead of SHUTDOWN at the ExecStop line.
But when I executed below command, systemd still killed my redis process.
sudo systemctl stop redis#6379
And there's no output by:
sudo journalctl -f -u redis#6379
I wonder how could this happen ?
PS:
I replaced redis-cli with an non-exist one:
ExecStop=/tmp/nonexist
journalctl showed error log like:
Sep 05 20:51:57 myhost systemd[96306]: Failed at step EXEC spawning /tmp/nonexist: No such file or directory
Sep 05 20:51:57 myhost systemd[1]: redis#6379.service: control process exited, code=exited status=203
Sep 05 20:51:57 myhost systemd[1]: Unit redis#6379.service entered failed state.
Sep 05 20:51:57 myhost systemd[1]: redis#6379.service failed.
But the running redis process was still killed.

Changing systemd.service TimeoutSec value to “infinity” has no effect

My app.service file's [Service] part is the following:-
[Service]
Type=forking
Restart=no
IgnoreSIGPIPE=no
GuessMainPID=no
ExecStart=/opt/app/appl_init.d start
ExecStop=/opt/app/appl_init.d stop
TimeoutSec=infinity
After which I installed the app, and the file is correctly copied to /usr/lib/systemd/system/app.service.
I have run systemctl daemon-reload, but it seems to have no effect on the start up time! It fails just as I run systemctl start app or systemctl reload app.service with the following error:-
Job for app.service failed because a fatal signal was delivered to the control process. See "systemctl status app.service" and "journalctl -xe" for details
Output of systemctl status app is:-
● app.service - ApplicationTest
Loaded: loaded (/opt/app/appl_init.d; enabled; vendor preset: disabled)
Active: failed (Result: signal) since Tue 2017-03-21 01:55:22 EDT; 1min 4s ago
Docs: man:app(8)
Process: 4126 ExecStart=/opt/app/appl_init.d start (code=killed, signal=KILL)
Mar 21 01:55:22 centosvm systemd[1]: Starting ApplicationTest...
Mar 21 01:55:22 centosvm systemd[1]: app.service start operation timed out. Terminating.
Mar 21 01:55:22 centosvm systemd[1]: app.service stop-final-sigterm timed out. Killing.
Mar 21 01:55:22 centosvm systemd[1]: app.service: control process exited, code=killed status=9
Mar 21 01:55:22 centosvm systemd[1]: Failed to start ApplicationTest.
Mar 21 01:55:22 centosvm systemd[1]: Unit app.service entered failed state.
Mar 21 01:55:22 centosvm systemd[1]: app.service failed.
Another queer thing that I noticed is when I run systemctl show app.service -p TimeoutSec, I don't get any result; it's blank?
I have tried doing a systemctl reboot, but still, no dice.
Of course, when I change the value to anything else like TimeoutSec=5min, then it works perfectly fine. But I really need this application to take up infinity.
Where am I going wrong?
TimeoutSec=0 fixed the problem.
Apparently, if you are using a version of systemd older than 229, you will need to use 0 instead of infinity to disable the timeout.

Setting up systemctl for uwsgi

I'm trying to set up uwsgi service as /etc/systemd/system/emperor.uwsgi.service
[Unit]
Description=uWSGI Emperor
After=syslog.target
[Service]
ExecStart=/root/uwsgi/uwsgi --ini /etc/uwsgi/emperor.ini
# Requires systemd version 211 or newer
RuntimeDirectory=uwsgi
Restart=always
KillSignal=SIGQUIT
Type=notify
StandardError=syslog
NotifyAccess=all
[Install]
WantedBy=multi-user.target
When trying to start it, I get the following error:
ubuntu#ip-172-31-16-133:~$ sudo systemctl start emperor.uwsgi.service
Job for emperor.uwsgi.service failed because the control process exited with error code. See "systemctl status emperor.uwsgi.service" and "journalctl -xe" for details.
This is the output for when I checked the status:
ubuntu#ip-172-31-16-133:~$ sudo systemctl status emperor.uwsgi.service
● emperor.uwsgi.service - uWSGI Emperor
Loaded: loaded (/etc/systemd/system/emperor.uwsgi.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Jan 30 11:16:05 ip-172-31-16-133 systemd[1]: Stopped uWSGI Emperor.
Jan 30 11:16:05 ip-172-31-16-133 systemd[1]: Starting uWSGI Emperor...
Jan 30 11:16:05 ip-172-31-16-133 systemd[1]: emperor.uwsgi.service: Main process exited, code=exited
Jan 30 11:16:05 ip-172-31-16-133 systemd[1]: Failed to start uWSGI Emperor.
Jan 30 11:16:05 ip-172-31-16-133 systemd[1]: emperor.uwsgi.service: Unit entered failed state.
Jan 30 11:16:05 ip-172-31-16-133 systemd[1]: emperor.uwsgi.service: Failed with result 'exit-code'.
Jan 30 11:16:05 ip-172-31-16-133 systemd[1]: emperor.uwsgi.service: Service hold-off time over, sche
Jan 30 11:16:05 ip-172-31-16-133 systemd[1]: Stopped uWSGI Emperor.
Jan 30 11:16:05 ip-172-31-16-133 systemd[1]: emperor.uwsgi.service: Start request repeated too quick
Jan 30 11:16:05 ip-172-31-16-133 systemd[1]: Failed to start uWSGI Emperor.
I've had similar issues. It seems systemd swallows some output when failing to start a (UWSGI) service. Here are a couple of things to check to figure out what's causing the issue:
Check systemd journal: journalctl -b -u $service
Try to run the service manually: simply run the cmdline specified after ExecStart= in the systemd service file; so in your example: /root/uwsgi/uwsgi --ini /etc/uwsgi/emperor.ini
Either of these should shed some light as to whether the service fails to start.

systemd cannot run service after running commands

I tried to run systemd using the commands systemctl enable photogrid.service & systemctl start photogrid.service in ubuntu 16
The nodejs app itself can run as expected. The service is to ensure that application will auto-start when application crash or server reboot.
The service apparently did not start. So I key in systemctl status photogrid.service to see what happened, the below is what I got from the terminal.
● photogrid.service - Photogrid
Loaded: loaded (/lib/systemd/system/photogrid.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Wed 2016-11-09 04:35:36 UTC; 7s ago
Process: 27523 ExecStart=/usr/local/bin/node /home/ubuntu/photogrid/app.js (code=exited, status=203/EXEC)
Main PID: 27523 (code=exited, status=203/EXEC)
Nov 09 04:35:36 ip-172-31-34-151 systemd[1]: photogrid.service: Main process exited, code=exited, status=203/EXEC
Nov 09 04:35:36 ip-172-31-34-151 systemd[1]: photogrid.service: Unit entered failed state.
Nov 09 04:35:36 ip-172-31-34-151 systemd[1]: photogrid.service: Failed with result 'exit-code'.
This the script that I wrote for the service under the path /lib/systemd/system/photogrid.service
[Unit]
Description=Photogrid
[Service]
Type=simple
Restart=always
RestartSec=10
Environment=NODE_ENV=production
ExecStart=/usr/local/bin/node /home/ubuntu/photogrid/app.js
[Install]
WantedBy=multi-user.target
Basically under ExecStart make sure you point to the correct nodejs executable. For my case it was in a different folder and not /usr/local/bin/node, to check where is your node executable. (Assuming you confirm you have downloaded and install it correctly in linux) use command which node to give you path direction.

Resources