I have a service file that refuses to actually start the service specified in the ExecStart directive.
If I run exactly the same command from the terminal prompt, everything works as expected.
I have checked all permissions on files etc. but nothing I do will launch the program.
Here is my service file:
[Unit]
Description=Start pjsip Pjsua client in background
After=network.target
[Service]
Type=simple
RestartSec=3
ExecStart=/usr/bin/screen -dmS Pjsua /usr/local/sbin/Pjsua --config-file /usr/local/sbin/pjsua.cfg
[Install]
WantedBy=multi-user.target
and here is the output from the log file:
May 06 20:27:07 vring systemd[1]: pjsip.service: Trying to enqueue job pjsip.service/start/replace
May 06 20:27:07 vring systemd[1]: pjsip.service: Installed new job pjsip.service/start as 2553
May 06 20:27:07 vring systemd[1]: pjsip.service: Enqueued job pjsip.service/start as 2553
May 06 20:27:07 vring systemd[1]: pjsip.service: About to execute: /usr/bin/screen -dmS Pjsua /usr/local/sbin/Pjsua --confif-file /usr/local/sbin/pjsua.cfg
May 06 20:27:07 vring systemd[1]: pjsip.service: Forked /usr/bin/screen as 11355
May 06 20:27:07 vring systemd[1]: pjsip.service: Changed dead -> running
May 06 20:27:07 vring systemd[1]: pjsip.service: Job pjsip.service/start finished, result=done
May 06 20:27:07 vring systemd[1]: Started Start pjsip Pjsua client in background.
May 06 20:27:07 vring systemd[1]: pjsip.service: Child 11355 belongs to pjsip.service
May 06 20:27:07 vring systemd[1]: pjsip.service: Main process exited, code=exited, status=0/SUCCESS
May 06 20:27:07 vring systemd[1]: pjsip.service: Changed running -> stop-sigterm
May 06 20:27:07 vring systemd[1]: pjsip.service: Child 11356 belongs to pjsip.service
May 06 20:27:07 vring systemd[1]: pjsip.service: cgroup is empty
May 06 20:27:07 vring systemd[1]: pjsip.service: Changed stop-sigterm -> dead
May 06 20:27:07 vring systemd[1]: pjsip.service: Collecting.
From the above it appears as though the process has exited and I cannot understand why as it has to be commanded to do so.
If I run the command specified for ExecStart from the command line it all works perfectly. The program stays alive until I actually command it to shut down.
This problem has been driving me nuts for most of a day and I am no closer to a resolution.
Any pointers as to what I may be doing wrong would be very much appreciated.
I have solved the problem!!!
Because the system forks off the screen command I needed to change the service type to 'forking'.
It all works perfectly now.
Often it's the stupid things that catch you out!
Related
So I have a systemd unit that needs to be monitored, restarted in case of a crash and also something done in case the unit fails. I'm working on an embedded system so this needs to be robust.
In my case we have a systemd service:
Description=Demo unit
Wants=multi-user.target
OnFailure=FailHandler#%N.service
[Service]
ExecStart=/bin/bash /home/root/demo.sh
Restart=on-failure
RestartSec=1
Type=simple
The bash I start:
echo "Started demo.sh"
current_date=`date`
sleep 10s
echo "${current_date} Demo was here" >> /home/root/demo.txt
exit 1
So far so good. The bash always exits with 1 afer 10 seconds and logs the time. The problem is that FailHandler is never called in that case. Now this is just a demo all of the applications are in C++ but the behavior is the same. Now if I manually set the wrong path to the bash file it unit fails but it starts the "OnFailure" part. Here's syslog output from having correct path:
2021-09-03T13:06:31.575094+00:00 hostname bash[1125]: Started demo.sh
2021-09-03T13:06:41.629450+00:00 hostname systemd[1]: demo.service: Main process exited, code=exited, status=1/FAILURE
2021-09-03T13:06:41.644681+00:00 hostname systemd[1]: demo.service: Failed with result 'exit-code'.
2021-09-03T13:06:41.818089+00:00 hostname systemd[1]: demo.service: Service RestartSec=100ms expired, scheduling restart.
2021-09-03T13:06:41.824005+00:00 hostname systemd[1]: demo.service: Scheduled restart job, restart counter is at 1.
2021-09-03T13:06:41.850933+00:00 hostname bash[1179]: Started demo.sh
2021-09-03T13:06:51.870376+00:00 hostname systemd[1]: demo.service: Main process exited, code=exited, status=1/FAILURE
2021-09-03T13:06:51.872611+00:00 hostname systemd[1]: demo.service: Failed with result 'exit-code'.
2021-09-03T13:06:52.117479+00:00 hostname systemd[1]: demo.service: Service RestartSec=100ms expired, scheduling restart.
2021-09-03T13:06:52.136102+00:00 hostname systemd[1]: demo.service: Scheduled restart job, restart counter is at 2.
2021-09-03T13:06:52.163865+00:00 hostname bash[1221]: Started demo.sh
Here's output from when path is incorrect:
2021-09-03T13:07:46.582269+00:00 hostnaem bash[1446]: /bin/bash: /ahome/root/daemo.sh: No such file or directory
2021-09-03T13:07:46.588715+00:00 hostnaem systemd[1]: daemo.service: Main process exited, code=exited, status=127/n/a
2021-09-03T13:07:46.590356+00:00 hostnaem systemd[1]: daemo.service: Failed with result 'exit-code'.
2021-09-03T13:07:46.694616+00:00 hostnaem systemd[1]: daemo.service: Service RestartSec=100ms expired, scheduling restart.
2021-09-03T13:07:46.701519+00:00 hostnaem systemd[1]: daemo.service: Scheduled restart job, restart counter is at 1.
2021-09-03T13:07:46.720879+00:00 hostnaem systemd[1]: daemo.service: Start request repeated too quickly.
2021-09-03T13:07:46.721405+00:00 hostnaem systemd[1]: daemo.service: Failed with result 'exit-code'.
2021-09-03T13:07:46.722723+00:00 hostnaem systemd[1]: daemo.service: Triggering OnFailure= dependencies.
2021-09-03T13:07:46.804815+00:00 hostnaem FailHandler.sh[1457]: Failed application: daemo
2021-09-03T13:07:46.822342+00:00 hostnaem bash[1457]: error: cannot stat /etc/logrotate.d/daemo: No such file or directory
2021-09-03T13:07:46.841577+00:00 hostnaem FailHandler.sh[1457]: ERROR: Failed logrotate for daemo crash
2021-09-03T13:07:46.977003+00:00 hostnaem systemd[1]: FailHandler#daemo.service: Succeeded.
I understand from the syslog that it starts the FailHandler whenever number of restarts reaches StartLimitBurst=1 within 100ms but is there a way that it starts anytime the application exits with an error code?
Thank you man. I took one look at the link you sent and it landed. The solution in my case was:
ExecStopPost=/bin/bash -c 'if [ "$$EXIT_STATUS" != 0 ]; then systemctl start FailHandler#%N.service; fi'
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 2 years ago.
Improve this question
I created a service in my Ubuntu instance by adding it to systemd like:
sudo systemctl enable myservice.service
Created symlink /etc/systemd/system/multi-user.target.wants/myservice.service → /etc/systemd/system/myservice.service.
the setting inside my myservice.service are:
[Unit]
Description=myserviceService
[Service]
Restart=always
Type=simple
ExecStart=/home/myservice-app/core/core
[Install]
WantedBy=multi-user.target
This is the error when I check status:
Feb 02 06:51:16 myservice-ubuntu-app systemd[1]: myservice.service: Main process exited, code=exited, status=1/FAILURE
Feb 02 06:51:16 myservice-ubuntu-app systemd[1]: myservice.service: Failed with result 'exit-code'.
Feb 02 06:51:16 myservice-ubuntu-app systemd[1]: myservice.service: Scheduled restart job, restart counter is at 5.
Feb 02 06:51:16 myservice-ubuntu-app systemd[1]: Stopped myserviceService.
Feb 02 06:51:16 myservice-ubuntu-app systemd[1]: myservice.service: Start request repeated too quickly.
Feb 02 06:51:16 myservice-ubuntu-app systemd[1]: myservice.service: Failed with result 'exit-code'.
Feb 02 06:51:16 myservice-ubuntu-app systemd[1]: Failed to start myservice Service.
What am I doing wrong? The service runs properly in foreground with no errors, its just the service that struggles to run.
If your Go program is able to start normally (meaning not as a service, but manually from command line), then check if this is a policy issue.
For instance, a SELinux policy could prevent your Go binary to start if it is not installed in a system path (like /usr/local/bin).
Or the service definition uses relative instead of absolute paths (same here).
Or with the wrong user.
When I logged in to my VM (Ubuntu 18.04) for second time, its showing error:
# systemctl status puppetserver.service
puppetserver.service - puppetserver Service Loaded: loaded
(/lib/systemd/system/puppetserver.service; enabled; vendor preset: enabled)
**Active**: failed (Result: exit-code) since Wed 2019-10-02 11:42:52 UTC; 2min 31s ago
Process: 23034
ExecStart=/opt/puppetlabs/server/apps/puppetserver/bin/puppetserver
start (code=exited, status=1/FAILURE)
Oct 02 11:42:52 puppet-master systemd[1]: puppetserver.service:
Control process exited, code=exited status=1 Oct 02 11:42:52
puppet-master systemd[1]: puppetserver.service: Failed with result
'exit-code'. Oct 02 11:42:52 puppet-master systemd[1]: Failed to start
puppetserver Service. Oct 02 11:42:52 puppet-master systemd[1]:
puppetserver.service: Service hold-off time over, scheduling restart.
Oct 02 11:42:52 puppet-master systemd[1]: puppetserver.service:
Scheduled restart job, restart counter is at 5. Oct 02 11:42:52
puppet-master systemd[1]: Stopped puppetserver Service. Oct 02
11:42:52 puppet-master systemd[1]: puppetserver.service: Start request
repeated too quickly. Oct 02 11:42:52 puppet-master systemd[1]:
puppetserver.service: Failed with result 'exit-code'. Oct 02 11:42:52
puppet-master systemd[1]: Failed to start puppetserver Service.
Is there a way to identify the issue?
To troubleshoot more, you can run:
journalctl -xe -u puppetserver
And also check the /var/log/puppetlabs/puppetserver/puppetserver.log file for more info, but based on the error message:
Start request repeated too quickly
It seems that you didn't have patience to restart (as it can take few minutes).
The best to restart I recommend to you:
systemctl restart puppet*
systemctl restart puppet pxp-agent
In this way, the services manage the dependency between all puppet* services.
Note: Server Fault, part of Stack Exchange (includes Stack Overflow) provides support for managing information technology systems in a business environment.
After I installed couchdb， I could get the welcome information
$ curl localhost:5984
{"couchdb":"Welcome","version":"2.1.2","features":["scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
But I can't check the status by systemctl
$ systemctl status couchdb.service
● couchdb.service
Loaded: not-found (Reason: No such file or directory)
Active: failed (Result: start-limit-hit) since 一 2018-12-03 14:52:14 CST; 6min ago
Main PID: 30946 (code=killed, signal=USR2)
12月 03 14:52:14 gpuhuawei systemd[1]: couchdb.service: Unit entered failed state.
12月 03 14:52:14 gpuhuawei systemd[1]: couchdb.service: Failed with result 'signal'.
12月 03 14:52:14 gpuhuawei systemd[1]: couchdb.service: Service hold-off time over, scheduling restart.
12月 03 14:52:14 gpuhuawei systemd[1]: Stopped Apache CouchDB.
12月 03 14:52:14 gpuhuawei systemd[1]: couchdb.service: Start request repeated too quickly.
12月 03 14:52:14 gpuhuawei systemd[1]: Failed to start Apache CouchDB.
12月 03 14:52:14 gpuhuawei systemd[1]: couchdb.service: Unit entered failed state.
12月 03 14:52:14 gpuhuawei systemd[1]: couchdb.service: Failed with result 'start-limit-hit'.
12月 03 14:53:53 gpuhuawei systemd[1]: Stopped Apache CouchDB.
12月 03 14:53:53 gpuhuawei systemd[1]: Stopped Apache CouchDB.
When I run couchdb by command line, I got
$ couchdb
{"init terminating in do_boot",{{badmatch,{error,{bad_return,{{couch_app,start,[normal,["/etc/couchdb/default.ini","/etc/couchdb/local.ini"]]},{'EXIT',{{badmatch,{error,{error,eacces}}},[{couch_server_sup,start_server,1,[{file,"couch_server_sup.erl"},{line,56}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,273}]}]}}}}}},[{couch,start,0,[{file,"couch.erl"},{line,18}]},{init,start_it,1,[]},{init,start_em,1,[]}]}}
[1] 2288 user-defined signal 2 couchdb
My work enviroment
$ uname -a
Linux gpuhuawei 4.15.0-34-generic #37~16.04.1-Ubuntu SMP Tue Aug 28 10:44:06 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
This is a bit late, but the "start-limit-hit" message is a red herring. I have seen something very similar with a Moodle installation using MySQL and it's actually saying that you (or the service start process) have tried to restart the database too many times or too soon after a failed attempt to start. Basically, this start-limit-hit message is saying "stop trying to do the same thing and expecting different results".
The actual issue will be further up in the syslog. Unhelpfully, service status does not return enough lines of the error messages to actually see what is wrong. Try a service start, and go and look in the actual syslog, and you will see the series of start attempts and a line just above each one hopefully will tell you the actual issue. In my case here, you can see that the problem is the mount point containing the database is missing - thanks, Azure. For one attempt of service start, it tries to start 5 times in quick succession, failing each time because the data dir was not mounted, and on the sixth it fails with the start-limit-hit.
Always back up your data/ and etc/ directories prior to upgrading CouchDB.
We recommend that you overwrite your etc/default.ini file with the version provided by the new release. New defaults sometimes contain mandatory changes to enable default functionality. Always places your customization in etc/local.ini or any etc/local.d/*.ini file.
(I was followed this and it worked)
https://docs.couchdb.org/en/3.0.0/install/upgrading.html
Trying to create a start up script for my nodejs app which runs on port 3000.
Issue: The node-app.server script is not working and I think it's because ExecStart pathway is wrong. When I go to the server IP in Chrome nothing shows.
The node app was created with npm generator, and I normally use npm start to start the app. I've added path to bin/wwww, here:
[Unit]
Description=tweetMonster twtiter server - making your environment variables rad
Documentation=https://example.com
After=network.target
[Service]
Environment=NODE_PORT=3000
Type=simple
User=ubuntu
ExecStart=/home/ubuntu/twitter-server/bin/www.js
Restart=on-failure
[Install]
WantedBy=multi-user.target
My app is running on Ubuntu 18 at /home/ubuntu/twitter-server . And if do ls:
/twitter-server$ ls
app.js node_modules package.json routes
bin package-lock.json public views
Please help!
ERROR TERMINAL:
Nov 01 05:40:25 ip-172-31-22-207 systemd[1]: node-app.service: Main process exited, code=exited, status=203/EXEC
Nov 01 05:40:25 ip-172-31-22-207 systemd[1]: node-app.service: Failed with result 'exit-code'.
Nov 01 05:40:26 ip-172-31-22-207 systemd[1]: node-app.service: Service hold-off time over, scheduling restart.
Nov 01 05:40:26 ip-172-31-22-207 systemd[1]: node-app.service: Scheduled restart job, restart counter is at 5.
Nov 01 05:40:26 ip-172-31-22-207 systemd[1]: Stopped hello_env.js - making your environment variables rad.
Nov 01 05:40:26 ip-172-31-22-207 systemd[1]: node-app.service: Start request repeated too quickly.
Nov 01 05:40:26 ip-172-31-22-207 systemd[1]: node-app.service: Failed with result 'exit-code'.
Nov 01 05:40:26 ip-172-31-22-207 systemd[1]: Failed to start hello_env.js - making your environment variables rad.
Nov 01 06:20:11 ip-172-31-22-207 systemd[1]: /etc/systemd/system/node-app.service:10: Executable path is not absolute: "node /home/ubuntu/twitter-server/bin/www.js"
Nov 01 06:24:35 ip-172-31-22-207 systemd[1]: /etc/systemd/system/node-app.service:10: Executable path is not absolute: "node /home/ubuntu/twitter-server/bin/www.js"
root#ip-172-31-22-207:/etc/systemd/system#
There are 2 issues with your service config.
First, wrap the value of Environment with double quotes:
Environment="NODE_PORT=3000"
Second,
You need to use node to run the ExecStart script. /home/ubuntu/twitter-server/bin/www.js is no command in itself.
Do,
ExecStart=/bin/bash -c '$$(which node) /home/ubuntu/twitter-server/bin/www.js'
I recommend this package (service-systemd) for a simple program
Sometimes you just want an "old style" daemon for simple services.
Sometimes you have to deploy in small devices (like a RaspberryPi) and you can't use Docker and all the band.