The Upstart script using the start-stop-daemon we've been using for Play 1.2.7 is now unable to stop/restart Play since Play 1.3 due to it having an incorrect PID.
Framework version: 1.3.0 on Ubuntu 12.04.5 LTS
Reproduction steps:
Setup an upstart script (playframework.conf) for a Play application
Play application starts successfully on server reboot Run 'sudo
status playframework' will return playframework start/running,
process 28912 - At this point process 28912 doesn't exist
vi {playapplicationfolder}/server.pid shows 28927
'stop playframework'
then fails due to unknown pid 28912 'status playframework' results in
playframework stop/killed, process 28912
Only way to restart play framework after this point is to either find the actual process and kill it then start play using the usual 'play start' command manually. Or restart the server.
This has broken our deployments scripts now as we used to install the new version of our app, then do play restart before reconnecting to the load balancer.
Upstart Script:
#Upstart script for a play application that binds to an unprivileged user.
# put this into a file like /etc/init/playframework
# you can then start/stop it using either initctl or start/stop/restart
# e.g.
# start playframework
description "PlayApp"
author "-----"
version "1.0"
env PLAY_BINARY=/opt/play/play
env JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
env HOME=/opt/myapp/latest
env USER=ubuntu
env GROUP=admin
env PROFILE=prod
start on (filesystem and net-device-up IFACE=lo) or runlevel [2345]
stop on runlevel [!2345]
limit nofile 65536 65536
respawn
respawn limit 10 5
umask 022
expect fork
pre-start script
test -x $PLAY_BINARY || { stop; exit 0; }
test -c /dev/null || { stop; exit 0; }
chdir ${HOME}
rm ${HOME}/server.pid || true
/opt/configurer.sh
end script
pre-stop script
exec $PLAY_BINARY stop $HOME
end script
post-stop script
rm ${HOME}/server.pid || true
end script
script
exec start-stop-daemon --start --exec $PLAY_BINARY --chuid $USER:$GROUP --chdir $HOME -- start $HOME -javaagent:/opt/newrelic/newrelic.jar --%$PROFILE -Dprecompiled=true --http.port=8080 --https.port=4443
end script
We've tried specifying the PID file in the start-stop-daemon as per: http://man.he.net/man8/start-stop-daemon however this also didnt seem to have any effect.
I have found some threads on similar issues https://askubuntu.com/questions/319199/upstart-tracking-wrong-pid-of-process-not-respawning but have been unable to find a way round this so far. I have tried changing fork to daemon but the same issue remains. I also can't see what has changed between Play 1.2.7 and 1.3 to cause this.
Another SO post has also asked a similar question but not had an answer as yet: https://stackoverflow.com/questions/23117345/upstart-gets-wrong-pid-after-launching-celery-with-start-stop-daemon
This is because getJavaVersion() spawns a subprocess, which bumps the PID count, which breaks Upstart, the latter which expects Play to fork exactly none, once or twice, depending on which expect stanza you use.
I've fixed this in a pull request.
I'm curious of how su -c "cmd arg" and tomcat works within a shell script.
Manually from the cmd line, I ran:
su -m tomcat -c /path/to/tomcat/catalina.sh start
This didn't work. However, if I put the cmd in quotes, it worked
su -m tomcat -c "/path/catalina.sh start"
Within a shell script (trying to run chkconfig so it starts automatically), I have something like this:
START="/path/catalina.sh start"
...
...
su -m tomcat -c "$START"
The command is in quotes since I found out manually that quotes are required.
When I run the script, the output comes back as:
Using CATALINA_BASE: /apps/local/apache-tomcat-7.0.42
Using CATALINA_HOME: /apps/local/apache-tomcat-7.0.42
Using CATALINA_TMPDIR: /apps/local/apache-tomcat-7.0.42/temp
Using JRE_HOME: /usr/java/default
Using CLASSPATH: /apps/local/apache-tomcat-7.0.42/bin/bootstrap.jar:/apps/local/apache-tomcat-7.0.42/bin/tomcat-juli.jar
Usage: catalina.sh ( commands ... )
commands:
debug Start Catalina in a debugger
debug -security Debug Catalina with a security manager
jpda start Start Catalina under JPDA debugger
run Start Catalina in the current window
run -security Start in the current window with security manager
start Start Catalina in a separate window
start -security Start in a separate window with security manager
stop Stop Catalina, waiting up to 5 seconds for the process to end
stop n Stop Catalina, waiting up to n seconds for the process to end
stop -force Stop Catalina, wait up to 5 seconds and then use kill -KILL if still running
stop n -force Stop Catalina, wait up to n seconds and then use kill -KILL if still running
configtest Run a basic syntax check on server.xml - check exit code for result
version What version of tomcat are you running?
Note: Waiting for the process to end and use of the -force option require that $CATALINA_PID is defined
This output is like if I manually ran catalina.sh w/o any start/stop arguments. Why is the start/stop option being ignored in the shell script? Yes, I replaced catalina.sh with startup.sh and shutdown.sh and that made the script work but I'm still curious why inside a script, su -c "cmd arg" seemed to ignore the argument portion of the command.
Thanks in advance for your explanation. I'll shoot myself if it's something as easy as shell expansion and quote removal unless I do something to make it not lose the quote.
I use the following in my tomcat init files:
...
su -l $TOMCAT_USER -c $CATALINA_HOME/bin/startup.sh
...
The -l creates a login shell and allows you to pass what you need.
This might be related also with "setenv.sh"
If you created setenv.sh, and have lines starts with "set X=Y", then you will get this problem, because of "set". Remove "set" and try to restart tomcat again.
https://ubuntuforums.org/showthread.php?t=2032330
This is my upstart file /etc/init/myapp.conf on my Ubuntu 12.04.
description "Example of starting Node with Upstart and Forever"
start on filesystem or runlevel [2345]
stop on runlevel [06]
expect fork
console output
setuid lwood
env HOME=/home/lwood
script
cd $HOME
exec forever -a -l /home/lwood/myapp/applog.log \
start /home/lwood/myapp/app.js
end script
pre-stop script
cd $HOME
exec forever stop /home/lwood/myapp/app.js
end script
Do you know why $ sudo restart myapp is only stopping the service? It does not start the service after stopping it. So it has the same effect as $ sudo stop myapp.
Doing $ sudo stop myapp and then $ sudo start myapp works fine though. Wondering why restarting behaves different.
Can you output anything? Output your environment, and specifically your path variable and see if they are different from when you can start it, to when it fails.
When it fails, is there an output file and what is the error?
My guess is that the path is not yet fully set up.
I am using node's forever module to keep my node server running. Forever however terminates when there is a system restart. Is there any way I can automatically start the node server (with forever) when the system restarts?
I would suggest using crontab. It's easy to use.
How to
To start editing run the following replacing the "testuser" with your desired runtime user for the node process. If you choose a different user other than yourself, you will have to run this with sudo.
$ crontab -u testuser -e
If you have never done this before, it will ask you which editor you wish to edit with. I like vim, but will recommend nano for ease of use.
Once in the editor add the following line:
#reboot /usr/local/bin/forever start /your/path/to/your/app.js
Save the file. You should get some feedback that the cron has been installed.
For further confirmation of the installation of the cron, execute the following (again replacing "testuser" with your target username) to list the currently installed crons:
$ crontab -u testuser -l
Note that in my opinion, you should always use full paths when executing binaries in cron.
Also, if the path to your forever script is not correct, run which forever to get the full path.
Given that forever calls node, you may also want to provide the full path to node:
#reboot /usr/local/bin/forever start -c /usr/local/bin/node /your/path/to/your/app.js
Further Reading
crontab Man Page
Ubuntu Cron HowTo
You can use forever-service for doing this.
npm install -g forever-service
forever-service install test
This will provision app.js in the current directory as a service via forever. The service will automatically restart every time system is restarted. Also when stopped it will attempt a graceful stop. This script provisions the logrotate script as well.
Github url: https://github.com/zapty/forever-service
NOTE: I am the author of forever-service.
Install PM2 globally using NPM
npm install pm2 -g
Start your script with pm2
pm2 start app.js
generate an active startup script
pm2 startup
NOTE: pm2 startup is for startting the PM2 when the system reboots. PM2 once started, restarts all the processes it had been managing before the system went down.
In case you want to disable the automatic startup, simply use pm2 unstartup
If you want the startup script to be executed under another user, just use the -u <username> option and the --hp <user_home>:
This case is valid for Debian.
Add the following to /etc/rc.local
/usr/bin/sudo -u {{user}} /usr/local/bin/forever start {{app path}}
{{user}} replaces your username.
{{app path}} replaces your app path. For example, /var/www/test/app.js
An alternative crontab method inspired by this answer and this blog post.
1. Create a bash script file (change bob to desired user).
vi /home/bob/node_server_init.sh
2. Copy and paste this inside the file you've just created.
#!/bin/sh
export NODE_ENV=production
export PATH=/usr/local/bin:$PATH
forever start /node/server/path/server.js > /dev/null
Make sure to edit the paths above according to your config!
3. Make sure the bash script can be executed.
chmod 700 /home/bob/node_server_init.sh
4. Test the bash script.
sh /home/bob/node_server_init.sh
5. Replace "bob" with the runtime user for node.
crontab -u bob -e
6. Copy and paste (change bob to desired user).
#reboot /bin/sh /home/bob/node_server_init.sh
Save the crontab.
You've made it to the end, your prize is a reboot (to test) :)
Copied answer from the attached question.
You can use PM2, it's a production process manager for Node.js applications with a built-in load balancer.
Install PM2
$ npm install pm2 -g
Start an application
$ pm2 start app.js
If you using express then you can start your app like
pm2 start ./bin/www --name="app"
Listing all running processes:
$ pm2 list
It will list all process. You can then stop / restart your service by using ID or Name of the app with following command.
$ pm2 stop all
$ pm2 stop 0
$ pm2 restart all
To display logs
$ pm2 logs ['all'|app_name|app_id]
You need to create a shell script in the /etc/init.d folder for that. It's sort of complicated if you never have done it but there is plenty of information on the web on init.d scripts.
Here is a sample a script that I created to run a CoffeeScript site with forever:
#!/bin/bash
#
# initd-example Node init.d
#
# chkconfig: 345
# description: Script to start a coffee script application through forever
# processname: forever/coffeescript/node
# pidfile: /var/run/forever-initd-hectorcorrea.pid
# logfile: /var/run/forever-initd-hectorcorrea.log
#
# Based on a script posted by https://gist.github.com/jinze at https://gist.github.com/3748766
#
# Source function library.
. /lib/lsb/init-functions
pidFile=/var/run/forever-initd-hectorcorrea.pid
logFile=/var/run/forever-initd-hectorcorrea.log
sourceDir=/home/hectorlinux/website
coffeeFile=app.coffee
scriptId=$sourceDir/$coffeeFile
start() {
echo "Starting $scriptId"
# This is found in the library referenced at the top of the script
start_daemon
# Start our CoffeeScript app through forever
# Notice that we change the PATH because on reboot
# the PATH does not include the path to node.
# Launching forever or coffee with a full path
# does not work unless we set the PATH.
cd $sourceDir
PATH=/usr/local/bin:$PATH
NODE_ENV=production PORT=80 forever start --pidFile $pidFile -l $logFile -a -d --sourceDir $sourceDir/ -c coffee $coffeeFile
RETVAL=$?
}
restart() {
echo -n "Restarting $scriptId"
/usr/local/bin/forever restart $scriptId
RETVAL=$?
}
stop() {
echo -n "Shutting down $scriptId"
/usr/local/bin/forever stop $scriptId
RETVAL=$?
}
status() {
echo -n "Status $scriptId"
/usr/local/bin/forever list
RETVAL=$?
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status
;;
restart)
restart
;;
*)
echo "Usage: {start|stop|status|restart}"
exit 1
;;
esac
exit $RETVAL
I had to make sure the folder and PATHs were explicitly set or available to the root user since init.d scripts are ran as root.
Use the PM2
Which is the best option to run the server production server
What are the advantages of running your application this way?
PM2 will automatically restart your application if it crashes.
PM2 will keep a log of your unhandled exceptions - in this case, in a file at /home/safeuser/.pm2/logs/app-err.log.
With one command, PM2 can ensure that any applications it manages restart when the server reboots. Basically, your node application will start as a service.
ref: https://www.digitalocean.com/community/tutorials/how-to-use-pm2-to-setup-a-node-js-production-environment-on-an-ubuntu-vps
Forever was not made to get node applications running as services. The right approach is to either create an /etc/inittab entry (old linux systems) or an upstart (newer linux systems).
Here's some documentation on how to set this up as an upstart:
https://github.com/cvee/node-upstart
crontab does not work for me on CentOS x86 6.5. #reboot seems to be not working.
Finally I got this solution:
Edit: /etc/rc.local
sudo vi /etc/rc.local
Add this line to the end of the file. Change USER_NAME and PATH_TO_PROJECT to your own. NODE_ENV=production means the app runs in production mode. You can add more lines if you need to run more than one node.js app.
su - USER_NAME -c "NODE_ENV=production /usr/local/bin/forever start /PATH_TO_PROJECT/app.js"
Don't set NODE_ENV in a separate line, your app will still run in development mode, because forever does not get NODE_ENV.
# WRONG!
su - USER_NAME -c "export NODE_ENV=production"
Save and quit vi (press ESC : w q return). You can try rebooting your server. After your server reboots, your node.js app should run automatically, even if you don't log into any account remotely via ssh.
You'd better set NODE_ENV environment in your shell. NODE_ENV will be set automatically when your account USER_NAME logs in.
echo export NODE_ENV=production >> ~/.bash_profile
So you can run commands like forever stop/start /PATH_TO_PROJECT/app.js via ssh without setting NODE_ENV again.
I wrote a script that does exactly this:
https://github.com/chovy/node-startup
I have not tried with forever, but you can customize the command it runs, so it should be straight forward:
/etc/init.d/node-app start
/etc/init.d/node-app restart
/etc/init.d/node-app stop
The problem with rc.local is that the commands are accessed as root which is different than logging to as a user and using sudo.
I solved this problem by adding a .sh script with the startup commands i want to etc/profile.d. Any .sh file in profile.d will load automatically and any command will be treated as if you used the regular sudo.
The only downside to this is the specified user needs to loggin for things to start which in my situation was always the case.
I tried lots of the above answers. None of them worked for me. My app is installed in /home and as user, not as root. This probably means that when the above mentioned start scripts run, /home is not mounted yet, so the app is not started.
Then I found these instructions by Digital Ocean:
https://www.digitalocean.com/community/tutorials/how-to-use-pm2-to-setup-a-node-js-production-environment-on-an-ubuntu-vps
Using PM2 as explained was very simple and works perfectly: My virtual servers had two physical crashes since - downtime was only about a minute.
complete example crontab (located at /etc/crontab) ..
#!/bin/bash
# edit this file with .. crontab -u root -e
# view this file with .. crontab -u root -l
# put your path here if it differs
PATH=/root/bin:/root/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
# * * * * * echo "executes once every minute" > /root/deleteme
#reboot cd /root/bible-api-dbt-server; npm run forever;
#reboot cd /root/database-api-server; npm run forever;
#reboot cd /root/mailer-api-server; npm run forever;
I have found my own solution by using serve & npm as follows:
Install serve package: npm install -g serve
Then have the command serve -s /var/www/sitename to execute on reboot.
This is what works for me on my VPS.
You can use the following command in your shell to start your node forever:
forever app.js //my node script
You need to keep in mind that the server on which your app is running should always be kept on.
I have the following in a .conf file in the /etc/init/ directory of my CentOS server:
#!upstart
description "shortnr server for fmc.io"
author "Felix Milea-Ciobanu"
start on startup
stop on shutdown
respawn
respawn limit 10 30
script
export HOME="/root"
exec /usr/local/bin/node /var/www/fmc.io/nodejs/app.js >> /var/www/fmc.io/logs/shortnr.upstart.log 2>&1
end script
pre-start script
echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Starting" >> /var/www/fmc.io/logs/shortnr.upstart.log
end script
pre-stop script
echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Stopping" >> /var/www/fmc.io/logs/shortnr.upstart.log
end script
It's a pretty simple and straight forward upstart script. I named this service shortnr, after the nodejs software that the script starts up.
At the command line if I type in start shortnr I get something along the lines of shortnr start/running, process 28350.
However, I can't seem to access the nodejs server; If I do ps aux | grep shortnr at the command shell, nothing comes up.
If I do stop shortnr after running start, I get stop: Unknown instance:, meaning that the original service never started up.
The log file that I setup in the Upstart script looks something like this:
[2012-10-05T17:00:17.174Z] (sys) Starting
[2012-10-05T17:00:17.181Z] (sys) Starting
[2012-10-05T17:00:17.190Z] (sys) Starting
[2012-10-05T17:00:17.197Z] (sys) Starting
[2012-10-05T17:00:17.204Z] (sys) Starting
Basically the script is trying to start multiple times a second when I issued the start command, meaning that the service must be crashing on start or something and trying to respawn?
However, if I copy the command after exec and paste it in the shell prompt, the nodejs script starts up and runs properly.
So that means something must be wrong with my Upstart script.
If I try start/stop the service with the initctl command, I get the same results.
I'm running CentOS 6.3 and Upstart 0.6.5
Anyone have any idea what could be causing this or how to fix my script?
While I couldn't figure out the answer to my problem, I just ended up using forever instead: https://github.com/nodejitsu/forever-monitor
I am also running into similar problems on CentOS 6.3, Upstart 0.6.5 and Node.Js 0.10.5. I specifically upgraded Node so I could use the daemon module and be able to put the daemonized Node app under Upstart control.
Here's my /etc/init/job-worker.conf:
description "job-worker under Upstart/init control"
start on job-worker-start-event
stop on job-worker-stop-event
expect daemon
script
#setuid myuser
exec /root/BasicJobWorker/bin/basic-job-worker
#sleep 5
end script
respawn
respawn limit 10 5
And here's my basic-job-worker script:
\#!/usr/bin/env node
// this code is run twice
// see implementation notes below
console.log(process.pid);
// after this point, we are a daemon
require('daemon')();
// different pid because we are now forked
// original parent has exited
console.log(process.pid);
var BasicJobWorker = require('../lib/basic-job-worker.js');
new BasicJobWorker().boot();
I have tried using "expect fork", "expect daemon" as well as no expect at all. In all cases the job is respawned too fast and it is eventually stopped.