UPSTART script non root not working - linux

I'm trying to run a nodejs application using upstart as a non root user.
But somehow parts of the script will not run : for instance:
if I run it like a root user(below example) NODE_ENV never gets called/set
the only way to called is with "sudo initctl stop pdcapp"
sudo nameofApp start|stop would not work
When called sudo initctl stop nameofApp the pre-stop script will not echo to the log file
if I try to runit like a non root user it would not even start
isn't a more cleaner easier way of doing this (systemd) I've looked a various tutorials around and apparently this is how they've doneit. so what am I missing here?
This is the .conf file under /etc/init/
env FULL_PATH="/srv/pd/sept011100/dev"
env NODE_PATH="/usr/local/nodeJS/bin/node"
env NODE_ENV=production
start on filesystem or runlevel [2345]
stop on [!2345]
script
export NODE_ENV #this variable is never set
echo $$ > /var/run/PD.pid
cd $FULL_PATH
# the command below will not work
#exec sudo -u nginx "$NODE_PATH server.js >> /var/log/PD/pdapp.log 2>&1"
exec $NODE_PATH server.js >> /var/log/PD/pdapp.log 2>&1
end script
pre-start script
echo "[`date`] (sys) Starting" >> /var/log/PD/pdapp.log
end script
pre-stop script
rm /var/run/pdapp.pid
echo "[`date`] (sys) Stopping" >> /var/log/PDC/pdapp.log
end script
in /var/log/messages I get this when I stop the application, otherwise I get nothing in the logfile
Sep 2 18:23:14 547610-redhat-dev2 init: pdcapp pre-stop process (6903) terminated with status 1
Sep 2 18:23:14 547610-redhat-dev2 init: pdcapp main process (6899) terminated with status 143
any Ideas why is this not working I'm running redhat 6.5

Red Hat has a super old version of Upstart that is probably full of bugs because they never contributed to Upstart, despite using it (Fedora switched to systemd right after RHEL 6 was released, before they even really tried it out well).

Related

Unable to start Node on system reboot Ubuntu Crontab

I have tried this with adding the forever start code in /etc/rc.local didn't work.
When I use the #reboot keyword in /etc/rc.local it says #reboot cannot be found.
So I went back to using crontab Here is my crontab script. All other crontabs are working except the reboot one. In syslog, it says
Jun 4 09:51:12 ip-172-31-28-35 /usr/sbin/irqbalance: Balancing is ineffective on systems with a single cache domain. Shutting down
Jun 4 09:51:12 ip-172-31-28-35 cron[959]: (CRON) STARTUP (fork ok)
Jun 4 09:51:12 ip-172-31-28-35 cron[959]: (CRON) INFO (Running #reboot jobs)
Jun 4 09:51:12 ip-172-31-28-35 CRON[1005]: (ubuntu) CMD (/usr/bin/sudo -u ubuntu /usr/local/bin/forever start home/ubuntu/chat2/index.js)
Which shows that the reboot command in my cron tab is working but for some reason forever is still not starting node. After reboot , I run forever list and it says No forever processes running
I am assuming the problem is somehow with the node and forever paths. I am new to this and dont know which exact path to use on this statement in crontab.
I have also tried the following:
#reboot /usr/local/bin/forever start -c /usr/local/bin/node /home/ubuntu/chat2/index.js
and
#reboot /usr/local/bin/forever start /home/ubuntu/chat2/index.js
None of these are working.
If I run which forever it says
/usr/local/bin/forever
If I run which node it says
/usr/local/bin/node
If I get the full path of my index.js app file, by doing readlink -f index.js in my chat2 directory it says
/home/ubuntu/chat2/index.js
I just want to run this command every time my system reboots. I want to start my node app. The following line works perfect when I cd to the chat2 directory manually. I want this to work on reboot itself.
forever -m5000 -w start index.js
You can create a service with you code instead of using cron. Actually I prefer that because you can stop or start it whenever you want and you can also run it on the system reboot or start.
So:
1- Create a service in /etc/init.d/name_of_file
#!/bin/bash
#/etc/init.d/name_of_file
### BEGIN INIT INFO
# Provides: name
# Required-Start: $syslog
# Required-Stop: $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: add service
# Description:
#
### END INIT INFO
# Some things that run always
case "$1" in
start)
echo "Starting app_name "
touch /var/lock/app_name
cd /where/is/your/file
node index.js &
;;
stop)
echo " Stopping "
rm /var/lock/app_name
sudo pkill -f node
;;
status)
if [ -e /var/lock/app_name ]
then
echo "app_name is running"
else
echo "app_name is not running"
fi
;;
*)
echo "Usage:service app_name{start|stop|status}"
exit 1
;;
esac
exit 0
So after that you have created a service for running you nodejs application.
You have to give running permission to that script
chmod +x /etc/init.d/app_name
Now the only thing you have to do is configure this to run on boot.
Run:
sudo update-rc.d app_name defaults
And then every time you reboot you computer the service will start itself.
Suggest redirect stdout/stderr to file to debug why your script in crontab not work:
/usr/local/bin/forever start -c /usr/local/bin/node /home/ubuntu/chat2/index.js >/tmp/forever.log 2>&1 &
See log file for details after reboot.
You also can try pm2 , like forever but support buildin system start script generate, and will launch your apps after reboot.

Upstart node.js working directory

Starting Node.js with Upstart, when trying to access files within Node.js it cannot access them without using the full path. I need it to use the working directory.
start on startup
stop on shutdown
script
echo $$ > /var/run/mynodeapp.pid
exec sudo -u mynodeapp node server.js >> /var/log/mynodeapp.sys.log 2>&1
end script
pre-start script
echo "Starting" >> /var/log/mynodeapp.sys.log
end script
pre-stop script
rm /var/run/mynodeapp.pid
echo "Stopping" >> /var/log/mynodeapp.sys.log
end script
The solution is to change directory within the script. In my case, the user is mynodeapp and the node files are in the users directory (/home/mynodeapp/).
script
chdir /home/mynodeapp/
echo $$ > /var/run/mynodeapp.pid
exec sudo -u mynodeapp node server.js >> /var/log/mynodeapp.sys.log 2>&1
end script
I have yet to find out what $$ means on the echo line or 2>&1. Maybe somebody could chime in with this if they know!
You should use the chdir stanza as per the Upstart docs: http://upstart.ubuntu.com/cookbook/#chdir

Upstart script for node.js app

I'm having trouble starting an Upstart script.
Here's the script (app.conf in /etc/init/)
description "node.js server"
author "kvz"
start on startup
stop on shutdown
script
# We found $HOME is needed. Without it, we ran into problems
export HOME="/root"
exec sudo -u /usr/local/bin/node \
/var/www/vhosts/travelseguro.com/node/app.js \
2>&1 >> /var/log/node.log
end script
When I run sudo start app, I get:
start: Unknown job: app
How can I make this work?
I was having the same problem running on the latest Amazon (AWS) linux which is Redhat based.
I have my upstart file in /etc/init called node.conf and when I ran sudo start node I would get a similar error to you start: Unknown job: node.
It turns out that the job won't start if there's an error in your .conf file. So I started out by commenting out all the lines and slowly building up to find the error. The error message isn't very clear and makes it look like upstart can't find your conf file.
Tailing your '/var/log/messages' will help you debug as Upstart logs to there (It may be somewhere different on Ubuntu. Mine said init: /etc/init/node-upstart.conf:8: Unknown stanza which helped me get to the bottom of it. In my particular case I was declaring variables incorrectly.
See on AskUbuntu for a similar thread.
Here's my edited working script:
<!-- language: lang-sh -->
#!upstart
# using upstart http://upstart.ubuntu.com/getting-started.html and node forever https://github.com/nodejitsu/forever/
# to run server
# redhat has additional sudo restrictions, you must comment out 'Default requiretty' from /etc/sudoers
#startup monitoring script derived from http://stackoverflow.com/questions/11084279/node-js-setup-for-easy-deployment-and-updating
description "node.js server"
author "jujhar"
env PROGRAM_NAME="node"
env FULL_PATH="/home/httpd/buto-middleman/public"
env FILE_NAME="forever.js"
env NODE_PATH="/usr/local/bin/node"
env USERNAME="springloops"
start on startup
stop on shutdown
script
export HOME="/root"
export NODE_ENV=staging #development/staging/production
echo $$ > /var/run/$PROGRAM_NAME.pid
cd $FULL_PATH
#exec sudo -u $USERNAME $NODE_PATH $FULL_PATH/$FILE_NAME >> /var/log/$PROGRAM_NAME.sys.log 2>&1
exec $NODE_PATH $FULL_PATH/$FILE_NAME >> /var/log/$PROGRAM_NAME.sys.log 2>&1
end script
pre-start script
# Date format same as (new Date()).toISOString() for consistency
echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Starting" >> /var/log/$PROGRAM_NAME.sys.log
end script
pre-stop script
rm /var/run/$PROGRAM_NAME.pid
echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Stopping" >> /var/log/$PROGRAM_NAME.sys.log
end script
-- Edit 2013-06-01 --
If you're on Centos or Amazon Linux like me, take a look at this init.d script.
-- Edit 2013-10-14 --
Here's a link to a gist of an init.d script that I actually use in production on Amazon Linux(Redhat Based). I simply keep it in my project under an init.d folder and then symlink to it in the /etc/init.d folder and now it's a daemon/service!
-- Edit 2014-06-05 --
Check out this awesome blog artcile by Jeff Dickey on Node.js in production using systemd which is much cleaner and easier than all the stuff we've been doing here (IMHO). He also uses Ansible to control his cluster (which I love) but you don't have to go that far if you're not ready.
After a few attempts I implemented working .conf file for upstart which works as a service with automatic start after reboot and restart (respawn) in case of crash. Also it can start my app with unprivileged user permissions. The name of the file is /etc/init/my-app.conf.
To start / stop service please use
sudo start my-app / sudo stop my-app
If you have an error like
start: Unknown job: my-app
exec the following command
sudo initctl reload-configuration
My /etc/init/my-app.conf file:
#my-app
description "node.js my-app website"
env FULL_PATH="/home/myuser/app.prod/app"
env NODE_PATH="/usr/bin/node"
start on filesystem or runlevel [2345]
stop on [!2345]
script
export HOME="/root"
export NODE_ENV=production
echo $$ > /var/run/my-app.pid
cd $FULL_PATH
#Use exec below if you want to launch it under myuser,
#don't forget to create /var/log/my-app.sys.log with appropriate permissions
#exec sudo -u myuser sh -c "$NODE_PATH server.js >> /var/log/my-app.sys.log 2>&1"
exec $NODE_PATH server.js >> /var/log/my-app.sys.log 2>&1
end script
pre-start script
echo "[`date`] (sys) Starting" >> /var/log/my-app.sys.log
end script
pre-stop script
rm /var/run/my-app.pid
echo "[`date`] (sys) Stopping" >> /var/log/my-app.sys.log
end script
#uncomment respawn if you want to restart your service in case of crash
#respawn
#respawn limit 50 30
I do recommend to uncomment respawn after you will make sure that everything works ok.
UPDATE
I improved my script (please keep in mind that it works not under root but under regular user zn ):
#znapi.conf
description "node.js zn api"
env FULL_PATH="/home/zn/app.prod"
env NODE_PATH="/usr/bin/node"
env LOG="/var/log/znapi.log"
env PIDFILE="/var/run/znapi.pid"
# Start up when the system hits any normal runlevel, and
#start on filesystem or runlevel [2345]
#start when mongod started
start on started mongod
# shuts down when the system goes to shutdown or reboot.
stop on runlevel [06]
respawn
respawn limit 50 5
pre-start script
# check that log file exist or create it
test -f $LOG || { touch $LOG; chown zn:zn $LOG; }
# Date format same as (new Date()).toISOString() for consistency
echo "[`date`] (sys) Starting" >> $LOG
end script
script
export NODE_ENV=production
exec start-stop-daemon --start -m -p $PIDFILE -c zn -d $FULL_PATH -x server.js >> $LOG 2>&1
end script
pre-stop script
rm $PIDFILE
echo "[`date`] (sys) Stopping" >> $LOG
end script

Upstart script for nodejs daemon in CentOS doesn't work (crashes on start)

I have the following in a .conf file in the /etc/init/ directory of my CentOS server:
#!upstart
description "shortnr server for fmc.io"
author "Felix Milea-Ciobanu"
start on startup
stop on shutdown
respawn
respawn limit 10 30
script
export HOME="/root"
exec /usr/local/bin/node /var/www/fmc.io/nodejs/app.js >> /var/www/fmc.io/logs/shortnr.upstart.log 2>&1
end script
pre-start script
echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Starting" >> /var/www/fmc.io/logs/shortnr.upstart.log
end script
pre-stop script
echo "[`date -u +%Y-%m-%dT%T.%3NZ`] (sys) Stopping" >> /var/www/fmc.io/logs/shortnr.upstart.log
end script
It's a pretty simple and straight forward upstart script. I named this service shortnr, after the nodejs software that the script starts up.
At the command line if I type in start shortnr I get something along the lines of shortnr start/running, process 28350.
However, I can't seem to access the nodejs server; If I do ps aux | grep shortnr at the command shell, nothing comes up.
If I do stop shortnr after running start, I get stop: Unknown instance:, meaning that the original service never started up.
The log file that I setup in the Upstart script looks something like this:
[2012-10-05T17:00:17.174Z] (sys) Starting
[2012-10-05T17:00:17.181Z] (sys) Starting
[2012-10-05T17:00:17.190Z] (sys) Starting
[2012-10-05T17:00:17.197Z] (sys) Starting
[2012-10-05T17:00:17.204Z] (sys) Starting
Basically the script is trying to start multiple times a second when I issued the start command, meaning that the service must be crashing on start or something and trying to respawn?
However, if I copy the command after exec and paste it in the shell prompt, the nodejs script starts up and runs properly.
So that means something must be wrong with my Upstart script.
If I try start/stop the service with the initctl command, I get the same results.
I'm running CentOS 6.3 and Upstart 0.6.5
Anyone have any idea what could be causing this or how to fix my script?
While I couldn't figure out the answer to my problem, I just ended up using forever instead: https://github.com/nodejitsu/forever-monitor
I am also running into similar problems on CentOS 6.3, Upstart 0.6.5 and Node.Js 0.10.5. I specifically upgraded Node so I could use the daemon module and be able to put the daemonized Node app under Upstart control.
Here's my /etc/init/job-worker.conf:
description "job-worker under Upstart/init control"
start on job-worker-start-event
stop on job-worker-stop-event
expect daemon
script
#setuid myuser
exec /root/BasicJobWorker/bin/basic-job-worker
#sleep 5
end script
respawn
respawn limit 10 5
And here's my basic-job-worker script:
\#!/usr/bin/env node
// this code is run twice
// see implementation notes below
console.log(process.pid);
// after this point, we are a daemon
require('daemon')();
// different pid because we are now forked
// original parent has exited
console.log(process.pid);
var BasicJobWorker = require('../lib/basic-job-worker.js');
new BasicJobWorker().boot();
I have tried using "expect fork", "expect daemon" as well as no expect at all. In all cases the job is respawned too fast and it is eventually stopped.

Upstart: Error when using command substitution in post-start script stanza during startup sequence

I'm seeing an issue in upstart where using command substitution inside a post-start script stanza causes an error (syslog reports "terminated with status 1"), but only during the initial system startup.
I've tried using just about every startup event hook under the sun. local-filesystems and net-device-up worked without error about 1/100 tries, so it looks like a race condition. It works just fine on manual start/stop. The command substitutions I've seen trigger the error are a simple cat or date, and I've tried using both the $() way and the backtick way. I've also tried using sleep in pre-start to beat the race condition but that did nothing.
I'm running Ubuntu 11.10 on VMWare with a Win7 host. Spent too many hours troubleshooting this already... Anyone got any ideas?
Here is my .conf file for reference:
start on runlevel [2345]
stop on runlevel [016]
env NODE_ENV=production
env MYAPP_PIDFILE=/var/run/myapp.pid
respawn
exec start-stop-daemon --start --make-pidfile --pidfile $MYAPP_PIDFILE --chuid node-svc --exec /usr/local/n/versions/0.6.14/bin/node /opt/myapp/live/app.js >> /var/log/myapp/audit.node.log 2>&1
post-start script
MYAPP_PID=`cat $MYAPP_PIDFILE`
echo "[`date -u +%Y-%m-%dT%T.%3NZ`] + Started $UPSTART_JOB [$MYAPP_PID]: PROCESS=$PROCESS UPSTART_EVENTS=$UPSTART_EVENTS" >> /var/log/myapp/audit.upstart.log
end script
post-stop script
MYAPP_PID=`cat $MYAPP_PIDFILE`
echo "[`date -u +%Y-%m-%dT%T.%3NZ`] - Stopped $UPSTART_JOB [$MYAPP_PID]: PROCESS=$PROCESS UPSTART_STOP_EVENTS=$UPSTART_STOP_EVENTS EXIT_SIGNAL=$EXIT_SIGNAL EXIT_STATUS=$EXIT_STATUS" >> /var/log/myapp/audit.upstart.log
end script
The most likely scenario I can think of is that $MYAPP_PIDFILE has not been created yet.
Because you have not specified an 'expect' stanza, the post-start is run as soon as the main process has forked and execed. So, as you suspected, there is probably a race between start-stop-daemon running node and writing that pidfile and /bin/sh forking, execing, and forking again to exec cat $MYAPP_PIDFILE.
The right way to do this is to rewrite your post-start as such:
post-start script
for i in 1 2 3 4 5 ; do
if [ -f $MYAPP_PIDFILE ] ; then
echo ...
exit 0
fi
sleep 1
done
echo "timed out waiting for pidfile"
exit 1
end script
Its worth noting that in Upstart 1.4 (included first in Ubuntu 12.04), upstart added logging ability, so there's no need to redirect output into a special log file. All console output defaults to /var/log/upstart/$UPSTART_JOB.log (which is rotated by logrotate). So those echos could just be bare echos.

Resources