In my usecase, I am trying to write an upstart script for grafana service. Here is my content,
#grafana upstart script
description "start and stop grafana server"
start on (net-device-up
and local-filesystems
and runlevel [2345])
stop on runlevel [016]
respawn
respawn limit 5 30
console output
exec service grafana-server stop
But this is not woking all the time. Do I need remove any pid after stopping the service?
Also is this the right way to create upstart conf with service. Because I tried to do the same for nginx service. But it fails because we need to remove the pid after stopping the service.
So in general how can we write upstart script for services?
I am using amazon linux [ec2]. Could someone help me with this?
You can just look at any script in the folder /etc/init.d and change the instructions according your your needs. The you can use the systemctl (or the appropiate command for your distribution) to make it run on the desired init levels
Couldn't seem to find a direct answer around here.
I'm not sure if I should run ./myBinary as a Cron process or if I should run "go run myapp.go"
What's an effective way to make sure that it is always running?
Sorry I'm used to Apache and Nginx.
Also what are best practices for deploying a Go app? I want everything (preferably) all served on the same server. Just like how my development environment is like.
I read something else that used S3, but, I really don't want to use S3.
Use the capabilities your init process provides. You're likely running system with either Systemd or Upstart. They've both got really easy descriptions of services and can ensure your app runs with the right privileges, is restarted when anything goes down, and that the output is are handled correctly.
For quick Upstart description look here, your service description is likely to be just:
start on runlevel [2345]
stop on runlevel [!2345]
setuid the_username_your_app_runs_as
exec /path/to/your/app --options
For quick Systemd description look here, your service is likely to be just:
[Unit]
Description=Your service
[Service]
User=the_username_your_app_runs_as
ExecStart=/path/to/your/app --options
[Install]
WantedBy=multi-user.target
You can put it in an inifiny loop, such as:
#! /bin/sh
while true; do
go run myapp.go
sleep 2 # Just in case
done
Hence, once the app dies due some reason, it will be run again.
You can put it in a script and run it in background using:
$ nohup ./my-script.sh >/dev/null 2>&1 &
You may want to go for virtual terminal utility like screen here. Example:
screen -S myapp # create screen with name myapp
cd ... # to your app directory
go run myapp.go # or go install and then ./myappfrom go bin dir
Ctrl-a+d # to go out of screen
If you want to return to the screen:
screen -r myapp
EDIT: this solution will persist the process when you go out of terminal, but won't restart it when it'll crash.
We have a series of node.js scripts on an Ubuntu (13.10) server that we want to keep up and running as much as possible, and restart automatically in the event of a server reboot. We've tried a few different techniques, but have yet to find a solution that works.
Setup: None of the scripts run on port 80, instead we run them on ports above 8000.
Node.js files are currently running in /usr/lib/sites/path/Node
We have set them up and running individually using Forever while running in the context of a well privileged (but not root) user, simply by running the following from the context of the folder containing the scripts:
forever start server_process.js
We want to run these scripts at server start up, and have some ability later (if needed) to restart them.
Upstart sounds like it should be the solution, but we've not yet managed to get it working. The following script starts, then stops without indicating why...
description "Our app"
version "1.0"
author "Nautilytics"
start on startup
stop on shutdown
expect fork
env FOREVER_PATH=/usr/bin/forever
env APPLICATION_DIRECTORY=/usr/lib/sites/path/Node
env APPLICATION_START=ourapp.js
env LOG_PATH=/var/log/ourapp.log
chdir /usr/lib/sites/path/Node
script
exec $FOREVER_PATH start --sourceDir $APPLICATION_DIRECTORY -f -v $APPLICATION_START >> $LOG_PATH 2&>1
end script
Through blunt trial and error, a couple of times we have been able to get errors that indicate that other files (required by ourapp.js) could not be found, as if the chdir has not worked or passed through into the forever start.
After scouring the internet for a solution, we decided to stick with nodejs and not use forever for this task. It was much easier and with respawn it does just about everything we needed forever to do for us.
start on runlevel [2345]
stop on shutdown
respawn
script
exec sudo nodejs /usr/lib/sites/path/Node/ourapp.js 2>&1 >> /var/log/ourapp.log
end script
A bit more secure upstart config script might be (hardened-nodejs.conf):
description "Managing and monitoring nodejs application"
# start when filesystem is mounted networking is up
start on (filesystem and net-device-up IFACE!=lo)
# stop on shutting down the system
stop on runlevel [016]
# application environment
# staging and development instances should use hardened-nodejs.override to define environment
env NODE_ENV=production
# respawn the job up to 10 times within a 5 second period.
# If the job exceeds these values, it will be stopped and marked as failed.
respawn
respawn limit 10 5
# ssl-cert group can read certificates
setuid www-data
setgid ssl-cert
exec /usr/bin/nodejs /usr/lib/sites/path/Node/ourapp.js 2>&1 >> /var/log/ourapp.log
One of solutions to run nodejs application listening on a low port might be to use capabilities tools.
If serving over SSL user or group must have read access to the certificate. Check ssl-cert package and this answer for basic concepts.
Have you checked process.cwd()? It could be that your process is running elsewhere.
I've been setting up my server recently and today I had to restart it... then I realised all of my Node apps I had running weren't running anymore. I'm using Node Forever module to keep the apps running, but then I realised I still need to have them starting when my server restarts or shut downs and powers up again.
I have been researching the best way to do this, but what I'm trying just doesn't seem to work. I've created an Upstart script in my /etc/init/ folder on my Ubuntu Server 10.04LTS remote server and tried restarting and it doesn't seem to do anything. Nothing is getting listed when I run forever list.
Here is my current Upstart script I was trying out today:
#/etc/init/myapp.conf
start on (local-filesystems and net-device-up IFACE=eth0)
stop on shutdown
script
exec sudo /usr/local/bin/node /var/www/myapp/myapp.forever.js
end script
I use Forever in a Node script as I find it easier to configure it how I want. It's confirmed that the script runs just fine if I do this outside the script, there is just something wrong with the Upstart script itself. It seems to have the same permissions as all the other Upstart scripts in /etc/init/ folder.
As an additional note, I have gone through almost all the answers I could find here on StackOverflow, and that it how I got together the script that I have at present.
UPDATE:
With Tom's answer, I have now tried:
#/etc/init/myapp.conf
start on (local-filesystems and net-device-up IFACE=eth0)
stop on shutdown
exec sudo /usr/local/bin/node /var/www/myapp/myapp.forever.js
But it's still not working.
So I don't know why this isn't running when I restart my server. Please help!
This is not a very happy setup. The way upstart works is that it starts your process running it using the process id for the start command. Forever JS works similarly, it is probably inspired by Upstart.
When you try to run forever.js with upstart, the forever process you create in your upstart script exits immediately after starting. Upstart counts on having the process continue to run.
When I tried to run forever using upstart, I wound up with five different forever process running because upstart thought it had failed to start forever, and it retried five times.
Did you try doing it without the start script lines?
description "my server"
author "name"
start on (local-filesystems and net-device-up IFACE=eth0)
stop on shutdown
#respawn if you were not using forever
exec sudo /usr/local/bin/node myapp.forever.js
Source: http://caolanmcmahon.com/posts/deploying_node_js_with_upstart
I've opted to use an #reboot statement in the user's crontab file, which will execute forever on server restarts.
#reboot forever start app.js
Additional Reading - http://www.cyberciti.biz/faq/linux-execute-cron-job-after-system-reboot/
What is the best way to deploy Node.js?
I have a Dreamhost VPS (that's what they call a VM), and I have been able to install Node.js and set up a proxy. This works great as long as I keep the SSH connection that I started node with open.
2016 answer: nearly every Linux distribution comes with systemd, which means forever, monit, PM2, etc. are no longer necessary - your OS already handles these tasks.
Make a myapp.service file (replacing 'myapp' with your app's name, obviously):
[Unit]
Description=My app
[Service]
ExecStart=/var/www/myapp/app.js
Restart=always
User=nobody
# Note Debian/Ubuntu uses 'nogroup', RHEL/Fedora uses 'nobody'
Group=nogroup
Environment=PATH=/usr/bin:/usr/local/bin
Environment=NODE_ENV=production
WorkingDirectory=/var/www/myapp
[Install]
WantedBy=multi-user.target
Note if you're new to Unix: /var/www/myapp/app.js should have #!/usr/bin/env node on the very first line and have the executable mode turned on chmod +x myapp.js.
Copy your service file into the /etc/systemd/system folder.
Tell systemd about the new service with systemctl daemon-reload.
Start it with systemctl start myapp.
Enable it to run on boot with systemctl enable myapp.
See logs with journalctl -u myapp
This is taken from How we deploy node apps on Linux, 2018 edition, which also includes commands to generate an AWS/DigitalOcean/Azure CloudConfig to build Linux/node servers (including the .service file).
Use Forever. It runs Node.js programs in separate processes and restarts them if any dies.
Usage:
forever start example.js to start a process.
forever list to see list of all processes started by forever
forever stop example.js to stop the process, or forever stop 0 to stop the process with index 0 (as shown by forever list).
I've written about my deployment method here: Deploying node.js apps
In short:
Use git post-receive hook
Jake for the build tool
Upstart as a service wrapper for node
Monit to monitor and restart applications it they go down
nginx to route requests to different applications on the same server
pm2 does the tricks.
Features are: Monitoring, hot code reload, built-in load balancer, automatic startup script, and resurrect/dump processes.
You can use monit, forever, upstart or systemd to start your server.
You can use Varnish or HAProxy instead of Nginx (Nginx is known not to work with websockets).
As a quick and dirty solution you can use nohup node your_app.js & to prevent your app terminating with your server, but forever, monit and other proposed solutions are better.
I made an Upstart script currently used for my apps:
description "YOUR APP NAME"
author "Capy - http://ecapy.com"
env LOG_FILE=/var/log/node/miapp.log
env APP_DIR=/var/node/miapp
env APP=app.js
env PID_NAME=miapp.pid
env USER=www-data
env GROUP=www-data
env POST_START_MESSAGE_TO_LOG="miapp HAS BEEN STARTED."
env NODE_BIN=/usr/local/bin/node
env PID_PATH=/var/opt/node/run
env SERVER_ENV="production"
######################################################
start on runlevel [2345]
stop on runlevel [016]
respawn
respawn limit 99 5
pre-start script
mkdir -p $PID_PATH
mkdir -p /var/log/node
end script
script
export NODE_ENV=$SERVER_ENV
exec start-stop-daemon --start --chuid $USER:$GROUP --make-pidfile --pidfile $PID_PATH/$PID_NAME --chdir $APP_DIR --exec $NODE_BIN -- $APP >> $LOG_FILE 2>&1
end script
post-start script
echo $POST_START_MESSAGE_TO_LOG >> $LOG_FILE
end script
Customize all before #########, create a file in /etc/init/your-service.conf and paste it there.
Then you can:
start your-service
stop your-service
restart your-service
status your-service
I've written a pretty comprehensive guide to deploying Node.js, with example files:
Tutorial: How to Deploy Node.js Applications, With Examples
It covers things like http-proxy, SSL and Socket.IO.
Here's a longer article on solving this problem with systemd: http://savanne.be/articles/deploying-node-js-with-systemd/
Some things to keep in mind:
Who will start your process monitoring? Forever is a great tool, but it needs a monitoring tool to keep itself running. That's a bit silly, why not just use your init system?
Can you adequately monitor your processes?
Are you running multiple backends? If so, do you have provisions in place to prevent any of them from bringing down the others in terms of resource usage?
Will the service be needed all the time? If not, consider socket activation (see the article).
All of these things are easily done with systemd.
If you have root access you would better set up a daemon so that it runs safe and sound in the background. You can read how to do just that for Debian and Ubuntu in blog post Run Node.js as a Service on Ubuntu.
Forever will do the trick.
#Kevin: You should be able to kill processes fine. I would double check the documentation a bit. If you can reproduce the error it would be great to post it as an issue on GitHub.
Try this: http://www.technology-ebay.de/the-teams/mobile-de/blog/deploying-node-applications-with-capistrano-github-nginx-and-upstart.html
A great and detailed guide for deploying Node.js apps with Capistrano, Upstart and Nginx
As Box9 said, Forever is a good choice for production code. But it is also possible to keep a process going even if the SSH connection is closed from the client.
While not necessarily a good idea for production, this is very handy when in the middle of long debug sessions, or to follow the console output of lengthy processes, or whenever is useful to disconnect your SSH connection, but keep the terminal alive in the server to reconnect later (like starting the Node.js application at home and reconnecting to the console later at work to check how things are going).
Assuming that your server is a *nix box, you can use the screen command from the shell to do keep the process running even if the client SSH is closed. You can download/install screen from the web if not already installed (look for a package for your distribution if Linux, or use MacPorts if OS X).
It works as following:
When you first open the SSH connection, type 'screen' - this will start your screen session.
Start working as normal (i.e. start your Node.js application)
When you are done, close your terminal. Your server process(es) will continue running.
To reconnect to your console, ssh back to the server, login, and enter 'screen -r' to reconnect. Your old console context will pop back ready for you to resume using it.
To exit screen, while connected to the server, type 'exit' on the console prompt - that will drop you onto the regular shell.
You can have multiple screen sessions running concurrently like this if you need, and you can connect to any of it from any client. Read the documentation online for all the options.
Forever is a good option for keeping apps running (and it's npm installable as a module which is nice).
But for more serious 'deployment' -- things like remote management of deploying, restarting, running commands etc -- I would use capistrano with the node extension.
https://github.com/loopj/capistrano-node-deploy
https://paastor.com is a relatively new service that does the deploy for you, to a VPS or other server. There is a CLI to push code. Paastor has a free tier, at least it did at the time of posting this.
In your case you may use the upstart daemon. For a complete deployment solution, I may suggest capistrano. Two useful guides are How to setup Node.js env and How to deploy via capistrano + upstart.
Try node-deploy-server. It is a complex toolset for deploying an application onto your private servers. It is written in Node.js and uses npm for installation.