I'm currently running a python script as a systemd service. Is there any way to "switch" into the service and take control of the script? The script has a menu but runs calculations in the background in another thread. Most likely not, so is there a way to run a python script 24/7, start on boot, restart on crash etc (just like systemd service) but be able to take control of it after I connect to the server via SSH, so I can manipulate the app?
One solution you could try would be to edit the systemd configuration to launch the process in screen or tmux, then attach that when logging in via SSH.
For instance, in the systemd unit, you might have:
[Service]
Type=single
ExecStart=tmux new "command"
Using Type=single will treat the tmux command as the main process, which would be killed if you stop it with systemctl stop systemprocess
The ExecStart=tmux new "command" creates a new tmux session with the command inside of it.
You can then attach to it using tmux attach as the same user the systemd unit is running as (I believe this is root by default).
I have used various snippets of code to build a system which
listens to a port for incoming TCP data (using a perl script), writes this data to a log file.
calls and runs a PHP script to consume the log file and write it to an RDS MySQL DB
I have a GPS device configured to send the data to the elastic IP of my AWS EC2 Server
It works fine, and when i run via SSH
perl portlistener.pl
it does it's job fine, happily working away.
The only way I can stop the script running is by closing the terminal window, ending my SSH session. What I need to do, is keep it running at all times, and to implement a start, stop and restart facility. Do i need to create a daemon?
I know PHP, but until now have never worked with Perl. I'm also not that familiar with command line, other than installing updates, navigating and editing single files etc.
Thanks in advance for any help, or for pointing me in the right direction.
Solved it I think!
installed CPAN http://www.thegeekstuff.com/2008/09/how-to-install-perl-modules-manually-and-using-cpan-command/
Using CPAN, installed Deamon::Control
Then created a new program as below (portlistener_launcher.pl), and ran it as SU.
#!/usr/bin/perl
use strict;
use warnings;
use Daemon::Control;
$ENV{PHP_FCGI_CHILDREN} = 10;
$ENV{PHP_FCGI_MAX_REQUESTS} = 1000;
Daemon::Control->new({
name => 'portlistener',
program => 'perl /home/ec2-user/portlistener/portlistener.pl',
fork => 2,
pid_file => '/var/run/portlistener.pid',
stdout_file => '/var/log/portlistener.log',
stderr_file => '/var/log/portlistener.log',
})->run;
There's probably a neater way of doing it, but it seems to work, and I can stop/start it like so:
perl portlistener_launcher.pl start
If the terminal window is the only task, you can use the nohup command, e.g.
http://linux.101hacks.com/unix/nohup-command/
To terminate the listener you can kill an appropriate running process or processes.
An implementation of daemon does not ensure its permanent running. It can crash or might be killed from someone. To guarantee the permanent daemon running you must implement a 24x7 monitoring of this daemon and automatic restarting of it.
I'm deploying a node web application as an upstart service using grunt and monitoring it using monit. However:
My upstart and monit configuration duplicate each other a little bit
Upstart doesn't do variable expansion inside env stanzas
I can't find a way to configure monit dynamically (ala upstart .override files)
My question
This means I'm looking for a grunt plugin or other tool that I can use to generate the uptstart .conf and monit conf.d/ files. Can you please help me find one (or suggest a better way of robustly running my node web app)?
A rather crude solution?
To be honest an underscore template of the upstart and monit files would probably be sufficient, and that's what I'll wrap up into a grunt plugin if there isn't a ready-made solution, but this feels like a problem that other people must have run into as well so I imagine there's a solution out there, I just can't find it.
Detail
A bit more detail to illustrate the three problems. My upstart conf file looks like this:
setuid node
setgid node
# ...
script
mkdir -p /home/node/.my-app
echo $$ > /home/node/.my-app/upstart.pid
/usr/local/bin/node /var/node/my-app/server.js >> /var/node/my-app/logs/console.log 2>&1
end script
# ...
And my monit configuration looks like this:
check process node with pidfile /home/node/.my-app/upstart.pid
start program = "/sbin/start my-app" with timeout 60 seconds
stop program = "/sbin/stop my-app"
if failed host localhost port 17394 protocol http
and request "/monit.html"
then restart
if 3 restarts within 5 cycles then timeout
As you can see, the PID file path and config directory is duplicated across the two (problem 1) and I'd love to parameterise the host, port and request URL in the monit file (problem 3).
For problem 2 (no variable expansion in upstart's env stanza, bug report here) there are a couple of workarounds that work for variables used inside *script blocks, which are interpreted as bash scripts, but they don't seem to work in the conf file itself. I think this makes it impossible to specify the user id the app should run as in a configuration file?
Those techniques I just mentioned:
Method 1: Don't use env - echo the variables in the pre-script to a file and then source it later
Method 2: Duplicate the variable expansion in all the script bodies where it is needed
Method 3: Store the variables in a file and cat them in using command substitution
...or suggest a better way of robustly running my node web app
Use PM2 by Unitech instead.
I run all my node apps using PM2. It works perfectly and is easy to configure. It has built-in functionality for autogenerating of startup script. It gives you logging, monitoring and easy maintenance.
Here is a good article showing off the highlights.
Install
npm install -g pm2
Start app
pm2 start app.js
Show running apps
pm2 list
Make a startup script
pm2 startup [ubuntu|centos|systemd]
More details in the readme on their github page
I need to have some processes start when the computer boots and run forever. These are not actually daemons, ie. they do not fork or demonize but they do not exit. I am currently using cron to start them using the #reboot directive like this:
#reboot /path/to/myProcess >>/logs/myProcess.log
Could this cause any problems with the cron daemon? I thought I could try nohup ... & to detach the new process from cron, like this:
#reboot nohup /path/to/myProcess >>/logs/myProcess.log &
Is this required at all?
Is there some other, preferred method to start processes at system boot? I know all Linux distributions provide config files and means to run a program as a service but I am looking for a method that is not Linux distribution specific.
http://www.somacon.com/p38.php
This article answers my question. It suggests that running daemons this way spawns two extra processes, a cron and a shell process, that live for as long as your daemon.
I tested this with linux and following the instructions I was able to get rid of the cron process but not the zombie shell process.
What is the best way to deploy Node.js?
I have a Dreamhost VPS (that's what they call a VM), and I have been able to install Node.js and set up a proxy. This works great as long as I keep the SSH connection that I started node with open.
2016 answer: nearly every Linux distribution comes with systemd, which means forever, monit, PM2, etc. are no longer necessary - your OS already handles these tasks.
Make a myapp.service file (replacing 'myapp' with your app's name, obviously):
[Unit]
Description=My app
[Service]
ExecStart=/var/www/myapp/app.js
Restart=always
User=nobody
# Note Debian/Ubuntu uses 'nogroup', RHEL/Fedora uses 'nobody'
Group=nogroup
Environment=PATH=/usr/bin:/usr/local/bin
Environment=NODE_ENV=production
WorkingDirectory=/var/www/myapp
[Install]
WantedBy=multi-user.target
Note if you're new to Unix: /var/www/myapp/app.js should have #!/usr/bin/env node on the very first line and have the executable mode turned on chmod +x myapp.js.
Copy your service file into the /etc/systemd/system folder.
Tell systemd about the new service with systemctl daemon-reload.
Start it with systemctl start myapp.
Enable it to run on boot with systemctl enable myapp.
See logs with journalctl -u myapp
This is taken from How we deploy node apps on Linux, 2018 edition, which also includes commands to generate an AWS/DigitalOcean/Azure CloudConfig to build Linux/node servers (including the .service file).
Use Forever. It runs Node.js programs in separate processes and restarts them if any dies.
Usage:
forever start example.js to start a process.
forever list to see list of all processes started by forever
forever stop example.js to stop the process, or forever stop 0 to stop the process with index 0 (as shown by forever list).
I've written about my deployment method here: Deploying node.js apps
In short:
Use git post-receive hook
Jake for the build tool
Upstart as a service wrapper for node
Monit to monitor and restart applications it they go down
nginx to route requests to different applications on the same server
pm2 does the tricks.
Features are: Monitoring, hot code reload, built-in load balancer, automatic startup script, and resurrect/dump processes.
You can use monit, forever, upstart or systemd to start your server.
You can use Varnish or HAProxy instead of Nginx (Nginx is known not to work with websockets).
As a quick and dirty solution you can use nohup node your_app.js & to prevent your app terminating with your server, but forever, monit and other proposed solutions are better.
I made an Upstart script currently used for my apps:
description "YOUR APP NAME"
author "Capy - http://ecapy.com"
env LOG_FILE=/var/log/node/miapp.log
env APP_DIR=/var/node/miapp
env APP=app.js
env PID_NAME=miapp.pid
env USER=www-data
env GROUP=www-data
env POST_START_MESSAGE_TO_LOG="miapp HAS BEEN STARTED."
env NODE_BIN=/usr/local/bin/node
env PID_PATH=/var/opt/node/run
env SERVER_ENV="production"
######################################################
start on runlevel [2345]
stop on runlevel [016]
respawn
respawn limit 99 5
pre-start script
mkdir -p $PID_PATH
mkdir -p /var/log/node
end script
script
export NODE_ENV=$SERVER_ENV
exec start-stop-daemon --start --chuid $USER:$GROUP --make-pidfile --pidfile $PID_PATH/$PID_NAME --chdir $APP_DIR --exec $NODE_BIN -- $APP >> $LOG_FILE 2>&1
end script
post-start script
echo $POST_START_MESSAGE_TO_LOG >> $LOG_FILE
end script
Customize all before #########, create a file in /etc/init/your-service.conf and paste it there.
Then you can:
start your-service
stop your-service
restart your-service
status your-service
I've written a pretty comprehensive guide to deploying Node.js, with example files:
Tutorial: How to Deploy Node.js Applications, With Examples
It covers things like http-proxy, SSL and Socket.IO.
Here's a longer article on solving this problem with systemd: http://savanne.be/articles/deploying-node-js-with-systemd/
Some things to keep in mind:
Who will start your process monitoring? Forever is a great tool, but it needs a monitoring tool to keep itself running. That's a bit silly, why not just use your init system?
Can you adequately monitor your processes?
Are you running multiple backends? If so, do you have provisions in place to prevent any of them from bringing down the others in terms of resource usage?
Will the service be needed all the time? If not, consider socket activation (see the article).
All of these things are easily done with systemd.
If you have root access you would better set up a daemon so that it runs safe and sound in the background. You can read how to do just that for Debian and Ubuntu in blog post Run Node.js as a Service on Ubuntu.
Forever will do the trick.
#Kevin: You should be able to kill processes fine. I would double check the documentation a bit. If you can reproduce the error it would be great to post it as an issue on GitHub.
Try this: http://www.technology-ebay.de/the-teams/mobile-de/blog/deploying-node-applications-with-capistrano-github-nginx-and-upstart.html
A great and detailed guide for deploying Node.js apps with Capistrano, Upstart and Nginx
As Box9 said, Forever is a good choice for production code. But it is also possible to keep a process going even if the SSH connection is closed from the client.
While not necessarily a good idea for production, this is very handy when in the middle of long debug sessions, or to follow the console output of lengthy processes, or whenever is useful to disconnect your SSH connection, but keep the terminal alive in the server to reconnect later (like starting the Node.js application at home and reconnecting to the console later at work to check how things are going).
Assuming that your server is a *nix box, you can use the screen command from the shell to do keep the process running even if the client SSH is closed. You can download/install screen from the web if not already installed (look for a package for your distribution if Linux, or use MacPorts if OS X).
It works as following:
When you first open the SSH connection, type 'screen' - this will start your screen session.
Start working as normal (i.e. start your Node.js application)
When you are done, close your terminal. Your server process(es) will continue running.
To reconnect to your console, ssh back to the server, login, and enter 'screen -r' to reconnect. Your old console context will pop back ready for you to resume using it.
To exit screen, while connected to the server, type 'exit' on the console prompt - that will drop you onto the regular shell.
You can have multiple screen sessions running concurrently like this if you need, and you can connect to any of it from any client. Read the documentation online for all the options.
Forever is a good option for keeping apps running (and it's npm installable as a module which is nice).
But for more serious 'deployment' -- things like remote management of deploying, restarting, running commands etc -- I would use capistrano with the node extension.
https://github.com/loopj/capistrano-node-deploy
https://paastor.com is a relatively new service that does the deploy for you, to a VPS or other server. There is a CLI to push code. Paastor has a free tier, at least it did at the time of posting this.
In your case you may use the upstart daemon. For a complete deployment solution, I may suggest capistrano. Two useful guides are How to setup Node.js env and How to deploy via capistrano + upstart.
Try node-deploy-server. It is a complex toolset for deploying an application onto your private servers. It is written in Node.js and uses npm for installation.