ExpressJS Server Goes Offline Every Night - 502 Bad Gateway - node.js

I have a website with Nginx installed as a reserve proxy for an ExpressJS server (proxies to port 3001). This uses Node and ReactJS for my frontend application.
This is simply a testing website currently, and isn't known or used by any users. I have this installed on a Digital Ocean Droplet with Ubuntu.
Every morning when I wake up, I load my website and see 502 Bad Gateway. The problem is, I don't know how to find out how this happened. I have PM2 installed which should automatically restart my ExpressJS server but it hasn't done so, and when I run pm2 list, my application is still showing online:
When I run pm2 logs, I get the following error (I am running this as an Administrator):
So I'll run pm2 restart all to restart the app, but then I don't see any crash information. However on this occasion when taking this screenshot, there were a couple of unusual requests. /robots.txt, /sitemap.xml and /.well-known/security.txt, but nothing indicating a crash:
When I look at my Nginx error.log file, all I can see is the following:
There is, however, something obscure within my access.log ([09/Oct/2018:06:33:19 +0000]) but I have no idea what this means:
If I run curl localhost:3001 whilst the server is offline, I will receive a connection error message. This works fine after I run pm2 restart all.
I'm completely stuck with this and even the smallest bit of help would be appreciated greatly, even if it's just to tell me I'm barking up the wrong tree completely and need to look elsewhere - thank you.

I think you should check this github thread, it seems like it could help you.
Basically, after few hours, a Nodejs server stop functioning, and the poor nginx can not forward its requests, as the service listening to the forward port is dead. So it triggers a 502 error.
It was all due to a memory leak, that leads to a massive garbage collection, then to the server to crash. Check your memory consumption, you could have some surprises. And try to debug your app code, a piece (dependency) at the time.
Updated answer:
So, i will add another branch to my question as it seems it has not helped you so far.
You could try to get rid of pm2, and use systemd to manage your app life cycle.
Create a service file
sudo vim /lib/systemd/system/appname.service
this is a simple file i used myself for a random ExpressJS app:
[Unit]
Description=YourApp Site Server
[Service]
ExecStart=/home/user/appname/index.js
Restart=always
Environment=PATH=/usr/bin:/usr/local/bin
Environment=NODE_ENV=production
WorkingDirectory=/home/user/appname
[Install]
WantedBy=multi-user.target
Note that it will try to restart if it fails somehow Restart=always
Manage it with systemd
Register the new service with:
sudo systemctl daemon-reload
Now start your app from systemd with:
sudo systemctl start appname
from now on you should be able to manage your app life cycle with the usual systemd commands.
You could add stdout and stderr to syslog to understand what you app is doing
StandardOutput=syslog
StandardError=syslog
Hope it helps more

You cannot say when exactly NodeJS will crash, or will do big GC, or will stun for other reason.
Easiest way to cover such issues is to do health check and restart an app. This should not be an issue when working with cluster.
Please look at health check module implementation, you may try to use it, or write some simple shell script to do the check

Related

I have hosted my Nodejs Application on Amazon AWS. Now it shows 502 bad gateway after some time

I started the server using Command prompt by the command "sudo node server.js"
Then I closed the command prompt.
Now after some time, the website automatically shows 502 bad gateway error.
I had also used the command "sudo node server.js &" but it still not works.
Can you help me find out what could be the possible problem.
Thanks in advance!
The problem is that you app is not running, here is why.
When start the application in the command line, the application is handled by that bash instance. When you close that instance, so does the application.
The solution to this is to use a package like pm2 which will make sure that your application is always running.

Systemd service for Node server can't connect to MongoDB

I have created a Systemd unit file to run Mongodb on system startup. I have also created a systemd service to run my node application on startup after mongodb is started. The service for mongodb works fine, but for some reason my service for the Node application tries to run and then gives the error: "MongoError: failed to connect to server [localhost:27017] on first connect". If I start the mongodb service using $systemctl start mongodb and then start my Node application using $/usr/bin/node /node_app_slot/server.js It seems to work fine. So the problem seems to be with my systemd unit file for my Node server.
I used this for the mongodb systemd service https://gist.github.com/jwilm/5842956
And here is my node_server.service:
[Unit]
Wants=network.target mongodb.service
After=network.target mongodb.service
[Service]
ExecStart=/usr/bin/node /node_app_slot/server.js
[Install]
WantedBy=multi-user.target
Im doing this on an intel edison set up in Access Point mode using hostapd. The OS is Yocto and is up to date with the latest release.
I can't see where Im going wrong. I will really appreciate it if someone could guide me in the right direction!
Thanks!
Your systemd syntax is correct, although you may want to use the network-online.target instead of network.target.
A workaround to consider is to add a 5 second sleep before your MongoDB app starts. Since this is happening at server boot time, the extra 5 seconds are unlikely to make a practical difference but may solve your problem.
ExecStartPre=/bin/sleep 5
It is curious that you are copy/pasting a MongoDB systemd service file from the internet when MongoDB ships with their own systemd service files. You didn't mention your OS version, systemd version or MongoDB version, but I would still recommend referencing the official MongoDB systemd configuration files if aren't using a version of MongoDB that ships with the files.
I had similar problem. The problem may be the fact that you try to connect before the network service has finished starting. You can take a look at here to see if it solves your problem:
https://unix.stackexchange.com/questions/126009/cause-a-script-to-execute-after-networking-has-started

Using a Cron Job to check if my mod_wsgi / apache server is running and restart

my group and I are running a server that is based upon Django and uses mod_wsgi to run an Apache server. We will not be working on this project after it is over, so I am attempting to set up cronjob similar functionality to check if the apache server has shut down(system restart or power failure), and if it has, will restart the server for me. I've found documentation on how to check if an apache server is down and restart the server if it is, but our server uses https and thus our start command is pretty verbose.
Can I simply use the functionality provided in these examples:
https://askubuntu.com/questions/277389/cron-job-to-restart-apache
https://www.digitalocean.com/community/tutorials/how-to-use-a-simple-bash-script-to-restart-server-programs
Or do I need a much more complicated process to make this happen?
The command we use to initially start the server is
python manage.py runmodwsgi --host 0.0.0.0 --port 8001 --https-port 8000 --ssl-certificate (certificate Location) --server-name (Domain Name)
I'm pretty new to Linux and using both Mod-wsgi as well as Apache so any help is greatly appreciated.
I suppose it is not good way to resolve this problem.
I recommend you use monit (https://mmonit.com/). It is cool program for checking services.
apt-get install monit
Apache restart configuration directives:
check process httpd with pidfile /var/run/httpd.pid
group apache
start program = "/etc/init.d/httpd start"
stop program = "/etc/init.d/httpd stop"
if failed host 127.0.0.1 port 80
protocol http then restart
if 5 restarts within 5 cycles then timeout
You are better off using the --setup-only option to mod_wsgi-express or the Django integration for it, to generate the configuration but not run it. Then as others have mentioned, integrate it into the system service manager.
The two commands for starting and stopping the Apache/mod_wsgi instance would be apachectl start and apachectl stop, where apachectl is that which was generated when running with the additional --setup-only option.
When running it as a system service, also make sure you use the --server-root option to specify a more persistent location for the generated configuration. Do not use the default under /tmp if running for anything but temporary development sessions as some Linux systems will remove files under /tmp causing things to start failing after a while.
Also, since under a service manager it would generally be starting as root, particularly if listening on port 80 is a requirement, ensure you use the --user and --group options to specify what user/group your Python web application should run as.
Read:
https://pypi.python.org/pypi/mod_wsgi
for more details of the --setup-only option and start-server commands for generating the configuration. Because you are using the Django integration, you will need to use the --setup-only option.
For more informed helped, bring your issue to the mod_wsgi mailing list. The mod_wsgi-express way of running Apache/mod_wsgi is new enough that unlikely that anyone here is really going to know much about it.
There is no need to do this at all. There is no reason to start up Apache manually; once it's installed as a system service, Ubuntu will start it up automatically on restart or crash.
You should reflect on why you feel the need to do this for Apache specifically, and not any of the other system services you depend on, such as the database.

Getting 502 error in browser for a NodeJS app run with Forever

I have a SailsJS app set up on a Webfaction server. Everything works nicely (site can be accessed through browser, console works) when I run the app via any of the following commands, with and without the --prod param:
sails console,
sails lift,
node app.js
However, when I try to run the app with forever using forever app.js I get a 502 error, as if nodejs server isn't even running. When I run forever list I can see app.js listed among running processes.
How can I have my app run with forever?
Forever is considered outdated by many in the Node community, and thankfully, has been replaced by several other fantastic (dare I say, better) tools.
If you're running a newer flavor of Ubuntu, you can always install systemd and kick off the application that way. If you're seeking something more streamlined, Phusion Passenger might be your ticket. It has a long track record of successes, and I wouldn't hesitate to toss it into production.
I managed to solve this issue; the problem occurred due to SailsJS migration prompt which shows up when you start the server. Running app.js with forever worked, but the server didn't start because the script hanged waiting for a prompt reply. If you encounter this issue just make sure you have your migrate option set in model config to avoid running into migration prompt.

How do I run a Node.js application as its own process?

What is the best way to deploy Node.js?
I have a Dreamhost VPS (that's what they call a VM), and I have been able to install Node.js and set up a proxy. This works great as long as I keep the SSH connection that I started node with open.
2016 answer: nearly every Linux distribution comes with systemd, which means forever, monit, PM2, etc. are no longer necessary - your OS already handles these tasks.
Make a myapp.service file (replacing 'myapp' with your app's name, obviously):
[Unit]
Description=My app
[Service]
ExecStart=/var/www/myapp/app.js
Restart=always
User=nobody
# Note Debian/Ubuntu uses 'nogroup', RHEL/Fedora uses 'nobody'
Group=nogroup
Environment=PATH=/usr/bin:/usr/local/bin
Environment=NODE_ENV=production
WorkingDirectory=/var/www/myapp
[Install]
WantedBy=multi-user.target
Note if you're new to Unix: /var/www/myapp/app.js should have #!/usr/bin/env node on the very first line and have the executable mode turned on chmod +x myapp.js.
Copy your service file into the /etc/systemd/system folder.
Tell systemd about the new service with systemctl daemon-reload.
Start it with systemctl start myapp.
Enable it to run on boot with systemctl enable myapp.
See logs with journalctl -u myapp
This is taken from How we deploy node apps on Linux, 2018 edition, which also includes commands to generate an AWS/DigitalOcean/Azure CloudConfig to build Linux/node servers (including the .service file).
Use Forever. It runs Node.js programs in separate processes and restarts them if any dies.
Usage:
forever start example.js to start a process.
forever list to see list of all processes started by forever
forever stop example.js to stop the process, or forever stop 0 to stop the process with index 0 (as shown by forever list).
I've written about my deployment method here: Deploying node.js apps
In short:
Use git post-receive hook
Jake for the build tool
Upstart as a service wrapper for node
Monit to monitor and restart applications it they go down
nginx to route requests to different applications on the same server
pm2 does the tricks.
Features are: Monitoring, hot code reload, built-in load balancer, automatic startup script, and resurrect/dump processes.
You can use monit, forever, upstart or systemd to start your server.
You can use Varnish or HAProxy instead of Nginx (Nginx is known not to work with websockets).
As a quick and dirty solution you can use nohup node your_app.js & to prevent your app terminating with your server, but forever, monit and other proposed solutions are better.
I made an Upstart script currently used for my apps:
description "YOUR APP NAME"
author "Capy - http://ecapy.com"
env LOG_FILE=/var/log/node/miapp.log
env APP_DIR=/var/node/miapp
env APP=app.js
env PID_NAME=miapp.pid
env USER=www-data
env GROUP=www-data
env POST_START_MESSAGE_TO_LOG="miapp HAS BEEN STARTED."
env NODE_BIN=/usr/local/bin/node
env PID_PATH=/var/opt/node/run
env SERVER_ENV="production"
######################################################
start on runlevel [2345]
stop on runlevel [016]
respawn
respawn limit 99 5
pre-start script
mkdir -p $PID_PATH
mkdir -p /var/log/node
end script
script
export NODE_ENV=$SERVER_ENV
exec start-stop-daemon --start --chuid $USER:$GROUP --make-pidfile --pidfile $PID_PATH/$PID_NAME --chdir $APP_DIR --exec $NODE_BIN -- $APP >> $LOG_FILE 2>&1
end script
post-start script
echo $POST_START_MESSAGE_TO_LOG >> $LOG_FILE
end script
Customize all before #########, create a file in /etc/init/your-service.conf and paste it there.
Then you can:
start your-service
stop your-service
restart your-service
status your-service
I've written a pretty comprehensive guide to deploying Node.js, with example files:
Tutorial: How to Deploy Node.js Applications, With Examples
It covers things like http-proxy, SSL and Socket.IO.
Here's a longer article on solving this problem with systemd: http://savanne.be/articles/deploying-node-js-with-systemd/
Some things to keep in mind:
Who will start your process monitoring? Forever is a great tool, but it needs a monitoring tool to keep itself running. That's a bit silly, why not just use your init system?
Can you adequately monitor your processes?
Are you running multiple backends? If so, do you have provisions in place to prevent any of them from bringing down the others in terms of resource usage?
Will the service be needed all the time? If not, consider socket activation (see the article).
All of these things are easily done with systemd.
If you have root access you would better set up a daemon so that it runs safe and sound in the background. You can read how to do just that for Debian and Ubuntu in blog post Run Node.js as a Service on Ubuntu.
Forever will do the trick.
#Kevin: You should be able to kill processes fine. I would double check the documentation a bit. If you can reproduce the error it would be great to post it as an issue on GitHub.
Try this: http://www.technology-ebay.de/the-teams/mobile-de/blog/deploying-node-applications-with-capistrano-github-nginx-and-upstart.html
A great and detailed guide for deploying Node.js apps with Capistrano, Upstart and Nginx
As Box9 said, Forever is a good choice for production code. But it is also possible to keep a process going even if the SSH connection is closed from the client.
While not necessarily a good idea for production, this is very handy when in the middle of long debug sessions, or to follow the console output of lengthy processes, or whenever is useful to disconnect your SSH connection, but keep the terminal alive in the server to reconnect later (like starting the Node.js application at home and reconnecting to the console later at work to check how things are going).
Assuming that your server is a *nix box, you can use the screen command from the shell to do keep the process running even if the client SSH is closed. You can download/install screen from the web if not already installed (look for a package for your distribution if Linux, or use MacPorts if OS X).
It works as following:
When you first open the SSH connection, type 'screen' - this will start your screen session.
Start working as normal (i.e. start your Node.js application)
When you are done, close your terminal. Your server process(es) will continue running.
To reconnect to your console, ssh back to the server, login, and enter 'screen -r' to reconnect. Your old console context will pop back ready for you to resume using it.
To exit screen, while connected to the server, type 'exit' on the console prompt - that will drop you onto the regular shell.
You can have multiple screen sessions running concurrently like this if you need, and you can connect to any of it from any client. Read the documentation online for all the options.
Forever is a good option for keeping apps running (and it's npm installable as a module which is nice).
But for more serious 'deployment' -- things like remote management of deploying, restarting, running commands etc -- I would use capistrano with the node extension.
https://github.com/loopj/capistrano-node-deploy
https://paastor.com is a relatively new service that does the deploy for you, to a VPS or other server. There is a CLI to push code. Paastor has a free tier, at least it did at the time of posting this.
In your case you may use the upstart daemon. For a complete deployment solution, I may suggest capistrano. Two useful guides are How to setup Node.js env and How to deploy via capistrano + upstart.
Try node-deploy-server. It is a complex toolset for deploying an application onto your private servers. It is written in Node.js and uses npm for installation.

Resources