When using forever to run a node.js program as a daemon
i.e.
forever start myNodeTask
If the daemon (myNodeTask) decides it needs to exit, what is the correct way to do so?
If I just call process.exit() the program does terminate but it doesn't delete the forever log file, which leads me to believe that I need the program to exit in a more forever-friendly manner.
The node tasks I'm running are plain tcp servers that stream data to connected clients, not web servers.
The forever module always keeps the log files, even after a process has finished. There is no forever-friendly manner to delete those files.
But, you could use the forever-monitor module, which allow you to programatically use forever (from the docs):
var forever = require('forever-monitor'),
fs = require('fs');
var child = new (forever.Monitor)('your-filename.js', {
max: 3,
silent: true,
options: []
});
child.on('exit', function () {
console.log('your-filename.js has exited after 3 restarts');
// here you can delete your log file
fs.unlink('path_to_your_log_file', function (err) {
// do something amazing
});
});
child.start();
Related
I am trying to create some setup and teardown logic for an expressjs server. Here's my entry code:
import fs from "fs";
import express from "express";
import { setRoutes } from "./routes";
let app = express();
const server = app.listen(8080, function () {
console.log(`š§ Mock Server is now running on port : ${8080}`);
});
app = setRoutes(app);
function stop() {
fs.rmdirSync("./uploads", { recursive: true });
fs.mkdirSync("uploads");
console.log("\nš§¹ Uploads folder purged");
server.on("close", function () {
console.log("ā¬ Shutting down server");
process.exit();
});
server.close();
}
process.on("SIGINT", stop);
// Purge sw images on restart
process.once("SIGUSR2", function () {
fs.rmdirSync("./uploads/swimages", { recursive: true });
console.log("š§¹ Software Images folder purged");
process.kill(process.pid, "SIGUSR2");
});
The npm script to start this up is "start": "FORCE_COLOR=3 nodemon index.js --exec babel-node".
The setup and restart logic works as expected. I get š§ Mock Server is now running on port : 8080 logged to console on startup. When I save a file, nodemon restarts the server, and the code in process.once is executed. When I want to shut it all down, I ctrl + c in the terminal. The cleanup logic from within the stop function is run. However, the process bever fully exits. In the terminal, am still stuck in the process, and I have to hit ctrl + c again to fully exit the process. It looks like this:
As far as I know there are no open connections (other questions mentioned that if there is a keep-alive connection still open, the server will not close properly, but as far as I can tell, that is not the case). I have tried different variations of server.close(callback), server.on('close', callback), process.exit(), process.kill(process.pid), etc, but nothing seems to fully exit the process.
Note that if I simply run node index.js, I do not have this issue. The cleanup logic runs, and the process exits to completion without issue. It seems to be an issue when using nodemon only.
I don't want other developers to have to wait for cleanup logic to run and then hit ctrl + c again. What am I missing to run my cleanup logic and fully exit the process in the terminal?
There is an open connection for sure. Check this package that can tell you which one: https://www.npmjs.com/package/wtfnode
I built a home-grown Netlify. It answers webhooks from Github and executes a bash script to deploy a cute React SPA.
It works great but when run under a PM2 process manager environment, it restarts once the child process closes/exits. I've tested many things - I've even built a test endpoint that feeds it fake data as repeatedly committing fake deploys got tiresome.
Here's some basic code:
const childProcess = require('child_process');
async function processDeploy(deploy) {
console.log(`Processing commit: ${deploy.commit.id}...`);
childProcess.execFile('./deploy.sh', function (error, stdout, stderr) {
if (!error) {
// a function to report the result of the commit
postDeploy(stdout, stderr, deploy.commit, { state: 'deployed' });
} else {
postDeploy('{}', error, deploy.commit, { state: 'error' });
}
});
}
There's a bunch of supporting functions that keep track of the queue and so on to serve a basic status frontend.
I've spent a long time picking through Node and PM2 logs only to find... Nothing. And I think it's partly due to my suspicion that there isn't actually an error - PM2 executes the script, the script exits once completed, and PM2 decides that it's time to kill the process.
The process then restarts as it's set to restart on kill/die.
So, how does PM2 differ to npm start when running my Express-based app? More importantly, how do I fix it...
In case it's relevant, this is my ecosystem.config.js for PM2:
module.exports = {
apps: [
{
name: 'App',
script: 'bin/www',
exec_mode: 'fork',
instances: 1,
autorestart: true,
watch: false,
ignore_watch: ['node_modules', '.log'],
max_memory_restart: '4G',
},
],
};
Update:
I've tried Node versions 10, 12, and 14.
I've switched away from child_process.execFile to child_process.spawn
I've investigated memory issues and scoured PM2's Github issues.
I am now confident that PM2 is catching the child process's event and confusing it with the parent process's.
Terminal output:
Deploy child process exited with code 0 and signal null // child process event listener for exit
Deploy child process closed with code 0 // child process event listener for close
Processing commit completed.
Queue empty.
Something is trying to kill me. // parent process event listener for SIGINT
I've been browsing around but to no success. I've found some npm packages like nodemon and forever but documentation doesn't explain how to call a restart inside the script properly.
I've found this code snippet on another question but I'm not using Express or other frameworks since the script is using a pulling service not a response one.
This is code I've made so far using internal Node.js dependencies but no luck.
'use strict'
process.on('uncaughtException', (error) => {
console.error('Global uncaughtException error caught')
console.error(error.message)
console.log('Killing server with restart...')
process.exit(0)
})
process.on('exit', () => {
console.log('on exit detected')
const exec = require('child_process').exec
var command = 'node app.js'
exec(command, (error, stdout, stderr) => {
console.log(`error: ${error.message}`)
console.log(`stdout: ${stdout}`)
console.log(`stderr: ${stderr}`)
})
})
setTimeout(() => {
errorTriggerTimeBomb() // Dummy error for testing triggering uncaughtException
}, 3000)
Just to note I'm running the server on Termux, a Linux terminal app for android. I know it's better to run from desktop but I'm always at a WiFi or mobile data area and that I don't like leaving my PC on overnight.
A typical restart using something like nodemon or forever would be triggered by calling process.exit() in your script.
This would exit the current script and then the monitoring agent would see that it exited and would restart it for you. It's the same principal as if it crashed on its own, but if you're trying to orchestrate it shutting down, you just exit the process and then the monitoring agent will restart it.
I have a home automation server that is being monitored using forever. If it crashes forever will automatically restart it. I also have it set so that at 4am every morning, it will call process.exit() and then forever will automatically restart it. I do this to prevent any memory leak accumulation over a long period of time and 30 seconds of down time in the middle of the night for my application is no big deal.
I'm using forever to run my Node.js server and I very like it's start/stop/restart command line commands. Recently I decided that I need system that will email me about all server crashes. Unfortunately forever doesn't support such kind of functionality out of the box.
So I decided to take forever-monitor (the tool like forever but with programmatic approach) and nodemailer. Here is a script I wrote:
var forever = require('forever-monitor');
var nodemailer = require('nodemailer');
var sendmailTransport = require('nodemailer-sendmail-transport');
var transporter = nodemailer.createTransport(sendmailTransport({
path: '/usr/sbin/sendmail'
}));
var error = '';
function sendMail(subject, text) {
var message = {
// sender info
from: '"Server name" <no-reply#example.com>',
// Comma separated list of recipients
to: '"My name" <myemail#example.com>',
// Subject of the message
subject: subject,
// plaintext body
text: text
};
transporter.sendMail(message, function (err) {
if (err) {
console.log('Error during trial email with server error report');
console.log(err);
}
});
}
var child = new (forever.Monitor)('app.js', {
outFile: './logs/stdout.log',
errFile: './logs/stderr.log'
});
child.on('stderr', function (data) {
error += data;
});
// Check for errors every second
setInterval(function () {
if (error) {
sendMail('Your server crashed', error);
error = '';
}
}, 1000);
child.start();
I called this file start.js. All it does is: spawns new process (app.js, which runs my Express web-server) with forever-monitor, I listen for stderr event of this child process and collect all errors into variable, then I check this variable every second and if it doesn't empty I send an email and clear variable. This code works fine.
Now to start my server I need execute something like nohup node start.js &. The bad thing that there is no easy way to stop or restart my server. Previously I did it with forever stopall and forever restartall which is very convenient.
My next thought was try to combine forever and forever-monitor so I tried to start server like this forever start start.js. Server started just like I supposed. But restarting didn't work anymore. Executing forever stopall and then forever start start.js didn't work too. Server started to complain that port is already busy. I assume it happens because forever stops start.js process, but the child app.js process spawned by forever-monitor still was occupying the port.
So my question is: how to run Node.js server, receive emails about it's crashes, automatically restart it when it crashed and have possibility to easily start/stop/restart it like with forever command line utility? Solution can use forever, forever-monitor or both of them, or actually any library you think may work for this kind of problem.
I think you should fork forever's git repository and add emails notifications in source code. This way you can still use forever CLI without additional scripts.
If you're not familiar with concept check out: https://help.github.com/articles/fork-a-repo/
Seems like such a feature have been already requested but without much reply.
With working solution you can either add your (forked) repo to your package.json or contribute to forever project by requesting pull. Maybe thanks to you forever will have email notifications.
I have a node.js app which runs until user quits the process. I'm trying to run this app as a background process with forever, however, my app originally outputs something to stdout:
process.stdout.clearLine();
process.stdout.cursorTo(0);
process.stdout.write('...');
and if I run this app with forever
forever start -l forever.log -o out.log -e err.log app.js
it does run in the background but there's nothing in out.log file. If I look at err.log I see that problem is in printing something to stdout:
$ cat err.log
/Users/err/Sites/js/test/app.js:105
process.stdout.clearLine();
^
TypeError: Object #<Socket> has no method 'clearLine'
at null.<anonymous> (/Users/err/Sites/js/test/app.js:105:20)
at wrapper [as _onTimeout] (timers.js:252:14)
at Timer.listOnTimeout [as ontimeout] (timers.js:110:15)
I also tried to use forever-monitor and wrote a small script:
var config = require('./config');
if (config.runInBackground) {
var forever = require('forever-monitor');
var child = new (forever.Monitor)('app.js', {
max: 1,
silent: true,
outFile: 'monitor.out.log',
errFile: 'monitor.err.log'
});
child.on('stdout', function(e) {
console.log(e);
});
child.on('exit', function () {
console.log('exited.');
});
child.start();
} else {
// do the regular process.stdout.write('...');
}
But my script exists without writing any file or starting a background process.
How can I run my node app in the background and write the original stdout to some log file? Or even better, is it possible to have an option (let's say in config.js file) to run this app as a background process or not, and if so, all stdout stuff should be written to a file?
You may need to specify an absolute path for your files.
That said, there are alternatives to forever that might work depending on your operating system. node-windows and node-mac both offer what you're looking for. node-linux isn't ready yet, but the project has some init.d scripts that could be configured to run processes with logging.
Disclosure: I am the author of all three of these.