I have nodejs process that listens for an connection. I want to let foreverjs restart it on any connection error this way:
amqpConnection.on('error', function (err) {
console.error(err);
process.exit(1);
});
But this don't work. After error I see my process in this state:
~> forever list
info: Forever processes running
data: uid command script logfile
data: [0] OpQF /usr/bin/node my-script.js my-script-log.log STOPPED
Log file has connection error and two attempts to restart script.
error: Script restart attempt #2
{ [Error: connect ECONNREFUSED]
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect' }
error: Forever detected script exited with code: 1
Why forever stops trying to restart my process?
I should have read warnings that forever prints!
Script that not stay running more than --minUptime (1000ms by default) considered "spinning". One should set --spinSleepTime to make spinning scripts to be restarted. When this parameter is omitted spinning script won't be restarted.
I have reproduced your problem and found that forever successfully restart your process if some delay (1sec) take place before process.exit.
It's look that forever don't restart process when it crash too fast.
setTimeout(function() {
process.exit(1);
}, 1000);
Related
Nodejs was running on PM2 for a long time. And there is a corn which clears PM2 logs everyday
0 0 * * * find /home/user/.pm2/logs* -mtime +2 -exec rm -rf {} \;
Below error occurred for 1000 times and then pm2 stopped working and then when I reloaded the instance it was working fine as usual.
What could be the reason for this error?
/home/user/.nvm/versions/node/v12.16.1/lib/node_modules/pm2/lib/ProcessContainer.js:167
throw err;
^
[Error: ENOENT: no such file or directory, open '/home/user/.pm2/logs/out.log'] {
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/home/user/.pm2/logs/out.log'
}
2020-09-09T00:55:33: PM2 log: App name:app id:1 disconnected
2020-09-09T00:55:33: PM2 log: App exited with code [1] via signal [SIGINT]
2020-09-09T00:55:33: PM2 log: App starting in -cluster mode-
2020-09-09T00:55:33: PM2 log: App online
/home/user/.nvm/versions/node/v12.16.1/lib/node_modules/pm2/lib/ProcessContainer.js:167
throw err;
^
[Error: ENOENT: no such file or directory, open '/home/user/.pm2/logs/out.log'] {
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/home/user/.pm2/logs/out.log'
}
2020-09-09T00:55:33: PM2 log: App name:app id:1 disconnected
2020-09-09T00:55:33: PM2 log: App exited with code [1] via signal [SIGINT]
2020-09-09T00:55:33: PM2 log: App starting in -cluster mode-
2020-09-09T00:55:33: PM2 log: App online
/home/user/.nvm/versions/node/v12.16.1/lib/node_modules/pm2/lib/ProcessContainer.js:167
throw err;
^
[Error: ENOENT: no such file or directory, open '/home/user/.pm2/logs/out.log'] {
errno: -2,
code: 'ENOENT',
syscall: 'open',
path: '/home/user/.pm2/logs/out.log'
}
You're trying to clean up logfiles by just using rm . You Can't Do That™. On Linux and other UNIX-derived OSs, the program writing the log file holds it open even when you rm it from a directory. The file doesn't actually disappear until the program writing it closes it (or terminates). This is true for all sorts of software, not just pm2.
If all you want is to clear out the log files and start over, the command pm2 flush is for you.
If you want to save old logs for a limited period of time, you should investigate pm2's log rotation addon.
Commands like this may do the trick for you
# install the pm2 addon ... notice it says pm2 install, not npm install
pm2 install pm2-logrotate
# rotate when a logfile fills to ten megabytes
pm2 set pm2-logrotate:max_size 10M
# gzip compress the rotated logs to save space
pm2 set pm2-logrotate:compress true
# force rotation at 00:00 each day even if logfile is not full
pm2 set pm2-logrotate:rotateInterval '0 0 * * *'
You can say pm2 config pm2-logrotate to get it to show you its current settings.
Can you try the below two steps and check this if it works.
pm2 start bin/www -i 0 // this will start a cluster and the main app
pm2 stop 0 // this will allow the cluster to keep running
Adding
out_file: "/dev/null",
error_file: "/dev/null"
to the ecosystem.config.json file worked for me. It is basically disabling the logs. This is good for testing if the reason is in the logging functionality.
This is a workaround. If it works the the rotation of logging suggested in the answer of #O.Jones should work too.
I am trying to run my reactjs project on background with this command.
nohup npm start > /dev/null 2>&1 &
output should be redirected and nohup means do not terminate process when the stty is cut off.
However my npm node is not running at all. this is from nohup log.
Starting the development server...
events.js:291
throw er; // Unhandled 'error' event
^
Error: EBADF: bad file descriptor, read
Emitted 'error' event on ReadStream instance at:
at internal/fs/streams.js:202:14
at FSReqCallback.wrapper [as oncomplete] (fs.js:528:5) {
errno: -9,
code: 'EBADF',
syscall: 'read'
}
you can use forever. you can check more advanced scenarios from the link. below is the simplest way of using it
sudo npm install forever -g
then
forever start app.js
I have a VM with a RabbitMQ instance running, and a separate VM with a Node process that will consume things off of a queue.
After some time of inactivity, my Node process will just quit and display an error. I have forever keeping the Node process alive but here is what the error looks like:
data: server.js:18171 - error: Script restart attempt #33
data: server.js:18171 - [rabbitmq] Waiting for messages in savvy_shipping_to_logic. To exit press CTRL+C
data: server.js:18171 - Error: read ETIMEDOUT
data: server.js:18171 - at errnoException (net.js:901:11)
data: server.js:18171 - at TCP.onread (net.js:556:19)
data: server.js:18171 - error: Forever detected script exited with code: 8
data: server.js:18171 - error: Script restart attempt #34
data: server.js:18171 - [rabbitmq] Waiting for messages in savvy_shipping_to_logic. To exit press CTRL+C
data: server.js:18171 - Error: read ETIMEDOUT
data: server.js:18171 - at errnoException (net.js:901:11)
data: server.js:18171 - at TCP.onread (net.js:556:19)
data: server.js:18171 - error: Forever detected script exited with code: 8
data: server.js:18171 - error: Script restart attempt #35
both VMs are Ubuntu 14.04 running on Azure. I should note that I did not experience the same issues locally on my Macbook Pro running El Cap. I feel like I'm missing some flag?
I'm connecting to amqp://my.example.com
you may need to set a heartbeat in your connection setting to keep the connection alive. try adding a heartbeat of 30 seconds, for example, and see if that helps.
you may also want to increase the connection timeout.
I'm trying to run the forever function for node.js but I get below warnings;
C:\serv>forever start SERVER.js
warn: --minUptime not set. Defaulting to: 1000ms
warn: --spinSleepTime not set. Your script will exit if it does not stay up f
or at least 1000ms
info: Forever processing file: SERVER.js
How to set --minUptime and --spinSleepTime to remove these warnings
Installed forever package with npm install forever -g
These are only warnings. You could go on ignoring them, if you want.
But if you want to explicitly set them, forever --help tells you how to do so. Just start forever with:
forever start --minUptime 1000 --spinSleepTime 1000 SERVER.js
The documentation is not very exhaustive, a bit of additional information.
In the following examples we will use two scripts:
fail-fast.js:
process.exit(1);
fail-slow.js:
setTimeout(() => { process.exit(1); }, 2000);
1) using defaults
forever fail-fast.js
fail-fast.js script will execute only once, then no other start attempts will be made.
forever fail-slow.js
fail-slow.js script will be restarted indefinitely, as it stays up more than 1000ms (default value of minUptime if not specified). You can limit the number of restarts with the -m parameter.
2) setting only minUptime
forever --minUptime 10000 fail-fast.js
forever --minUptime 10000 fail-slow.js
Both fail-fast.js and fail-slow.js will be never be restarted, because we extended minUptime to 10 seconds and now fail-slow.js is considered spinning.
3) setting spinSleepTime
Whenever you set spinSleepTime (with or without minUptime), your process will restart even if it is considered 'spinning'.
forever --spinSleepTime 30000 fail-fast.js
forever --spinSleepTime 30000 fail-slow.js
Both scripts will be restarted forever, waiting spinSleepTime milliseconds between restarts.
In short:
When stop
if hadRunTime >= minUptime
restart
else if spinSleepTime != 0
wait spinSleepTime
restart
else
stop and no restart
#Megadix The answer has something wrong with spinSleepTime.
fail-fast.js will restart wating spinSleepTime,but fail-slow.js will restart immediately,no waiting! It can be Proved by:
console.log((new Date()).getTime());
setTimeout(() => {
process.exit(1);
}, 2000);
output like:
1468812185697
error: Forever detected script exited with code: 1
error: Script restart attempt #1
1468812187766
error: Forever detected script exited with code: 1
error: Script restart attempt #2
1468812189834
error: Forever detected script exited with code: 1
error: Script restart attempt #3
1468812191901
error: Forever detected script exited with code: 1
error: Script restart attempt #4
1468812193977
error: Forever detected script exited with code: 1
error: Script restart attempt #5
1468812196039
error: Forever detected script exited with code: 1
error: Script restart attempt #6
1468812198107
error: Forever detected script exited with code: 1
error: Script restart attempt #7
1468812200172
error: Forever detected script exited with code: 1
forever start --minUptime 1234 --spinSleepTime 3421 SERVER.js
https://github.com/nodejitsu/forever#usage
I'm assuming you are using express module that is way after you using - forever start SERVER.js - your server is not running, because express module run server in path ./bin/www - so you should use this command - forever start ./bin/www --name="SERVER" - name is the name of your js file, by default app.js
I have a node server running on AWS and I am using forever to keep it running in the background.
I have been very happy with forever so far; it's super easy to set up and use. However, even though my app is very simple, sometimes it crashes with this error:
Express server listening on port 3001
events.js:72
throw er; // Unhandled 'error' event
^
Error: connect ECONNREFUSED
at errnoException (net.js:901:11)
at Object.afterConnect [as oncomplete] (net.js:892:19)
error: Forever detected script exited with code: 8
error: Forever restarting script for 4 time
When I look back at the logs, I can see that this has happened 4 more times, and each time forever manages to restart the server. However, when it happens for the 5th time, for some reason forever fails to restart the server and my app becomes unresponsive.
Express server listening on port 3001
events.js:72
throw er; // Unhandled 'error' event
^
Error: connect ECONNREFUSED
at errnoException (net.js:901:11)
at Object.afterConnect [as oncomplete] (net.js:892:19)
error: Forever detected script exited with code: 8
/usr/lib/node_modules/forever/node_modules/forever-monitor/node_modules/broadway/node_modules/eventemitter2/lib/ev$
throw arguments[1]; // Unhandled 'error' event
^
Error: Cannot stop process that is not running.
at /usr/lib/node_modules/forever/node_modules/forever-monitor/lib/forever-monitor/monitor.js:332:26
at process._tickCallback (node.js:415:13)
Has anyone else experienced similar problems?
You have to use the option -MAX like this:
forever -MAX 10000000000 Mypath/ToMyFile.js
Also put a -minUptime for the minimum time the server can stay up, if it crash before that laps of time then it won't go up again so set it to 1.
forever --minUptime (number-in-milisecond-here) -MAX (number-of-restart-here) path/file.js
Just try this:
forever --minUptime 1 -MAX 1000000000 YourPath/ToYourFile.js
More documentation there
https://www.npmjs.com/package/forever
To allow a max of 100 restarts when the server crashes
forever start -m 100 index.js