pm2 reload ecosystem.config.js causing many restarts on application - node.js

I am experiencing problems reloading the application using the ecosystem.config.js file. When the application is started for the first time, it starts correctly, but when I refresh/reload the application using the ecosystem.config.js file, the application restarts several times causing an error.
My SO is Ubuntu Xenial, PM2 version is 3.2.2 and Node v10.13.0. The application uses the latest version from Express module (4.16.4).
If I reload the application with "pm2 reload app_name", this problem doesn't occur.
The ecosystem.config.js content:
module.exports = {
apps: [{
script: "./index.js",
instances: "max",
exec_mode: "cluster",
kill_timeout: "2000",
env: {
NODE_ENV: "development",
},
env_production: {
NODE_ENV: "production",
}
}]
}
When I run the first time:
$ pm2 reload ecosystem.config.js
[PM2][WARN] Applications index not running, starting...
[PM2] App [index] launched (2 instances)
node#ubuntu:/data/$ pm2 logs
[TAILING] Tailing last 15 lines for [all] processes (change the value with >--lines option)
/home/node/.pm2/pm2.log last 15 lines:
PM2 | 2018-11-23T13:14:30: PM2 log: App [index:0] starting in -cluster >mode-
PM2 | 2018-11-23T13:14:31: PM2 log: App [index:0] online
PM2 | 2018-11-23T13:14:31: PM2 log: App [index:1] starting in -cluster >mode-
PM2 | 2018-11-23T13:14:31: PM2 log: App [index:1] online
When I reload the application by name (ex: pm2 reload app_name), the application contiue runnig, but I see some timeouts to kill process:
PM2 | 2018-11-23T14:01:02: PM2 log: pid=11296 msg=failed to kill - retrying in 100ms
PM2 | 2018-11-23T14:01:02: PM2 log: Process with pid 11289 still alive after 6000ms, sending it SIGKILL now...
PM2 | 2018-11-23T14:01:02: PM2 log: pid=11296 msg=failed to kill - retrying in 100ms
PM2 | 2018-11-23T14:01:02: PM2 log: Process with pid 11296 still alive after 6000ms, sending it SIGKILL now...
PM2 | 2018-11-23T14:01:02: PM2 log: App name:index id:_old_0 disconnected
PM2 | 2018-11-23T14:01:02: PM2 log: App [index:_old_0] exited with code [0] via signal [SIGKILL]
PM2 | 2018-11-23T14:01:02: PM2 log: App name:index id:_old_1 disconnected
PM2 | 2018-11-23T14:01:02: PM2 log: App [index:_old_1] exited with code [0] via signal [SIGKILL]
PM2 | 2018-11-23T14:01:02: PM2 log: pid=11289 msg=process killed
PM2 | 2018-11-23T14:01:02: PM2 log: pid=11296 msg=process killed
But, even though timeouts occur the application is running.
When I execute "pm2 reload ecosystem.config.js", the PM2 restart the application several times and one instance fail:
0|index | at Module.load (internal/modules/cjs/loader.js:598:32)
0|index | at tryModuleLoad (internal/modules/cjs/loader.js:537:12)
0|index | at Function.Module._load (internal/modules/cjs/loader.js:529:3)
0|index | at Object. (/usr/lib/node_modules/pm2/lib/ProcessContainerFork.js:48:21)
0|index | Error: listen EADDRINUSE :::3001
I believe the problem is related to some timeout to properly terminate the http connection of the Express module, but I'm still investigating this.

It has been fixed on the lastest PM2 version, please update:
npm install pm2#latest -g
pm2 update
Make sure you pm2 delete all and then start back your application again, it will work then when doing reload or restart

Related

How to tell to pm2 to ignore all errors in scripts locpated at node_modules and subfolders?

I am testing an APP under server enviroment:
Use of aaPanel or btPanel under ubuntu
Use of pm2 module
The app is running but restart on error
I am using custom error managment and log functions
I want pm2 just ignore all console.log(error) and errors ocurred in scripts located at node_modules and subfolder
Anny idea?
Thanks for your time and for any help
This is a example of some error log from pm2:
2021-03-12T01:52:51: PM2 log: App [MyApp] exited with code [0] via signal [SIGINT]
And this is a example of error in out log from pm2:
Error: /www/wwwroot/apps/MyApp/1.0.0.2/node_modules/bcrypt/lib/binding/napi-v3/bcrypt_lib.node: invalid ELF header
at Object.Module._extensions..node (internal/modules/cjs/loader.js:1122:18)
at Module.load (internal/modules/cjs/loader.js:928:32)
at Function.Module._load (internal/modules/cjs/loader.js:769:14)
at Module.require (internal/modules/cjs/loader.js:952:19)
at Module.Hook._require.Module.require (/www/server/nvm/versions/node/v14.16.0/lib/node_modules/pm2/node_modules/require-in-the-middle/index.js:80:39)
at require (internal/modules/cjs/helpers.js:88:18)
at Object.<anonymous> (/www/wwwroot/apps/MyApp/1.0.0.2/node_modules/bcrypt/bcrypt.js:6:16)
at Module._compile (internal/modules/cjs/loader.js:1063:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)
at Module.load (internal/modules/cjs/loader.js:928:32)
This is a log from console pm2 console in aaPanel or btPanel:
2021-03-12T00:05:50: PM2 log: ===============================================================================
2021-03-12T00:05:50: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
2021-03-12T00:05:50: PM2 log: Time : Fri Mar 12 2021 00:05:50 GMT+0000 (Coordinated Universal Time)
2021-03-12T00:05:50: PM2 log: PM2 version : 4.5.5
2021-03-12T00:05:50: PM2 log: Node.js version : 14.16.0
2021-03-12T00:05:50: PM2 log: Current arch : x64
2021-03-12T00:05:50: PM2 log: PM2 home : /root/.pm2
2021-03-12T00:05:50: PM2 log: PM2 PID file : /root/.pm2/pm2.pid
2021-03-12T00:05:50: PM2 log: RPC socket file : /root/.pm2/rpc.sock
2021-03-12T00:05:50: PM2 log: BUS socket file : /root/.pm2/pub.sock
2021-03-12T00:05:50: PM2 log: Application log path : /root/.pm2/logs
2021-03-12T00:05:50: PM2 log: Worker Interval : 30000
2021-03-12T00:05:50: PM2 log: Process dump file : /root/.pm2/dump.pm2
2021-03-12T00:05:50: PM2 log: Concurrent actions : 2
2021-03-12T00:05:50: PM2 log: SIGTERM timeout : 1600
2021-03-12T00:05:50: PM2 log: ===============================================================================
2021-03-12T00:20:11: PM2 log: App [MyApp] starting in -fork mode-
2021-03-12T00:20:11: PM2 log: App [MyApp] online
2021-03-12T00:21:59: PM2 log: Stopping app:1.0.0.2 id:0
2021-03-12T00:21:59: PM2 log: App [MyApp] exited with code [0] via signal [SIGINT]
2021-03-12T00:21:59: PM2 log: pid=95350 msg=process killed
2021-03-12T00:22:24: PM2 log: App [MyApp] starting in -fork mode-
2021-03-12T00:22:24: PM2 log: App [MyApp] online
2021-03-12T00:40:41: PM2 log: Stopping app:MyApp id:0
2021-03-12T00:40:41: PM2 log: App [MyApp] exited with code [0] via signal [SIGINT]
2021-03-12T00:40:41: PM2 log: pid=96990 msg=process killed
Aditional Notes:
The app run flawless under local enviroment
The pm2 is the problem some awfull bad config maybe!
I run and test the app using 'nodemon index.js' all ok
I run and test the app using 'node index.js' all ok
I run and test the app using 'npm start' all ok
I run npm update all ok
I run npm audit all ok

Build folder not recognized in pm2 service

I'm running my node+react app on vps ubuntu machine using ssh.
I used nginx, pm2 for running my node js server, but when I check my logs it shows the following error:
App [server:0] starting in -fork mode-
PM2 | 2020-09-16T07:05:03: PM2 log: App [server:0] online
PM2 | 2020-09-16T07:06:18: PM2 log: Reloading logs...
PM2 | 2020-09-16T07:06:18: PM2 log: Reloading logs for process id 0
PM2 | 2020-09-16T07:38:58: PM2 log: Stopping app:server id:0
PM2 | 2020-09-16T07:38:58: PM2 log: App [server:0] exited with code [0] via signal [SIGINT]
PM2 | 2020-09-16T07:38:58: PM2 log: pid=6888 msg=process killed
PM2 | 2020-09-16T07:38:58: PM2 log: App [server:0] starting in -fork mode-
PM2 | 2020-09-16T07:38:58: PM2 log: App [server:0] online
0|server | Error: ENOENT: no such file or directory, stat '/root/repos/IRS-Ecommerce/IRS-
Ecommerce/build/index.html'
/root/.pm2/logs/server-out.log last 15 lines:
0|server | Server is up and running at port : 8080
0|server | MongoDB connected successfully!
My node js server file where I'm sending my react app in response:
app.use(express.static("IRS-Ecommerce/build"));
app.get("*", (req, res) => {
res.sendFile(path.join(__dirname, "IRS-Ecommerce", "build", "index.html"));
});
I've run npm build in my react app and build folder is present there, but it is showing this error that index.html is not there. I've checked the path, it is correct.
When I access public IP, my react app shows up which means it is working. But why am I seeing this error here?
/root/repos/IRS-Ecommerce/IRS- Ecommerce/build/index.html double IRS-
Ecommerce orz

Nodejs pm2 is always restarting when I reboot Ubuntu

I have an express nodejs server running in Ubuntu LTS with pm2. The server runs fine, but when I restart Ubuntu, the server is always rebooting.
I use an ecossystem.config like this:
module.exports = {
apps : [{
name: 'gTimeTracking',
script: 'index.js',
args: 'one two',
instances: 1,
autorestart: true,
watch: true,
max_memory_restart: '1G',
env: {
NODE_ENV: 'development'
},
env_production: {
NODE_ENV: 'production'
}
}]
};
I started the server with this command:
pm2 start ecosystem.config.js --env production
And with pm2 save
and I have this infinite error on Ubuntu reboot
0|gTimeTracking | Server running since: Mon Jul 01 2019 09:36:43 GMT+0200 (CEST)
PM2 | Change detected on path logs/logger-01-07-2019-09.log for app gTimeTracking - restarting
PM2 | Stopping app:gTimeTracking id:0
PM2 | App [gTimeTracking:0] exited with code [0] via signal [SIGINT]
PM2 | pid=16255 msg=process killed
PM2 | App [gTimeTracking:0] starting in -fork mode-
PM2 | App [gTimeTracking:0] online
0|gTimeTracking | Server running since: Mon Jul 01 2019 09:36:44 GMT+0200 (CEST)
PM2 | Change detected on path logs/logger-01-07-2019-09.log for app gTimeTracking - restarting
PM2 | Stopping app:gTimeTracking id:0
PM2 | App [gTimeTracking:0] exited with code [0] via signal [SIGINT]
PM2 | pid=16274 msg=process killed
PM2 | App [gTimeTracking:0] starting in -fork mode-
PM2 | App [gTimeTracking:0] online
Last time I had this problem I had to reinstall many times pm2 to relaunch the server, but now this method doesn't works and isn't an stable solution
What could be wrong?
I had to use pm2 cleardump for solve the problem with pm2 delete all and pm2 kill it didn't works for me. (I didn't have to change anything about the loggers path)

pm2 Daemon Dies After a Few Hours

I have a Node.js/Express app that implements a set of REST APIs and I'm attempting to use pm2 to manage its deployment. The app starts fine (using pm2 start ecosystem.config.js) and remains available for a few hours, but the pm2 daemon always dies eventually without any errors in the logs.
A few notes:
I'm running in a CentOS 7 shared hosting environment.
The /var/log directory is empty and journalctl doesn't return any entries.
I've verified that the system isn't rebooting.
The only pm2 module I have installed is pm2-logrotate.
I'm trapping and logging SIGINT, SIGTERM, SIGQUIT, and SIGABRT signals, but that logic never seems to get hit (it does if I run pm2 stop).
If I run pm2 list it just restarts the daemon and shows an empty app list.
Here's my ecosystem.config.js:
module.exports = {
apps: [
{
kill_timeout: 60000,
listen_timeout: 10000,
log: 'logs/my-app.log',
name: 'my-app',
script: 'dist/index.js',
wait_ready: true,
instances: 1,
autorestart: true,
watch: false,
max_memory_restart: '1G',
env: {
NODE_ENV: 'development'
},
env_production: {
NODE_ENV: 'production'
}
}
]
};
Here's pm2.log:
2019-04-24T19:20:24: PM2 log: ===============================================================================
2019-04-24T19:20:24: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
2019-04-24T19:20:24: PM2 log: Time : Wed Apr 24 2019 19:20:24 GMT-0700 (Mountain Standard Time)
2019-04-24T19:20:24: PM2 log: PM2 version : 3.5.0
2019-04-24T19:20:24: PM2 log: Node.js version : 10.5.0
2019-04-24T19:20:24: PM2 log: Current arch : x64
2019-04-24T19:20:24: PM2 log: PM2 home : /home/myuser/.pm2
2019-04-24T19:20:24: PM2 log: PM2 PID file : /home/myuser/.pm2/pm2.pid
2019-04-24T19:20:24: PM2 log: RPC socket file : /home/myuser/.pm2/rpc.sock
2019-04-24T19:20:24: PM2 log: BUS socket file : /home/myuser/.pm2/pub.sock
2019-04-24T19:20:24: PM2 log: Application log path : /home/myuser/.pm2/logs
2019-04-24T19:20:24: PM2 log: Process dump file : /home/myuser/.pm2/dump.pm2
2019-04-24T19:20:24: PM2 log: Concurrent actions : 2
2019-04-24T19:20:24: PM2 log: SIGTERM timeout : 1600
2019-04-24T19:20:24: PM2 log: ===============================================================================
2019-04-24T19:20:24: PM2 log: App [pm2-logrotate:0] starting in -fork mode-
2019-04-24T19:20:24: PM2 log: App [pm2-logrotate:0] online
2019-04-24T19:20:24: PM2 log: App [my-app:1] starting in -fork mode-
2019-04-24T19:20:28: PM2 log: App [my-app:1] online
Here's pm2-logrotate-out.log:
"/home/myuser/.pm2/logs/my-app-out-1__2019-04-25_00-00-00.log" has been created
"/home/myuser/my-app/logs/my-app-1__2019-04-25_00-00-00.log" has been created
Any idea what's causing this issue or how I can debug it further?
It turns out that this was caused by resource limiting imposed by my hosting provider. I'm still confused about why nothing was logged to indicate what happened, but I'm marking this as answered since I've found the root cause.

node-sails app is getting stop in every 24 hours or earlier on server

I have two sails js apps on two servers but my both apps on one server are getting stop in every 24 hours, I usually check both the app in the morning and it shows the following after running
pm2 list or pm2 restart all:-
[PM2] Spawning PM2 daemon
[PM2] PM2 Successfully daemonized
[PM2][WARN] No process found
and every time I had to go to their directory and run the command:- pm2 start app.js individually, but both the apps on another server are working correctly.
Please suggest what should I do?
Thanks
Now my service is stoped and after running pm2 logs its showing:
[PM2] Spawning PM2 daemon
[PM2] PM2 Successfully daemonized
########### Starting streaming logs for [all] process
Can not find log files, try to reload logs by executing "pm2 reload <name|all>".
PM2: 2015-07-17 03:27:22: App name:app id:0 online
PM2: child_process.js:935
PM2: throw errnoException(process._errno, 'spawn');
PM2: ^
PM2: Error: spawn ENOMEM
PM2: at errnoException (child_process.js:988:11)
PM2: at ChildProcess.spawn (child_process.js:935:11)
PM2: at exports.spawn (child_process.js:723:9)
PM2: at Object.exports.execFile (child_process.js:607:15)
PM2: at exports.exec (child_process.js:578:18)
PM2: at Object.parseHg [as parse] (/usr/local/lib/node_modules/pm2/node_modules/vizion/lib/hg.js:36:2)
PM2: at /usr/local/lib/node_modules/pm2/node_modules/vizion/lib/vizion.js:16:24
PM2: at /usr/local/lib/node_modules/pm2/node_modules/vizion/lib/identify.js:18:12
PM2: at /usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:151:21
PM2: at /usr/local/lib/node_modules/pm2/node_modules/vizion/lib/identify.js:12:16
PM2: 2015-07-20 06:23:46: [PM2][WORKER] Started with refreshing interval: 30000
PM2: 2015-07-20 06:23:46: [[[[ PM2/God daemon launched ]]]]
PM2: 2015-07-20 06:23:46: BUS system [READY] on port /root/.pm2/pub.sock
PM2: 2015-07-20 06:23:46: RPC interface [READY] on port /root/.pm2/rpc.sock
and after running pm2 list or pm2 restart all its showing:
[PM2] Spawning PM2 daemon
[PM2] PM2 Successfully daemonized
[PM2][WARN] No process found
plz help I am totally stuck in this.
Thanks

Resources