Last couple of weeks, I am struggling with pm2 100% CPU usage which hangs my node server.
I tried reading logs but I didn't find any issue there.
My node version: 6.9.1
PM2 is : 2.4.4.
OS: Ubuntu 14.04
On an average my cpu usage is : ~5
Manually I am restarting all apps - pm2 restart all.
pm2.log:
fluid_admin#instance-2:~$ tail -15 .pm2/pm2.log
2017-04-12 09:55:30: Starting execution sequence in -fork mode- for app name:fluid-prod id:0
2017-04-12 09:55:30: App name:fluid-prod id:0 online
2017-04-14 13:53:21: Stopping app:fluid-prod id:0
2017-04-14 13:53:21: Stopping app:nedbserver id:1
2017-04-14 13:53:21: App [nedbserver] with id [1] and pid [32557], exited with code [0] via signal [SIGINT]
2017-04-14 13:53:21: pid=32574 msg=failed to kill - retrying in 100ms
2017-04-14 13:53:21: pid=32557 msg=process killed
2017-04-14 13:53:21: Starting execution sequence in -fork mode- for app name:nedbserver id:1
2017-04-14 13:53:21: App [fluid-prod] with id [0] and pid [32574], exited with code [0] via signal [SIGINT]
2017-04-14 13:53:21: App name:nedbserver id:1 online
2017-04-14 13:53:21: pid=32574 msg=process killed
2017-04-14 13:53:21: Starting execution sequence in -fork mode- for app name:fluid-prod id:0
2017-04-14 13:53:21: App name:fluid-prod id:0 online
App is running on GAE, OS is : Ubuntu 14.04.
My CPU usage when PM2 is high
I have moved from forever to pm2 6 months back. till recent time, it was working file but now I am frequently having this issue.
I don't know deal this problem. Can some one help me how to debug this issue.
It happened again. Output of TOP commands
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25059 fluid_a+ 20 0 1260364 105532 8236 R 99.7 17.5 58:27.27 node fluid/server/tools/www +
1 root 20 0 33520 3208 1792 S 0.0 0.5 0:04.82 /sbin/init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [kthreadd]
Script to start my app:
pm2 stop fluid-prod
pm2 start \
-n fluid-prod \
-e /path/to/fluid-error.log \
-o /path/to/fluid-out.log \
$(dirname $0)/www -- --max-memory-restart 200M --env=production
Related
I have an express nodejs server running in Ubuntu LTS with pm2. The server runs fine, but when I restart Ubuntu, the server is always rebooting.
I use an ecossystem.config like this:
module.exports = {
apps : [{
name: 'gTimeTracking',
script: 'index.js',
args: 'one two',
instances: 1,
autorestart: true,
watch: true,
max_memory_restart: '1G',
env: {
NODE_ENV: 'development'
},
env_production: {
NODE_ENV: 'production'
}
}]
};
I started the server with this command:
pm2 start ecosystem.config.js --env production
And with pm2 save
and I have this infinite error on Ubuntu reboot
0|gTimeTracking | Server running since: Mon Jul 01 2019 09:36:43 GMT+0200 (CEST)
PM2 | Change detected on path logs/logger-01-07-2019-09.log for app gTimeTracking - restarting
PM2 | Stopping app:gTimeTracking id:0
PM2 | App [gTimeTracking:0] exited with code [0] via signal [SIGINT]
PM2 | pid=16255 msg=process killed
PM2 | App [gTimeTracking:0] starting in -fork mode-
PM2 | App [gTimeTracking:0] online
0|gTimeTracking | Server running since: Mon Jul 01 2019 09:36:44 GMT+0200 (CEST)
PM2 | Change detected on path logs/logger-01-07-2019-09.log for app gTimeTracking - restarting
PM2 | Stopping app:gTimeTracking id:0
PM2 | App [gTimeTracking:0] exited with code [0] via signal [SIGINT]
PM2 | pid=16274 msg=process killed
PM2 | App [gTimeTracking:0] starting in -fork mode-
PM2 | App [gTimeTracking:0] online
Last time I had this problem I had to reinstall many times pm2 to relaunch the server, but now this method doesn't works and isn't an stable solution
What could be wrong?
I had to use pm2 cleardump for solve the problem with pm2 delete all and pm2 kill it didn't works for me. (I didn't have to change anything about the loggers path)
I am experiencing problems reloading the application using the ecosystem.config.js file. When the application is started for the first time, it starts correctly, but when I refresh/reload the application using the ecosystem.config.js file, the application restarts several times causing an error.
My SO is Ubuntu Xenial, PM2 version is 3.2.2 and Node v10.13.0. The application uses the latest version from Express module (4.16.4).
If I reload the application with "pm2 reload app_name", this problem doesn't occur.
The ecosystem.config.js content:
module.exports = {
apps: [{
script: "./index.js",
instances: "max",
exec_mode: "cluster",
kill_timeout: "2000",
env: {
NODE_ENV: "development",
},
env_production: {
NODE_ENV: "production",
}
}]
}
When I run the first time:
$ pm2 reload ecosystem.config.js
[PM2][WARN] Applications index not running, starting...
[PM2] App [index] launched (2 instances)
node#ubuntu:/data/$ pm2 logs
[TAILING] Tailing last 15 lines for [all] processes (change the value with >--lines option)
/home/node/.pm2/pm2.log last 15 lines:
PM2 | 2018-11-23T13:14:30: PM2 log: App [index:0] starting in -cluster >mode-
PM2 | 2018-11-23T13:14:31: PM2 log: App [index:0] online
PM2 | 2018-11-23T13:14:31: PM2 log: App [index:1] starting in -cluster >mode-
PM2 | 2018-11-23T13:14:31: PM2 log: App [index:1] online
When I reload the application by name (ex: pm2 reload app_name), the application contiue runnig, but I see some timeouts to kill process:
PM2 | 2018-11-23T14:01:02: PM2 log: pid=11296 msg=failed to kill - retrying in 100ms
PM2 | 2018-11-23T14:01:02: PM2 log: Process with pid 11289 still alive after 6000ms, sending it SIGKILL now...
PM2 | 2018-11-23T14:01:02: PM2 log: pid=11296 msg=failed to kill - retrying in 100ms
PM2 | 2018-11-23T14:01:02: PM2 log: Process with pid 11296 still alive after 6000ms, sending it SIGKILL now...
PM2 | 2018-11-23T14:01:02: PM2 log: App name:index id:_old_0 disconnected
PM2 | 2018-11-23T14:01:02: PM2 log: App [index:_old_0] exited with code [0] via signal [SIGKILL]
PM2 | 2018-11-23T14:01:02: PM2 log: App name:index id:_old_1 disconnected
PM2 | 2018-11-23T14:01:02: PM2 log: App [index:_old_1] exited with code [0] via signal [SIGKILL]
PM2 | 2018-11-23T14:01:02: PM2 log: pid=11289 msg=process killed
PM2 | 2018-11-23T14:01:02: PM2 log: pid=11296 msg=process killed
But, even though timeouts occur the application is running.
When I execute "pm2 reload ecosystem.config.js", the PM2 restart the application several times and one instance fail:
0|index | at Module.load (internal/modules/cjs/loader.js:598:32)
0|index | at tryModuleLoad (internal/modules/cjs/loader.js:537:12)
0|index | at Function.Module._load (internal/modules/cjs/loader.js:529:3)
0|index | at Object. (/usr/lib/node_modules/pm2/lib/ProcessContainerFork.js:48:21)
0|index | Error: listen EADDRINUSE :::3001
I believe the problem is related to some timeout to properly terminate the http connection of the Express module, but I'm still investigating this.
It has been fixed on the lastest PM2 version, please update:
npm install pm2#latest -g
pm2 update
Make sure you pm2 delete all and then start back your application again, it will work then when doing reload or restart
I start pm2on booting my raspberry pi 3 with ONE of theses shell scripts (rc.local executed as root during boot)
As root user
code
startPm2Root() {
#delete to avoid all pm2 issues
rm -rf "$PM2_HOME"
pkill -f pm2
USER="root"
HOME="/root"
PM2_HOME="$HOME/.pm2"
export USER HOME PM2_HOME
pm2 startup systemd &
if [ -f /home/pi/app/newVersion.tar.gz ]; then
bash /home/pi/app/deploy.sh
else
pm2 kill && pm2 start /home/pi/app/processes.json &
fi
pm2 save
}
problem/log
+ USER=root
+ HOME=/root
+ PM2_HOME=/root/.pm2
+ export USER HOME PM2_HOME
+ [ -f /home/pi/app/newVersion.tar.gz ]
+ pm2 startup systemd
+ pm2 save
+ pm2 kill
[PM2] Spawning PM2 daemon with pm2_home=/root/.pm2
[PM2] Spawning PM2 daemon with pm2_home=/root/.pm2
[PM2] Init System found: systemd
Platform systemd
Template
[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target
[Service]
Type=forking
User=root
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Environment=PATH=/usr/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PM2_HOME=/root/.pm2
PIDFile=/root/.pm2/pm2.pid
ExecStart=/usr/lib/node_modules/pm2/bin/pm2 resurrect
ExecReload=/usr/lib/node_modules/pm2/bin/pm2 reload all
ExecStop=/usr/lib/node_modules/pm2/bin/pm2 kill
[Install]
WantedBy=multi-user.target
Target path
/etc/systemd/system/pm2-root.service
Command list
[ 'chmod +x /etc/systemd/system/pm2-root.service',
'systemctl enable pm2-root',
'systemctl start pm2-root',
'systemctl daemon-reload',
'systemctl status pm2-root' ]
[PM2] Writing init configuration in /etc/systemd/system/pm2-root.service
[PM2] Making script booting at startup...
>>> Executing chmod +x /etc/systemd/system/pm2-root.service
[DONE]
>>> Executing systemctl enable pm2-root
[DONE]
>>> Executing systemctl start pm2-root
[PM2] PM2 Successfully daemonized
[PM2] Stopping PM2...
[PM2][WARN] No process found
[PM2] All processes have been stopped and deleted
[PM2] PM2 stopped
+ pm2 start /home/pi/app/processes.json
[PM2] Spawning PM2 daemon with pm2_home=/root/.pm2
[DONE]
>>> Executing systemctl daemon-reload
[DONE]
>>> Executing systemctl status pm2-root
● pm2-root.service - PM2 process manager
Loaded: loaded (/etc/systemd/system/pm2-root.service; enabled)
Active: active (running) since Fri 2017-06-30 16:36:29 UTC; 199ms ago
Docs: https://pm2.keymetrics.io/
Main PID: 1034 (PM2 v2.5.0: God)
CGroup: /system.slice/pm2-root.service
└─1034 PM2 v2.5.0: God Daemon (/root/.pm2)
Jun 30 16:36:27 grassberry pm2[1021]: [PM2] Spawning PM2 daemon with pm2_home=/root/.pm2
Jun 30 16:36:29 grassberry pm2[1021]: [PM2] PM2 Successfully daemonized
Jun 30 16:36:29 grassberry pm2[1021]: [PM2] Resurrecting
Jun 30 16:36:29 grassberry pm2[1021]: [PM2] Restoring processes located in /root/.pm2/dump.pm2
Jun 30 16:36:29 grassberry pm2[1021]: [PM2][ERROR] No processes saved; DUMP file doesn't exist
Jun 30 16:36:29 grassberry pm2[1021]: ┌──────────┬────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────────┐
Jun 30 16:36:29 grassberry pm2[1021]: │ App name │ id │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ watching │
Jun 30 16:36:29 grassberry pm2[1021]: └──────────┴────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────────┘
Jun 30 16:36:29 grassberry pm2[1021]: Use `pm2 show <id|name>` to get more details about an app
Jun 30 16:36:29 grassberry systemd[1]: Started PM2 process manager.
[DONE]
+---------------------------------------+
[PM2] Freeze a process list on reboot via:
$ pm2 save
[PM2] Remove init script via:
$ pm2 unstartup systemd
As pi
code
startPm2Pi() {
#delete to avoid all pm2 issues
rm -rf "$PM2_HOME"
pkill -f pm2
USER="pi"
HOME="/home/pi"
PM2_HOME="$HOME/.pm2"
export USER HOME PM2_HOME
pm2 startup systemd -u pi & #also tried it without -u
if [ -f /home/pi/app/newVersion.tar.gz ]; then
bash /home/pi/app/deploy.sh
else
pm2 kill && pm2 -u pi start /home/pi/app/processes.json &
fi
pm2 save
}
problem
Error: connect EACCES /home/pi/.pm2/rpc.sock
Also it looks like I got multiple processes running:
root#gb:/home/pi# ps -aux | grep pm2
root 946 1.8 3.4 87908 32172 ? Sl 16:36 0:01 node /usr/bin/pm2 start /home/pi/app/processes.json
root 947 2.1 3.3 87872 31500 ? Sl 16:36 0:01 node /usr/bin/pm2 save
root 990 1.6 2.9 81372 28096 ? Ssl 16:36 0:01 PM2 v2.5.0: God Daemon (/root/.pm2)
root 1034 1.5 3.0 81892 28396 ? Ssl 16:36 0:01 PM2 v2.5.0: God Daemon (/root/.pm2)
root 1040 1.5 2.9 81136 28160 ? Ssl 16:36 0:01 PM2 v2.5.0: God Daemon (/root/.pm2)
pi 1090 7.5 2.9 81520 28216 ? Ssl 16:37 0:01 PM2 v2.5.0: God Daemon (/home/pi/.pm2)
Looks stable so far, I removed the startup script, I think it is either the one (startup) or the other (start).
Removed all fragments from startup for pi
sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 unstartup systemd -u pi --hp /home/pi
and root
sudo pm2 unstartup
And stripped the code to:
startPm2Root() {
#delete to avoid all pm2 issues
rm -rf "$PM2_HOME"
pkill -f pm2
USER="root"
HOME="/root"
PM2_HOME="$HOME/.pm2"
export USER HOME PM2_HOME
#pm2 startup systemd & <= removed this
pm2 kill && pm2 start /home/pi/app/processes.json &
#pm2 save <= removed this, launches a second instance somehow
}
I have two sails js apps on two servers but my both apps on one server are getting stop in every 24 hours, I usually check both the app in the morning and it shows the following after running
pm2 list or pm2 restart all:-
[PM2] Spawning PM2 daemon
[PM2] PM2 Successfully daemonized
[PM2][WARN] No process found
and every time I had to go to their directory and run the command:- pm2 start app.js individually, but both the apps on another server are working correctly.
Please suggest what should I do?
Thanks
Now my service is stoped and after running pm2 logs its showing:
[PM2] Spawning PM2 daemon
[PM2] PM2 Successfully daemonized
########### Starting streaming logs for [all] process
Can not find log files, try to reload logs by executing "pm2 reload <name|all>".
PM2: 2015-07-17 03:27:22: App name:app id:0 online
PM2: child_process.js:935
PM2: throw errnoException(process._errno, 'spawn');
PM2: ^
PM2: Error: spawn ENOMEM
PM2: at errnoException (child_process.js:988:11)
PM2: at ChildProcess.spawn (child_process.js:935:11)
PM2: at exports.spawn (child_process.js:723:9)
PM2: at Object.exports.execFile (child_process.js:607:15)
PM2: at exports.exec (child_process.js:578:18)
PM2: at Object.parseHg [as parse] (/usr/local/lib/node_modules/pm2/node_modules/vizion/lib/hg.js:36:2)
PM2: at /usr/local/lib/node_modules/pm2/node_modules/vizion/lib/vizion.js:16:24
PM2: at /usr/local/lib/node_modules/pm2/node_modules/vizion/lib/identify.js:18:12
PM2: at /usr/local/lib/node_modules/pm2/node_modules/async/lib/async.js:151:21
PM2: at /usr/local/lib/node_modules/pm2/node_modules/vizion/lib/identify.js:12:16
PM2: 2015-07-20 06:23:46: [PM2][WORKER] Started with refreshing interval: 30000
PM2: 2015-07-20 06:23:46: [[[[ PM2/God daemon launched ]]]]
PM2: 2015-07-20 06:23:46: BUS system [READY] on port /root/.pm2/pub.sock
PM2: 2015-07-20 06:23:46: RPC interface [READY] on port /root/.pm2/rpc.sock
and after running pm2 list or pm2 restart all its showing:
[PM2] Spawning PM2 daemon
[PM2] PM2 Successfully daemonized
[PM2][WARN] No process found
plz help I am totally stuck in this.
Thanks
npm start will start my app just fine but when I do:
pm2 start app.js
I get:
[PM2] Spawning PM2 daemon
[PM2] PM2 Successfully daemonized
[PM2] Process app.js launched
┌──────────┬────┬──────┬──────┬────────┬─────────┬────────┬─────────────┬──────────┐
│ App name │ id │ mode │ pid │ status │ restart │ uptime │ memory │ watching │
├──────────┼────┼──────┼──────┼────────┼─────────┼────────┼─────────────┼──────────┤
│ app │ 0 │ fork │ 4681 │ online │ 0 │ 0s │ 11.508 MB │ disabled │
└──────────┴────┴──────┴──────┴────────┴─────────┴────────┴─────────────┴──────────┘
Use `pm2 show <id|name>` to get more details about an app
in the logs I get:
[PM2] Starting streaming logs for [all] process
PM2: 2015-06-04 17:12:37: App name:app id:0 exited with code 0
PM2: 2015-06-04 17:12:37: Starting execution sequence in -fork mode- for app name:app id:0
PM2: 2015-06-04 17:12:37: App name:app id:0 online
PM2: 2015-06-04 17:12:37: App name:app id:0 exited with code 0
PM2: 2015-06-04 17:12:37: Starting execution sequence in -fork mode- for app name:app id:0
PM2: 2015-06-04 17:12:37: App name:app id:0 online
PM2: 2015-06-04 17:12:37: App name:app id:0 exited with code 0
PM2: 2015-06-04 17:12:37: Starting execution sequence in -fork mode- for app name:app id:0
PM2: 2015-06-04 17:12:37: App name:app id:0 online
PM2: 2015-06-04 17:12:38: App name:app id:0 exited with code 0
PM2: 2015-06-04 17:12:38: Starting execution sequence in -fork mode- for app name:app id:0
PM2: 2015-06-04 17:12:38: App name:app id:0 online
PM2: 2015-06-04 17:12:38: App name:app id:0 exited with code 0
PM2: 2015-06-04 17:12:38: Starting execution sequence in -fork mode- for app name:app id:0
PM2: 2015-06-04 17:12:38: App name:app id:0 online
PM2: 2015-06-04 17:12:39: App name:app id:0 exited with code 0
PM2: 2015-06-04 17:12:39: Starting execution sequence in -fork mode- for app name:app id:0
PM2: 2015-06-04 17:12:39: App name:app id:0 online
PM2: 2015-06-04 17:12:39: App name:app id:0 exited with code 0
PM2: 2015-06-04 17:12:39: Script /home/user/app/app.js had too many unstable restarts (15). Stopped. "errored"
here is my package.json:
1 {
2 "name": "app",
3 "version": "0.0.0",
4 "private": true,
5 "scripts": {
6 "start": "node ./bin/www"
7 },
8 "dependencies": {
9 "body-parser": "~1.12.4",
10 "cookie-parser": "~1.3.5",
11 "debug": "~2.2.0",
12 "express": "~4.12.4",
13 "jade": "~1.9.2",
14 "morgan": "~1.5.3",
15 "serve-favicon": "~2.2.1",
16 "stylus": "0.42.3"
17 }
18 }
node version: v0.10.38
pm2 version: 0.12.15
How do I even debug this? I am not sure why it's failing, is there some other place I need to check? My app.js file?
First, I'd try node ./bin/www and just make sure that works correctly.
I'm not sure what the "official" way to figure this out is but this should work:
You could put an uncaught exception handler into your code which simply writes to a file or do something else.
process.on('uncaughtException', function(err) {
console.log('Caught exception: ' + err);
throw err;
});
Edit:
Based on your comment the reason you're having issues is because app.js is not the real entry point in your application. The real entry point is ./bin/www
So you need to tell PM2 to start that file rather than app.js like this:
pm2 start ./bin/www