I have a problem with pm2.
My start json is
{
"apps" : [{
"name" : "sails",
"script" : "app.js",
"log_date_format" : "DD-MM-YYYY HH:mm Z",
"ignoreWatch" : ["[\\/\\\\]\\./", "node_modules", ".tmp", "views", "assets", ".idea"],
"watch" : true
}]
}
Then i do
pm2 start dev.json
PM2 goes to infinity restarting with this message
PM2: 2015-02-11 14:56:39: Change detected for app name: sails - restarting
PM2: 2015-02-11 14:56:39: Process with pid 5766 still not killed, retrying...
PM2: 2015-02-11 14:56:39: App closed with code: 0
PM2: 2015-02-11 14:56:39: App name:sails id:0 exited
PM2: 2015-02-11 14:56:40: Process with pid 5766 killed
PM2: 2015-02-11 14:56:40: Starting execution sequence in -fork mode- for app name:sails id:0
PM2: 2015-02-11 14:56:40: App name:sails id:0 online
But no files was changed. It seems to be PM2 restarting after sails log out debug messages. Any ideas?
I'm solved this issue by switching a PM2 version frpm 0.12.5 to 0.11
I had a similar issue. Install the newest version that was released 3 days ago, 0.12.7 and see if the information in issue 969 link helps you out.
https://github.com/Unitech/PM2/issues/969
Related
I have a production only error appearing in my pm2 logs that is giving me absolutely no helpful information - the callstack is not deep enough to view where it started in my own codebase, and I know of no environment variables that can be modified for more verbose stack tracing
does anyone know of any settings that can effect the callstack
App [backgroundExcelIntegration:1] exited with code [0] via signal [SIGTERM]
App [backgroundExcelIntegration:1] starting in -fork mode-
TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received type number (18583)
at Object.writeFileSync (fs.js:1522:5)
at /usr/lib/node_modules/pm2/lib/God/ForkMode.js:258:12
at wrapper (/usr/lib/node_modules/pm2/node_modules/async/internal/once.js:12:16)
at next (/usr/lib/node_modules/pm2/node_modules/async/waterfall.js:96:20)
at /usr/lib/node_modules/pm2/node_modules/async/internal/onlyOnce.js:12:16
at WriteStream.<anonymous> (/usr/lib/node_modules/pm2/lib/Utility.js:186:13)
at WriteStream.emit (events.js:400:28)
at WriteStream.emit (domain.js:537:15)
at internal/fs/streams.js:340:12
at FSReqCallback.oncomplete (fs.js:180:23)
App [backgroundExcelIntegration:1] online
pid=18547 msg=process killed
Deleting process 1
Stopping app:backgroundExcelIntegration id:1
App [backgroundExcelIntegration:1] exited with code [0] via signal [SIGINT]
pid=18583 msg=process killed
Exited peacefully
the answer ended up having nothing to do with node or typescript, it was a PM2 error, after observing pm2 monit it was discovered the server was crashing every 88 seconds (why 88 i have no idea), seemingly others have experienced similar issue and resolved it by running pm2 update
we ran pm2 update and the issue resolved itself
I have a NodeJs+Express+Mongodb app and I'm trying to run it on my vps. There was a crashing problem with mongo so I removed "mongo.conf" file. Usually my app start crashing after few minutes. But after this change, everything was good for 24h then my server crash and restart every 1 second :
pm2.log (loop)
2021-04-25T09:46:58: PM2 log: App [server:1] exited with code [1] via signal [SIGINT]
2021-04-25T09:46:58: PM2 log: App [server:1] starting in -fork mode-
2021-04-25T09:46:58: PM2 log: App [server:1] online
2021-04-25T09:47:00: PM2 log: App [server:1] exited with code [1] via signal [SIGINT]
2021-04-25T09:47:00: PM2 log: App [server:1] starting in -fork mode-
2021-04-25T09:47:00: PM2 log: App [server:1] online
2021-04-25T09:47:02: PM2 log: App [server:1] exited with code [1] via signal [SIGINT]
2021-04-25T09:47:02: PM2 log: App [server:1] starting in -fork mode-
2021-04-25T09:47:02: PM2 log: App [server:1] online
2021-04-25T09:47:34: PM2 log: App [server:1] exited with code [1] via signal [SIGINT]
2021-04-25T09:47:34: PM2 log: App [server:1] starting in -fork mode-
2021-04-25T09:47:34: PM2 log: App [server:1] online
server-error.log (loop)
Deprecation warning: use moment.updateLocale(localeName, config) to change an existing locale. moment.defineLocale(localeName, config) should only be used for creating a new locale See http://momentjs.com/guides/#/warnings/define-locale/ for more info.
(node:1022) Warning: Accessing non-existent property 'MongoError' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
TypeError: Cannot read property 'db' of undefined
at /home/debian/app/server.js:987:21
at /home/debian/app/node_modules/mongodb/lib/utils.js:684:9
at /home/debian/app/node_modules/mongodb/lib/mongo_client.js:226:23
at connectCallback (/home/debian/app/node_modules/mongodb/lib/operations/connect.js:366:5)
at /home/debian/app/node_modules/mongodb/lib/operations/connect.js:599:14
at Object.connectHandler [as callback] (/home/debian/app/node_modules/mongodb/lib/core/sdam/topology.js:285:11)
at Timeout._onTimeout (/home/debian/app/node_modules/mongodb/lib/core/sdam/topology.js:443:25)
at listOnTimeout (node:internal/timers:557:17)
at processTimers (node:internal/timers:500:7)
I don't understand what's wrong. Everything works fine on local. Every page works great online and suddenly everything crash and start reboot in loop ?
If that is an environmental variable not seen by your server application, try restart by refreshing with the following
pm2 restart server --update-env
I have been using pm2 for some time now. Recently, I needed to add a custom log directory to my Express4 project called "actionLog". Since it is a directory that gets updated with log files and I don't want pm2 to restart the app on log file changes, I wanted pm2 to ignore watching that directory. After updating pm2 to latest, here is the command I used:
pm2 start app.js --watch --ignore-watch="actionLog"
I get the following error streaming through the pm2 logs:
PM2 Error: watch ENOSPC
PM2 at exports._errnoException (util.js:746:11)
PM2 at FSWatcher.start (fs.js:1172:11)
PM2 at Object.fs.watch (fs.js:1198:11)
PM2 at createFsWatchInstance (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:37:15)
PM2 at setFsWatchListener (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:80:15)
PM2 at EventEmitter.NodeFsHandler._watchWithNodeFs (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:228:14)
PM2 at EventEmitter.NodeFsHandler._handleFile (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:255:21)
PM2 at EventEmitter.<anonymous> (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:468:21)
PM2 at FSReqWrap.oncomplete (fs.js:95:15)
PM2 Error: watch ENOSPC
PM2 at exports._errnoException (util.js:746:11)
PM2 at FSWatcher.start (fs.js:1172:11)
PM2 at Object.fs.watch (fs.js:1198:11)
PM2 at createFsWatchInstance (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:37:15)
PM2 at setFsWatchListener (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:80:15)
PM2 at EventEmitter.NodeFsHandler._watchWithNodeFs (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:228:14)
PM2 at EventEmitter.NodeFsHandler._handleFile (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:255:21)
PM2 at EventEmitter.<anonymous> (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:468:21)
PM2 at FSReqWrap.oncomplete (fs.js:95:15)
PM2 Error: watch ENOSPC
I have also tried using the command:
pm2 start bin/www --watch --ignore-watch="actionLog"
This also generated the same error.
Once I have the correct ignore-watch parameter, I will update the json config file I use for starting pm2. At the moment, using this config file with the ignore-watch is also causing errors, but instead of a detailed stack trace as above, I only see the following in the pm2 logs:
PM2: 2016-02-23 04:05:34: Starting execution sequence in -fork mode- for app name:aadm id:2
PM2: 2016-02-23 04:05:34: App name:aadm id:2 online
PM2: 2016-02-23 04:05:35: Change detected on path actionLog/userAction.log for app aadm - restarting
PM2: 2016-02-23 04:05:35: Stopping app:aadm id:2
PM2: 2016-02-23 04:05:35: App name:aadm id:2 exited with code SIGTERM
PM2: 2016-02-23 04:05:35: Process with pid 5102 killed
PM2: 2016-02-23 04:05:35: Starting execution sequence in -fork mode- for app name:aadm id:2
PM2: 2016-02-23 04:05:35: App name:aadm id:2 online
I have looked at some of the reports of ignore-watch problems such as:
https://github.com/Unitech/PM2/issues/1288
https://github.com/Unitech/PM2/issues/918
https://github.com/Unitech/PM2/issues/1275
How to auto reload project nodejs use pm2
Express 4 + pm2 watch not working
Unfortunately, I am still stuck. Any ideas?
You have two options:
Increase the systems inotify max watch limit:
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
or
Try to decrease you watch count by additionally ignoring node_modules:
pm2 start bin/www --watch --ignore-watch="actionLog node_modules"
If you don't have sudo rights option 2 is for you.
Turns out even though the errors were triggered by ignore-watch, they were not caused by it. The following command fixed the issue:
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
I found this solution here: https://stackoverflow.com/a/32600959/2234029
I also tried "npm dedupe" -as suggested on that thread- which didn't help.
I had the same problem.
It was driving me crazy, then I discovered there were many sessions of PM2 running in my system.
A command like ps aux | grep pm2 shows you all instances running.
So I killed manually each instance (shown by the command above) one by one with:
kill -9 <pid1> <pid2> <pid3> <...>
At last I restarted the process:
pm2 start ecosystem.config.js
and then everything went working smoothly as written in my config file (without any "watch" option).
The actual question is: why those processes where running on my server, but I can't answer this.
I think it is ignore_watch ie with an underline
I am getting an error while starting pm2 in cluster mode.
I ran this command -
pm2 start <filename> -i 4
Only one instance gets created.
I am using Windows 7, Node 0.12.0 and pm2 0.14.0
This is my error log
2015-06-18 10:52:04: [PM2][WORKER] Started with refreshing interval:
30000 2015-06-18 10:52:04: [[[[ PM2/God daemon launched ]]]]
2015-06-18 10:52:04: BUS system [READY] on port \.\pipe\pub.sock
2015-06-18 10:52:04: RPC interface [READY] on port \.\pipe\rpc.sock
2015-06-18 10:52:04: Starting execution sequence in -cluster mode- for
app name:server id:0 2015-06-18 10:52:04: App name:server id:0 online
2015-06-18 10:52:04: Starting execution sequence in -cluster mode- for
app name:server id:1 2015-06-18 10:52:04: Trace: { [Error: spawn
EBADF] code: 'EBADF', errno: 'EBADF', syscall: 'spawn' }
at Object.God.logAndGenerateError (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God\Methods.js:30:15)
at Object.nodeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God\ClusterMode.js:52:11)
at Object.executeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:145:9)
at ex (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:361:18)
at C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:365:16
at Worker.cluOnline (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:188:17)
at Worker.g (events.js:199:16)
at Worker.emit (events.js:104:17)
at online (cluster.js:449:12)
at Worker.onmessage (cluster.js:436:7) 2015-06-18 10:52:04: Starting execution sequence in -cluster mode- for app name:server id:2
2015-06-18 10:52:05: Trace: { [Error: spawn EBADF] code: 'EBADF',
errno: 'EBADF', syscall: 'spawn' }
at Object.God.logAndGenerateError (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God\Methods.js:30:15)
at Object.nodeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God\ClusterMode.js:52:11)
at Object.executeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:145:9)
at ex (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:361:18)
at C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:362:25
at nodeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:146:29)
at Object.nodeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God\ClusterMode.js:53:14)
at Object.executeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:145:9)
at ex (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:361:18)
at C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:365:16
2015-06-18 10:52:05: Starting execution sequence in -cluster mode- for
app name:server id:3 2015-06-18 10:52:05: Trace: { [Error: spawn
EBADF] code: 'EBADF', errno: 'EBADF', syscall: 'spawn' }
at Object.God.logAndGenerateError (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God\Methods.js:30:15)
at Object.nodeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God\ClusterMode.js:52:11)
at Object.executeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:145:9)
at ex (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:361:18)
at C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:362:25
at nodeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:146:29)
at Object.nodeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God\ClusterMode.js:53:14)
at Object.executeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:145:9)
at ex (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:361:18)
at C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:362:25
I met this problem today too.
If I run "pm2 logs" before any other pm2 operation, and then start applications in cluster mode ("pm2 start app.js -i 4"), everything looks fine. It seems pm2 need some time for initialization. I have no idea about the real reason.
Looking forward for better solution too.
The solution is quite simple: restart the pc or server.
On Windows, don't use clusters:
pm2 start "filename.js"
Trying to get a nodejs app up and running with pm2.
We use SASS and rebuild the to css files on file change with "node-sass-middleware". It all works perfectly until pm2 starts auto restarting the app every time the sass middleware rebuilds a css file... which, is what we tell it to do with:
pm2 start app.js --watch
So, after some reading of the docs it seems you are meant to be able to configure pm2 with a json file and it has an option of "ignore_watch".
So, following the guide on github: https://github.com/Unitech/PM2/blob/master/ADVANCED_README.md#json-app-declaration we set a config json file up to look as follows:
{
"name" : "ourApp",
"script" : "server.js",
"watch" : true,
"ignore_watch" : ["public/css"],
"env": {
"NODE_ENV": "development"
}
}
The problem is, as soon as we try to run it the pm2 logs spit out 1000's of lines of log errors all looking a little like this:
PM2: 2015-05-09 15:32:00: Error: watch ENOSPC
PM2: at errnoException (fs.js:1024:11)
PM2: at FSWatcher.start (fs.js:1056:11)
PM2: at Object.fs.watch (fs.js:1081:11)
PM2: at createFsWatchInstance (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:24:15)
PM2: at setFsWatchListener (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:47:19)
PM2: at EventEmitter.NodeFsHandler._watchWithNodeFs (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:177:15)
PM2: at EventEmitter.NodeFsHandler._handleFile (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:201:8)
PM2: at EventEmitter.<anonymous> (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:353:12)
PM2: at Object.oncomplete (fs.js:107:15)
PM2: 2015-05-09 15:32:00: Error: watch ENOSPC
PM2: at errnoException (fs.js:1024:11)
PM2: at FSWatcher.start (fs.js:1056:11)
PM2: at Object.fs.watch (fs.js:1081:11)
PM2: at createFsWatchInstance (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:24:15)
PM2: at setFsWatchListener (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:47:19)
PM2: at EventEmitter.NodeFsHandler._watchWithNodeFs (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:177:15)
PM2: at EventEmitter.NodeFsHandler._handleFile (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:201:8)
PM2: at EventEmitter.<anonymous> (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:353:12)
PM2: at Object.oncomplete (fs.js:107:15)
PM2: 2015-05-09 15:32:00: Process with pid 9155 still not killed, retrying...
PM2: 2015-05-09 15:32:00: Process with pid 9155 still alive after 800ms
thanks to the answer below
This seems to be the culprit and apparently in linux world ENOSPC means no disk space left.. but with only 2.1gig being used on the entire dev server with a total HDD space of 25gig this cannot be the issue.
PM2: 2015-05-09 15:32:00: Error: watch ENOSPC
Further more, when we run pm2 from cli as mentioned at the top is all works perfectly...
We really need this to work as it is causing havoc for the dev team.
Has anyone else hit this issue?
Thanks,
John
ENOSPC means no space left on the device. So check to see if one of your file systems is full. Assuming your on Linux or another Unix-like operating system, /tmp and /var are likely candidates.