I have a NodeJs+Express+Mongodb app and I'm trying to run it on my vps. There was a crashing problem with mongo so I removed "mongo.conf" file. Usually my app start crashing after few minutes. But after this change, everything was good for 24h then my server crash and restart every 1 second :
pm2.log (loop)
2021-04-25T09:46:58: PM2 log: App [server:1] exited with code [1] via signal [SIGINT]
2021-04-25T09:46:58: PM2 log: App [server:1] starting in -fork mode-
2021-04-25T09:46:58: PM2 log: App [server:1] online
2021-04-25T09:47:00: PM2 log: App [server:1] exited with code [1] via signal [SIGINT]
2021-04-25T09:47:00: PM2 log: App [server:1] starting in -fork mode-
2021-04-25T09:47:00: PM2 log: App [server:1] online
2021-04-25T09:47:02: PM2 log: App [server:1] exited with code [1] via signal [SIGINT]
2021-04-25T09:47:02: PM2 log: App [server:1] starting in -fork mode-
2021-04-25T09:47:02: PM2 log: App [server:1] online
2021-04-25T09:47:34: PM2 log: App [server:1] exited with code [1] via signal [SIGINT]
2021-04-25T09:47:34: PM2 log: App [server:1] starting in -fork mode-
2021-04-25T09:47:34: PM2 log: App [server:1] online
server-error.log (loop)
Deprecation warning: use moment.updateLocale(localeName, config) to change an existing locale. moment.defineLocale(localeName, config) should only be used for creating a new locale See http://momentjs.com/guides/#/warnings/define-locale/ for more info.
(node:1022) Warning: Accessing non-existent property 'MongoError' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
TypeError: Cannot read property 'db' of undefined
at /home/debian/app/server.js:987:21
at /home/debian/app/node_modules/mongodb/lib/utils.js:684:9
at /home/debian/app/node_modules/mongodb/lib/mongo_client.js:226:23
at connectCallback (/home/debian/app/node_modules/mongodb/lib/operations/connect.js:366:5)
at /home/debian/app/node_modules/mongodb/lib/operations/connect.js:599:14
at Object.connectHandler [as callback] (/home/debian/app/node_modules/mongodb/lib/core/sdam/topology.js:285:11)
at Timeout._onTimeout (/home/debian/app/node_modules/mongodb/lib/core/sdam/topology.js:443:25)
at listOnTimeout (node:internal/timers:557:17)
at processTimers (node:internal/timers:500:7)
I don't understand what's wrong. Everything works fine on local. Every page works great online and suddenly everything crash and start reboot in loop ?
If that is an environmental variable not seen by your server application, try restart by refreshing with the following
pm2 restart server --update-env
Related
I have a production only error appearing in my pm2 logs that is giving me absolutely no helpful information - the callstack is not deep enough to view where it started in my own codebase, and I know of no environment variables that can be modified for more verbose stack tracing
does anyone know of any settings that can effect the callstack
App [backgroundExcelIntegration:1] exited with code [0] via signal [SIGTERM]
App [backgroundExcelIntegration:1] starting in -fork mode-
TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received type number (18583)
at Object.writeFileSync (fs.js:1522:5)
at /usr/lib/node_modules/pm2/lib/God/ForkMode.js:258:12
at wrapper (/usr/lib/node_modules/pm2/node_modules/async/internal/once.js:12:16)
at next (/usr/lib/node_modules/pm2/node_modules/async/waterfall.js:96:20)
at /usr/lib/node_modules/pm2/node_modules/async/internal/onlyOnce.js:12:16
at WriteStream.<anonymous> (/usr/lib/node_modules/pm2/lib/Utility.js:186:13)
at WriteStream.emit (events.js:400:28)
at WriteStream.emit (domain.js:537:15)
at internal/fs/streams.js:340:12
at FSReqCallback.oncomplete (fs.js:180:23)
App [backgroundExcelIntegration:1] online
pid=18547 msg=process killed
Deleting process 1
Stopping app:backgroundExcelIntegration id:1
App [backgroundExcelIntegration:1] exited with code [0] via signal [SIGINT]
pid=18583 msg=process killed
Exited peacefully
the answer ended up having nothing to do with node or typescript, it was a PM2 error, after observing pm2 monit it was discovered the server was crashing every 88 seconds (why 88 i have no idea), seemingly others have experienced similar issue and resolved it by running pm2 update
we ran pm2 update and the issue resolved itself
I have been using pm2 for some time now. Recently, I needed to add a custom log directory to my Express4 project called "actionLog". Since it is a directory that gets updated with log files and I don't want pm2 to restart the app on log file changes, I wanted pm2 to ignore watching that directory. After updating pm2 to latest, here is the command I used:
pm2 start app.js --watch --ignore-watch="actionLog"
I get the following error streaming through the pm2 logs:
PM2 Error: watch ENOSPC
PM2 at exports._errnoException (util.js:746:11)
PM2 at FSWatcher.start (fs.js:1172:11)
PM2 at Object.fs.watch (fs.js:1198:11)
PM2 at createFsWatchInstance (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:37:15)
PM2 at setFsWatchListener (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:80:15)
PM2 at EventEmitter.NodeFsHandler._watchWithNodeFs (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:228:14)
PM2 at EventEmitter.NodeFsHandler._handleFile (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:255:21)
PM2 at EventEmitter.<anonymous> (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:468:21)
PM2 at FSReqWrap.oncomplete (fs.js:95:15)
PM2 Error: watch ENOSPC
PM2 at exports._errnoException (util.js:746:11)
PM2 at FSWatcher.start (fs.js:1172:11)
PM2 at Object.fs.watch (fs.js:1198:11)
PM2 at createFsWatchInstance (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:37:15)
PM2 at setFsWatchListener (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:80:15)
PM2 at EventEmitter.NodeFsHandler._watchWithNodeFs (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:228:14)
PM2 at EventEmitter.NodeFsHandler._handleFile (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:255:21)
PM2 at EventEmitter.<anonymous> (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:468:21)
PM2 at FSReqWrap.oncomplete (fs.js:95:15)
PM2 Error: watch ENOSPC
I have also tried using the command:
pm2 start bin/www --watch --ignore-watch="actionLog"
This also generated the same error.
Once I have the correct ignore-watch parameter, I will update the json config file I use for starting pm2. At the moment, using this config file with the ignore-watch is also causing errors, but instead of a detailed stack trace as above, I only see the following in the pm2 logs:
PM2: 2016-02-23 04:05:34: Starting execution sequence in -fork mode- for app name:aadm id:2
PM2: 2016-02-23 04:05:34: App name:aadm id:2 online
PM2: 2016-02-23 04:05:35: Change detected on path actionLog/userAction.log for app aadm - restarting
PM2: 2016-02-23 04:05:35: Stopping app:aadm id:2
PM2: 2016-02-23 04:05:35: App name:aadm id:2 exited with code SIGTERM
PM2: 2016-02-23 04:05:35: Process with pid 5102 killed
PM2: 2016-02-23 04:05:35: Starting execution sequence in -fork mode- for app name:aadm id:2
PM2: 2016-02-23 04:05:35: App name:aadm id:2 online
I have looked at some of the reports of ignore-watch problems such as:
https://github.com/Unitech/PM2/issues/1288
https://github.com/Unitech/PM2/issues/918
https://github.com/Unitech/PM2/issues/1275
How to auto reload project nodejs use pm2
Express 4 + pm2 watch not working
Unfortunately, I am still stuck. Any ideas?
You have two options:
Increase the systems inotify max watch limit:
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
or
Try to decrease you watch count by additionally ignoring node_modules:
pm2 start bin/www --watch --ignore-watch="actionLog node_modules"
If you don't have sudo rights option 2 is for you.
Turns out even though the errors were triggered by ignore-watch, they were not caused by it. The following command fixed the issue:
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p
I found this solution here: https://stackoverflow.com/a/32600959/2234029
I also tried "npm dedupe" -as suggested on that thread- which didn't help.
I had the same problem.
It was driving me crazy, then I discovered there were many sessions of PM2 running in my system.
A command like ps aux | grep pm2 shows you all instances running.
So I killed manually each instance (shown by the command above) one by one with:
kill -9 <pid1> <pid2> <pid3> <...>
At last I restarted the process:
pm2 start ecosystem.config.js
and then everything went working smoothly as written in my config file (without any "watch" option).
The actual question is: why those processes where running on my server, but I can't answer this.
I think it is ignore_watch ie with an underline
I am getting an error while starting pm2 in cluster mode.
I ran this command -
pm2 start <filename> -i 4
Only one instance gets created.
I am using Windows 7, Node 0.12.0 and pm2 0.14.0
This is my error log
2015-06-18 10:52:04: [PM2][WORKER] Started with refreshing interval:
30000 2015-06-18 10:52:04: [[[[ PM2/God daemon launched ]]]]
2015-06-18 10:52:04: BUS system [READY] on port \.\pipe\pub.sock
2015-06-18 10:52:04: RPC interface [READY] on port \.\pipe\rpc.sock
2015-06-18 10:52:04: Starting execution sequence in -cluster mode- for
app name:server id:0 2015-06-18 10:52:04: App name:server id:0 online
2015-06-18 10:52:04: Starting execution sequence in -cluster mode- for
app name:server id:1 2015-06-18 10:52:04: Trace: { [Error: spawn
EBADF] code: 'EBADF', errno: 'EBADF', syscall: 'spawn' }
at Object.God.logAndGenerateError (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God\Methods.js:30:15)
at Object.nodeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God\ClusterMode.js:52:11)
at Object.executeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:145:9)
at ex (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:361:18)
at C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:365:16
at Worker.cluOnline (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:188:17)
at Worker.g (events.js:199:16)
at Worker.emit (events.js:104:17)
at online (cluster.js:449:12)
at Worker.onmessage (cluster.js:436:7) 2015-06-18 10:52:04: Starting execution sequence in -cluster mode- for app name:server id:2
2015-06-18 10:52:05: Trace: { [Error: spawn EBADF] code: 'EBADF',
errno: 'EBADF', syscall: 'spawn' }
at Object.God.logAndGenerateError (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God\Methods.js:30:15)
at Object.nodeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God\ClusterMode.js:52:11)
at Object.executeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:145:9)
at ex (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:361:18)
at C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:362:25
at nodeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:146:29)
at Object.nodeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God\ClusterMode.js:53:14)
at Object.executeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:145:9)
at ex (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:361:18)
at C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:365:16
2015-06-18 10:52:05: Starting execution sequence in -cluster mode- for
app name:server id:3 2015-06-18 10:52:05: Trace: { [Error: spawn
EBADF] code: 'EBADF', errno: 'EBADF', syscall: 'spawn' }
at Object.God.logAndGenerateError (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God\Methods.js:30:15)
at Object.nodeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God\ClusterMode.js:52:11)
at Object.executeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:145:9)
at ex (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:361:18)
at C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:362:25
at nodeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:146:29)
at Object.nodeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God\ClusterMode.js:53:14)
at Object.executeApp (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:145:9)
at ex (C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:361:18)
at C:\Users\pbaid\AppData\Roaming\npm\node_modules\pm2\lib\God.js:362:25
I met this problem today too.
If I run "pm2 logs" before any other pm2 operation, and then start applications in cluster mode ("pm2 start app.js -i 4"), everything looks fine. It seems pm2 need some time for initialization. I have no idea about the real reason.
Looking forward for better solution too.
The solution is quite simple: restart the pc or server.
On Windows, don't use clusters:
pm2 start "filename.js"
I have a problem with pm2.
My start json is
{
"apps" : [{
"name" : "sails",
"script" : "app.js",
"log_date_format" : "DD-MM-YYYY HH:mm Z",
"ignoreWatch" : ["[\\/\\\\]\\./", "node_modules", ".tmp", "views", "assets", ".idea"],
"watch" : true
}]
}
Then i do
pm2 start dev.json
PM2 goes to infinity restarting with this message
PM2: 2015-02-11 14:56:39: Change detected for app name: sails - restarting
PM2: 2015-02-11 14:56:39: Process with pid 5766 still not killed, retrying...
PM2: 2015-02-11 14:56:39: App closed with code: 0
PM2: 2015-02-11 14:56:39: App name:sails id:0 exited
PM2: 2015-02-11 14:56:40: Process with pid 5766 killed
PM2: 2015-02-11 14:56:40: Starting execution sequence in -fork mode- for app name:sails id:0
PM2: 2015-02-11 14:56:40: App name:sails id:0 online
But no files was changed. It seems to be PM2 restarting after sails log out debug messages. Any ideas?
I'm solved this issue by switching a PM2 version frpm 0.12.5 to 0.11
I had a similar issue. Install the newest version that was released 3 days ago, 0.12.7 and see if the information in issue 969 link helps you out.
https://github.com/Unitech/PM2/issues/969
I'm running a simple express app, it is pulling static files from S3, and runs fine locally, but on Heroku it crashes constantly on a warning about permission denied but with no further info.
Is there a way I could find out what is causing the permission error?
Heroku log from deploy to crash:
2011-12-26T22:41:14+00:00 heroku[slugc]: Slug compilation started
2011-12-26T22:41:19+00:00 heroku[api]: Deploy 20d0578 by jeffandersen#gmail.com
2011-12-26T22:41:19+00:00 heroku[api]: Release v20 created by jeffandersen#gmail.com
2011-12-26T22:41:20+00:00 heroku[web.1]: State changed from crashed to created
2011-12-26T22:41:20+00:00 heroku[web.1]: State changed from created to starting
2011-12-26T22:41:21+00:00 heroku[slugc]: Slug compilation finished
2011-12-26T22:41:22+00:00 heroku[web.1]: Starting process with command `node server.js`
2011-12-26T22:41:23+00:00 app[web.1]: info - socket.io started
2011-12-26T22:41:23+00:00 app[web.1]: warn - error raised: Error: EACCES, Permission denied
2011-12-26T22:41:24+00:00 heroku[web.1]: State changed from starting to crashed
2011-12-26T22:41:25+00:00 heroku[web.1]: Process exited
Are you binding the server port to the environment variable injected by Heroku, $PORT?