I have a production only error appearing in my pm2 logs that is giving me absolutely no helpful information - the callstack is not deep enough to view where it started in my own codebase, and I know of no environment variables that can be modified for more verbose stack tracing
does anyone know of any settings that can effect the callstack
App [backgroundExcelIntegration:1] exited with code [0] via signal [SIGTERM]
App [backgroundExcelIntegration:1] starting in -fork mode-
TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received type number (18583)
at Object.writeFileSync (fs.js:1522:5)
at /usr/lib/node_modules/pm2/lib/God/ForkMode.js:258:12
at wrapper (/usr/lib/node_modules/pm2/node_modules/async/internal/once.js:12:16)
at next (/usr/lib/node_modules/pm2/node_modules/async/waterfall.js:96:20)
at /usr/lib/node_modules/pm2/node_modules/async/internal/onlyOnce.js:12:16
at WriteStream.<anonymous> (/usr/lib/node_modules/pm2/lib/Utility.js:186:13)
at WriteStream.emit (events.js:400:28)
at WriteStream.emit (domain.js:537:15)
at internal/fs/streams.js:340:12
at FSReqCallback.oncomplete (fs.js:180:23)
App [backgroundExcelIntegration:1] online
pid=18547 msg=process killed
Deleting process 1
Stopping app:backgroundExcelIntegration id:1
App [backgroundExcelIntegration:1] exited with code [0] via signal [SIGINT]
pid=18583 msg=process killed
Exited peacefully
the answer ended up having nothing to do with node or typescript, it was a PM2 error, after observing pm2 monit it was discovered the server was crashing every 88 seconds (why 88 i have no idea), seemingly others have experienced similar issue and resolved it by running pm2 update
we ran pm2 update and the issue resolved itself
Related
hope somebody can help me, Im pretty new dealing with prod enviroment, Im trying to deploy my nodejs backend on aws ec2 linux instance. Using pm2 to run the enviroment but after y use pm2 start ecosystem.config.js --env production command and I log the output I've got this logs:
at process.emit (events.js:327:22)
at emit (internal/child_process.js:876:12)
at processTicksAndRejections (internal/process/task_queues.js:85:21)
Warning: connect.session() MemoryStore is not
designed for a production environment, as it will leak
memory, and will not scale past a single process.
Error: bind EACCES null:80
at listenOnMasterHandle (net.js:1380:18)
at rr (internal/cluster/child.js:132:12)
at Worker.<anonymous> (internal/cluster/child.js:99:7)
at process.onInternalMessage (internal/cluster/utils.js:47:8)
at process.emit (events.js:327:22)
at emit (internal/child_process.js:876:12)
at processTicksAndRejections (internal/process/task_queues.js:85:21)
I have a NodeJs+Express+Mongodb app and I'm trying to run it on my vps. There was a crashing problem with mongo so I removed "mongo.conf" file. Usually my app start crashing after few minutes. But after this change, everything was good for 24h then my server crash and restart every 1 second :
pm2.log (loop)
2021-04-25T09:46:58: PM2 log: App [server:1] exited with code [1] via signal [SIGINT]
2021-04-25T09:46:58: PM2 log: App [server:1] starting in -fork mode-
2021-04-25T09:46:58: PM2 log: App [server:1] online
2021-04-25T09:47:00: PM2 log: App [server:1] exited with code [1] via signal [SIGINT]
2021-04-25T09:47:00: PM2 log: App [server:1] starting in -fork mode-
2021-04-25T09:47:00: PM2 log: App [server:1] online
2021-04-25T09:47:02: PM2 log: App [server:1] exited with code [1] via signal [SIGINT]
2021-04-25T09:47:02: PM2 log: App [server:1] starting in -fork mode-
2021-04-25T09:47:02: PM2 log: App [server:1] online
2021-04-25T09:47:34: PM2 log: App [server:1] exited with code [1] via signal [SIGINT]
2021-04-25T09:47:34: PM2 log: App [server:1] starting in -fork mode-
2021-04-25T09:47:34: PM2 log: App [server:1] online
server-error.log (loop)
Deprecation warning: use moment.updateLocale(localeName, config) to change an existing locale. moment.defineLocale(localeName, config) should only be used for creating a new locale See http://momentjs.com/guides/#/warnings/define-locale/ for more info.
(node:1022) Warning: Accessing non-existent property 'MongoError' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
TypeError: Cannot read property 'db' of undefined
at /home/debian/app/server.js:987:21
at /home/debian/app/node_modules/mongodb/lib/utils.js:684:9
at /home/debian/app/node_modules/mongodb/lib/mongo_client.js:226:23
at connectCallback (/home/debian/app/node_modules/mongodb/lib/operations/connect.js:366:5)
at /home/debian/app/node_modules/mongodb/lib/operations/connect.js:599:14
at Object.connectHandler [as callback] (/home/debian/app/node_modules/mongodb/lib/core/sdam/topology.js:285:11)
at Timeout._onTimeout (/home/debian/app/node_modules/mongodb/lib/core/sdam/topology.js:443:25)
at listOnTimeout (node:internal/timers:557:17)
at processTimers (node:internal/timers:500:7)
I don't understand what's wrong. Everything works fine on local. Every page works great online and suddenly everything crash and start reboot in loop ?
If that is an environmental variable not seen by your server application, try restart by refreshing with the following
pm2 restart server --update-env
Relevant stack-trace below - from Heroku:
2020-04-28T17:49:11.997756+00:00 app[web.1]: Your app is listening on port 14385
2020-04-28T17:49:12.491486+00:00 heroku[web.1]: State changed from starting to up
2020-04-28T17:49:41.748150+00:00 heroku[web.1]: State changed from up to crashed
2020-04-28T17:49:41.649885+00:00 app[web.1]: _stream_readable.js:660
2020-04-28T17:49:41.649896+00:00 app[web.1]: dest.on('unpipe', onunpipe);
2020-04-28T17:49:41.649897+00:00 app[web.1]: ^
2020-04-28T17:49:41.649897+00:00 app[web.1]:
2020-04-28T17:49:41.649898+00:00 app[web.1]: TypeError: dest.on is not a function
2020-04-28T17:49:41.649899+00:00 app[web.1]: at ReadStream.Readable.pipe (_stream_readable.js:660:8)
2020-04-28T17:49:41.649900+00:00 app[web.1]: at SendStream.stream (/app/node_modules/send/index.js:798:10)
2020-04-28T17:49:41.649900+00:00 app[web.1]: at SendStream.send (/app/node_modules/send/index.js:707:8)
2020-04-28T17:49:41.649900+00:00 app[web.1]: at /app/node_modules/send/index.js:774:12
2020-04-28T17:49:41.649901+00:00 app[web.1]: at FSReqCallback.oncomplete (fs.js:172:5)
From AWS (EC2 using a Bitnami instance for NodeJS apps):
Your app is listening on port 8080
_stream_readable.js:666
dest.on('unpipe', onunpipe);
^
TypeError: dest.on is not a function
at ReadStream.Readable.pipe (_stream_readable.js:666:8)
at SendStream.stream (/opt/bitnami/apps/demo/htdocs/node_modules/send/index.js:798:10)
at SendStream.send (/opt/bitnami/apps/demo/htdocs/node_modules/send/index.js:707:8)
at /opt/bitnami/apps/coronavirus-demo/htdocs/node_modules/send/index.js:774:12
at FSReqCallback.oncomplete (fs.js:167:5)
error: Forever detected script exited with code: 1
error: Script restart attempt #1
I see that we seem to be dying on _stream_readable.js:666 where 'dest.on' isn't defined - and I'm way into the weeds of Node at this point - so I'm not sure what I'm looking for, but I find that we're defining this method further down in the file:
https://github.com/nodejs/node/blob/master/lib/_stream_readable.js:852 -
Is this as simple as we're evaluating this _stream_readable file top-down and we haven't defined on() as of line 666 and that's why it's failing?
But why would it work locally? The app runs fine with Nanoexpress server on a Macbook Pro - same package.json and lock files - no devDependencies that make any difference (the only devDependencies are like Mocha and Sinon and Chai for tests)
I've run the app locally with NODE_ENV=production and NPM_CONFIG_PRODUCTION=true (it's a React & Express [Nanoexpress] app) and it boots and I can get what I need running just fine.
It's only when I push to Heroku / AWS that I'm getting this failure. Feels like we're trying to use a function before it's defined...
nanoexpress author here.
nanoexpress works fine on Heroku on my projects, but on AWS did not test, but nanoexpress does not support Lambda/Serverless.
Please check first your code or try PRO version as now it's Free
2018-05-14T09:12:46.770011+00:00 app[api]: Starting process with command node musicology.js by user leshaber24#yandex.ru
2018-05-14T09:12:50.704836+00:00 heroku[run.5756]: Awaiting client
2018-05-14T09:12:50.760658+00:00 heroku[run.5756]: Starting process with command node musicology.js
2018-05-14T09:12:50.767575+00:00 heroku[run.5756]: State changed from starting to up
2018-05-14T09:12:56.951346+00:00 heroku[run.5756]: Client connection closed. Sending SIGHUP to all processes
2018-05-14T09:12:57.484760+00:00 heroku[run.5756]: Process exited with status 129
2018-05-14T09:12:57.499531+00:00 heroku[run.5756]: State changed from up to complete
These are the logs coming from Heroku. My app is here: https://github.com/Alesha24/mysicology_telegram_bot
I've already fixed an error with SIGKILL command, now (after doing node musicology.js) error is of status 129 and moreover this one:
Error R13 (Attach error) -> Failed to attach to process
This error also appears to be unclear for me. Any ideas of way, how can I fix it?
Trying to get a nodejs app up and running with pm2.
We use SASS and rebuild the to css files on file change with "node-sass-middleware". It all works perfectly until pm2 starts auto restarting the app every time the sass middleware rebuilds a css file... which, is what we tell it to do with:
pm2 start app.js --watch
So, after some reading of the docs it seems you are meant to be able to configure pm2 with a json file and it has an option of "ignore_watch".
So, following the guide on github: https://github.com/Unitech/PM2/blob/master/ADVANCED_README.md#json-app-declaration we set a config json file up to look as follows:
{
"name" : "ourApp",
"script" : "server.js",
"watch" : true,
"ignore_watch" : ["public/css"],
"env": {
"NODE_ENV": "development"
}
}
The problem is, as soon as we try to run it the pm2 logs spit out 1000's of lines of log errors all looking a little like this:
PM2: 2015-05-09 15:32:00: Error: watch ENOSPC
PM2: at errnoException (fs.js:1024:11)
PM2: at FSWatcher.start (fs.js:1056:11)
PM2: at Object.fs.watch (fs.js:1081:11)
PM2: at createFsWatchInstance (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:24:15)
PM2: at setFsWatchListener (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:47:19)
PM2: at EventEmitter.NodeFsHandler._watchWithNodeFs (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:177:15)
PM2: at EventEmitter.NodeFsHandler._handleFile (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:201:8)
PM2: at EventEmitter.<anonymous> (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:353:12)
PM2: at Object.oncomplete (fs.js:107:15)
PM2: 2015-05-09 15:32:00: Error: watch ENOSPC
PM2: at errnoException (fs.js:1024:11)
PM2: at FSWatcher.start (fs.js:1056:11)
PM2: at Object.fs.watch (fs.js:1081:11)
PM2: at createFsWatchInstance (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:24:15)
PM2: at setFsWatchListener (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:47:19)
PM2: at EventEmitter.NodeFsHandler._watchWithNodeFs (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:177:15)
PM2: at EventEmitter.NodeFsHandler._handleFile (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:201:8)
PM2: at EventEmitter.<anonymous> (/usr/local/lib/node_modules/pm2/node_modules/chokidar/lib/nodefs-handler.js:353:12)
PM2: at Object.oncomplete (fs.js:107:15)
PM2: 2015-05-09 15:32:00: Process with pid 9155 still not killed, retrying...
PM2: 2015-05-09 15:32:00: Process with pid 9155 still alive after 800ms
thanks to the answer below
This seems to be the culprit and apparently in linux world ENOSPC means no disk space left.. but with only 2.1gig being used on the entire dev server with a total HDD space of 25gig this cannot be the issue.
PM2: 2015-05-09 15:32:00: Error: watch ENOSPC
Further more, when we run pm2 from cli as mentioned at the top is all works perfectly...
We really need this to work as it is causing havoc for the dev team.
Has anyone else hit this issue?
Thanks,
John
ENOSPC means no space left on the device. So check to see if one of your file systems is full. Assuming your on Linux or another Unix-like operating system, /tmp and /var are likely candidates.