Backend Setup:
Node.js/Express server running on port 5000
AWS S3 (for images)
AWS RDS Mysql (for data)
Hi guys,
I'm a frontend guy, so I don't know much about backend, I just started learning.
I have a Node.js/Express server on port 5000 that I use to connect to an S3 bucket and an RDS Mysql database, that is enough for the React frontend I'm developing at the moment.
I use nodemon to deal with disconnects and reconnect automatically. It used to work just fine 24/7, but I must have changed something during the last 3 months and now it crashes several times a day and I have to restart the server manually. It's not a huge deal but I would like to learn how to deal with it.
Here's the error I keep getting:
[nodemon] restarting due to changes...
[nodemon] starting `node server.js`
Listening on port 5000..
Connection Established Successfully
Error: read ECONNRESET
at TCP.onStreamRead (internal/stream_base_commons.js:205:27)
--------------------
at Protocol._enqueue (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/mysql/lib/protocol/Protocol.js:144:48)
at Connection.query (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/mysql/lib/Connection.js:198:25)
at /Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/routes/meProfileImageData.js:10:12
at Layer.handle [as handle_request] (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/layer.js:95:5)
at next (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/route.js:137:13)
at Route.dispatch (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/route.js:112:3)
at Layer.handle [as handle_request] (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/layer.js:95:5)
at /Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/index.js:281:22
at Function.process_params (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/index.js:335:12)
at next (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/index.js:275:10) {
errno: 'ECONNRESET',
code: 'ECONNRESET',
syscall: 'read',
fatal: true
}
Error: Cannot enqueue Query after fatal error.
at Protocol._validateEnqueue (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/mysql/lib/protocol/Protocol.js:212:16)
at Protocol._enqueue (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/mysql/lib/protocol/Protocol.js:138:13)
at Connection.query (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/mysql/lib/Connection.js:198:25)
at /Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/routes/meProfileImageData.js:10:12
at Layer.handle [as handle_request] (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/layer.js:95:5)
at next (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/route.js:137:13)
at Route.dispatch (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/route.js:112:3)
at Layer.handle [as handle_request] (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/layer.js:95:5)
at /Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/index.js:281:22
at Function.process_params (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/index.js:335:12) {
code: 'PROTOCOL_ENQUEUE_AFTER_FATAL_ERROR',
fatal: false
}
Obviously the second part happens when I try to connect without knowing the server has already crashed.
There are plenty of posts on stackoverflow about similar ECONNRESET problems, so it's very possible that my exact problem has already been replied somewhere else, but to be honest I'm just not good enough at this yet to assimilate the answer by myself, I could use some pointers as to what files the problem seems to be in or whatever you could tell me.
I have seen in several posts that the best way to deal with this kind of problem is to set error listeners, and I'm happy to do just that, but to be honest I don't know where to start because the problem happens when I'm not using the server usually, like at night or when I'm not even in front of the computer. I have no idea how to reproduce the problem on my own, it just happens by itself whenever it happens, so that's why it's so hard for me to know where to start. I looked at the logs on my S3 and RDS Mysql and didn't see anything that seemed directly connected to this.
Could you offer me some pointers based on what you see here? I have some possible ideas:
could it be the router at index.js causing this?
when I look at my RDS Mysql database panel I see at the moment there are 3 active connections, even though I stopped working like 2 hours ago. Could it be that those connections timing out are causing this? (read this in another post)
could it be meProfileImageData.js that should get error listening first?
That's all I have for now, any ideas or pointers are highly welcome, and I'll be glad to run any tests or install any necessary additional software to troubleshoot this if you feel like putting in the time to help me.
Thanks
EDIT 1: had a new error today that seems to point at connections being closed automatically and Node.js not dealing with it before there is an attempt to connect:
Error: read ETIMEDOUT
at TCP.onStreamRead (internal/stream_base_commons.js:205:27)
--------------------
at Protocol._enqueue (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/mysql/lib/protocol/Protocol.js:144:48)
at Connection.query (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/mysql/lib/Connection.js:198:25)
at /Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/routes/meProfileImageData.js:10:12
at Layer.handle [as handle_request] (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/layer.js:95:5)
at next (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/route.js:137:13)
at Route.dispatch (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/route.js:112:3)
at Layer.handle [as handle_request] (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/layer.js:95:5)
at /Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/index.js:281:22
at Function.process_params (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/index.js:335:12)
at next (/Volumes/Content/Desktop/Dropbox/HugoPrivate/React/ladswork-server/node_modules/express/lib/router/index.js:275:10) {
errno: 'ETIMEDOUT',
code: 'ETIMEDOUT',
syscall: 'read',
fatal: true
Related
The redis get calls sometimes fails with the following error. Once this happens, as a follow through, many of the requests fail/timeout intermittently with a 502 status.
{ Error: Redis connection to X.X.X.X:6379 failed - write ECONNRESET
at afterWriteDispatched (internal/stream_base_commons.js:78:25)
at writeGeneric (internal/stream_base_commons.js:73:3)
at Socket._writeGeneric (net.js:714:5)
at Socket._write (net.js:726:8)
at doWrite (_stream_writable.js:415:12)
at writeOrBuffer (_stream_writable.js:399:5)
at Socket.Writable.write (_stream_writable.js:299:11)
at RedisClient.write (/workspace/node_modules/redis/index.js:949:43)
at RedisClient.internal_send_command (/workspace/node_modules/redis/index.js:885:14)
at RedisClient.internal_send_command_trace [as internal_send_command] (/workspace/node_modules/#google-cloud/trace-agent/build/src/plugins/plugin-redis.js:91:50)
at RedisClient.get (/workspace/node_modules/redis/lib/commands.js:46:25)
at Promise (internal/util.js:274:30)
at new Promise (<anonymous>)
at RedisClient.get (internal/util.js:273:12)
at ConversationQueueServices.startTimeoutToClearQueueForVisitor (/workspace/ls/Services/ConversationQueueServices.js:306:57)
at PeoplesService.onVisitorSubscriberLeft (/workspace/ls/Services/PeoplesService.js:51:31)
at app.post (/workspace/app.js:196:27)
at Layer.handle [as handle_request] (/workspace/node_modules/express/lib/router/layer.js:95:5)
at next (/workspace/node_modules/express/lib/router/route.js:137:13)
at Route.dispatch (/workspace/node_modules/express/lib/router/route.js:112:3)
at Layer.handle [as handle_request] (/workspace/node_modules/express/lib/router/layer.js:95:5)
at /workspace/node_modules/express/lib/router/index.js:281:22 errno: 'ECONNRESET', code: 'ECONNRESET', syscall: 'write' }
From the looks of it, ECONNRESET indicates the connection is closed by redis instance. There is no surge of traffic of any sort. cpu, memory and number of connections are as usual.
Will moving to updated redis npm client may help? Currently using 3.0.2 redis client version
Any help is much appreciated. Thanks in advance.
ECONNRESET error is thrown usually when the other end of the connection which could be TCP connected node that is communicating with the server or calling out and the socket timeout results in no response received which results in the sender closing it with the timeout exception handling type.
You can try to add retries or increase the timeout for the redis connection configurations.
Try running through this tutorial which might be helpful for troubleshooting the Redis Setup.
Please check a similar example with details node-js-best-practice-exception-handling.
I am not sure how I go about fixing this issue, for some reason, just every now and again my nodeJS app seems to fire a spawn error which I can't figure out how to fix or resolve.
Error: spawn ENOMEM
at ChildProcess.spawn (internal/child_process.js:406:11)
at spawn (child_process.js:553:9)
at Object.execFile (child_process.js:237:17)
at exec (child_process.js:158:25)
at CheckIP (/(PATH)/(APPNAME)/app.js:57:3)
at Layer.handle [as handle_request] (/(PATH)/(APPNAME)/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (/(PATH)/(APPNAME)/node_modules/express/lib/router/index.js:317:13)
at /(PATH)/(APPNAME)/node_modules/express/lib/router/index.js:284:7
at Function.process_params (/(PATH)/(APPNAME)/node_modules/express/lib/router/index.js:335:12)
at next (/(PATH)/(APPNAME)/node_modules/express/lib/router/index.js:275:10)
at cookieParser (/(PATH)/(APPNAME)/node_modules/cookie-parser/index.js:71:5)
at Layer.handle [as handle_request] (/(PATH)/(APPNAME)/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (/(PATH)/(APPNAME)/node_modules/express/lib/router/index.js:317:13)
at /(PATH)/(APPNAME)/node_modules/express/lib/router/index.js:284:7
at Function.process_params (/(PATH)/(APPNAME)/node_modules/express/lib/router/index.js:335:12)
at next (/(PATH)/(APPNAME)/node_modules/express/lib/router/index.js:275:10)
I am wondering is it possible to have a script that watches for this error and if it happens run:
systemctl stop APPNAME
systemctl start APPNAME
I know that does not fix the problem but until I can work out what the issue is I need a temp solution.
from everything I am reading seems it that the spawn is a memory thing? Not 100% sure, if that is the case could I have a script that releases memory as the app continues to run?Swap
The issue is related to the Memory. More precisely it is linked with Swap Memory.
If you want to automatically restart your node app. You can run it with help of PM2 and configure it's restart strategies
If you are using Ubantu and want to update swap size you can follow instructions mentioned here
I have a node and express api deployed on Heroku and the api works fine for normal GET requests but when I try to Get or Post a request that involves querying a Postgre database from heroku add ons, such as registering a new user , it gives me error with this error log:
2020-09-13T18:48:39.984122+00:00 app[web.1]: TypeError: str.charAt is not a function
2020-09-13T18:48:39.984146+00:00 app[web.1]: at parse (/app/node_modules/pg-connection-string/index.js:13:11)
2020-09-13T18:48:39.984147+00:00 app[web.1]: at new ConnectionParameters (/app/node_modules/pg/lib/connection-parameters.js:56:42)
2020-09-13T18:48:39.984148+00:00 app[web.1]: at new Client (/app/node_modules/pg/lib/client.js:19:33)
2020-09-13T18:48:39.984148+00:00 app[web.1]: at BoundPool.newClient (/app/node_modules/pg-pool/index.js:201:20)
2020-09-13T18:48:39.984149+00:00 app[web.1]: at BoundPool.connect (/app/node_modules/pg-pool/index.js:195:10)
2020-09-13T18:48:39.984150+00:00 app[web.1]: at BoundPool.query (/app/node_modules/pg-pool/index.js:343:10)
2020-09-13T18:48:39.984150+00:00 app[web.1]: at /app/routes/UserAuth/jwtAuth.js:19:33
2020-09-13T18:48:39.984150+00:00 app[web.1]: at Layer.handle [as handle_request] (/app/node_modules/express/lib/router/layer.js:95:5)
2020-09-13T18:48:39.984151+00:00 app[web.1]: at next (/app/node_modules/express/lib/router/route.js:137:13)
2020-09-13T18:48:39.984151+00:00 app[web.1]: at Route.dispatch (/app/node_modules/express/lib/router/route.js:112:3)
Two possibilities due to which you might be getting this error :
1.You are trying to access some other datatype as a string.
2.You are passing this as a callback, but str.charAt() returns a character and not function.
Check for both scenarios and since you have not shared code snippet, so could not suggest the exact code implemented solution.
I have a docker image running a node process. It is listening to a particular port. When i am trying to access the port using curl or through browser request, I am getting following error -
TypeError: mime.lookup is not a function
at ServerResponse.contentType (/E:/Saurabh/node_modules/express/lib/response.js:592:12)
at ServerResponse.send (/E:/Saurabh/node_modules/express/lib/response.js:145:14)
at app.get (/E:/Saurabh/apis/index.js:26:6)
at Layer.handle [as handle_request] (/E:/Saurabh/node_modules/express/lib/router/layer.js:95:5)
at next (/E:/Saurabh/node_modules/express/lib/router/route.js:137:13)
at Route.dispatch (/E:/Saurabh/node_modules/express/lib/router/route.js:112:3)
at Layer.handle [as handle_request] (/E:/Saurabh/node_modules/express/lib/router/layer.js:95:5)
at /E:/Saurabh/node_modules/express/lib/router/index.js:281:22
at Function.process_params (/E:/Saurabh/node_modules/express/lib/router/index.js:335:12)
at next (/E:/Saurabh/node_modules/express/lib/router/index.js:275:10)
If i run same code without docker, then it works as expected.
What can be the issue?
As for npm documentation on mime:
Version 2 is a breaking change from 1.x as the semver implies. Specifically:
lookup() renamed to getType()
If you prefer the legacy version of this module please npm install mime#^1.
Just change the way you call the method to mime.getType(path) or uninstall mime npm uninstall mime and reinstall it as npm install mime#^1.
Using MEAN.JS is possible to bring a Server Error changing this:
http://localhost:3000/#!/
To:
http://localhost:3000/%C0#!/
Error: Bad Request
at SendStream.error (/home/username/Project/node_modules/express/node_modules/send/index.js:239:16)
at SendStream.pipe (/home/username/Project/node_modules/express/node_modules/send/index.js:411:32)
at serveStatic (/home/username/Project/node_modules/express/node_modules/serve-static/index.js:110:12)
at Layer.handle [as handle_request] (/home/username/Project/node_modules/express/lib/router/layer.js:82:5)
at trim_prefix (/home/username/Project/node_modules/express/lib/router/index.js:269:13)
at /home/username/Project/node_modules/express/lib/router/index.js:236:9
at Function.proto.process_params (/home/username/Project/node_modules/express/lib/router/index.js:311:12)
at /home/username/Project/node_modules/express/lib/router/index.js:227:12
at Function.match_layer (/home/ec2-user/Project/node_modules/express/lib/router/index.js:294:3)
at next (/home/username/Project/node_modules/express/lib/router/index.js:188:10)
We can see the username of the server and this is dangerous...
Any suggestion to solve this bug?
The solution is easy. Only need to modify the message of file app/views/500.server.view.html