At some random times, on our production server, we face such type of timeouts and the server crashes.
Error: ESOCKETTIMEDOUT
at ClientRequest.<anonymous> (/usr/src/app/node_modules/request/request.js:816:19)
at Object.onceWrapper (events.js:272:13)
at ClientRequest.emit (events.js:180:13)
at ClientRequest.emit (domain.js:422:20)
at TLSSocket.emitTimeout (_http_client.js:703:34)
at Object.onceWrapper (events.js:272:13)
at TLSSocket.emit (events.js:180:13)
at TLSSocket.emit (domain.js:422:20)
at TLSSocket.Socket._onTimeout (net.js:396:8)
at ontimeout (timers.js:466:11)
at tryOnTimeout (timers.js:304:5)
at Timer.listOnTimeout (timers.js:267:5)
I have gone though a lot of blogs and QAs. To summarize, it's read timeout when the caller doesn't get next bytes of response within some specific time limit. From the offical docs (request-npm), we can see -
Read timeout: Time to wait for a server to send response headers (and start the response body) before aborting the request.
Connection timeout: Sets the socket to timeout after timeout milliseconds of inactivity. Note that increasing the timeout beyond the OS-wide TCP connection timeout will not have any effect (the default in Linux can be anywhere from 20-120 seconds)
However I am trying to reproduce it in local linux machine with a HTTP request to my own Node JS server. I am able to get ETIMEDOUT which is connection timeout but I can't get ESOCKETTIMEDOUT error reproduced. In my destination server I had used this simple script with Express JS as an attempt to get this error.
app.use((req, res, next) => {
res.write('sample string');
// Do nothing, let the req wait
});
Therefore does anybody knows a simple approach to get this reproduced? Also I would like to know how to handle this error instead of having it as uncaught exception.
Thanks!
Related
I have an express server that accepts POST requests with binary bodies. I pipe the request into a temp file. The client (ffmpeg in this case) sends the POST with a binary body and immediately closes the connection without waiting for a response. Express is randomly throwing ECONNRESET/aborted errors (around 1 every 5 uploaded files). I presume it's only throwing when the TCP connection is closed before the stream has fully flushed. How can I tell express to not throw when the connection is closed earlier than it thinks it should be?
{
"code": "ECONNRESET",
"name": "Error",
"message": "aborted",
"stack": "Error: aborted\n at connResetException (node:internal/errors:691:14)\n at abortIncoming (node:_http_server:596:17)\n at socketOnClose (node:_http_server:590:3)\n at Socket.emit (node:events:402:35)\n at Socket.emit (node:domain:475:12)\n at TCP.<anonymous> (node:net:687:12)"
}
I am having a fetchError while trying to delete multiple files in a single bucket. My application requires finding older objects and then deleting any files a user has uploaded to the cloud storage.
So when I loop through files to delete, I seem to get a fetchError when there are too many files to delete.
At first it was working fine(testing phase) but now I notice that I get a fetchError when there is too many files to delete at the same time.
error:
{ FetchError: request to https://storage.googleapis.com/storage/v1/b/bucketName/o/filePath? failed, reason: socket hang up
at ClientRequest.<anonymous> (/srv/node_modules/node-fetch/lib/index.js:1453:11)
at emitOne (events.js:116:13)
at ClientRequest.emit (events.js:211:7)
at TLSSocket.socketErrorListener (_http_client.js:401:9)
at emitOne (events.js:116:13)
at TLSSocket.emit (events.js:211:7)
at emitErrorNT (internal/streams/destroy.js:66:8)
at _combinedTickCallback (internal/process/next_tick.js:139:11)
at process._tickDomainCallback (internal/process/next_tick.js:219:9)
message: 'request to https://storage.googleapis.com/storage/v1/b/bucketName/o/filePath? failed, reason: socket hang up',
type: 'system',
errno: 'ECONNRESET',
code: 'ECONNRESET' }
my deleteFunction:
I query a collection, then loop the querysnapshot to get the individual documentsnapshots where I get the image and video filePaths which I pass into deleteFile(myBucketName, mediaFilePath)
const gcs = new Storage();
async function deleteFile(bucketName, fileName) {
await gcs.bucket(bucketName).file(fileName).delete();
}
I am thinking, should I update the meta-data of the images to be deleted and listen in another function for onMetaDataChange for storage and then delete it? Would that be any better?
This doesn't seem to be an issue with Storage or Cloud Functions. Seems to be an issue with your implementation to handle multiple-requests as the error is related to the socket. (Here's is explained better). You can use batch requests or this answer could help you.
I am reading a csv file and I am extracting data and making a post request every 30 seconds(setTimeout help) with data extracted with request module, after making 3 requests it shows this error
Error: socket hang up
at createHangUpError (_http_client.js:200:15)
at Socket.socketOnEnd (_http_client.js:292:23)
at emitNone (events.js:72:20)
at Socket.emit (events.js:166:7)
at endReadableNT (_stream_readable.js:913:12)
at nextTickCallbackWith2Args (node.js:442:9)
at process._tickCallback (node.js:356:17)
Depending on your implementation there could be multiple things that could cause this.
I would use something like Async Parallel for these kind of tasks.
Give it a try.
I've read all the related questions to this one on here, I've searched through Github issues and I've read as much as possible about Express to try and figure this one out. As far is I've read most people say it is something to do with the headers being set after they've been sent usually with res.set() or something similar but I'm not doing that anywhere in my code.
This error only occurs shortly after I've deployed my SailsJS app to Modulus:
Error occurred in session middleware :: 'Error: Error getting collection: sessions\n at /mnt/data/2/node_modules/connect-mongo/lib/connect-mongo.js:160:19\n at Db.collection (/mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/db.js:524:27)\n at MongoStore._get_collection (/mnt/data/2/node_modules/connect-mongo/lib/connect-mongo.js:158:17)\n at MongoStore.set (/mnt/data/2/node_modules/connect-mongo/lib/connect-mongo.js:274:12)\n at Session.save (/mnt/data/2/node_modules/sails/node_modules/express/node_modules/connect/node_modules/express-session/session/session.js:63:25)\n at ServerResponse.end (/mnt/data/2/node_modules/sails/node_modules/express/node_modules/connect/node_modules/express-session/index.js:263:21)\n at ServerResponse.res.end (/mnt/data/2/node_modules/sails/node_modules/express/node_modules/connect/node_modules/compression/index.js:116:15)\n at ServerResponse.res.send (/mnt/data/2/node_modules/sails/node_modules/express/lib/response.js:175:8)\n at ServerResponse.res.json (/mnt/data/2/node_modules/sails/node_modules/express/lib/response.js:223:15)\n at storedUrlCheckServiceDone (/mnt/data/2/api/controllers/ProcessController.js:44:18)\n at tryCatcher (/mnt/data/2/node_modules/bluebird/js/main/util.js:24:31)\n at Promise._settlePromiseFromHandler (/mnt/data/2/node_modules/bluebird/js/main/promise.js:454:31)\n at Promise._settlePromiseAt (/mnt/data/2/node_modules/bluebird/js/main/promise.js:530:18)\n at Promise._settlePromises (/mnt/data/2/node_modules/bluebird/js/main/promise.js:646:14)\n at Async._drainQueue (/mnt/data/2/node_modules/bluebird/js/main/async.js:175:16)\n at Async._drainQueues (/mnt/data/2/node_modules/bluebird/js/main/async.js:185:10)'
http.js:690
throw new Error('Can\'t set headers after they are sent.');
^
Error: Can't set headers after they are sent.
at ServerResponse.OutgoingMessage.setHeader (http.js:690:11)
at ServerResponse.res.setHeader (/mnt/data/2/node_modules/sails/node_modules/express/node_modules/connect/lib/patch.js:133:22)
at ServerResponse.res.set.res.header (/mnt/data/2/node_modules/sails/node_modules/express/lib/response.js:577:10)
at ServerResponse.res.send (/mnt/data/2/node_modules/sails/node_modules/express/lib/response.js:142:12)
at Object.<anonymous> (/mnt/data/2/node_modules/sails/lib/hooks/http/middleware/defaults.js:56:24)
at Object.immediate._onImmediate (timers.js:372:16)
at processImmediate [as _immediateCallback] (timers.js:354:15)
[2015-04-13T03:23:03.264Z] Application CRASH detected. Exit code 8.
It's something to do with the sessions and connect-mongo but I have no idea how to figure it out. Usually it happens once or twice straight after deployment but sometimes it'll happen up to 5 or 6 times in a row straight after deployment as well.
I've also seen this error occasionally as well around the same time so it might be related:
/mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/connection/base.js:246
throw message;
^
Error: Error setting TTL index on collection : sessions
at /mnt/data/2/node_modules/connect-mongo/lib/connect-mongo.js:169:23
at /mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/db.js:1497:46
at /mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/db.js:1630:20
at /mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/cursor.js:170:22
at /mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/cursor.js:714:39
at Cursor.close (/mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/cursor.js:1009:5)
at commandHandler (/mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/cursor.js:714:21)
at /mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/db.js:1903:9
at Server.Base._callHandler (/mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/connection/base.js:453:41)
at /mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/connection/server.js:488:18
at MongoReply.parseBody (/mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js:68:5)
at null.<anonymous> (/mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/connection/server.js:446:20)
at emit (events.js:95:17)
at null.<anonymous> (/mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:207:13)
at emit (events.js:98:17)
at Socket.<anonymous> (/mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/connection/connection.js:440:22)
[2015-04-13T03:24:15.011Z] Application CRASH detected. Exit code 8.
[2015-04-13T03:24:22.048Z] Application restarted with pid 14613
I'm using Node v0.10.37, Sails v0.10.5 and Passport for authentication and just the mongo adapter for session storage.
I'd really appreciate any help or insights you can give as I've spent nearly a week trying to figure this out before posting here and even had Newrelic's support team looking at and they couldn't figure it out either.
Thanks
Update
I'm not seeing this error locally, only on Modulus.
This is a shortened down version of my controller.
I basically have a Auth Service that checks if the users is authenticated, if they are I pass the results off to another promise which checks the db for some of their data, if it find a record then it returns it if not then it calls a 3rd party API and does some processing, once thats done it returns and creates a new record, then returns the results back to the browser. If they aren't authenticated then I send back an error.
There are 4 calls to different Sails services in total in the controller, which all return a promise.
text: function (req, res) {
// Check that the host is authenticated
AuthService.hostname(req).done( authServiceDone, authServiceFail );
function authServiceDone (site) {
// Check the DB for the origin
return urlService.check(req, site.owner).done(UrlCheckServiceDone, UrlCheckServiceFail );
}
function authServiceFail (error) {
return res.json({
error : error
});
}
}
I have my MongoDB running on a Cloud VM , and NodeJS Server running on another Azure Cloud VM. I use Mongoose in the Application for MongoDB Connect. Keep getting this timeout issue every day? Any suggestions
events.js:72
throw er; // Unhandled 'error' event
^
Error: failed to connect to [XXXXX.cloudapp.net:27017]
at null.<anonymous> (/app/satwebnodev2/satwebnodev2/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/server.js:546:74)
at emit (events.js:106:17)
at null.<anonymous> (/app/satwebnodev2/satwebnodev2/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:150:15)
at emit (events.js:98:17)
at Socket.<anonymous> (/app/satwebnodev2/satwebnodev2/node_modules/mongoose/node_modules/mongodb/lib/mongodb/connection/connection.js:533:10)
at Socket.emit (events.js:95:17)
at net.js:440:14
at process._tickCallback (node.js:419:13)
Refer this link :- http://www.slideshare.net/VictoriaMalaya1/mongo-db-in-windows-azure
MongoDB in Windows Azure:-
Drawbacks and workarounds : connection timeout for .Net driver MongoDB Client Uses .Net driver socket timeout exception Windows Azure terminates an inactive connection after roughly 4 minutes of idling MongoDB suggests: Set the max idle connection time on the driver: MongoDefaults.MaxConnectionIdleTime = TimeSpan.FromMinutes(1); My experience: Add following options to connection string: maxIdleTimeMS=60000;connectTimeoutMS=60000;socketTimeoutMS=60000 Sample of connection string: connectionString="mongodb://localhost/mydb ?maxIdleTimeMS=60000;connectTimeoutMS=60000;socketTimeoutMS=60000"