I have serverless API which saves and fetch data in Redis mostly it runs fine but randomly throws 502 error (socket hang up) which I can't even handle inside the lambda.
"errorType": "Error",
"errorMessage": "socket hang up",
"code": "ECONNRESET",
"stack": [
"Error: socket hang up",
" at createHangUpError (_http_client.js:323:15)",
" at TLSSocket.socketOnEnd (_http_client.js:426:23)",
" at TLSSocket.emit (events.js:203:15)",
" at endReadableNT (_stream_readable.js:1143:12)",
" at process._tickCallback (internal/process/next_tick.js:63:19)"
]
}```
I am using:
Nodejs 12.x
Redis: 2.8.0
AWS Lambda functions have a timeout of 3-900 seconds, with 3 seconds being the default. If your Lambda does not have the timeout configured, and it takes more than 3 seconds to execute, you will get these errors.
https://docs.aws.amazon.com/lambda/latest/dg/resource-model.html
Related
today I'm able to put a record in Kinesis, but this operation takes 30 ms (which is too much for my use case), and I would like to not wait for the kinesis response (= fire-and-forget)
Today I use the PutRecordCommand solution:
const putRecordCommand = new PutRecordCommand({
Data: Buffer.from(JSON.stringify(report)),
PartitionKey: requestId,
StreamName: STREAM_NAME,
});
await kinesisClient.send(putRecordCommand);
If I remove the await key word, and the lambda ends before the kinesis client retrieves a response, I get the following error:
{
"errorType": "Runtime.UnhandledPromiseRejection",
"errorMessage": "Error [ERR_HTTP2_STREAM_CANCEL]: The pending stream has been canceled (caused by: Client network socket disconnected before secure TLS connection was established)",
"reason": {
"errorType": "Error",
"errorMessage": "The pending stream has been canceled (caused by: Client network socket disconnected before secure TLS connection was established)",
"code": "ERR_HTTP2_STREAM_CANCEL",
"cause": {
"errorType": "Error",
"errorMessage": "Client network socket disconnected before secure TLS connection was established",
"code": "ECONNRESET",
"host": "kinesis.eu-west-1.amazonaws.com",
"port": "443",
"stack": [
"Error: Client network socket disconnected before secure TLS connection was established",
" at connResetException (node:internal/errors:692:14)",
" at TLSSocket.onConnectEnd (node:_tls_wrap:1587:19)",
" at TLSSocket.emit (node:events:539:35)",
" at endReadableNT (node:internal/streams/readable:1345:12)",
" at processTicksAndRejections (node:internal/process/task_queues:83:21)"
]
},
"$metadata": {
"attempts": 1,
"totalRetryDelay": 0
},
"stack": [
"Error [ERR_HTTP2_STREAM_CANCEL]: The pending stream has been canceled (caused by: Client network socket disconnected before secure TLS connection was established)",
" at new NodeError (node:internal/errors:372:5)",
" at closeSession (node:internal/http2/core:1136:20)",
" at ClientHttp2Session.destroy (node:internal/http2/core:1530:5)",
" at TLSSocket.socketOnError (node:internal/http2/core:2981:13)",
" at TLSSocket.emit (node:events:527:28)",
" at emitErrorNT (node:internal/streams/destroy:157:8)",
" at emitErrorCloseNT (node:internal/streams/destroy:122:3)",
" at processTicksAndRejections (node:internal/process/task_queues:83:21)"
]
},
"promise": {},
"stack": [
"Runtime.UnhandledPromiseRejection: Error [ERR_HTTP2_STREAM_CANCEL]: The pending stream has been canceled (caused by: Client network socket disconnected before secure TLS connection was established)",
" at process.<anonymous> (file:///var/runtime/index.mjs:775:15)",
" at process.emit (node:events:527:28)",
" at emit (node:internal/process/promises:140:20)",
" at processPromiseRejections (node:internal/process/promises:274:27)",
" at processTicksAndRejections (node:internal/process/task_queues:97:32)"
]
}
Questions:
Is there another Kinesis "command" from AWS SDK v3 that allows me to fire-and-forget?
Or an option to use on the kinesis client in order to make it work without await?
Notes:
Env: AWS Lambda with NodeJS v16
Dependency: AWS SDK v3 v3.154
async/await programming style
We are using AWS Lambda to call our API's. We recently upgraded our Lambda Node version from 8.10 to Node 10.x version. We have added process event for unhandled rejection and also written code in try-catch block but none of the code resolves our issue. Please find the error block that we are getting when we hit the API through Lambda.
Please note that we are using promise-request NPM package for request.
Tried code with Try catch and handling unhandled event using Process.event
{
"errorType": "Runtime.UnhandledPromiseRejection",
"errorMessage": "RequestError: Error: Client network socket disconnected before secure TLS connection was established",
"stack": [
"Runtime.UnhandledPromiseRejection: RequestError: Error: Client network socket disconnected before secure TLS connection was established",
" at process.on (/var/runtime/index.js:37:15)",
" at process.emit (events.js:203:15)",
" at process.EventEmitter.emit (domain.js:448:20)",
" at /var/task/node_modules/bluebird/js/release/debuggability.js:199:33",
" at activeFireEvent (/var/task/node_modules/bluebird/js/release/debuggability.js:242:44)",
" at fireRejectionEvent (/var/task/node_modules/bluebird/js/release/debuggability.js:632:14)",
" at Promise._notifyUnhandledRejection (/var/task/node_modules/bluebird/js/release/debuggability.js:65:9)",
" at Timeout._onTimeout (/var/task/node_modules/bluebird/js/release/debuggability.js:44:14)",
" at ontimeout (timers.js:436:11)",
" at tryOnTimeout (timers.js:300:5)",
" at listOnTimeout (timers.js:263:5)",
" at Timer.processTimers (timers.js:223:10)"
],
"reason": {
"errorType": "RequestError",
"errorMessage": "Error: Client network socket disconnected before secure TLS connection was established",
"stack": [
"RequestError: Error: Client network socket disconnected before secure TLS connection was established",
" at new RequestError (/var/task/node_modules/request-promise-core/lib/errors.js:14:15)",
" at Request.plumbing.callback (/var/task/node_modules/request-promise-core/lib/plumbing.js:87:29)",
" at Request.RP$callback [as _callback] (/var/task/node_modules/request-promise-core/lib/plumbing.js:46:31)",
" at self.callback (/var/task/node_modules/request/request.js:185:22)",
" at Request.emit (events.js:198:13)",
" at Request.EventEmitter.emit (domain.js:448:20)",
" at Request.onRequestError (/var/task/node_modules/request/request.js:881:8)",
" at ClientRequest.emit (events.js:198:13)",
" at ClientRequest.EventEmitter.emit (domain.js:448:20)",
" at TLSSocket.socketErrorListener (_http_client.js:392:9)",
" at TLSSocket.emit (events.js:198:13)",
" at TLSSocket.EventEmitter.emit (domain.js:448:20)",
" at emitErrorNT (internal/streams/destroy.js:91:8)",
" at emitErrorAndCloseNT (internal/streams/destroy.js:59:3)",
" at process._tickCallback (internal/process/next_tick.js:63:19)"
],
}
}
Upgrading to Node.js 12.x, which is possible now, avoids the issue.
Time and time again, we get this error:
{ Error: Redis connection to some-redis-server:6379 failed - read ETIMEDOUT
at TCP.onStreamRead (internal/stream_base_commons.js:171:27) errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'read' }
{ AbortError: Redis connection lost and command aborted. It might have been processed.
at RedisClient.flush_and_error (/usr/src/app/node_modules/redis/index.js:362:23)
at RedisClient.connection_gone (/usr/src/app/node_modules/redis/index.js:664:14)
at Socket.<anonymous> (/usr/src/app/node_modules/redis/index.js:289:14)
at Object.onceWrapper (events.js:281:20)
at Socket.emit (events.js:193:13)
at TCP._handle.close (net.js:614:12)
code: 'UNCERTAIN_STATE',
command: 'GET'
args: [ 'some-key' ] }
We are using https://www.npmjs.com/package/redis to access redis.
Some details of the setup:
Currently 25GB of data stored in Redis
Writes = a rate of approximately 2MB/sec.
Max number of concurrent clients connected: 400
I don't know where to really start looking, so any help would be appreciated.
Consumer stops fetching after getting following error
{ Error: Local: Broker transport failure at Error (native) origin: 'local', message: 'broker transport failure', code: -1, errno: -1, stack: 'Error: Local: Broker transport failure\n at Error (native)' }
All kafka servers are healthy even then I am getting the transport failure .
After this error consumers stops fetching further messages
I'm doing http request using request.js, and I have run my program for a very long time this is the first time got this kind of error, it seems related to openssl and TLS issue. Does anybody have idea?
2017-02-20T14:27:34.330Z - error: Error: write EPROTO 140125135710016:error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number:../deps/openssl/openssl/ssl/s3_pkt.c:365:
at exports._errnoException (util.js:1022:11)
at WriteWrap.afterWrite (net.js:801:14) 2017-02-20T14:27:34.332Z - warn: worker warning, msg={"code":"EPROTO","errno":"EPROTO","syscall":"write"}
2017-02-20T14:27:34.348Z - error: uncaughtException: Cannot read property 'getNegotiatedProtocol' of null ,
version=v6.9.5,
argv=[/usr/bin/nodejs, /home/bda/tm/worker.js],rss=234672128, heapTotal=135876608, heapUsed=107730496, loadavg=[0.43798828125, 0.5263671875, 0.53857421875], uptime=906959,
trace=[column=36, file=_tls_wrap.js, function=TLSSocket._finishInit, line=588, method=_finishInit, native=false, column=38, file=_tls_wrap.js, function=TLSWrap.ssl.onhandshakedone, line=433, method=ssl.onhandshakedone, native=false], stack=[TypeError: Cannot read property 'getNegotiatedProtocol' of null, at TLSSocket._finishInit (_tls_wrap.js:588:36), at TLSWrap.ssl.onhandshakedone (_tls_wrap.js:433:38)]