Time and time again, we get this error:
{ Error: Redis connection to some-redis-server:6379 failed - read ETIMEDOUT
at TCP.onStreamRead (internal/stream_base_commons.js:171:27) errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'read' }
{ AbortError: Redis connection lost and command aborted. It might have been processed.
at RedisClient.flush_and_error (/usr/src/app/node_modules/redis/index.js:362:23)
at RedisClient.connection_gone (/usr/src/app/node_modules/redis/index.js:664:14)
at Socket.<anonymous> (/usr/src/app/node_modules/redis/index.js:289:14)
at Object.onceWrapper (events.js:281:20)
at Socket.emit (events.js:193:13)
at TCP._handle.close (net.js:614:12)
code: 'UNCERTAIN_STATE',
command: 'GET'
args: [ 'some-key' ] }
We are using https://www.npmjs.com/package/redis to access redis.
Some details of the setup:
Currently 25GB of data stored in Redis
Writes = a rate of approximately 2MB/sec.
Max number of concurrent clients connected: 400
I don't know where to really start looking, so any help would be appreciated.
Related
Any help would be appreciated...
Not sure why I get this error every now and then. I've search off and on for a solution but keep coming up empty.
I run my Node App as a service on Centos7. It can run for a few days ... or even a month before getting this error.
I use a Pooled connection to Postgres and am inserting data into several different databases every few seconds.
events.js:377
throw er; // Unhandled 'error' event
^
error: terminating connection due to administrator command
at Parser.parseErrorMessage (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg-protocol/dist/parser.js:287:98)
at Parser.handlePacket (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg-protocol/dist/parser.js:126:29)
at Parser.parse (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg-protocol/dist/parser.js:39:38)
at Socket.<anonymous> (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg-protocol/dist/index.js:11:42)
at Socket.emit (events.js:400:28)
at addChunk (internal/streams/readable.js:290:12)
at readableAddChunk (internal/streams/readable.js:265:9)
at Socket.Readable.push (internal/streams/readable.js:204:10)
at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
Emitted 'error' event on Client instance at:
at Client._handleErrorEvent (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg/lib/client.js:319:10)
at Client._handleErrorMessage (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg/lib/client.js:330:12)
at Connection.emit (events.js:400:28)
at /home/beenth12/public_html/ws/wxbox/node/node_modules/pg/lib/connection.js:114:12
at Parser.parse (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg-protocol/dist/parser.js:40:17)
at Socket.<anonymous> (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg-protocol/dist/index.js:11:42)
[... lines matching original stack trace ...]
at Socket.Readable.push (internal/streams/readable.js:204:10) {
length: 116,
severity: 'FATAL',
code: '57P01',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'postgres.c',
line: '3193',
routine: 'ProcessInterrupts'
}
The documentation will inform you that SQLSTATE 57P01 is admin_shutdown. So somebody shutdown the database, and it is not surprising that your database connection got terminated.
I have my node app deployed inside Kubernetes cluster, when the app tries to connect to Redis outside from the Kubernetes, it throws following error:
2019-09-04T06:21:55.320Z ioredis:redis status[101.37.199.xx:6379]: connecting -> connect
2019-09-04T06:21:55.321Z ioredis:redis write command[101.37.199.xx:6379]: 32 -> auth([ 'xxxxxx' ])
2019-09-04T06:21:55.321Z ioredis:redis write command[101.37.199.xx:6379]: 32 -> select([ '32' ])
2019-09-04T06:21:55.321Z ioredis:connection error: Error: write EPIPE
at afterWriteDispatched (internal/stream_base_commons.js:146:25)
at writeGeneric (internal/stream_base_commons.js:137:3)
at Socket._writeGeneric (net.js:698:11)
at Socket._write (net.js:710:8)
at doWrite (_stream_writable.js:417:12)
at writeOrBuffer (_stream_writable.js:401:5)
at Socket.Writable.write (_stream_writable.js:301:11)
at Redis.sendCommand (/app/node_modules/ioredis/built/redis/index.js:601:33)
at Redis.select (/app/node_modules/ioredis/built/commander.js:124:21)
at Socket.<anonymous> (/app/node_modules/ioredis/built/redis/event_handler.js:32:18) {
errno: 'EPIPE',
code: 'EPIPE',
syscall: 'write'
}
[ioredis] Unhandled error event: Error: write EPIPE
at afterWriteDispatched (internal/stream_base_commons.js:146:25)
at writeGeneric (internal/stream_base_commons.js:137:3)
at Socket._writeGeneric (net.js:698:11)
at Socket._write (net.js:710:8)
at doWrite (_stream_writable.js:417:12)
at writeOrBuffer (_stream_writable.js:401:5)
at Socket.Writable.write (_stream_writable.js:301:11)
at Redis.sendCommand (/app/node_modules/ioredis/built/redis/index.js:601:33)
at Redis.select (/app/node_modules/ioredis/built/commander.js:124:21)
at Socket.<anonymous> enter code here(/app/node_modules/ioredis/built/redis/event_handler.js:32:18)
2019-09-04T06:21:55.324Z ioredis:redis queue command[101.37.199.xx:6379]: 32 -> info([])
2019-09-04T06:21:55.324Z ioredis:redis status[101.37.199.xx:6379]: connect -> close
I've tried to deploy redis-cli inside the kubernetes cluster, it can access the redis service.
you have to expose redis as a service in your cluster
I have a Lambda function that is set to timeout at 5 minutes, and occasionally when I invoke the function using the node SDK, I am getting a timeout error after 5 minutes.
However, when I have checked the CloudWatch logs after getting the client-side error, and am seeing the function successfully complete after ~3:30 minutes of execution time.
I have also noticed that the client side will log the timeout 2 minutes before the CloudWatch logs read that the function completed (ie, client side error timestamps at 4:08 and the CloudWatch successful end timestamps at 4:10), but I am assuming that is a lag between log writes, not run time. Thought I would mention it just in case that is something worth thinking more about.
From what I can tell, when the function runs under two minutes, it is consistently getting a positive response.
Code looks like this:
let aws_config = new AWS.Config({
region: 'us-east-1',
credentials: credentials,
httpOptions: {
proxy: process.env.HTTP_PROXY,
timeout: 300000
}
})
var lambda = new AWS.Lambda(aws_config);
lambda.invoke(params, (err, data: Result) => {
...
}
Error looks like this:
{ TimeoutError: Connection timed out after 300000ms
at ClientRequest.<anonymous> (.../node_modules/aws-sdk/lib/http/node.js:83:34)
at Object.onceWrapper (events.js:273:13)
at ClientRequest.emit (events.js:182:13)
at ClientRequest.EventEmitter.emit (domain.js:442:20)
at Socket.emitRequestTimeout (_http_client.js:661:40)
at Object.onceWrapper (events.js:273:13)
at Socket.emit (events.js:182:13)
at Socket.EventEmitter.emit (domain.js:442:20)
at Socket._onTimeout (net.js:449:8)
at ontimeout (timers.js:425:11)
message: 'Connection timed out after 300000ms',
code: 'TimeoutError',
time: 2018-10-05T20:08:35.719Z,
region: 'us-east-1',
hostname: 'lambda.us-east-1.amazonaws.com',
retryable: true } 'TimeoutError: Connection timed out after 300000ms\n at ClientRequest.<anonymous> (.../node_modules/aws-sdk/lib/http/node.js:83:34)\n at Object.onceWrapper (events.js:273:13)\n at ClientRequest.emit (events.js:182:13)\n at ClientRequest.EventEmitter.emit (domain.js:442:20)\n at Socket.emitRequestTimeout (_http_client.js:661:40)\n at Object.onceWrapper (events.js:273:13)\n at Socket.emit (events.js:182:13)\n at Socket.EventEmitter.emit (domain.js:442:20)\n at Socket._onTimeout (net.js:449:8)\n at ontimeout (timers.js:425:11)'
Have you looked into
context.callbackWaitsForEmptyEventLoop = false;
Here'e a related SO posting:
Why does AWS Lambda function always time out?
Please help solve the problem:
When I start the nats server listening 4222 port:
./gnatsd --addr localhost --port 4222, then
./ws-tcp-relay localhost:4222 -p 4223 and try to connect through websockets using websocket-nats like this (in service.js for example):
'use strict';
const nats = require('websocket-nats').connect('ws://localhost:4223');
I get this error:
node server.js
events.js:188
throw err;
^
Error: Unhandled "error" event. (Could not connect to server: Error: getaddrinfo ENOTFOUND localhost:4223 localhost:4223:4223)
at Client.emit (events.js:186:19)
at Socket.<anonymous> (/home/dev/projects/node-pub/node_modules/websocket-nats/lib/nats.js:404:14)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at emitErrorNT (internal/streams/destroy.js:64:8)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
But, if I try to connect to nats directly using tcp client port 4222, the connection establishes succesfull:
const nats = require('nats').connect('nats://localhost:4222');
I installed meanjs in my OpenSUSE distro, however when I run npm start, here's what I get:
+ Important warning: config.domain is empty. It should be set to the fully qualified domain of the app.
Unhandled rejection MongoError: not authorized on mean-dev to execute command { listIndexes: "sessions", cursor: {} }
at Function.MongoError.create (/home/tasher/Projects/meanjs_test/node_modules/mongodb-core/lib/error.js:31:11)
at queryCallback (/home/tasher/Projects/meanjs_test/node_modules/mongodb-core/lib/cursor.js:212:36)
at /home/tasher/Projects/meanjs_test/node_modules/mongodb-core/lib/connection/pool.js:469:18
at _combinedTickCallback (internal/process/next_tick.js:73:7)
at process._tickCallback (internal/process/next_tick.js:104:9)
From previous event:
at MongoStore.setAutoRemoveAsync (/home/tasher/Projects/meanjs_test/node_modules/connect-mongo/src/index.js:129:40)
at MongoStore.handleNewConnectionAsync (/home/tasher/Projects/meanjs_test/node_modules/connect-mongo/src/index.js:121:18)
at MongoStore (/home/tasher/Projects/meanjs_test/node_modules/connect-mongo/src/index.js:96:26)
at Object.module.exports.initSession (/home/tasher/Projects/meanjs_test/config/lib/express.js:120:12)
at Object.module.exports.init (/home/tasher/Projects/meanjs_test/config/lib/express.js:243:8)
at /home/tasher/Projects/meanjs_test/config/lib/app.js:25:23
at /home/tasher/Projects/meanjs_test/config/lib/mongoose.js:35:21
at process._tickCallback (internal/process/next_tick.js:109:7)
Unhandled rejection MongoError: not authorized on mean-dev to execute command { listIndexes: "sessions", cursor: {} }
at Function.MongoError.create (/home/tasher/Projects/meanjs_test/node_modules/mongodb-core/lib/error.js:31:11)
at queryCallback (/home/tasher/Projects/meanjs_test/node_modules/mongodb-core/lib/cursor.js:212:36)
at /home/tasher/Projects/meanjs_test/node_modules/mongodb-core/lib/connection/pool.js:469:18
at _combinedTickCallback (internal/process/next_tick.js:73:7)
at process._tickCallback (internal/process/next_tick.js:104:9)
From previous event:
at MongoStore.setAutoRemoveAsync (/home/tasher/Projects/meanjs_test/node_modules/connect-mongo/src/index.js:129:40)
at MongoStore.handleNewConnectionAsync (/home/tasher/Projects/meanjs_test/node_modules/connect-mongo/src/index.js:121:18)
at MongoStore (/home/tasher/Projects/meanjs_test/node_modules/connect-mongo/src/index.js:96:26)
at module.exports (/home/tasher/Projects/meanjs_test/config/lib/socket.io.js:73:20)
at Object.module.exports.configureSocketIO (/home/tasher/Projects/meanjs_test/config/lib/express.js:214:38)
at Object.module.exports.init (/home/tasher/Projects/meanjs_test/config/lib/express.js:258:14)
at /home/tasher/Projects/meanjs_test/config/lib/app.js:25:23
at /home/tasher/Projects/meanjs_test/config/lib/mongoose.js:35:21
at process._tickCallback (internal/process/next_tick.js:109:7)
error: uncaughtException: getaddrinfo ENOTFOUND linux-hcjz.suse date=Mon Aug 28 2017 22:46:50 GMT+0100 (WEST), pid=4497, uid=1000, gid=100, cwd=/home/tasher/Projects/meanjs_test, execPath=/usr/bin/node6, version=v6.11.1, argv=[/usr/bin/node6, /home/tasher/Projects/meanjs_test/server.js], rss=97284096, heapTotal=69029888, heapUsed=48863040, external=20182576, loadavg=[0.6513671875, 0.61669921875, 0.43505859375], uptime=3056
Error: getaddrinfo ENOTFOUND linux-hcjz.suse
at errnoException (dns.js:28:10)
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:76:26)
I have re-installed meanjs twice but in vain. Mongo is also running. By the way:
tasher#linux-hcjz:~/Projects/meanjs_test> node -v
v6.11.1
tasher#linux-hcjz:~/Projects/meanjs_test> npm6 -v
3.10.10
Any thoughts?
try adding execution permission to your mean-dev
chmod 755 path_for_package/mean-dev
replace path path_for_package with your path