What causes this Node/Postgres FATAL Error - node.js

Any help would be appreciated...
Not sure why I get this error every now and then. I've search off and on for a solution but keep coming up empty.
I run my Node App as a service on Centos7. It can run for a few days ... or even a month before getting this error.
I use a Pooled connection to Postgres and am inserting data into several different databases every few seconds.
events.js:377
throw er; // Unhandled 'error' event
^
error: terminating connection due to administrator command
at Parser.parseErrorMessage (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg-protocol/dist/parser.js:287:98)
at Parser.handlePacket (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg-protocol/dist/parser.js:126:29)
at Parser.parse (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg-protocol/dist/parser.js:39:38)
at Socket.<anonymous> (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg-protocol/dist/index.js:11:42)
at Socket.emit (events.js:400:28)
at addChunk (internal/streams/readable.js:290:12)
at readableAddChunk (internal/streams/readable.js:265:9)
at Socket.Readable.push (internal/streams/readable.js:204:10)
at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
Emitted 'error' event on Client instance at:
at Client._handleErrorEvent (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg/lib/client.js:319:10)
at Client._handleErrorMessage (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg/lib/client.js:330:12)
at Connection.emit (events.js:400:28)
at /home/beenth12/public_html/ws/wxbox/node/node_modules/pg/lib/connection.js:114:12
at Parser.parse (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg-protocol/dist/parser.js:40:17)
at Socket.<anonymous> (/home/beenth12/public_html/ws/wxbox/node/node_modules/pg-protocol/dist/index.js:11:42)
[... lines matching original stack trace ...]
at Socket.Readable.push (internal/streams/readable.js:204:10) {
length: 116,
severity: 'FATAL',
code: '57P01',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'postgres.c',
line: '3193',
routine: 'ProcessInterrupts'
}

The documentation will inform you that SQLSTATE 57P01 is admin_shutdown. So somebody shutdown the database, and it is not surprising that your database connection got terminated.

Related

How do I globally handle all type of Exception in NESTJS

error: Error: Duplicate entry 'Story of Lawyer' for key 'book.book-title-unique'
at Packet.asError (E:\Work\nest_best_practice\node_modules\mysql2\lib\packets\packet.js:728:17)
at Query.execute (E:\Work\nest_best_practice\node_modules\mysql2\lib\commands\command.js:29:26)
at PoolConnection.handlePacket (E:\Work\nest_best_practice\node_modules\mysql2\lib\connection.js:456:32)
at PacketParser.onPacket (E:\Work\nest_best_practice\node_modules\mysql2\lib\connection.js:85:12)
at PacketParser.executeStart (E:\Work\nest_best_practice\node_modules\mysql2\lib\packet_parser.js:75:16)
at Socket. (E:\Work\nest_best_practice\node_modules\mysql2\lib\connection.js:92:25)
at Socket.emit (node:events:527:28)
at addChunk (node:internal/streams/readable:315:12)
at readableAddChunk (node:internal/streams/readable:289:9)
at Socket.Readable.push (node:internal/streams/readable:228:10) {
code: 'ER_DUP_ENTRY',
errno: 1062,
sqlState: '23000',
sqlMessage: "Duplicate entry 'Story of Lawyer' for key 'book.book-title-unique'",
sql: "INSERT INTO book(id, title, desc, createdAt, updatedAt, image, pdf) VALUES (DEFAULT, 'Story of Lawyer', 'No Description', DEFAULT, DEFAULT, 'software-developer-copy-6d00.jpg', 'NEST CLI-3339.pdf')"
}
Have you tried using an exception filter
https://docs.nestjs.com/exception-filters

Mongodb 5 node driver detailed schema validation error

Using mongodb 5 improved json schema validation errors (https://www.mongodb.com/developer/article/mongodb-5-0-schema-validation/), we should be able to have a detailed validation error when e.g inserting incorrect data in a document.
However, it does not seem to be working using latest mongodb node driver 4.2.
Here's the error I have :
MongoServerError: BSON field 'insert.documents.0' is the wrong type 'array', expected type 'object'
at MessageStream.messageHandler (/workspace/node_modules/mongodb/src/cmap/connection.ts:741:20)
at MessageStream.emit (events.js:400:28)
at MessageStream.emit (domain.js:470:12)
at processIncomingData (/workspace/node_modules/mongodb/src/cmap/message_stream.ts:167:12)
at MessageStream._write (/workspace/node_modules/mongodb/src/cmap/message_stream.ts:64:5)
at writeOrBuffer (internal/streams/writable.js:358:12)
at MessageStream.Writable.write (internal/streams/writable.js:303:10)
at Socket.ondata (internal/streams/readable.js:726:22)
at Socket.emit (events.js:400:28)
at Socket.emit (domain.js:470:12) {
ok: 0,
code: 14,
codeName: 'TypeMismatch'
Is there a way to have the detailed error message ?

mysql2 connection.connect() hangs in typescript jest test on macOS

I am trying to create a mysql2 connection to a local mysql-Database in a jest-Test on macOS using typescript. And the call to connect() hangs.
Environment:
MacOS: High Sierra (10.13.6)
MySQL: 8.0.18
Typescript: 3.7.2
mysql2: 2.0.1
nodejs: v10.16.3
I am using the promise-Wrapper. The code looks like this:
import { createConnection, ConnectionOptions, Connection } from 'mysql2/promise'
...
async openConnection(options: ConnectionOptions): Promise<Connection> {
console.log('About to create mysql connection using options ', options);
const connection = await createConnection(options);
console.log('Connection created => about to call connect()');
await connection.connect();
console.log('Connection created => connect() returned'); // this line is never reached!
....
return connection;
}
I already added the debug-Option and this is the output:
Dieters-MBP-6:db-typescript rehdie$ node_modules/.bin/jest src/tests/mysql/mysql-connection.spec.ts --testNamePattern="Should open connection"
console.log src/lib/mysql/driver/mysql-driver.ts:21
About to create mysql connection using options { user: 'mysql-test',
password: '1q2w3e4r',
host: 'localhost',
port: 3306,
database: 'mysql_test_db',
charset: 'latin1_swedish_ci',
connectTimeout: 5000,
debug: true }
console.log node_modules/mysql2/lib/connection.js:425
Add command: ClientHandshake
console.log node_modules/mysql2/lib/connection.js:381
raw: 0a382e302e31380018000000277919626469296e00ffffff0200ffc715000000000000000000006d6335316a3c3d454220053a0063616368696e675f736861325f70617373776f726400
console.error console.js:274
Trace:
at Connection.handlePacket (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:387:17)
at PacketParser.Connection.packetParser.p [as onPacket] (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:70:12)
at PacketParser.executeStart (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/packet_parser.js:75:16)
at Socket.Connection.stream.on.data (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:77:25)
at Socket.emit (events.js:198:13)
at addChunk (_stream_readable.js:288:12)
at readableAddChunk (_stream_readable.js:269:11)
at Socket.Readable.push (_stream_readable.js:224:10)
at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
console.log node_modules/mysql2/lib/connection.js:395
0 undefined ==> ClientHandshake#unknown name(0,,78)
console.log node_modules/mysql2/lib/commands/client_handshake.js:96
Server hello packet: capability flags:3355443199=(long password, found rows, long flag, connect with db, no schema, compress, odbc, local files, ignore space, protocol 41, interactive, ssl, ignore sigpipe, transactions, reserved, secure connection, multi statements, multi results, ps multi results, plugin auth, connect attrs, plugin auth lenenc client data, can handle expired passwords, session track, deprecate eof, ssl verify server cert, remember options)
console.log node_modules/mysql2/lib/commands/client_handshake.js:41
Sending handshake packet: flags:11203535=(long password, found rows, long flag, connect with db, odbc, local files, ignore space, protocol 41, ignore sigpipe, transactions, reserved, secure connection, multi results, plugin auth, plugin auth lenenc client data, session track)
console.log node_modules/mysql2/lib/connection.js:253
0 24 <== ClientHandshake#unknown name(1,,104)
console.log node_modules/mysql2/lib/connection.js:257
0 24 <== 64000001cff3aa00000000000800000000000000000000000000000000000000000000006d7973716c2d746573740014a13dd11599725225bcae39464cc3700a5fa085fc6d7973716c5f746573745f6462006d7973716c5f6e61746976655f70617373776f726400
console.log node_modules/mysql2/lib/connection.js:381
raw: fe63616368696e675f736861325f70617373776f726400277919626469296e6d6335316a3c3d454220053a00
console.error console.js:274
Trace:
at Connection.handlePacket (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:387:17)
at PacketParser.Connection.packetParser.p [as onPacket] (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:70:12)
at PacketParser.executeStart (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/packet_parser.js:75:16)
at Socket.Connection.stream.on.data (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:77:25)
at Socket.emit (events.js:198:13)
at addChunk (_stream_readable.js:288:12)
at readableAddChunk (_stream_readable.js:269:11)
at Socket.Readable.push (_stream_readable.js:224:10)
at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
console.log node_modules/mysql2/lib/connection.js:395
0 24 ==> ClientHandshake#unknown name(2,,48)
console.log node_modules/mysql2/lib/connection.js:253
0 24 <== ClientHandshake#unknown name(3,,36)
console.log node_modules/mysql2/lib/connection.js:257
0 24 <== 20000003374930d53dd94bc2dbe6b54e4b83cd628658a1984656eb6ee4f31d7df596ce36
console.log node_modules/mysql2/lib/connection.js:381
raw: 0103
console.error console.js:274
Trace:
at Connection.handlePacket (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:387:17)
at PacketParser.Connection.packetParser.p [as onPacket] (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:70:12)
at PacketParser.executeStart (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/packet_parser.js:75:16)
at Socket.Connection.stream.on.data (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:77:25)
at Socket.emit (events.js:198:13)
at addChunk (_stream_readable.js:288:12)
at readableAddChunk (_stream_readable.js:269:11)
at Socket.Readable.push (_stream_readable.js:224:10)
at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
console.log node_modules/mysql2/lib/connection.js:395
0 24 ==> ClientHandshake#unknown name(4,,6)
console.log node_modules/mysql2/lib/connection.js:381
raw: 000000024000000010010e0d6d7973716c5f746573745f6462
console.error console.js:274
Trace:
at Connection.handlePacket (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:387:17)
at PacketParser.Connection.packetParser.p [as onPacket] (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:70:12)
at PacketParser.executeStart (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/packet_parser.js:75:16)
at Socket.Connection.stream.on.data (/Users/rehdie/development/projects/db-typescript/node_modules/mysql2/lib/connection.js:77:25)
at Socket.emit (events.js:198:13)
at addChunk (_stream_readable.js:288:12)
at readableAddChunk (_stream_readable.js:269:11)
at Socket.Readable.push (_stream_readable.js:224:10)
at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
console.log node_modules/mysql2/lib/connection.js:395
0 24 ==> ClientHandshake#unknown name(5,maybeOK,29)
console.log src/lib/mysql/driver/mysql-driver.ts:24
Connection created => about to call connect()
RUNS src/tests/mysql/mysql-connection.spec.ts
When I perform a show process list in mysql, I see a new process with Command 'Sleep' (state is empty and info is null).
Any ideas?
You need to ensure that node is using the system time and not fake timers. Add the below line immediately after your imports in the test file
jest.useRealTimers();
Or better still you can set timers to real in your jest config
"timers":"real"

Intermittent redis connection error: 'UNCERTAIN STATE'

Time and time again, we get this error:
{ Error: Redis connection to some-redis-server:6379 failed - read ETIMEDOUT
at TCP.onStreamRead (internal/stream_base_commons.js:171:27) errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'read' }
{ AbortError: Redis connection lost and command aborted. It might have been processed.
at RedisClient.flush_and_error (/usr/src/app/node_modules/redis/index.js:362:23)
at RedisClient.connection_gone (/usr/src/app/node_modules/redis/index.js:664:14)
at Socket.<anonymous> (/usr/src/app/node_modules/redis/index.js:289:14)
at Object.onceWrapper (events.js:281:20)
at Socket.emit (events.js:193:13)
at TCP._handle.close (net.js:614:12)
code: 'UNCERTAIN_STATE',
command: 'GET'
args: [ 'some-key' ] }
We are using https://www.npmjs.com/package/redis to access redis.
Some details of the setup:
Currently 25GB of data stored in Redis
Writes = a rate of approximately 2MB/sec.
Max number of concurrent clients connected: 400
I don't know where to really start looking, so any help would be appreciated.

lambda.invoke call through Node AWS SDK is timing out at 5 minutes while Lambda is completing under 4 minutes

I have a Lambda function that is set to timeout at 5 minutes, and occasionally when I invoke the function using the node SDK, I am getting a timeout error after 5 minutes.
However, when I have checked the CloudWatch logs after getting the client-side error, and am seeing the function successfully complete after ~3:30 minutes of execution time.
I have also noticed that the client side will log the timeout 2 minutes before the CloudWatch logs read that the function completed (ie, client side error timestamps at 4:08 and the CloudWatch successful end timestamps at 4:10), but I am assuming that is a lag between log writes, not run time. Thought I would mention it just in case that is something worth thinking more about.
From what I can tell, when the function runs under two minutes, it is consistently getting a positive response.
Code looks like this:
let aws_config = new AWS.Config({
region: 'us-east-1',
credentials: credentials,
httpOptions: {
proxy: process.env.HTTP_PROXY,
timeout: 300000
}
})
var lambda = new AWS.Lambda(aws_config);
lambda.invoke(params, (err, data: Result) => {
...
}
Error looks like this:
{ TimeoutError: Connection timed out after 300000ms
at ClientRequest.<anonymous> (.../node_modules/aws-sdk/lib/http/node.js:83:34)
at Object.onceWrapper (events.js:273:13)
at ClientRequest.emit (events.js:182:13)
at ClientRequest.EventEmitter.emit (domain.js:442:20)
at Socket.emitRequestTimeout (_http_client.js:661:40)
at Object.onceWrapper (events.js:273:13)
at Socket.emit (events.js:182:13)
at Socket.EventEmitter.emit (domain.js:442:20)
at Socket._onTimeout (net.js:449:8)
at ontimeout (timers.js:425:11)
message: 'Connection timed out after 300000ms',
code: 'TimeoutError',
time: 2018-10-05T20:08:35.719Z,
region: 'us-east-1',
hostname: 'lambda.us-east-1.amazonaws.com',
retryable: true } 'TimeoutError: Connection timed out after 300000ms\n at ClientRequest.<anonymous> (.../node_modules/aws-sdk/lib/http/node.js:83:34)\n at Object.onceWrapper (events.js:273:13)\n at ClientRequest.emit (events.js:182:13)\n at ClientRequest.EventEmitter.emit (domain.js:442:20)\n at Socket.emitRequestTimeout (_http_client.js:661:40)\n at Object.onceWrapper (events.js:273:13)\n at Socket.emit (events.js:182:13)\n at Socket.EventEmitter.emit (domain.js:442:20)\n at Socket._onTimeout (net.js:449:8)\n at ontimeout (timers.js:425:11)'
Have you looked into
context.callbackWaitsForEmptyEventLoop = false;
Here'e a related SO posting:
Why does AWS Lambda function always time out?

Resources