"connection terminated unexpectedly" error with Node, Postgres on AWS Lambda - node.js

Question

I have a number of Node functions running on AWS Lambda. These functions have been using the Node 8 runtime but AWS sent out an end-of-life notice saying that functions should be upgraded to the latest LTS. With that, I upgraded one on my functions to use Node 12. After being in production for a bit, I'm starting to see a ton of connection terminated unexpectedly errors when querying the database.
Here are the errors that I'm seeing:
The connection terminated unexpectedly error
And Error [ERR_STREAM_DESTROYED]: Cannot call write after a stream was destroyed - this seems to happen on the 1st or second invocation after seeing the connection terminated unexpectedly error.
I'm using Knex.js for querying the database. I was running older version of knex and node-postgres and recently upgraded to see if it would resolve the issue, but no luck. Here are the versions of knex and node-postgres that I'm currently running:
"knex": "^0.20.8"
"pg": "^7.17.1"
The only change I've made to this particular function is the upgrade to Node 12. I've also tried Node 10, but the same issue persists. Unfortunately, AWS won't let me downgrade to Node 8 to verify that it is indeed an issue. None of my other functions running on Node 8 are experiencing this issue.
I've researched knex, node-postgres and tarn.js (the Knex connection pooling library) to see if any related issues or solutions popped up, but so far, I haven't had any luck.
UPDATE:
Example of a handler. Note that this is happening on many different Lambdas, all running Node 12.
require('../../helpers/knex')
const { Rollbar } = require('#scoutforpets/utils')
const { Email } = require('#scoutforpets/notifications')
const { transaction: tx } = require('objection')
const Invoice = require('../../models/invoice')
// configure rollbar for error logging
const rollbar = Rollbar.configureRollbar(process.env.ROLLBAR_TOKEN)
/**
*
* #param {*} event
*/
async function handler (event) {
const { invoice } = event
const { id: invoiceId } = invoice
try {
return tx(Invoice, async Invoice => {
// send the receipt
await Email.Customer.paymentReceipt(invoiceId, true)
// convert JSON to model
const i = Invoice.fromJson(invoice)
// mark the invoice as having been sent
await i.markAsSent()
})
} catch (err) {
return err
}
}
module.exports.handler = rollbar.lambdaHandler(handler)

Answer 1

Starting with node.js 10 aws lambda make the handler async, so you have to adapt your code.
Docs : https://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html
The runtime passes three arguments to the handler method. The first
argument is the event object, which contains information from the
invoker. The invoker passes this information as a JSON-formatted
string when it calls Invoke, and the runtime converts it to an object.
When an AWS service invokes your function, the event structure varies
by service.
The second argument is the context object, which contains information
about the invocation, function, and execution environment. In the
preceding example, the function gets the name of the log stream from
the context object and returns it to the invoker.
The third argument, callback, is a function that you can call in
non-async functions to send a response. The callback function takes
two arguments: an Error and a response. When you call it, Lambda waits
for the event loop to be empty and then returns the response or error
to the invoker. The response object must be compatible with
JSON.stringify.
For async functions, you return a response, error, or promise to the
runtime instead of using callback.
exports.handler = async function(event, context, callback) {
console.log("EVENT: \n" + JSON.stringify(event, null, 2))
return context.logStreamName
}
Thx!

Answer 2

I think you need to set the right connection pooling config.
See the docs here: https://github.com/marcogrcr/sequelize/blob/patch-1/docs/manual/other-topics/aws-lambda.md
const { Sequelize } = require("sequelize");
let sequelize = null;
async function loadSequelize() {
const sequelize = new Sequelize(/* (...) */, {
// (...)
pool: {
/*
* Lambda functions process one request at a time but your code may issue multiple queries
* concurrently. Be wary that `sequelize` has methods that issue 2 queries concurrently
* (e.g. `Model.findAndCountAll()`). Using a value higher than 1 allows concurrent queries to
* be executed in parallel rather than serialized. Careful with executing too many queries in
* parallel per Lambda function execution since that can bring down your database with an
* excessive number of connections.
*
* Ideally you want to choose a `max` number where this holds true:
* max * EXPECTED_MAX_CONCURRENT_LAMBDA_INVOCATIONS < MAX_ALLOWED_DATABASE_CONNECTIONS * 0.8
*/
max: 2,
/*
* Set this value to 0 so connection pool eviction logic eventually cleans up all connections
* in the event of a Lambda function timeout.
*/
min: 0,
/*
* Set this value to 0 so connections are eligible for cleanup immediately after they're
* returned to the pool.
*/
idle: 0,
// Choose a small enough value that fails fast if a connection takes too long to be established.
acquire: 3000,
/*
* Ensures the connection pool attempts to be cleaned up automatically on the next Lambda
* function invocation, if the previous invocation timed out.
*/
evict: CURRENT_LAMBDA_FUNCTION_TIMEOUT
}
});
// or `sequelize.sync()`
await sequelize.authenticate();
return sequelize;
}
module.exports.handler = async function (event, callback) {
// re-use the sequelize instance across invocations to improve performance
if (!sequelize) {
sequelize = await loadSequelize();
} else {
// restart connection pool to ensure connections are not re-used across invocations
sequelize.connectionManager.initPools();
// restore `getConnection()` if it has been overwritten by `close()`
if (sequelize.connectionManager.hasOwnProperty("getConnection")) {
delete sequelize.connectionManager.getConnection;
}
}
try {
return await doSomethingWithSequelize(sequelize);
} finally {
// close any opened connections during the invocation
// this will wait for any in-progress queries to finish before closing the connections
await sequelize.connectionManager.close();
}
};
It's actually for sequelize, not knex, but I'm sure under the hood they work the same way.

Answer 3

I had this problem too, in my case it was cause i tried to connect db in production.
so, I added ssl to Pool, like this:
const pool = new Pool({
connectionString: connectionString,
ssl: {rejectUnauthorized: false},
});
Hope it helps you too...