I'm trying to write a Lambda function (in nodejs) which executes a simple SQL query. The database is an Aurora Postgres. I have set up function configuration for VPC access, worked fine with a MySQL database.
As you can see in the following, I'm using the "pg" node module (https://github.com/brianc/node-postgres).
app.js
// For example, this function times out
const { Client } = require('pg');
const config = {
user: '<MY_DB_USER>',
host: '<MY_HOST>',
database: '<MY_DB_NAME>',
password: '<MY_DB_PASSWORD>',
port: <MY_PORT>
};
var postgresClient = new Client(config);
module.exports.handler = (event, context, cb) => {
console.log('entered handler');
postgresClient
.connect()
.then(() => {
console.log('connected');
return postgresClient.end();
})
.then(() => cb())
.catch(cb);
};
Yet, when using a postgres database, the lambda function times out every time I invoke. I have set up the subnets and the Security Group.
I have also tried using Pool from the pg module, with no success.
Note that this functions runs locally with success... I can't figure out why the lambda fails when it is deployed.
Related
So i have this small Node JS app where i have the following script, which i invoke in my HTML index page, in order to connect to a Cloud SQL database in GCP and perform a specific query so i can pass the values to a dropdown later:
try {
pool = new Pool({
user: "postgres",
host: "/cloudsql/sfmcsms-d-970229:europe-west1:dsi-sfmc-sms-database",
database: "postgres",
password: "dsi-sfmc-sms-database",
port: "5432",
});
console.log("Connection successfull!");
pool.query("select * from ConfigParameter;", (error, results) => {
if (error) {
console.log(error);
}
qResult = results;
console.log(qResult);
//insert logic to populate dropdowns
});
} catch (err) {
console.log("Failed to start pool", err);
}
I'm still working on the logic to populate the dropdowns but for now, i'm focusing on establishing a successful connection first before i get to that. However, everytime i run the script, i seem to get this particular error:
ReferenceError: Pool is not defined
I've been looking around for some possible answers but no luck.
Before using Pool you have import first like this
const { Pool } = require('pg')
And obviously node-postgres should be installed
npm i pg
I was trying to make a post query the db (RDS) using handler.async.
However, I ran into the following issues.
Half of the time, the lambda function completes but the query is not successfully sent to RDS. The other half of the time, it will be completely send to lambda. Tried adding a setTimeout function to increase lambda execution time by 3 secs and the query will be sent all the time.
Also the log will shows the error:
INFO Error: Cannot enqueue Query after fatal error.
The following are my code:
var mysql = require('mysql');
var connection = mysql.createConnection({
host : '***',
user : '***',
password : '***',
database : '***'
});
exports.handler = async (event) => {
const sql = `INSERT INTO forms VALUES(777,2,3,4,5,6,7,8,9,10,11);`;
const query = (x) => {
return new Promise ((resolve,reject)=>{
resolve(connection.query(x, function (error, results, fields) {
console.log(error)
console.log(results)
console.log(fields)
}))})}
await query(sql)
}
With the timeout function,
var mysql = require('mysql');
var connection = mysql.createConnection({
host : '***',
user : '***',
password : '***',
database : '***'
});
exports.handler = async (event) => {
const sql = `INSERT INTO forms VALUES(777,2,3,4,5,6,7,8,9,10,11);`;
const query = (x) => {
return new Promise ((resolve,reject)=>{
resolve(connection.query(x, function (error, results, fields) {
console.log(error)
console.log(results)
console.log(fields)
}))})}
await query(sql)
await wait(3000)
}
const wait = (x) => {
return new Promise ((resolve,reject)=>{
setTimeout(()=>{resolve(console.log("delay"))}, x);
})
}
The first value is a primary key. A constant 777 is sent to check, if error shows duplicate primary key, it will mean that the query is successfully sent. If there is no error, it means that the query is unsuccessfully sent although lambda finishes.
execution result succeeded but shows:
START RequestId: e541fe4b-6927-4fbb-90b4-750f77e5f460 Version: $LATEST
2019-12-19T01:54:45.212Z e541fe4b-6927-4fbb-90b4-750f77e5f460 INFO Error: **Cannot enqueue Query after fatal error**.
at Protocol._validateEnqueue (/var/task/node_modules/mysql/lib/protocol/Protocol.js:212:16)
at Protocol._enqueue (/var/task/node_modules/mysql/lib/protocol/Protocol.js:138:13)
at Connection.query (/var/task/node_modules/mysql/lib/Connection.js:201:25)
at /var/task/index.js:14:24
at new Promise (<anonymous>)
at query (/var/task/index.js:13:10)
at Runtime.exports.handler (/var/task/index.js:20:7)
at Runtime.handleOnce (/var/runtime/Runtime.js:66:25) {
code: 'PROTOCOL_ENQUEUE_AFTER_FATAL_ERROR',
fatal: false
}2019-12-19T01:54:45.213Z e541fe4b-6927-4fbb-90b4-750f77e5f460 INFO undefined2019-12-19T01:54:45.213Z e541fe4b-6927-4fbb-90b4-750f77e5f460 INFO undefined2019-12-19T01:54:45.262Z e541fe4b-6927-4fbb-90b4-750f77e5f460 INFO delayEND RequestId: e541fe4b-6927-4fbb-90b4-750f77e5f460
REPORT RequestId: e541fe4b-6927-4fbb-90b4-750f77e5f460 Duration: 51.09 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 80 MB
May you please advise and also tell me which is the best way to execute it??
Managing RDBMS connections in any environment is not a trivial task. Lambda adds a layer of complexity here. You need to understand the distinction between warm and cold restarts, what it means for resources created outside of your handler function, when connection pools are appropriate, and when and how to release connections.
Persistent connections to a database are not particularly suitable in a microservices, FaaS environment like Lambda. That's one reason that Aurora Serverless supports an HTTP Data API (and hopefully other DB engines will too at some point).
Read How To: Manage RDS Connections from AWS Lambda Serverless Functions.
Also be aware of the new Amazon RDS Proxy with AWS Lambda.
In your particular case, the most obvious concern is that you are repeatedly creating DB connections but never releasing them (unless that is a built-in feature of the mysql package's query function that I'm not aware of).
You can increase the lambda timeout upto 15 minutes. But if you are calling the lambda through api gateway, the timeout is 29 seconds.
here is the code working for me.
const mysql = require('mysql');
const con = mysql.createConnection({
host: process.env.RDS_HOSTNAME,
user: process.env.RDS_USERNAME,
password: process.env.RDS_PASSWORD,
port: process.env.RDS_PORT,
connectionLimit: 10,
multipleStatements: true,// Prevent nested sql statements
debug: true
// ,database:'testdb1'
});
exports.handler = async (event) => {
try {
const data = await new Promise((resolve, reject) => {
con.connect(function (err) {
if (err) {
reject(err);
}
const sql = `INSERT INTO forms VALUES(777,2,3,4,5,6,7,8,9,10,11);`;
con.query(sql, function (err, result) {
if (err) {
console.log("Error->" + err);
reject(err);
}
resolve(result);
});
})
});
return {
statusCode: 200,
body: JSON.stringify(data)
}
} catch (err) {
return {
statusCode: 400,
body: err.message
}
}
};
reference: aws lambda with rds mysql DDL command not working
I have a function app that serves a node.js API. We are hitting the 900 concurrent connections limit with tedious connected to Azure SQL and realize we should add connection pools (unless there is a better recommendation of course).
Azure Functions + Azure SQL + Node.js and connection pooling between requests? seems to answer our prayers but wanted to validate how you can use a single connection pool with Azure functions
Is the best practice to put "let pool = new ConnectionPool(poolConfig, connectionConfig);" above mode.exports on all functions? Is that not creating a new pool every time an individual function is called?
Microsoft doesn't have clear documentation on this for node.js unfortunately so any help would be greatly appreciated!
To make the whole Function app share one single pool, we need to put the initialization part to a shared module. Christiaan Westerbeek had posted a wonderful solution using mssql, there's not so much difference between a Function app and a web app in this respect.
I recommend using mssql(use tedious and generic-pool internally) instead of tedious-connection-pool which seems not updated for 2 years.
Put the connection code in poolConfig.js under a SharedLib folder.
const sql = require('mssql');
const config = {
pool:{
max:50 // default: 10
},
user: '',
password: '',
server: '',
database: '',
options: {
encrypt: true // For Azure Sql
}
};
const poolPromise = new sql.ConnectionPool(config).connect().then(pool => {
console.log('Connected to MSSQL');
return pool;
})
.catch(err => console.log('Database Connection Failed! Bad Config: ', err));
module.exports = {
sql, poolPromise
}
And load the module to connect to sql. We use await to get ConnectionPool the function should be async(default for v2 js function).
const { poolPromise } = require('../SharedLib/poolConfig');
module.exports = async function (context, req) {
var pool = await poolPromise;
var result = await pool.request().query("");
...
}
Note that if Function app is scaled out with multiple instances, new pool will be created for each instance as well.
I have an AWS RDS which is publicly accessible and I want to connect to that RDS using AWS Lambda. I am using API Gateway to create a Rest API named "/hello", which needs to return the RDS connection status.
The Lambda code is given below.
var mysql = require('mysql');
var ApiBuilder = require('claudia-api-builder'),
api = new ApiBuilder();
var con = mysql.createConnection({
host: "host",
user: "user",
password: "password"
});
module.exports = api;
api.get('/hello', function () {
//any code written here, works perfectly, when this is called
con.connect(function(err) { //this part doesn't work
if (err) return err;
return "connected";
});
});
The database is publicly accessible, even from my local machine. The Lambda function also has the required permissions with AWS RDS.
The response I get on the browser is as follows
{}
I checked the cloud watch logs, and it is given below.
08:16:11
START RequestId: bf804be1-4797-11e8-8b3f-5b09118631a6 Version: $LATEST
08:16:11
END RequestId: bf804be1-4797-11e8-8b3f-5b09118631a6
08:16:11
REPORT RequestId: bf804be1-4797-11e8-8b3f-5b09118631a6 Duration: 34.28 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 26 MB
The Lambda function doesn't go past con.connect(...) where there is no error being shown anywhere (not even on RDS logs). The API Gateway works, because any code before con.connect(...) works perfectly. What could be the issue here?
con.connect(function(err) {
if (err) return err;
return "connected";
});
This is an asynchronous call. Before connect callback function is called lambda will complete and exit.
Change this to
con.connect(function(err) {
if (err) context.done(err);
context.done(null, "connected");
});
As #ConfusedCoder explained, this is caused due to con.connect(...) being an asynchronous call. Using the context in Lambda is not an option here, as Lambda is being invoked using API Gateway, and that there is no exports.myhandler being used in the scenario. Therefore the context object cannot be used to control the flow of Lamda.
exports.myHandler = function(event, context) {
...
}
I tried playing with promises, async await and other techniques until I found a node package named sync-mysql. This makes synchronous queries to a mysql database, where AWS Lambda does not proceed without executing the database call. The updated code is given below, using 'sync-mysql'.
var ApiBuilder = require('claudia-api-builder'), api = new ApiBuilder();
var MySql = require('sync-mysql');
var connection = new MySql({
host: "host",
user: "user",
password: "password"
});
module.exports = api;
api.get('/hello', function(request) {
return connection.query('SELECT * FROM DB.DummyTable');
});
My file db/index.js
const { Pool } = require('pg');
const pool = new Pool;
module.exports = {
query: (text, params, callback) => {
return pool.query(text,params,callback);
}
};
In my main file main.js I do:
const db = require('./db/index');
What command can I run on db to figure out if node-postgres is able to connect to my Postgres setup correctly?
To simply test if you can connect from node.js to pgsql database you can use the following snippet:
const { Pool } = require('pg')
const pool = new Pool()
pool.query('SELECT NOW()', (err, res) => {
console.log(err, res)
pool.end()
})
// or you can use async/await instead of callbacks
const res = await pool.query('SELECT NOW()')
console.log(res)
await pool.end()
This should return the response in form of pg.Result object containing current datetime.
node-postgres uses the same environment variables as libpq to connect to a PostgreSQL server, so to run the above code you can invoke it like so:
PGUSER=postgres PGHOST=127.0.0.1 PGPASSWORD=mysecretpassword PGDATABASE=postgres PGPORT=5432 node script.js
But you have to provide connection details to you database instance.
The default values for the environment variables used are:
PGHOST='localhost'
PGUSER=process.env.USER
PGDATABASE=process.env.USER
PGPASSWORD=null
PGPORT=5432
You can also provide the connection details programmatically, directly to either the Pool or Client instances. You can also use the connection string URI.
You can read more in the node-postgres documentation in the "connecting" section.