how to use node pg in AWS Lambda? - node.js

UPDATE/SOLVED ... the problem was that pgpass plugin was looking for $HOME/.pgpass, and $HOME wasn't defined on lambda, causing failure of join(undefined, '.pgpass'). Not really worth crediting myself with an answer for that ....
I have the following test code in a lambda handler (compiled from typescript):
const db = new Pool()
const {rows} = await db.query('select 1 as x')
console.log('rows', JSON.stringify(rows))
The connection is created via PGHOST PGPORT PGUSER PGDATABASE in the environment. I know the configuration is "almost right" because the first time I tried it I got:
{"errorMessage":"error: no pg_hba.conf entry for host \"10.1.1.249\",
user \"foo\", database \"bar\", SSL off\n at
Connection.parseE (/var/task/handler.js:9209:11)\n at
Connection.parseMessage (/var/task/handler.js:9034:19)\n at
Socket. (/var/task/handler.js:8777:22)\n at emitOne
(events.js:96:13)\n at Socket.emit (events.js:188:7)\n at
readableAddChunk (_stream_readable.js:176:18)\n at
Socket.Readable.push (_stream_readable.js:134:10)\n at TCP.onread
(net.js:547:20)"}
That seemed hopeful. I changed the postgres config to allow from the subnet, and tried again. However, now I get:
2018-02-06 18:14:21.183 (-05:00) 76761ca5-0b93-11e8-8783-a74d098c9f4a select
2018-02-06 18:14:21.202 (-05:00) 76761ca5-0b93-11e8-8783-a74d098c9f4a TypeError: Path must be a string. Received undefined
at assertPath (path.js:7:11)
at Object.join (path.js:1211:7)
at Object.module.exports.getFileName (/var/task/handler.js:32434:16)
at module.exports (/var/task/handler.js:32355:23)
at Connection.<anonymous> (/var/task/handler.js:31255:9)
at emitOne (events.js:96:13)
at Connection.emit (events.js:188:7)
at Socket.<anonymous> (/var/task/handler.js:8781:12)
at emitOne (events.js:96:13)
at Socket.emit (events.js:188:7)
END RequestId: 76761ca5-0b93-11e8-8783-a74d098c9f4a
What is going on? It seemed as if it had got to the database and connected the first time... Now it fails on the query. Is there some option I should be using?

First, you are not calling pool.connect to acquire a client from the pool.
Second, the current Node.js runtime 6.10 on AWS Lambda does not support async functions. (Unless you transpile your code down to ES5 using Babel directly or some other boilerplate that uses Babel such as es2017-lambda-boilerplate)
Having said that, you'll have to use either callbacks or promises (then/catch). For example:
const {Pool} = require("pg");
const db = new Pool({
host: "localhost",
user: "database-user",
// ... rest of config options
});
db.connect((err, client, release) => {
if (err) {
console.error("Error acquiring client.", err.stack);
} else {
client.query("SELECT 1 AS x", (err, result) => {
release();
if (err) {
console.error("Error executing query.", err.stack);
} else {
console.log(result.rows);
}
})
}
})

Related

Retry connection postgres Nodejs

I am using a postgres database for my express web server.
I am using the 'pg' library to execute queries on this database.
Here is my connection method :
const db = new Client({
user: 'xxx',
host: 'xxx',
database: 'xxx',
password: 'xxx',
port: xxx,
})
db.connect(err => {
if (err) {
console.error('connection error', err.stack)
} else {
console.log('connected')
}
Then to execute a request I do this:
db.query(MY_REQUEST, function (err, data) {
if (err) throw err;
res.render('hello/world', {
title: 'Hello',
data: data.rows
});
});`
It all works perfectly. But after several minutes without using my website, my connection to the db times out, and I get the following error:
node:events:355
throw er; // Unhandled 'error' event
^
Error: Connection terminated unexpectedly
at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:132:73)
at Object.onceWrapper (node:events:484:28)
at Connection.emit (node:events:378:20)
at Socket.<anonymous> (/usr/src/app/node_modules/pg/lib/connection.js:58:12)
at Socket.emit (node:events:378:20)
at TCP.<anonymous> (node:net:665:12)
Emitted 'error' event on Client instance at:
at Client._handleErrorEvent (/usr/src/app/node_modules/pg/lib/client.js:319:10)
at Connection.<anonymous> (/usr/src/app/node_modules/pg/lib/client.js:149:16)
at Object.onceWrapper (node:events:484:28)
[... lines matching original stack trace ...]
at TCP.<anonymous> (node:net:665:12)
How could I do to reconnect automatically when the connection is cut or when a request fails?
You should attach an error-handler in order to prevent the unhandled error crashing your app. It's as simple as:
db.on('error', e => {
console.error('DB error', e);
});
As to why the error happens we need more details, looks like it could be a connection reset due to idle-timeout?
You can create a function to control if you're connected to database or not, before you continue with your main function.
Create a function for controlling database connection status, reconnecting etc. and before you run a database related function, first start that middle function and wait for result, after that you can continue using database again.
If you want(which should be prefered way mostly), create that middle function as an async function and return a promise, when using that function wait for that function.

Getting "InternalServerError" while using Mongo with Node js

I am new to Node js programming. Trying to write a simple user create using.. code along with the stack track is given below. Appreciate any help as to what is causing this..
users-add.mjs:
'use strict';
const util = require('util');
const restify = require('restify-clients');
console.log("starting...");
var client = restify.createJsonClient({url: 'http://localhost:'+process.env.PORT, version: '*'});
console.log("starting...2 ");
client.basicAuth('them', 'D4ED43C0-8BD6-4FE2-B358-7C0E230D11EF');
client.post('/create-user', { username: "me", password: "w0rd", provider: "local",
familyName: "GUD", givenName: "RG", middleName: "Ud", emails: [], photos: []},
(err, req, res, obj) => {
console.log("starting...4 ");
if(err) {
console.log("starting...5");
console.error(err.stack);
}
else {
console.log("starting...6");
console.log('Created ' + util.inspect(obj));
}
});
console.log("starting...3 ");
Here is the error I'm getting.. Surprisingly the server side code is able to insert the record into MongoDB, and yet I get this error..
InternalServerError: {}
at Object.createHttpErr (/Users/raviguduru/node-web-dev/chapter8/users/node_modules/restify-clients/lib/helpers/errors.js:91:26)
at ClientRequest.onResponse (/Users/raviguduru/node-web-dev/chapter8/users/node_modules/restify-clients/lib/HttpClient.js:309:26)
at Object.onceWrapper (events.js:300:26)
at ClientRequest.emit (events.js:210:5)
at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:583:27)
at HTTPParser.parserOnHeadersComplete (_http_common.js:115:17)
at Socket.socketOnData (_http_client.js:456:22)
at Socket.emit (events.js:210:5)
at addChunk (_stream_readable.js:309:12)
at readableAddChunk (_stream_readable.js:290:11)
Pls help.
This problem is solved now.. The internal server error comes when we have MongoDBClient calls not wrapped in try/catch blocks. After I included try/catch blocks, they went away.

How can I get a stack trace into client code with the MongoDB Node.js driver?

I've noticed that MongoDB errors offer rather useless stack traces, pointing only within the driver's internals. The line of code in the client that triggered the error is not output:
const MongoClient = require('mongodb').MongoClient;
MongoClient.connect('...', { useNewUrlParser: true }, async (err, client) => {
try {
// some mongo client code...
// some more mongo client calls...
await client.db().collection('nonexistent').drop();
// yet more mongo client calls...
} catch (e) {
console.error(e);
}
client.close();
});
Here's the output from that:
MongoError: ns not found
at Connection.<anonymous> (/home/dandv/mongostack/node_modules/mongodb-core/lib/connection/pool.js:443:61)
at Connection.emit (events.js:200:13)
at processMessage (/home/dandv/mongostack/node_modules/mongodb-core/lib/connection/connection.js:364:10)
at TLSSocket.<anonymous> (/home/dandv/mongostack/node_modules/mongodb-core/lib/connection/connection.js:533:15)
at TLSSocket.emit (events.js:200:13)
at addChunk (_stream_readable.js:290:12)
at readableAddChunk (_stream_readable.js:271:11)
at TLSSocket.Readable.push (_stream_readable.js:226:10)
at TLSWrap.onStreamRead (internal/stream_base_commons.js:166:17) {
ok: 0,
errmsg: 'ns not found',
code: 26,
codeName: 'NamespaceNotFound',
name: 'MongoError',
[Symbol(mongoErrorContextSymbol)]: {}
}
If there's more than one mongo driver call in the try/catch block, it's impossible to tell which one generated the error.
Is there a way to have those stack traces lead up to client code?

Error connecting to heroku postgres db in nodejs

It seems after my database was upgraded to 10.2 I'm unable to connect.
I'm using the pg 7.4.2 npm package.
To be clear, I have been connecting without issue for 6 months using the same connection string which had ?ssl=true appended to it.
I get this error connecting via Pool or Client.
AssertionError [ERR_ASSERTION]: false == true
at Object.exports.connect (_tls_wrap.js:1099:3)
at Socket.<anonymous> (/home/e/express/testpg/node_modules/pg/lib/connection.js:94:23)
at Object.onceWrapper (events.js:219:13)
at Socket.emit (events.js:127:13)
at Socket.emit (domain.js:421:20)
at addChunk (_stream_readable.js:269:12)
at readableAddChunk (_stream_readable.js:256:11)
at Socket.Readable.push (_stream_readable.js:213:10)
at TCP.onread (net.js:598:20)
I'm now hardcoding the full postgres connection string so there is no issue with env variables.
I've tried adding/removing ?ssl=true to the end of the connection string and adding/removing ssl:true from the constructor. I've also tried with and without promises. Same error no matter what on both local and deployed to heroku.
imports:
import { Pool, Client } from 'pg'
method 1:
let pool = new Pool({
connectionString: csnew,
ssl: true
})
pool.connect().then( client => {
console.log('connected')
})
.catch(e=> {
console.log(e)
})
method 2:
let pgclient = new Client({
connectionString: csnew,
ssl: true
})
pgclient.connect().then( () => {
console.log('connected')
}).catch(e=> {
console.log(e)
})
That's because v7.4.2 broke its SSL support. Here's the open issue.
You need to use strictly v7.4.1 till the issue is resolved.
UPDATE
Version 7.4.3 fixed the issue.

Cloud Functions for Firebase Image download function Error

I am building a web application using Firebase and the new feature Cloud Functions for Firebase. I have created a function that takes a URL and downloads the image into a 64-bit encoded string as below using the node modules request and request-promise-native:
module.exports = {
downloadImageFromUrl: function (url) {
var options = {
method: 'GET',
uri: url,
resolveWithFullResponse: true,
simple: false,
family: 4
};
return rp.get(options)
.then(function (res) {
return "data:" + res.headers["content-type"] + ";base64," + new Buffer(res.body).toString('base64');
})
.catch(function (error) {
console.log("ERROR GETTING image", error);
return error;
});
}
};
The top function works perfectly running locally but once on firebase it gives the error:
RequestError: Error: getaddrinfo EAI_AGAIN lh6.googleusercontent.com:443
at new RequestError (/user_code/node_modules/request-promise/node_modules/request-promise-core/lib/errors.js:14:15)
at Request.plumbing.callback (/user_code/node_modules/request-promise/node_modules/request-promise-core/lib/plumbing.js:87:29)
at Request.RP$callback [as _callback] (/user_code/node_modules/request-promise/node_modules/request-promise-core/lib/plumbing.js:46:31)
at self.callback (/user_code/node_modules/request/request.js:188:22)
at emitOne (events.js:96:13)
at Request.emit (events.js:188:7)
at Request.onRequestError (/user_code/node_modules/request/request.js:884:8)
at emitOne (events.js:96:13)
at ClientRequest.emit (events.js:188:7)
at TLSSocket.socketErrorListener (_http_client.js:310:9)
at emitOne (events.js:96:13)
at TLSSocket.emit (events.js:188:7)
at connectErrorNT (net.js:1020:8)
at _combinedTickCallback (internal/process/next_tick.js:74:11)
at process._tickDomainCallback (internal/process/next_tick.js:122:9)
I am calling the function in the firebase auth trigger when a user is created as below:
exports.createUser = functions.auth.user().onCreate(event => {
if (event.data.photoURL) {
utils.downloadImageFromUrl(event.data.photoURL)
.then(function(res){
console.log("User Photo", res);
})
.catch(function(error){
console.log("Error", error);
})
}
});
Any help would be greatly appreciated.
Not entirely sure yet if this is the answer, but after reading the documentation, I read their free plan which says you cannot make any out bound requests. So I guess getting an image from a Url counts as an outbound request. After I start paying for their service, I will come back to verify if this was the problem.

Resources