Can you keep a PostgreSQL connection alive from within a Next.js API? - node.js

I'm using Next.js for my side project. I have a PostrgeSQL database hosted on ElephantSQL. Inside the Next.js project, I have a GraphQL API set up, using the apollo-server-micro package.
Inside the file where the GraphQL API is set up (/api/graphql), I import a database helper-module. Inside that, I set up a pool connection and export a function which uses a client from the pool to execute a query and return the result. This looks something like this:
// import node-postgres module
import { Pool } from 'pg'
// set up pool connection using environment variables with a maximum of three active clients at a time
const pool = new Pool({ max: 3 })
// query function which uses next available client to execute a single query and return results on success
export async function queryPool(query) {
let payload
// checkout a client
try {
// try executing queries
const res = await pool.query(query)
payload = res.rows
} catch (e) {
console.error(e)
}
return payload
}
The problem I'm running into, is that it appears as though the Next.js API doesn't (always) keep the connection alive but rather opens up a new one (either for every connected user or maybe even for every API query), which results in the database quickly running out of connections.
I believe that what I'm trying to achieve is possible for example in AWS Lambda (by setting context.callbackWaitsForEmptyEventLoop to false).
It is very possible that I don't have a proper understanding of how serverless functions work and this might not be possible at all but maybe someone can suggest me a solution.
I have found a package called serverless-postgres and I wonder if that might be able to solve it but I'd prefer to use the node-postgres package instead as it has much better documentation. Another option would probably be to move away from the integrated API functionality entirely and build a dedicated backend-server, which maintains the database connection but obviously this would be a last resort.

I haven't stress-tested this yet, but it appears that the mongodb next.js example, solves this problem by attaching the database connection to global in a helper function. The important bit in their example is here.
Since the pg connection is a bit more abstract than mongodb, it appears this approach just takes a few lines for us pg enthusiasts:
// eg, lib/db.js
const { Pool } = require("pg");
if (!global.db) {
global.db = { pool: null };
}
export function connectToDatabase() {
if (!global.db.pool) {
console.log("No pool available, creating new pool.");
global.db.pool = new Pool();
}
return global.db;
}
then in, eg, our API route, we can just:
// eg, pages/api/now
export default async (req, res) => {
const { pool } = connectToDatabase();
try {
const time = (await pool.query("SELECT NOW()")).rows[0].now;
res.end(`time: ${time}`);
} catch (e) {
console.error(e);
res.status(500).end("Error");
}
};

Related

next.js and mongodb coherence?

I googled a lot but still have no clear solution to my issue.
Connecting to MongoDB, usually you establish a connection and after the job is done you close it.
Since next.js (and probably node.js) is single threaded. Sometimes it happens that there are two requests processed async while one request established the connection to the database, the otherone is closing the exact same connection. So the first request runs into an Topology closed exception. I have the feeling that the mongodb driver client is shared.
Is there something I did not understood correct in this?
try {
await client.connect()
const database = client.db("test")
const collection = database.collection("test")
const newDataset = await collection.insertOne({})
return newDataset.insertedId.toString()
} finally {
await client.close();
}
As in the comments stated, ive seen a lot of examples & questions here on stackoverflow where in each received request (example below) a database connection is established. This has no benefits and is "bad" because it just takes time and makes no sense. E.g:
app.get("/", (req, res) => {
MongoClient.connect("...", (err, client) => {
// do what ever you want here
client.close();
});
});
If you application needs a database connection, establish the connection "in the startup phase" and keep the connection open. There is no reason to open and close the database connection for each request.
const mongodb = require("monogdb");
const express = require("express");
const app = express();
// some custom init stuff
// e.g. require your route handler etc.
mongodb.MongoClient("...", (err, client) => {
// do what ever you want with the db connection now
// e.g. monkey patch it, so you can use it in other files
// (There are better ways to handle that)
mongodb.client = client;
// or the better way
// pass it as function parameter
require("./routes")(app, client);
app.listen(8080, () => {
console.log("http server listening");
});
});
As you can see in the code above, we first create a database connection and then do other stuff. This has some advantages:
If your credentials are invalid, your application is not externeal reachable because the http server is not started
You have a single connection for all requests
Database queries are potential faster because you dont have to wait to establish first a db connection
NOTE: the code above was "inline coded" here and is not tested.
But i think its illustrated the concept behind my statement.

How to share a single promise based RabbitMQ connection across files or controllers in Node js instead of creating a new Connection each time?

amqplib library lets you create a rabbitmq connection and that object will be a segue to doing other things such as creating a channel and etc.
suppose that I'm going for a Producer/Consumer pattern, where each time a user hits a specific route, a job is produced and sent to the rabbitmq server where it's processed by certain consumers(workers).
app.post("/routethatdelegatesheavywork", async (req,res) => {
const amqpServerLink =
"link-to-cloudmq";
const connection = await amqp.connect(amqpServerLink);
const channel = await connection.createChannel();
//do other stuff with channel
})
while this "works", but i don't want to re-create that connection every time the controller is invoked since it makes the producer very slow and it's really not how it's supposed to be done.
here is where my problem comes:
how do i initialize one connection and re-use it every time i need it?
i have tried to create a connection outside controllers and use it when necessary but it's not possible since the connection is promise-based and await doesn't work on entry point and it has to be inside an async function to work.
although it is possible to run await without async using ESM (es modules) i don't want to do so since i have written all of the application using CommonJS (require("package")), changing that would require me to go through a lot of files and change every import/export according to ESM.
So, is there any other way to create one connection(that is promise based) and re-use it without having to migrate to ESM syntax?
Yes, remember that require in nodejs are singletons. Make a new amqpServerInterface module, and do
const amqpServerLink = "link-to-cloudmq"
const connection = amqp.connect(amqpServerLink)
function connect() {
return connection
}
module.exports = {
connect
}
Then in your controllers
const amqpServerInterface = require('amqpServerInterface')
app.post("/routethatdelegatesheavywork", async (req,res) => {
const connection = await amqpServerInterface.connect();
const channel = await connection.createChannel();
//do other stuff with channel
})
This will always return the same connection promise and will resolve to the save connection.

Database Connection using common module is not working [ mongoose and mongodb ]

I am trying to implement a common module for MongoDB connection using mongoose. and want to use the connection in other application for database operation. but facing issue when trying to use common database module. operation is halted / hanging after creating db connection. here is my codebase.
When I am using module specific dababase connection, then it is working fine, but when I am using common database connection it is hanging
Common DB Module
'use strict'
const mongoose = require('mongoose');
const DBOptions = require('./DBOption');
require("dotenv").config();
mongoose.Promise = global.Promise;
let isConnected;
const connectToDatabase = (MONGODB_URL) => {
if (isConnected) {
console.log('using existing database connection');
return Promise.resolve();
}
console.log('using new database connection');
console.log('DBOptions >> '+JSON.stringify(DBOptions));
return mongoose.connect(MONGODB_URL, DBOptions)
.then(db => {
console.log('db.connections[0].readyState >> '+db.connections[0].readyState);
isConnected = db.connections[0].readyState;
});
};
module.exports = connectToDatabase;
API Controller
const dbConnection = require('../DB/connection') // Internal Class
const DBConnection = require('as-common-util').connectToDatabase; // Common Class
/**
*
*/
app.get('/usr/alluser', async (req, res) => {
try {
//await dbConnection(process.env.MONGODB_URL) // This is working
await DBConnection(process.env.MONGODB_URL) // Code is hanging for this
let allUsers = await UserService.getAllUser()
console.log("All Users >> " + allUsers)
if (allUsers) {
return res.status(200).send(
new APIResponse({
success: true,
obj: allUsers
})
)
}
} catch (error) {
console.log(error)
}
})
It is hanging at following position
using new database connection
DBOptions >>
{"useNewUrlParser":true,"useUnifiedTopology":true,"useCreateIndex":true,"useFindAndModify":false,"autoIndex":false,"poolSize":10,"serverSelectionTimeoutMS":5000,"socketTimeoutMS":45000,"family":4}
db.connections[0].readyState >> 1
I am confused why same code is not working for common module.
This kind of pattern is not how Mongoose is meant to be used. Under the hood, Mongoose passes the underlying connection to the models in your module without the user really knowing anything about what is going on. That's why you can do magic stuff like MyModel.find() without ever having to create a model object yourself, or pass a db connection object to it.
If your db connection is in another module though, Mongoose won't be able to make those connections between your models and the MongoDB client connection since the models are no longer being registered on the mongoose object that is actually connected, and as a result, any requests you make using your models will break, since they will always be trying to connect through the object in your module.
There are other reasons why this won't, and shouldn't, work though. You're not supposed to be able to split a client. Doing so would make it unclear where communication along a client is coming from, or going to. You could change your function to make it return an established client connection. But your Mongoose models still wouldn't work. You would just be left with raw mongodb. If you want to do that, you might as well just uninstall Mongoose and use the mongodb library. Ultimately, you don't really gain anything from initializing the connection in a shared module. Initializing a connection is just a couple lines of code.
I doubt it's the connection that you want to share, rather it's the models (I'm guessing). You can put those in a shared module, and export them as a kind of connector function that injects the a given Mongoose instance into the models. See: Defining Mongoose Models in Separate Module.

Is declaring a Node.js redis client as a const in multiple helpers a safe way to use it?

This is a little hard articulate so I hope my title isn't too terrible.
I have a frontend/backend React/Node.js(REST API) Web app that I want to add Redis support to for storing retrieving app global settings and per-user specific settings (like language preference, last login, etc... simple stuff) So I was considering adding a /settings branch to my backend REST API to push/pull this information from a redis instance.
This is where my Node.js inexperience comes through. I'm looking at using the ioredis client and it seems too easy. If I have a couple of helpers (more than one .js which will call upon redis) will constructing the client as a const in each be safe to do? Or is there another recommended approach to reusing a single instance of it be the way to go?
Here's a sample of what I'm thinking of doing. Imagine if I had 3 helper modules that require access to the redis client. Should I declare them as const in each? Or centralize them in a single helper module, and get the client from it? Is there a dis-advantage to doing either?
const config = require('config.json');
const redis_url = config.redis_url;
//redis setup
const Redis = require('ioredis');
const redis = new Redis(redis_url);
module.exports = {
test
}
async function test(id) {
redis.get(id, function (err, result) {
if (err) {
console.error(err);
throw(err);
} else {
return result;
}
});
Thank you.
If no redis conflicts...
If the different "helper" modules you are referring to have no conflicts when interacting with redis, such as overwriting / using the same redis keys, then I can't see any reason not to use the same redis instance (as outlined by garlicman) and export this to the different modules in which it is used.
Otherwise use separate redis databases...
If you do require separate redis database connections, redis ships with 16 databases so you can specify which to connect to when creating a new instance - see below:
const redis = new Redis({ // SET UP CONFIG FOR CONNECTION TO REDIS
port: 6379, // Redis port
host: 127.0.0.1, // Redis host
family: 4, // 4 (IPv4) or 6 (IPv6)
db: 10, // Redis database to connect to
});
Normally what I would do (in Java say) is implement any explicit class with singleton access the hold the connection and any connection error/reconnect handling.
All modules in Node.js are already singletons I believe, but what I will probably go with will be a client class to hold it and my own access related methods. Something like:
const config = require('config.json');
const Redis = require("ioredis");
class Redis {
constructor(){
client = new Redis(config.redis_url);
}
get(key) {
return this.client.get(key);
}
set(key, value, ttl) {
let rp;
if (ttl === 0) {
rp = this.client.set(key, value);
}
else {
rp = this.client.set(key, value)
.then(function(res) {
this.client.expire(key, ttl);
});
}
return rp;
}
}
module.exports = new Redis;
I'll probably include a data_init() method to check and preload an initial key/value structure on first connect.

Frequent requests with React and Express GraphQL

I'm creating a website using React and Express GraphQL, with a MariaDB database. I've come across a problem with making frequent requests/subscriptions to the GraphQL API, however.
After the page in question loads in React, I call this:
componentDidMount() {
this.setState({
waitingTimer: setInterval(this.queryWaiting.bind(this), 1000)
});
}
where queryWaiting is a function that carries out a fetch request to my GraphQL server at localhost:3000/graphql. That URL itself is a proxy defined in my setupProxy.js file which proxies the URL from localhost:4000/graphql, so that I don't need to use CORS.
It's worth noting that I also clearInterval this waitingTimer in componentWillUnmount.
In my GraphQL server file, which I'm simply running with node, I set up my server like this:
var app = express();
app.use("/graphql", graphqlHTTP({
schema: schema,
rootValue: root,
graphiql: true,
}));
app.listen(4000);
Where schema is a valid schema, and root is my root resolver.
The API query that I'm repeatably calling is called getWaitingCount, and it acts like this:
getWaitingCount: () => {
return new Promise((resolve, reject) => {
var currentTime = Date.now() / 1000;
if (cachedWaitingTime + cachedWaitingInterval > currentTime) {
return cachedWaiting;
}
connection.query("SELECT COUNT(*) FROM profiles WHERE isWaiting=1;", function (error, results, fields) {
if (error)
reject(error);
else {
// Update cache values
cachedWaiting = results[0]["COUNT(*)"];
cachedWaitingTime = Date.now() / 1000;
resolve(cachedWaiting);
}
});
});
}
I've implemented some caching to reduce server load. The caching variables are defined elsewhere.
The problem I experience is that, after between 2-10 seconds, the POST requests for this specific query from my React app stop getting resolved, and I have to reload the page for them to start getting accepted again. Weirdly, other requests go through fine.
I've tried various solutions, such as switching to using the proxy method I'm using currently, and I've also considered the possibility of a built-in anti-DDOS mechanism in the Express code, but I can't find any evidence for that.
Any help with this issue is very much appreciated.
Turns out the best solution is to type out and explain your code :)
This code:
if (cachedWaitingTime + cachedWaitingInterval > currentTime) {
return cachedWaiting;
}
is inside a promise, so really, it should do:
resolve(cachedWaiting);
instead of returning it.
That is all. I am currently kicking myself.

Resources