I have a server which I've written using Express and node-postgres (pg). It creates its own DB pool:
const dbPool = new pg.Pool(dbConfig);
and runs SQL queries directly using this connection.
Now I'm adding a new table and corresponding REST API. I'd like to use sequelize and epilogue to reduce the boilerplate. Unfortunately, sequelize wants to create its own database connection pool:
const sequelize = new Sequelize(database, user, password, config);
Is it possible to re-use the existing connection pool or otherwise share it between my existing pg code and my new sequelize code?
Sequelize does not offer the option to pass a custom pool, but you can pass options that will get used to create the sequelize pool, such as min and max connections.
What I would do in your case is to check your total DB connection count, and make a repartition based on the expected usage of your two pools.
For example if you have 20 connections max on your database:
const dbPool = new pg.Pool({
max: 10
});
const sequelize = new Sequelize(/* ... */, {
// ...
pool: {
max: 10,
min: 0,
acquire: 30000,
idle: 10000
}
});
I would also suggest using environment variables to set the max connection on your sequelize pool and nod-pg pool, so that you could adapt easily your repartition if needed.
Related
I have a database which is shared amongst multiple tenants/users. However, I want to add row-level-security protection so that any given tenant can only see those rows that belong to them.
As such, for each tenant I have a user in PostgreSQL, such as "client_1" and "client_2". In each table, there is a column "tenant_id", the default value of which is "session_user".
Then, I have row level security as such:
CREATE POLICY policy_warehouse_user ON warehouse FOR ALL
TO PUBLIC USING (tenant_id = current_user);
ALTER TABLE warehouse ENABLE ROW LEVEL SECURITY;
This works great, and if I set the user "SET ROLE client_1" I can only access those rows in which the tenant_id = "client_1".
However, I am struggling with how to best set this up in the Node JS back-end. Imortantly, for each tenant, such as "client_1", there can be multiple users connected. So several users on our system, all who work at company X, will connect to the database as "client_1".
What I am currently doing is this:
let config = {
user: 'test_client2',
host: process.env.PGHOST,
database: process.env.PGDATABASE,
max: 10, //default value
password: 'test_client2',
port: process.env.PGPORT,
}
const pool = new Pool(config);
const client = await pool.connect()
await client.query('sql...')
client.release();
I feel like this might be a bad solution, especially since I am creating a new Pool each time a query is executed. So the question is, how can I best ensure that each user executes queries in the database using the ROLE that corresponds to their tenant?
Maybe you can have a setupDatabase method that returns the pool for your app this will be called once at app bootstrap:
function setUpDatabase {
let config = {
user: 'test_client2',
host: process.env.PGHOST,
database: process.env.PGDATABASE,
max: 10, //default value
password: 'test_client2',
port: process.env.PGPORT,
}
const pool = new Pool(config);
return pool
}
and then when you identify the tenant before executing the query you set the role
await client.query('set role $tenant', currentTenant);
// my assumption is that next line will use the role you set before
await client.query('select * from X where Y');
This is just a suggestion, I haven't tested it.
DB Team inserts new data into a table. If new data is inserted I do need to send messages.
Is there any way I could track new data using Nodejs. There is no specific duration for data insertion.
If your DB is remoteDB then you do require full-duplex connection.
After getting successful connections from the telnet from both servers do as follows:
const connection = await oracledb.getConnection({
user : user,
password : password,
connectString : connectString,
events : true
});
function myCallback(message) {
console.log('CQN Triggered');
}
const options = {
callback: myCallback, // method called by notifications
sql: `SELECT ID FROM table where STATUS= 0`, // query
timeout: 600,
qos : oracledb.SUBSCR_QOS_ROWIDS, // SUBSCR_QOS_QUERY: generate notifications when new rows with STATUS= 0 are found
clientInitiated : true //By Default it's false.
};
await connection.subscribe('mysub', options);
See the node-oracledb documentation on Continuous Query Notification, which lets your Node.js app be notified if data has changed in the database.
There are examples in cqn1.js and cqn2.js. When using Oracle Database and Oracle client libraries 19.4, or later, you'll find testing easier if you set the optional subscriptions attribute clientInitiated property to true:
const connection = await oracledb.getConnection({
user : "hr",
password : mypw, // mypw contains the hr schema password
connectString : "localhost/XEPDB1",
events : true
});
function myCallback(message) {
console.log(message);
}
const options = {
sql : `SELECT * FROM mytable`, // query of interest
callback : myCallback, // method called by notifications
clientInitated : true
};
await connection.subscribe('mysub', options);
You could also look at Advanced Queuing as another way to propagate messages, though you would still need to use something like a table trigger or CQN to initiate an AQ message.
I feel like I've tried everything. I have a cloud function that I am trying to connect to Cloud SQL (PostgreSQL engine). Before I do so, I pull connection string info from Secrets Manager, set that up in a credentials object, and call a pg (package) pool to run a database query.
Below is my code:
Credentials:
import { Pool } from 'pg';
const credentials: sqlCredentials = {
"host":"127.0.0.1",
"database":"myFirstDatabase",
"port":"5432",
"user":"postgres",
"password":"postgres1!"
}
const pool: Pool = new Pool(credentials);
await pool.query(`select CURRENT_DATE;`).catch(error => console.error(`error in pool.query: ${error}`));
Upon running the cloud function with this code, I get the following error:
error in pool.query: Error: connect ECONNREFUSED 127.0.0.1:5432
I have attempted to update the host to the private IP of the Cloud SQL instance, and also update the host to the Cloud SQL instance name on this environment, but that is to no avail. Any other ideas?
Through much tribulation, I figured out the answer. Given that there is NO documentation on how to solve this, I'm going to put the answer here in hopes that I can come back here in 2025 and see that it has helped hundreds. In fact, I'm setting a reminder in my phone right now to check this URL on November 24, 2025.
Solution: The host must be set as:
/cloudsql/<googleProjectName(notId)>:<region>:<sql instanceName>
Ending code:
import { Pool } from 'pg';
const credentials: sqlCredentials = {
"host":"/cloudsql/my-first-project-191923:us-east1:my-first-cloudsql-inst",
"database":"myFirstDatabase",
"port":"5432",
"user":"postgres",
"password":"postgres1!"
}
const pool: Pool = new Pool(credentials);
await pool.query(`select CURRENT_DATE;`).catch(error => console.error(`error in pool.query: ${error}`));
My app server uses a database per tenant and I need to connect to many databases at once. To do so with Sequelize I maintain a db pool where I add a connection to a Map:
addConnection(dbKey) {
const sequelize = new Sequelize(
dbKey,
"username",
"password",
{
host: localhost,
port: 5000,
schema: 'main',
dialect: 'postgres'
}
);
const models = {
sequelize: sequelize,
customerData: CustomerData(sequelize, DataTypes)
}
this.dbPool.set(dbKey, models);
}
This allows me to retrieve sequelize instances as needed. Currently I do not see any support for single sequelize instances managing connections to multiple databases. And because I am maintaining my own Map of connections it seems that I would need to perform the task of releasing idle connections.
I would like to close idle connections and drop them from the Map if they are not used for a certain interval. I have considered a few ways of doing so but do not know which are feasible. It really comes down to 2..
1 - set options.pool.idle for the Sequelize instance.
> The maximum time, in milliseconds, that a connection can be idle before being released.
AFAIK this does not do what I want, this only sets a timeout when there is a connection pool for the same DB so this clearly won't work.
2 - Use SetInterval to close connections and drop their reference from the Map object when idle.
a) When opening connection, set a timestamp along with connection.
b) Whenever the sequelize instance is used to make a query, update the timestamp.
c) Every minute or so, close and free any connections that have been idle for longer than some time period.
To do so it would be great if there were some internal field in Sequelize or getter that would let me see when the connection was last used. Looking at the API for the Sequelize and model instances, I do not see such a field or method. Is there a way to determine the time a connection was last used in the Sequelize library?
When you create a new Sequelize instance it creates a pool in the background by default.
You can look the options.pool object that is available for the instance initialization.
https://sequelize.org/api/v6/class/src/sequelize.js~sequelize#instance-constructor-constructor :
const sequelize = new Sequelize(
dbKey,
"username",
"password",
{
host: localhost,
port: 5000,
schema: 'main',
dialect: 'postgres',
pool: {
idle: 10000, // milliseconds
evict: 20000, // milliseconds
}
}
);
It has default values as follow:
public constructor(database: string, username: string, password: string, options: object)
Name
Type
Attribute
Description
options.pool
object
optional
sequelize connection pool configuration
options.pool.max
number
optional, default: 5
Maximum number of connection in pool
options.pool.min
number
optional, default: 0
Minimum number of connection in pool
options.pool.idle
number
optional, default: 10000
The maximum time, in milliseconds, that a connection can be idle before being released.
options.pool.acquire
number
optional, default: 60000
The maximum time, in milliseconds, that pool will try to get connection before throwing error
options.pool.evict
number
optional, default: 1000
The time interval, in milliseconds, after which sequelize-pool will remove idle connections.
options.pool.validate
Function
optional
A function that validates a connection. Called with client. The default function checks that client is an object, and that its state is not disconnected
options.pool.maxUses
number
optional, default: Infinity
The number of times a connection can be used before discarding it for a replacement, used for eventual cluster rebalancing.
Therefore it will automatically release and acquire connections as per the default values above or the values that you provide.
So, lets say there's no query executed for 10000 milliseconds then the connection is released as per default value of options.pool.idle.
Therefore when no query is triggered for some time then the sequelize instance is just present there without any live DB connection. So if that's fine you can keep it as it is. Or can use connection hooks below to identify the connection and maintain a counter to initialize/remove Sequelize instances from your Map.
You can look at the connection hooks docs here:
https://sequelize.org/master/manual/hooks.html#connection-hooks
I know that this question was asked already, but it seems that some more things have to be clarified. :)
Database is designed in the way that each user has proper privileges to read documents, so the connection pool needs to have a connection with different users, which is out of connection pool concept. Because of the optimization and the performance I need to call so-called "user preparation" which includes setting session variables, calculating and caching values in a cache, etc, and after then execute queries.
For now, I have two solutions. In the first solution, I first check that everything is prepared for the user and then execute one or more queries. In case it is not prepared then I need to call "user preparation", and then execute query or queries. With this solution, I lose a lot of performance because every time I have to do the checking and so I've decided for another solution.
The second solution includes "database pool" where each pool is for one user. Only at the first connection useCount === 0 (I do not use {direct: true}) I call "user preparation" (it is stored procedure that sets some session variables and prepares cache) and then execute sql queries.
User preparation I’ve done in the connect event within the initOptions parameter for initializing the pgPromise. I used the pg-promise-demo so I do not need to explain the rest of the code.
The code for pgp initialization with the wrapper of database pooling looks like this:
import * as promise from "bluebird";
import pgPromise from "pg-promise";
import { IDatabase, IMain, IOptions } from "pg-promise";
import { IExtensions, ProductsRepository, UsersRepository, Session, getUserFromJWT } from "../db/repos";
import { dbConfig } from "../server/config";
// pg-promise initialization options:
export const initOptions: IOptions<IExtensions> = {
promiseLib: promise,
async connect(client: any, dc: any, useCount: number) {
if (useCount === 0) {
try {
await client.query(pgp.as.format("select prepareUser($1)", [getUserFromJWT(session.JWT)]));
} catch(error) {
console.error(error);
}
}
},
extend(obj: IExtensions, dc: any) {
obj.users = new UsersRepository(obj);
obj.products = new ProductsRepository(obj);
}
};
type DB = IDatabase<IExtensions>&IExtensions;
const pgp: IMain = pgPromise(initOptions);
class DBPool {
private pool = new Map();
public get = (ct: any): DB => {
const checkConfig = {...dbConfig, ...ct};
const {host, port, database, user} = checkConfig;
const dbKey = JSON.stringify({host, port, database, user})
let db: DB = this.pool.get(dbKey) as DB;
if (!db) {
// const pgp: IMain = pgPromise(initOptions);
db = pgp(checkConfig) as DB;
this.pool.set(dbKey, db);
}
return db;
}
}
export const dbPool = new DBPool();
import diagnostics = require("./diagnostics");
diagnostics.init(initOptions);
And web api looks like:
GET("/api/getuser/:id", (req: Request) => {
const user = getUserFromJWT(session.JWT);
const db = dbPool.get({ user });
return db.users.findById(req.params.id);
});
I'm interested in whether the source code correctly instantiates pgp or should be instantiated within the if block inside get method (the line is commented)?
I've seen that pg-promise uses DatabasePool singleton exported from dbPool.js which is similar to my DBPool class, but with the purpose of giving “WARNING: Creating a duplicate database object for the same connection”. Is it possible to use DatabasePool singleton instead of my dbPool singleton?
It seems to me that dbContext (the second parameter in pgp initialization) can solve my problem, but only if it could be forwarded as a function, not as a value or object. Am I wrong or can dbContext be dynamic when accessing a database object?
I wonder if there is a third (better) solution? Or any other suggestion.
If you are troubled by this warning:
WARNING: Creating a duplicate database object for the same connection
but your intent is to maintain a separate pool per user, you can indicate so by providing any unique parameter for the connection. For example, you can include custom property with the user name:
const cn = {
database: 'my-db',
port: 12345,
user: 'my-login-user',
password: 'my-login-password'
....
my_dynamic_user: 'john-doe'
}
This will be enough for the library to see that there is something unique in your connection, which doesn't match the other connections, and so it won't produce that warning.
This will work for connection strings as well.
Please note that what you are trying to achieve can only work well when the total number of connections well exceeds the number of users. For example, if you can use up to 100 connections, with up to 10 users. Then you can allocate 10 pools, each with up to 10 connections in it. Otherwise, scalability of your system will suffer, as total number of connections is a very limited resource, you would typically never go beyond 100 connections, as it creates excessive load on the CPU running so many physical connections concurrently. That's why sharing a single connection pool scales much better.