currently I'm designing an app on top of Sequelize using NodeJS/TypeScript, and I'm wondering if it can cause performance issues not closing a connection.
For instance, in a micro service, I need data from 1 entity.
const resolver = async (,,{db}) => {
const entity1 = await db.models.Entity1.findOne()
return entity1
}
Is it required to close the connection after having called findOne?
My understanding is that the following config defines a number of concurrent connections and idle is a parameter making the connection manager closing the connection of idle ones:
module.exports = {
development: {
host: 'db.sqlite',
dialect: 'sqlite',
pool: {
max:5,
min:0,
idle:10000
}
},
test: {
host: 'test.sqlite',
dialect: 'sqlite',
pool: {
max:5,
min:0,
idle:10000
}
}
}
Any advice is welcome
Sequelize maintains an internal database connection pool, that's what the pool parameters are for, so this isn't necessary. Each call actually borrows a connection temporarily and then returns it to the pool when done.
Closing that connection manually may poison the pool and cause performance issues.
If you don't close the Sequelize connection, the micro-service will still run until the connection got timed out (idle time pool parameter).. I suggest to close Sequelize connection, at least in micro-services..
Related
We have the following on node-postgres documentation:
// number of milliseconds to wait before timing out when connecting a new client
// by default this is 0 which means no timeout
connectionTimeoutMillis?: int,
And then, a little bit later on the same documentation:
You must call the releaseCallback or client.release (which points to the releaseCallback) when you are finished with a client. If you forget to release the client then your application will quickly exhaust available, idle clients in the pool and all further calls to pool.connect will timeout with an error or hang indefinitely if you have connectionTimeoutMillis configured to 0.
So, I was expecting that if I set connectionTimeoutMillis to 1000, then, after 1s idle, it should automatically release the connection, even if I don't call client.release().
But running the code below it sits idle forever on PostgreSQL:
// test.js
// PostgreSQL v14.2
// Node.js v16.15.1
// node-postgres (pg) v8.7.3
const { Pool } = require('pg')
const pool = new Pool({
user: 'postgres',
password: 'postgres',
host: `localhost`,
port: 5432,
database: 'app_dev',
max: 10,
connectionTimeoutMillis: 1000,
idleTimeoutMillis: 1000
})
;(async function() {
const client = await pool.connect()
const {rows} = await client.query('SELECT NOW()')
console.log(rows[0])
})()
Am I missing something?
I think this is what you are missing
When your app is calling
const client = await pool.connect()
This is where the connectionTimeoutMillis is valid. i.e. it is the amount of time in milliseconds to wait when trying to get a client (a connection) to your PostgresDB. If you set this to 1000ms it means, if I haven't created a new client or returned an existing one from the pool within 1000ms - throw an error. So since you are after obtaining the client - that will not release it or implicitly return your client to the pool as you HAVE the client - therefore there is no waiting on a connection to be established.
The idleTimeoutMillis time refers to the amount of time that the client (connection) can sit in the pool without being called by your application code before being destroyed.
For example, say your pool has a size of 1 client. Your app uses the client and then releases it back to the pool. If your app doesn't request another client for more than 1000ms i.e. the client is idle in the pool for a time equal or greater than the idleTimeoutMillis - it will be destroyed.
I'm trying to initialize a pool of SQL Server connections for my nodejs web application to use. I've set the config to create 10 min connections but when I start the app with the below code. I only have 0 available connections which doesn't allow me to begin any transactions.
Any help would be greatly appreciated!
test.js
const sql = require('mssql');
require('dotenv').config();
const appPool = new sql.ConnectionPool({
user: process.env.DB_USER,
password: process.env.DB_PWD,
database: process.env.DB_NAME,
server: process.env.DB_HOST,
pool: {
min: 10,
max: 100,
acquireTimeoutMillis: 15000,
},
options: {
encrypt: true,
trustServerCertificate: false
}
});
appPool.connect().then(pool => {
console.log(`SERVER: Connected to the db and ${pool.available} connections are available!`);
});
Output
MINGW64 ~/Desktop/React Projects/dummy-project (master)
$ node test.js
SERVER: Connected to the db and 0 connections are available!
node-mssql uses tarn.js for connection pooling. The connections are not created preemptively. Instead, they're created as and when the connections are requested. Once the number of concurrent requests for connections exceed pool.min, the available connections in the pool stays at that level.
So, doing a transaction or running a query shouldn't be a problem, node-mssql will create the connection and fulfil the request.
If you really want to have pool.min connections in your pool, you can write a small script to fire some concurrent queries on your database which will warm up the pool for you.
I use mysql2 module in my NodeJS project.
I understand the concept of database pooling in mysql2 module
(https://www.npmjs.com/package/mysql2).
Before using pool, I used regular connection with mysql2.createConnection() function,
but after some time I got 'PROTOCOL_CONNECTION_LOST' error.
My code is:
db.js:
const mysql = require('mysql2');
const pool = mysql.createPool({
host: 'sql server',
user: 'username',
database: 'database',
password: 'pass',
port: 3306,
connectionLimit: 10,
queueLimit: 0
});
module.exports = pool;
And how I using it:
const db = require('db');
db.query(...);
The query() and execute() functions on pool instance automatically call release() function of pool instance, and that is very good (because I don't need to write that command manually after any query).
But I need to understand: if I work like that (with pool, instead of without pool), there is a guarentee
'PROTOCOL_CONNECTION_LOST' will not be thrown? Should I handle that or the pooling mechanism does it automatically?
I ask because I saw on the internet some code that, for example, re-create the connection.
With pool, I need to re-create the connection sometime?
Thanks a lot!
I'm trying to create an API using nodeJS, express and azure-mobile-apps to do some data synchronisation between an Ionic3 mobile app (which use an SQLite local database) and a Microsoft SQL Database.
My API has to create a synchronisation connection for each mobile application. Each application will be linked to a distant database. For example, if user_01 wants to synchronise his data, he's going to be linked to his client_01 database. So each time it'll have to, the API will create a new process running on a different port.
here is an example : https://zupimages.net/up/19/36/szhu.png
The problem is that i'm not able to create more than one connection with azure-mobile-apps. The first one always works, but the second, third etc are still using the first connection that i have instantiated. I've looked into the app stack and everything seems fine.
Is that an issue with azure-mobile-app, or did I misunderstand something with express ?
Thanks for your responses !
var azureMobileApps = require('azure-mobile-apps');
var express = require('express');
module.exports = {
async createConnection(client) {
try {
let app = express();
mobileApp = azureMobileApps({
homePage: true,
swagger: true,
data: {
server: '****',
user: client.User,
password: client.Password,
port: '1443',
database: client.Database,
provider: 'mssql',
dynamicSchema: false,
options: {
encrypt: false
}
}
});
await mobileApp.tables.import('./tables');
await mobileApp.tables.initialize();
app.listen(global.portCounter);
app.use(mobileApp);
console.log(app._router.stack);
console.log('Listening on port ',global.portCounter);
global.portCounter++;
} catch (error) {
console.log(error);
}
}
}
It's working now. The thing is, it's impossible to do multiple connection with the azure-mobile-apps SDK for nodeJS.
I had to use worker-thread which seems to isolate the memory in a sub-proccess.
Hope it can help somebody one day
There are a lot of examples of graceful stop for expressjs, how can I achieve the same for koajs?
I would like to disconnect database connections as well
I have a mongoose database connection, and 2 oracle db connection (https://github.com/oracle/node-oracledb)
I created an npm package http-graceful-shutdown (https://github.com/sebhildebrandt/http-graceful-shutdown) some time ago. This works perfectly with http, express and koa. As you want to add also your own cleanup stuff, I modified the package, so that you now can add your own cleanup function, that will be called on shutdown. So basically this package handles all http shutdown things plus calling your cleanup function (if provided in the options):
const koa = require('koa');
const gracefulShutdown = require('http-graceful-shutdown');
const app = new koa();
...
server = app.listen(...); // app can be an express OR koa app
...
// your personal cleanup function - this one takes one second to complete
function cleanup() {
return new Promise((resolve) => {
console.log('... in cleanup')
setTimeout(function() {
console.log('... cleanup finished');
resolve();
}, 1000)
});
}
// this enables the graceful shutdown with advanced options
gracefulShutdown(server,
{
signals: 'SIGINT SIGTERM',
timeout: 30000,
development: false,
onShutdown: cleanup,
finally: function() {
console.log('Server gracefulls shutted down.....')
}
}
);
I have answered a variation of "how to terminate a HTTP server" many times on different node.js support channels. Unfortunately, I couldn't recommend any of the existing libraries because they are lacking in one or another way. I have since put together a package that (I believe) is handling all the cases expected of graceful HTTP server termination.
https://github.com/gajus/http-terminator
The main benefit of http-terminator is that:
it does not monkey-patch Node.js API
it immediately destroys all sockets without an attached HTTP request
it allows graceful timeout to sockets with ongoing HTTP requests
it properly handles HTTPS connections
it informs connections using keep-alive that server is shutting down by setting a connection: close header
it does not terminate the Node.js process
Usage with Koa:
import Koa from 'koa';
import {
createHttpTerminator,
} from 'http-terminator';
const app = new Koa();
const server = app.listen();
const httpTerminator = createHttpTerminator({
server,
});
await httpTerminator.terminate();
To make sure the Oracle DB connections are closed nicely, you can use a connection pool and call pool.close() with a drainTime of 0 or greater. This will let the app relatively cleanly interrupt any operation that is currently using a connection. It allows freeing the DB end of the connections without the DB waiting for whatever timeout period to expire before it cleans itself up. Even with two connections this is a solution I'd look at, since it doesn't matter that the pool is small. You may need to set the Oracle Net out-of-band break detection as well, see Connections and High Availability.
Modern versions of node have support for AbortController, so no need for external libraries. A Simple example:
const app = new Koa();
const server = http.createServer(app.callback());
const controller = new AbortController();
server.listen({
host: 'localhost',
port: 80,
signal: controller.signal
});
// middleware... etc.
app.use(async (ctx) => {
ctx.body = 'Hello World';
});
// Later, when you want to close the server.
controller.abort();