We use Knex with generic pool as our Query Builder and Pool Manager for our Oracle 11.2 Database.
The problem we are facing is that some time Knex / generic-pool starts to accumulate connections and cant recycle them.
I tried to pass some parameters to Knex / Generic Pool to make them kill connections after some point, but looks like it did not worked out.
Packges version:
Knex: v0.13.0
Oracledb: v1.13.1
Generic Pool: v2.5.4
Knex configuration:
{
client: 'oracledb',
connection: {
user: DB_USER,
password: DB_PASSWORD,
host: `${DB_HOST}:${DB_PORT}`,
database: DB_NAME
},
debug: true,
fetchAsString: ['number', 'clob'],
acquireConnectionTimeout: 843600000,
pool: {
min: 2,
max: 150,
acquireTimeoutMillis: 100000,
evictionRunIntervalMillis: 120000,
maxWaitingClients: 100,
idleTimeoutMillis: 100000
}
}
Openshift print with environment variable DEBUG="Knex:*" showing a lot of clients waiting for connection
Try knex 0.14.2 some pool related problems were fixed in that. Also try to add some debug information when transactions are created/committed/rolled back. Open transactions will take connection from pool and does bot release it until transaction is ended. You can get information about pool and transacions bu running app with DEBUG=knex:* environment variable set.
Related
I am running a Node.js (Node v14.15.4) application with Sequelize (v6.6.2) as an ORM connecting to a PostgreSQL database and after several operations, i find that there is about 35 idle processes on my pgadmin dashboard, see below image for reference:
In my index file, i have setup Sequelize like below:
sequelize = new Sequelize(process.env[config.use_env_variable], {
logging: false,
pool: {
max: 15,
min: 0,
acquire: 30000,
idle: 10000,
evict: 10000
}
});
is there something that i am missing here? Because i understand that evict instructs sequelize to remove any idle processes after the specified amount of time.
The connections with application name as "pgAdmin 4 - CONN-*" are used by the query tool of pgAdmin. Check if you have any open instances of query tool.
In order to improve the pool design of an application, I would like to be notified (ideally with an event) when the pool size of an application is reached. This way, I can add a log, if this log occurs too often I will increase the pool size.
With a mongo client initialized this way :
const client = new MongoClient(url, {
poolSize: 10,
});
Is there a way to be notified when the 10 connections are reached within my application ?
Use connection pool events. These should be implemented by all recent MongoDB drivers.
Node documentation
Documentation/example in Ruby
For your question you would track the pool size using ConnectionCheckOut*/ConnectionCheckedIn events if your driver does not expose the pool size or the pool at all directly.
I'm currently trying to connect a Node JS app to a single database that I created using the Azure SQL Database. In order to connect to the database, I use Sequelize. In order to do that, I set up the firewall to accept my IP address as explained here, and I configured a config.json file like so
"username": "SERVER_ADMIN_NAME#MY_IP_ADDRESS",
"password": "ADMIN_PASSWORD",
"database": "DATABASE_NAME",
"host": "SERVER_NAME",
"port": 1433,
"dialect": "mssql",
"dialectOptions": {
"options": {
"encrypt": true
}
}
However, after running the application it fails to connect to the database and returns the following message
"Cannot open server '211' requested by the login. Client with IP address 'MY_IP_ADDRESS' is not allowed to access the server. To enable access, use the Windows Azure Management Portal or run sp_set_firewall_rule on the master database to create a firewall rule for this IP address or address range. It may take up to five minutes for this change to take effect."
I've already waited for more than 5 minutes but the result is still the same. Now, the first thing that came into my mind was how I provided the values for the config.json file. However, after checking the sys.database_firewall_rules using the following query
SELECT * FROM sys.database_firewall_rules;
The table was EMPTY. From here on I'm not really sure what I'm supposed to do. I was wondering if anybody could point out what I was missing? Thanks in advance!
You should not connect to Azure SQL Database using the IP address because it can change any time.
Could you try a connection like below using tedious driver?
var Sql = require('sequelize');
var sql = new Sql('dbname', 'UserName#server', 'password', {
host: 'server.database.windows.net',
dialect: 'mssql',
driver: 'tedious',
options: {
encrypt: true,
database: 'dbname'
},
port: 1433,
pool: {
max: 5,
min: 0,
idle: 10000
}
});
Make sure you are adding your public IP address not your local IP address to the firewall rules. To verify the firewall rules you have added already, please run the following query:
SELECT * FROM sys.firewall_rules;
The above query shows rules at the server level. You have created your rules at that level.
I am trying to connect node.js app to MongoDB having replica set but it's throwing an error when any write operations are performed.
It throws MongoError: not master.
It tries to write on secondary mongo instances.
I have the options as { db: { readPreference: secondaryPreferred } } and passing it to the function MongoClient.connect in the node.js code using native Mongo Driver.
The URL used to connect looks like mongodb://admin:pass#host_one:27017,host_two:27017,host_three:27017/dbName
Any help would be really appreciated.
Did you add in your replicaSet name?
mongodb://admin:pass#host_one:27017,host_two:27017,host_three:27017/dbName?replicaSet=my-replica-set
replicaSet=name
The driver verifies that the name of the replica set it connects to
matches this name. Implies that the hosts given are a seed list, and
the driver will attempt to find all members of the set. No default
value.
If this is not set it will be treated as a standalone node.
Maybe your replica set configuration is not correct.
To check the configuration run the rs.conf() command in your mongo servers. You need to have a mongo host running as primary member.
MongoError: Not master
This error seems like your primary member of replica set is not configured properly.
You can confirm this by entering into mongo shell of the host_one. If mongo shell prompt doesn't show PRIMARY, then it's not configured properly.
Mongo shell prompt of host_two and host_three should show SECONDARY after proper configuration.
Important : Run rs.initiate() on just one and only one mongod instance for the replica set.
You can execute this command on the primary member to make the configuration work properly.
rs.initiate();
cfg = {
_id: 'rs0',
members: [{
_id: 0,
host: 'host_one:27017',
priority: 2
}, {
_id: 1,
host: 'host_two:27017',
priority: 1
}, {
_id: 2,
host: 'host_three:27017',
priority: 1
}]
};
cfg.protocolVersion = 1;
rs.reconfig(cfg, {
force: true
});
Please note that priority value indicates the relative eligibility of a member to become a primary.
Specify higher values to make a member more eligible to become primary, and lower values to make the member less eligible. A member with a priority of 0 is ineligible to become primary.
You can again check your replica set configuration using this command
rs.conf()
Read preference is not applicable to writes. Writes must always be performed on the primary.
You should be connecting to replica set instead of directly to an individual node. See node.js mongodb how to connect to replicaset of mongo servers
I need some clarification about what the pool is and what it does. The docs say Sequelize will setup a connection pool on initialization so you should ideally only ever create one instance per database.
var sequelize = new Sequelize('database', 'username', 'password', {
host: 'localhost',
dialect: 'mysql'|'mariadb'|'sqlite'|'postgres'|'mssql',
pool: {
max: 5,
min: 0,
idle: 10000
},
// SQLite only
storage: 'path/to/database.sqlite'
});
When your application needs to retrieve data from the database, it creates a database connection. Creating this connection involves some overhead of time and machine resources for both your application and the database. Many database libraries and ORM's will try to reuse connections when possible, so that they do not incur the overhead of establishing that DB connection over and over again. The pool is the collection of these saved, reusable connections that, in your case, Sequelize pulls from. Your configuration of
pool: {
max: 5,
min: 0,
idle: 10000
}
reflects that your pool should:
Never have more than five open connections (max: 5)
At a minimum, have zero open connections/maintain no minimum number of connections (min: 0)
Remove a connection from the pool after the connection has been idle (not been used) for 10 seconds (idle: 10000)
tl;dr: Pools are a good thing that help with database and overall application performance, but if you are too aggressive with your pool configuration you may impact that overall performance negatively.
pool is draining error
I found this thread in my search for a Sequalize error was giving my node.js app: pool is draining. I could not for the life of me figure it out. So for those who follow in my footsteps:
The issue was that I was closing the database earlier than I thought I was, with the command sequelize.closeConnections(). For some reason, instead of an error like 'the database has been closed`, it was instead giving the obscure error 'pool is draining'.
Seems that you can try to put pool to false to avoid having the pool bing created. Here is the API details table :http://sequelize.readthedocs.org/en/latest/api/sequelize/
[options.pool={}]
Object
Should sequelize use a connection pool.
Default is true