options.pool.validate in sequelize node js - node.js

Postgres sequence name - post_seq
SELECT query to get the next sequence - select nextval(post_seq)
Using sequelize v5.x
pool configuration -
{
max: 10,
min: 1,
acquire: 30000,
idle: 10000,
validate: async pgClient => {
const result = await pgClient.query('SELECT pg_is_in_recovery()');
const isReadOnly = result.rows[0].pg_is_in_recovery;
console.log(isReadOnly, 'isReadOnly:src/utils/db.js')
return !isReadOnly;
}
}
Expectation -
options.pool.validate is called for all the queries running in the application including the above SELECT query to get the next sequence id
What's happening -
options.pool.validate is called only for non-SELECT queries
I am assuming this is the default behavior of sequelize. If that's the case, what would be the other way to force SELECT queries to use only writable connection? The reason for this expectation is that during AWS RDS Failover, the reader connection can't be used to run the above SELECT query since nextval() isn't just a select query. If there's a way to call options.pool.validate for this SELECT query, sequelize would discard that connection before making this nextval() query because of pool configuration used. As of now, the error I am getting in the server logs is as follows -
SequelizeDatabaseError: cannot execute nextval() in a read-only transaction\n
Couple of other points to note -
I am connecting to cluster writer endpoint in the nodejs application
I am using 'SELECT pg_is_in_recovery()' query to check whether the connection being used is read-only. If it's read-only, the connection is discarded by sequelize.
I have tried using useMaster:true in the pool config and it doesn't seem to help during the failover scenario. Probably, this is useful mainly in case of replication rather than a DR setup.

Related

node-mysql2: resultset not reflecting the latest results

I'm using node-mysql2 with a connection pool and a connection limit of 10. When I restart the application, the results are good - they match what I have on the db. But when I start inserting new records and redo the same select queries, then I get intermittent results missing the latest record I just added.
If I do check the database directly, I can see the records I just added through my application. It's only the application that cannot see it somehow.
I think this is a bug, but here's how I have my code setup:
module.exports.getDB = function (dbName) {
if (!(dbName in dbs)) {
console.log(`Initiating ${dbName}`);
let config = dbConfigs[dbName];
dbs[dbName] = mysql.createPool({
host: config.host,
port: config.port || 3306,
user: config.user,
password: config.password,
connectionLimit: 10,
database: config.database,
debug: config.debug
});
}
return dbs[dbName]; // I just initialize each database once
};
This is my select query:
let db = dbs.getDB('myDb');
const [rows] = await db.query(`my query`);
console.log(rows[0]); // this one starts to show my results inconsistently once I insert records
And this is my insert query:
module.exports = {
addNote: async function(action, note, userID, expID) {
let db = dbs.getDB('myDb');
await db.query(`INSERT INTO experiment_notes (experiment_id, action, created_by, note)
VALUES (?, ?, ?, ?)`, [expID, action, userID, note]);
}
};
If I set the connectionLimit to 1, I cannot reproduce the problem... at least not yet
Any idea what I'm doing wrong?
Setting your connection_limit to 1 has an interesting side-effect: it serializes all access from your node program to your database. Each operation, be it INSERT or SELECT, must run to completion before the next one starts because it has to wait for the one connection in the pool to free up.
It's likely that your intermittently missing rows are due to concurrent access to your DBMS from different connections in your pool. If you do a SELECT from one connection while MySQL is handling the INSERT from another connection, the SELECT won't always find the row being inserted. This is a feature. It's part of ACID (atomicity, consistency, isolation, durability). ACID is vital to making DBMSs scale up.
In more complex applications than the one you showed us, the same thing can happen when you use DBMS transactions and forget to COMMIT them.
Edit Multiple database connections, even connections from the same pool in the same program, work independently of each other. So, if you're performing a not-yet-committed transaction on one connection and a query on another connection, the query will (usually) reflect the database's state before the transaction started. The query cannot force the transaction to roll back unless it somehow causes a deadlock. But deadlocks generate error messages; you probably are not seeing any.
You can sometimes control what a query sees by preceding it, on the same connection, with SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; . That can, on a busy DBMS, improve query performance a little bit, and prevent some deadlocks, as long as you're willing to have your query see only part of a transaction. I use it for historical queries (what happened yesterday). It's documented here. The default, the one that explains what you see, is SET TRANSACTION LEVEL REPEATABLE READ;
But, avoid that kind of isolation-level stuff until you need it. (That advice comes under the general heading of "too smart is dumb.")

Unhandled rejection error: relation 'dbname' does not exist

I have a database created through Postgresql. I'm attempting to connect to the database via Knex.js.
In my text editor (atom) I have required knex
const knex = require('knex');
Second - I have begun to initialize the library like so:
const postgres = knex ({ client: 'pg', connection: { host: '127.0.0.1', port: '5432', user: 'is myself', password: '', database: 'mediumrare_database' } });
Third - I am attempting to pull data from that db like so:
postgres.select('*').from('mediumrare_database).then(data => {console.log(data)});
Finally, the error message I am receiving is as follows:
Unhandled rejection error: relation "mediumrare_database" does not exist
Your database is named mediumrare_database, and the table you are trying to get datas from is named the same way.
So it seems the problem is that you didn't created any table.
In SQL, your Knex commands would be:
SELECT * FROM mediumrare_database;, which means Return all the datas inside the 'mediumrare_database' table.
You first need to create your database structure (tables to store data) with a CREATE instruction. See https://knexjs.org/#Schema-createTable.
FROM is meant to be used on a table, as you already specified the database to connect to in the connection string.
I can only suggest you learn the basics of SQL before using Knex, which is an SQL Query Builder. You will run into a lot of problems if you don't understand the underlying system and language, because Knex will not teach you that :)
You can check the one of CodeCademy, they always have great ressources.
EDIT: So you created a vinyl_information table (comments on OP). Your Knex command should then be:
postgres
.select('*')
.from('vinyl_information')
.then(data => console.log(data))
I have a Discord server where we help each others, don't hesitate joining it if you need further help on the topic :) https://discord.gg/C2bVzgb

Nodejs + Cassandra driver --- getting error 'unconfigured table' when trying to create materialized view

I'm running on Nodejs 8.9 & the latest Datastax Cassandra driver.
Upon service startup I'm executing 2 queries, one which creates a table (in case is does not exist) and the other creates a materialized view.
The table creation query passes without any issues, but when I execute the query for the materialized view, I get 'unconfigured table' error.
I've tried to debug it, and saw (via terminal) that indeed the table does not appear in Cassandra after the query executes, it appears only after I stop the service entirely. I've tried closing the connection after creating the table and re-creating it, but I still get the same error.
This is how I execute the query:
try{
let respose = await client.execute(query, null, queryOptions);
}catch(error){
throw (error);
}
Changing the CONSISTENCY_POLICY did not help either.
Please advise.
Usually this should happen when the schema isn't in agreement between all nodes. By default driver should wait 10 seconds until agreement is reached. This time is controlled by protocolOptions.maxSchemaAgreementWaitSeconds parameter of the Client - try to increase this parameter & try.
Also, you need to check that your cluster is in agreement - please run nodetool describecluster as described in documentation.

Sync elasticsearch on connection with database - nodeJS

Aim: sync elasticsearch with postgres database
Why: sometimes newtwork or cluster/server break so future updates should be recorded
This article https://qafoo.com/blog/086_how_to_synchronize_a_database_with_elastic_search.html suggests that I should create a separate table updates that will sync elasticsearch's id, allowing to select new data (from database) since the last record (in elasticsearch). So I thought what if I could record elasticsearch's failure and successful connection: if client ponged back successfully (returned a promise), I could launch a function to sync records with my database.
Here's my elasticConnect.js
import elasticsearch from 'elasticsearch'
import syncProcess from './sync'
const client = new elasticsearch.Client({
host: 'localhost:9200',
log: 'trace'
});
client.ping({
requestTimeout: Infinity,
hello: "elasticsearch!"
})
.then(() => syncProcess) // successful connection
.catch(err => console.error(err))
export default client
This way, I don't even need to worry about running cron job (if question 1 is correct), since I know that cluster is running.
Questions
Will syncProcess run before export default client? I don't want any requests coming in while syncing...
syncProcess should run only once (since it's cached/not exported), no matter how many times I import elasticConnect.js. Correct?
Is there any advantages using the method with updates table, instead of just selecting data from parent/source table?
The articles' comments say "don't use timestamp to compare new data!".Ehhh... why? It should be ok since database is blocking, right?
For 1: As it is you have not warranty that syncProcess will have run by the time the client is exported. Instead you should do something like in this answer and export a promise instead.
For 2: With the solution I linked to in the above question, this would be taken care of.
For 3: An updates table would also catch record deletions, while simply selecting from the DB would not, since you don't know which records have disappeared.
For 4: The second comment after the article you linked to provides the answer (hint: timestamps are not strictly monotonic).

Mongoose: Read on ReplicaSet

I have a mongodb replica set from which I want to read data from primary and secondary db.
I have used this command to connect to the db:
mongoose.connect('mongodb://user:password#54.230.1.1,user:password#54.230.1.2,user:password#54.230.1.3/PanPanDB?replicaSet=rs0&readPreference=nearest');
It doesn't work.. My application continues to read from the primary.. Any suggestion please?
If you want to read from a secondary, you should set your read preference to either of:
secondaryPreferred - In most situations, operations read from secondary members but if no secondary members are available, operations read from the primary.
secondary - All operations read from the secondary members of the replica set.
Reading from nearest as per your example will select the nearest member by ping time (which could be either the primary or a secondary).
Caveats
When using any read preference other than primary, you need to be aware of potential issues with eventual consistency that may affect your application logic. For example, if you are reading from a secondary there may be changes on the primary that have not replicated to that secondary yet.
If you are concerned about stronger consistency when reading from secondaries you should review the Write Concern for Replica Sets documentation.
Since secondaries have to write the same data as the primary, reading from secondaries may not improve performance unless your application is very read heavy or is fine with eventual consistency.
Following the documentation found on MongoDB website and on Mongoose web site, you can add this instruction for configuring the ReadPreference on Mongoose:
var opts = { replSet: {readPreference: 'ReadPreference.NEAREST'} };
mongoose.connect('mongodb://###:#######:###/###', opts);
This has been tested using Mongoose version 3.8.9
As well as setting the connection URI (as you did) and the connection options (as Emas did), I also had to explicitly choose the server for each query, e.g.
var query = User.find({}).read("nearest");
query.exec(function(err, users) {
// ...
});
Mongoose use node package "mongodb", connection uri or opts is parsed by "mongodb". Here is mongodb connect opts and mongodb readPreference source code.
So, we can use mongoose like this:
var opts = {db: {readPreference: 'nearest'};
mongoose.connect(uri, opts);
Also, just use uri like this:
var uri = 'mongodb://###?readPreference=nearest';
mongoose.connect(uri, opts);
In mongoose 4.3.4 above take effect.
This is the proper instantiation in Mongoose v5.9.5:
const opts = {
readPreference: 'nearest',
}
mongoose.connect(MONGODB_CONNECTION, opts)
These are the different string values depending on the preference type you're looking for:
ReadPreference.PRIMARY = 'primary';
ReadPreference.PRIMARY_PREFERRED = 'primaryPreferred';
ReadPreference.SECONDARY = 'secondary';
ReadPreference.SECONDARY_PREFERRED = 'secondaryPreferred';
ReadPreference.NEAREST = 'nearest'
You can simply do that by using below code
var collection = db.collection(collectionName,{readPreference:'secondaryPreferred'});
http://p1bugs.blogspot.in/2016/06/scaling-read-query-load-on-mongodb.html

Resources