I am facing some difficulty in deciding how to implement a read operation using cassandra.
The case is that I have an array of id's, let's call it idArray.
After making the read, I am pushing the result in a result array (resultArray)
Now my problem being that would such a code be efficient at all ?
`for(i;i<idAArray.length;i++)
{
let query = "SELECT * FROM table WHERE \"id\"=?idArray[i]"
client.execute(query)
.then(result => resultArray.push(result));
}`
If running in parallel is an option, please specify how exactly ?
Thanks in advance !
If you provide callback to the execute call, then the code will be asynchronous, and you can issue multiple requests in parallel:
client.execute(query, [ 'someone' ], function(err, result) {
assert.ifError(err);
console.log('User with email %s', result.rows[0].email);
});
Depending on number of queries that you need to execute, you may need to tune connection pooling to allow more in-fligh requests per connection. It's also recommended to prepare your query.
More information is in documentation.
Related
I'm using a Node.js application which inserts and updates data on a Oracle database. There is a situation where I need to update a column value of a table for number of rows within a single API call. However, It is observed to be time consuming task and sometimes connection max pool size is reached, giving the below error. Increasing pool size was not helpful either.
Knex: Timeout acquiring a connection. The pool is probably full. Are you missing a .transacting(trx) call?
To mitigate this issue, a transaction is created for all updates and used Bluebird.js to perform update calls in a concurrent manner as given below.
knex.transaction((trx) => {
...........
...........
Bluebird.map(allObjects, (item) => {
.......
trx('Table_Name')
.where('ID', item.ID)
.update({abc: JSON.stringify(updatedAbc)})
........
},
{ concurrency: 10 }).then(()=>{ trx.commit });;
However, This also gives above mentioned issue. When calling endpoint consecutively, Above pool size exceeding issue arises. Is there a better way to solve this problem. Help is really appreciated. Thank you.
You are at least missing some return statements. Also there is no reason to try to update multiple lines concurrently in transaction, because in transaction all queries are executed sequentially and driver just queues them if one tries to do something like that.
// more or less fixed version of the code above:
knex.transaction((trx) => {
return Bluebird.map(d, (item) => {
return trx('Table_Name')
.where('ID', item.ID)
.update({abc: JSON.stringify(updatedAbc)});
}, { concurrency: 10 }).then(() => console.log("transaction completed with success")).catch(err => console.log("Transaction was rolled back", err)));
Better way to do it:
// throws an error if transaction did fail and was rolled back
await knex.transaction(async (trx) => {
for (let item of allObjects) {
await trx('Table_Name').where('ID', item.ID)
.update({abc: JSON.stringify(updatedAbc)});
}
});
I am having some issues with a Knex route for PostgreSQL. I am trying to insert into a database but only when the item is not in the database already. I am trying to use where not exists but it doesn't seem to be doing what I want it to. I appreciate any help you can give me.
Thank you!
app.post('/addcart', (req,res)=>{
const{customer_id, product_id,item_quantity}=req.body;
db('shopping_carts')
.insert({
customer_id:customer_id,
product_id:product_id,
item_quantity:item_quantity
})
.whereNotExists(db.select('*').from('shopping_carts').where('product_id',product_id))
.then(item=>{
console.log(item)
res.json(item)
})
.catch((err)=>{
if(err.column === 'customer_id'){
res.status(400).json({message:err})
console.log('test')
}else{
res.status(500).json({message:err})
// console.log(err.name);
}
})
})
You can't combine a whereNotExists query with an insert query, they don't support this due to it's complexity (and per #mikael, most db's don't support this). So knex ignores the whereNotExists call after the insert in your method chain.
You need to check for existence first, and then do the insert, via separate calls.
You could also write a raw query. Here's an example, it's not pretty:
https://github.com/tgriesser/knex/commit/e74f43cfe57ab27b02250948f8706d16c5d821b8
However, you will run into concurrency/lock issues when trying to do this. You're much better off making use of a unique key and letting the DB reject the insert. Then you can catch it:
.catch((err) => {
if (err.code === 23505) { res.status(500).json({message: 'duplicate'});
}
Edit, more info if you're curious. There's a very long thread on the topic here:
https://github.com/tgriesser/knex/issues/871
Edit: thread from #mikael regarding DB's and insert-where:
https://github.com/tgriesser/knex/issues/871
This is my code:
var Maria = require('mariasql');
c = new Maria({
host: '127.0.0.1',
user : 'root',
password : 'maria',
db : 'gim'
});
c.query('SELECT * FROM contact WHERE id = ? AND nom = ?', [4, 'dupont'], function(err, rows) {
if (err) {
throw err;
}
else {
function getResult() {
return rows;
}
}
});
c.end();
//get overs file
console.log(getResult());
I want resultant data but getResult() is not defined.
How can I get resultant with node.js?
Well there are several errors on your code.
Most of them related to basic JS programming concepts. So I will suggest you to look for some JS courses or tutorials.
When you define getResult function, you're doing it inside of a function's clousure. Which means that, this function will be only accessible inside that function.
The function where you've declared that getResult function, is actually a callback which is a function that will get executed as soon as some operation completes, normally an I/O intensive operation. Like in your case reading records from the database.
When the last statement gets reached and executed your query did not return, as of per node.js non-blocking I/O, the control of the program executes c.query... and this is where you provide that callback that we mentioned earlier, so it can continue the execution of the rest of statements, and when the c.query... returns with the records, execute the callback with them as result. But by then console.log... would have been executed.
I'm not sure if this is an understandable answer, in part because the concepts that I've tried to explain are really basic to the language that you're using, so one more time I would suggest to start by a tutorial/course. There are some good ones on CodeSchool and so many other sites.
After reading https://stackoverflow.com/a/14797359/4158593 : about nodejs single thread and that it takes the first parameter of async function, processes it and then uses the callback to respond when everything is ready. What confused me is what if I had multiple queries that need to be excused all at once and tell nodeJS to block other requests by adding them in a queue.
To do that I realised that I need to wrap my queries in another callback. And promises do that pretty well.
const psqlClient = psqlPool.connect();
return psqlClient.query(`SELECT username FROM usernames WHERE username=$1`, ['me'])
.then((data) => {
if(!data.rows[0].username) {
psqlClient.query(`INSERT INTO usernames (username) VALUES ('me')`);
}
else { ... }
});
This code is used during sign up, to check if username isn't taken before inserting. So it very important that nodejs puts other requests into a queue, and makes sure to select and insert at the same time. Because this code might allow people with the same username sent at the same time to select a username that has been already been taken, therefore two usernames will be inserted.
Questions
Does the code above executes queries all at once?
If 1 is correct, if I was to change the code like this
const psqlClient = psqlPool.connect();
return psqlClient.query(`SELECT username FROM usernames WHERE username=$1`, ['me'], function(err, reply) {
if(!reply.rows[0].username) {
psqlClient.query(`INSERT INTO usernames (username) VALUES ('me')`);
}
});
would that effect the behaviour?
If 1 is wrong, how should this be solved? I am going to need this pattern (mainly using select and insert/update one after another) for things like making sure that my XML sitemaps don't contain more than 50000 urls by storing the count for each file in my db which happens dynamically.
The only thing that can guarantee data integrity in your case is a single SELECT->INSERT query, which was discussed here many times.
Some examples:
Is SELECT or INSERT in a function prone to race conditions?
Get Id from a conditional INSERT
You should be able to find more of that here ;)
I also touched on this subject in a SELECT ⇒ INSERT example within pg-promise.
There is however an alternative, to make any repeated insert generate a conflict, in which case you can re-run your select to get the new record. But it is not always a suitable solution.
Here's a reference from the creator of node-postgres: https://github.com/brianc/node-postgres/issues/83#issuecomment-212657287. Basically queries are queued, but don't rely on them in production where you have many requests....
However you can use BEGIN and COMIT
var Client = require('pg').Client;
var client = new Client(/*your connection info goes here*/);
client.connect();
var rollback = function(client) {
//terminating a client connection will
//automatically rollback any uncommitted transactions
//so while it's not technically mandatory to call
//ROLLBACK it is cleaner and more correct
client.query('ROLLBACK', function() {
client.end();
});
};
client.query('BEGIN', function(err, result) {
if(err) return rollback(client);
client.query('INSERT INTO account(money) VALUES(100) WHERE id = $1', [1], function(err, result) {
if(err) return rollback(client);
client.query('INSERT INTO account(money) VALUES(-100) WHERE id = $1', [2], function(err, result) {
if(err) return rollback(client);
//disconnect after successful commit
client.query('COMMIT', client.end.bind(client));
});
});
});
Check out: https://github.com/brianc/node-postgres/wiki/Transactions
However this doesn't block the table. Here's a list of solutions: Update where race conditions Postgres (read committed)
I receive input from a MS SQL SELECT query using the tedious driver.
I have attached a listener to the "row" event of the reader:
request.on('row', function(columns) {
insert_row_other_db(columns);
});
I am writing the results to another database in the insert_row_other_db function.
But the rows arrive much faster than they can be written, and I want to open only one connection. What is a good way to de-asyncronyze the writes to the other db? I would like to write the rows one after the other.
Assuming you are able to receive a callback when insert_row_other_db completes, you can use the Async library to create a queue that you can use to schedule your inserts. The queue has a concurrency setting that allows you to limit the number of tasks that can run at once.
var async = require("async");
var insertQueue = async.queue(function (columns, callback) {
insert_row_other_db(columns, callback);
}, 1);
request.on("row", function (columns) {
insertQueue.push(columns);
});