Queries being run twice on sql.js - node.js

I'm using SQLite with sql.js on my project and I have been having some trouble with my implementation. Seems like the queries are being run on the database twice because for the for the INSERT statements I get 2 records in the DB.
The way I do it, I create the SQL and then pass it on to this method (the opts variable contains all of the data being put into the database):
prepareStatementAndCompileResults(db, sql, opts){
const stmt = db.prepare(sql);
const result = stmt.getAsObject(opts);
var rows = [];
if(!this.isEmpty(result)){ // isEmpty is a simple method that checks for empty objects
rows.push(result);
}
while(stmt.step()) {
var row = stmt.getAsObject();
rows.push(row);
}
this.saveToFile(db);
stmt.free();
return rows;
},
Here is a sample SQL INSERT that is being run twice
INSERT OR IGNORE INTO tag_event (tag_id, event_id, unique_string)
VALUES (:tag_id,:event_id, :unique);
Here is what the opts variable would look like for this query:
var opts = {
[':tag_id']: 1,
[':event_id']:1,
[':unique']: '1-1'
}

Because you're pushing it into row 2 time.
// if not empty will add to row
if(!this.isEmpty(result)){ // isEmpty is a simple method that checks for empty objects
rows.push(result);
}
// not sure what step() does but I'm assuming this will also run
while(stmt.step()) {
var row = stmt.getAsObject();
rows.push(row);
}
Verify by using a debugger or just console.log(rows) after the while loop before the save

So, what it turns out I needed to do was bind the variables to the prepared statement before getting the rather than binding them through getAsObject. This is much more efficient. My API response time on a local test went from 785ms to 14.5ms
prepareStatementAndCompileResults(db, sql, opts){
const rows = [];
const stmt = db.prepare(sql);
stmt.bind(opts);
while(stmt.step()) {
var row = stmt.getAsObject();
rows.push(row);
}
this.saveToFile(db);
stmt.free();
return rows;
},

Related

Create a new key and or array to existing object in node.js

I have an existing object called unsettled:
var unsettled =
{
processor: 1,
dayFrom: 10,
dayTo: 12
}
Im trying to add an object called pName as I do with Angular, but it is not working.
unsettled.pname = "something"
It does work if unsettled.pname already exists on the object, but not if I want to create it.
Also, after a sql Query where I get several results, I need to create an array. Same thing, when I do
const pool = await utils.poolPromise
const recordset = await pool.request()
.query(sqlString)
if (recordset.rowsAffected[0] > 0) {
unsettled.processors = recordset.recordset;
}
Is not working either (the array processors is not being created).
Thanks.
For the array, try:
if (recordset.rowsAffected[0] > 0) {
unsettled["processors"] = recordset.recordset;
}
As for as pName, please try
unsettled["pname"] = "something"

How to get itemcount where child value is false with firebase functions?

Edit:I wrote a query that should return all items where ticked is true. I try to set the count of tickedItemsQuery but it tells me count is not an function. numChildren is also not a function. How can I set the count of the query?
exports.tickedBoxesCount = functions.database.ref('/listItems/{list_id}').onWrite(event => {
const ref = event.data.ref.parent;
const list_id = event.params.list_id;
const tickediItemsQuery = ref.orderByChild("ticked").equalTo(true)
admin.database().ref(`/lists/${list_id}/tickedCount`).set(tickediItemsQuery.count());
});
This line of code creates a query:
const tickediItemsQuery = ref.orderByChild("ticked").equalTo(true)
Note that it does not execute the query, it merely defines it. So there's no way that you can know the number of matching children from merely this line.
To determine the count, you'll need to execute the query, and determine the number of children in the resulting snapshot:
return tickediItemsQuery.once("value").then(function(snapshot) {
return admin.database().ref(`/lists/${list_id}/tickedCount`).set(snapshot.numChildren());
});
Note that I also added necessary return statements, to ensure the Cloud Functions knows when your (asynchronous) read and write operations are done.

Delete Documents from CosmosDB based on condition through Query Explorer

What's the query or some other quick way to delete all the documents matching the where condition in a collection?
I want something like DELETE * FROM c WHERE c.DocumentType = 'EULA' but, apparently, it doesn't work.
Note: I'm not looking for any C# implementation for this.
This is a bit old but just had the same requirement and found a concrete example of what #Gaurav Mantri wrote about.
The stored procedure script is here:
https://social.msdn.microsoft.com/Forums/azure/en-US/ec9aa862-0516-47af-badd-dad8a4789dd8/delete-multiple-docdb-documents-within-the-azure-portal?forum=AzureDocumentDB
Go to the Azure portal, grab the script from above and make a new stored procedure in the database->collection you need to delete from.
Then right at the bottom of the stored procedure pane, underneath the script textarea is a place to put in the parameter. In my case I just want to delete all so I used:
SELECT c._self FROM c
I guess yours would be:
SELECT c._self FROM c WHERE c.DocumentType = 'EULA'
Then hit 'Save and Execute'. Viola, some documents get deleted. After I got it working in the Azure Portal I switched over the Azure DocumentDB Studio and got a better view of what was happening. I.e. I could see I was throttled to deleting 18 a time (returned in the results). For some reason I couldn't see this in the Azure Portal.
Anyway, pretty handy even if limited to a certain amount of deletes per execution. Executing the sp is also throttled so you can't just mash the keyboard. I think I would just delete and recreate the Collection unless I had a manageable number of documents to delete (thinking <500).
Props to Mimi Gentz #Microsoft for sharing the script in the link above.
HTH
I want something like DELETE * FROM c WHERE c.DocumentType = 'EULA'
but, apparently, it doesn't work.
Deleting documents this way is not supported. You would need to first select the documents using a SELECT query and then delete them separately. If you want, you can write the code for fetching & deleting in a stored procedure and then execute that stored procedure.
I wrote a script to list all the documents and delete all the documents, it can be modified to delete the selected documents as well.
var docdb = require("documentdb");
var async = require("async");
var config = {
host: "https://xxxx.documents.azure.com:443/",
auth: {
masterKey: "xxxx"
}
};
var client = new docdb.DocumentClient(config.host, config.auth);
var messagesLink = docdb.UriFactory.createDocumentCollectionUri("xxxx", "xxxx");
var listAll = function(callback) {
var spec = {
query: "SELECT * FROM c",
parameters: []
};
client.queryDocuments(messagesLink, spec).toArray((err, results) => {
callback(err, results);
});
};
var deleteAll = function() {
listAll((err, results) => {
if (err) {
console.log(err);
} else {
async.forEach(results, (message, next) => {
client.deleteDocument(message._self, err => {
if (err) {
console.log(err);
next(err);
} else {
next();
}
});
});
}
});
};
var task = process.argv[2];
switch (task) {
case "listAll":
listAll((err, results) => {
if (err) {
console.error(err);
} else {
console.log(results);
}
});
break;
case "deleteAll":
deleteAll();
break;
default:
console.log("Commands:");
console.log("listAll deleteAll");
break;
}
And if you want to do it in C#/Dotnet Core, this project may help: https://github.com/lokijota/CosmosDbDeleteDocumentsByQuery. It's a simple Visual Studio project where you specify a SELECT query, and all the matches will be a) backed up to file; b) deleted, based on a set of flags.
create stored procedure in collection and execute it by passing select query with condition to delete. The major reason to use this stored proc is because of continuation token which will reduce RUs to huge extent and will cost less.
##### Here is the python script which can be used to delete data from Partitioned Cosmos Collection #### This will delete documents Id by Id based on the result set data.
Identify the data that needs to be deleted before below step
res_list = "select id from id_del"
res_id = [{id:x["id"]}
for x in sqlContext.sql(res_list).rdd.collect()]
config = {
"Endpoint" : "Use EndPoint"
"Masterkey" : "UseKey",
"WritingBatchSize" : "5000",
'DOCUMENTDB_DATABASE': 'Database',
'DOCUMENTDB_COLLECTION': 'collection-core'
};
for row in res_id:
# Initialize the Python DocumentDB client
client = document_client.DocumentClient(config['Endpoint'], {'masterKey': config['Masterkey']})
# use a SQL based query to get documents
## Looping thru partition to delete
query = { 'query': "SELECT c.id FROM c where c.id = "+ "'" +row[id]+"'" }
print(query)
options = {}
options['enableCrossPartitionQuery'] = True
options['maxItemCount'] = 1000
result_iterable = client.QueryDocuments('dbs/Database/colls/collection-core', query, options)
results = list(result_iterable)
print('DOCS TO BE DELETED : ' + str(len(results)))
if len(results) > 0 :
for i in range(0,len(results)):
# print(results[i]['id'])
docID = results[i]['id']
print("docID :" + docID)
options = {}
options['enableCrossPartitionQuery'] = True
options['maxItemCount'] = 1000
options['partitionKey'] = docID
client.DeleteDocument('dbs/Database/colls/collection-core/docs/'+docID,options=options)
print ('deleted Partition:' + docID)

synchronous execution of database function in node.js

There are two collections products and stock in my database.Each product have multipple stock with different supplier and attributes of products.
I have selected the products from products collection and run a loop for each product. I need to append price and offer price from the stock collection to products i have selected already.
I think the for loop compleated its execution before executiong the find method on stock collection.I need to execute everything in the loop in serial manner(not asynchronous). Please check the code below and help me to solve this. I'm new in node.js and mongodb
collection = db.collection('products');
collection.find().toArray(function(err, abc) {
var finalout = [];
for( var listProducts in abc){
var subfinal = {
'_id' :abc[listProducts]['_id'],
'min_price' :abc[listProducts]['value']['price'],
'stock' :abc[listProducts]['value']['stock'],
'name' :abc[listProducts]['value']['name'],
'price' :'',
'offer_price' :'',
};
collection = db.collection('stock');
collection.find({"product":abc[listProducts]['_id'] ,"supplier":abc[listProducts]['value']['def_supplier']}).toArray(function(err, newprod) {
for( var row in newprod){
subfinal['price'] = newprod[row]['price'];
subfinal.offer_price = newprod[row]['offer_price'];
}
finalout.push(subfinal);
});
}
console.log(finalout);
});
Yes, the loop is running and starting the get requests on the database all at the same time. It is possible, like you said, to run them all sequentially, however it's probably not what you're looking for. Doing it this way will take more time, since each request to Mongo would need to wait for the previous one to finish.
Considering that each iteration of the loop doesn't depend on the previous ones, all you're really looking for is a way to know once ALL of the operations are finished. Keep in mind that these can end in any order, not necessarily the order in which they were initiated in the loop.
There's was also an issue in the creation of the subfinal variable. Since the same variable name is being used for all iterations, when they all come back they'll be using the final assignment of the variable (all results will be written on the same subfinal, which will have been pushed multiple times into the result). To fix that, I've wrapped the entire item processing iteration into another function to give each subfinal variable it's own scope.
There are plenty of modules to ease this kind of management, however here is one way to run some code only after all the Mongo calls are finished using a counter and callback that doesn't require any additional installs:
var finished = 0; // incremented every time an iteration is done
var total = Object.keys(abc).length; // number of keys in the hash table
collection = db.collection('products');
collection.find().toArray(function(err, abc) {
var finalout = [];
for( var listProducts in abc){
processItem(listProducts);
}
});
function processItem(listProducts) {
var subfinal = {
'_id' :abc[listProducts]['_id'],
'min_price' :abc[listProducts]['value']['price'],
'stock' :abc[listProducts]['value']['stock'],
'name' :abc[listProducts]['value']['name'],
'price' :'',
'offer_price' :'',
};
collection = db.collection('stock');
collection.find({"product":abc[listProducts]['_id'] ,"supplier":abc[listProducts]['value']['def_supplier']}).toArray(function(err, newprod) {
for( var row in newprod){
subfinal['price'] = newprod[row]['price'];
subfinal.offer_price = newprod[row]['offer_price'];
}
finalout.push(subfinal);
finished++;
if (finished === total) { // if this is the last one to finish.
allFinished(finalout);
}
});
}
function allFinished(finalout) {
console.log(finalout);
}
It could be written more concisely with less variables, but it should be easier to understand this way.

Re-Apply Linq to entities from last Iqueryable

here my issue:
I've an IQueryable object and I need to execute a new query on Db (just like a refresh) by launching the same contained query in my IQueryable obj.
An example:
myObj = objCtx.Person
.Where(p => p.IdPerson...)
.OrderBy(p => ...)
.Select(..some field..);
//...
// From another function I just want re-execute
// the same query
// -> Well, that retrieve full lambda
var et = this.myObj.Expression
// This doesn't work:
var anotherObj = objCtx.Person.Where(et);
//or..
var tmp = Expression.Lambda<Func<T, bool>>(et);
// This doesn't work too:
var anotherObj = objCtx.Person.Where(tmp);
Is it possibile to achieve? What am I missing?
Thanks
Ok:
I would be able to retrieve full query (select statement) from an IQueryable object and execute it in order to get all my updated data.
This is the original query:
this.myIQueryableObj = objCtx.Person
.Where(p => p.IdPerson...)
.OrderBy(p => ...)
.Select(..some field..);
//..And in some button click, I want execute again the above query, but all what I know is myIQueryableObjOnly. Can you help me?

Resources