I'd like to write a wrapper function function select(db: any, ids: number[]): Cat[] that returns an array of Cat rows fetched from the DB by ID. The function should return the entire array of rows.
Below is one approach I've written. Instead of calling db.each on every ID in a for-loop as I do below, is it possible to pass my entire ids: number[] array as a parameter to a db.all / db.each query function?
// dbmethods.ts
async function select(db: any, ids: number[]): Promise<Cat[]> {
let query = "SELECT * FROM cats_table WHERE id = ?;";
let cats_back: Cat[] = [];
for (let i = 0; i < ids.length; i++) {
let cat: Promise<Cat> = new Promise(async function (resolve, reject) {
await db.get(query, ids[i], (err: Error, row: any) => {
if (err) {
reject(err);
} else {
let cat: Cat = {
index: row.id,
cat_type: row.cat_type,
health: row.health,
num_paws: row.num_paws
};
resolve(cat);
}
});
});
cats_back.push(await cat);
}
return cats_back;
}
and
// index.ts
let ids = create_many_ids(10_000); // returns an array of unique ordered ints between 0 and 10K
let res = await select(db, ids);
console.log(res); // successfully prints my cats
Benchmarks on my select function above suggest that it takes 300ms to select 10_000 rows by ID. It seems to me that's a little long; 10K rows shouldn't take that long for sqlite's select by id functionality... How can I be more efficient?
SELECT * FROM cats_table WHERE id IN (SELECT value FROM json_each(?));
The query parameter is a string representing a JSON array of ids, e.g., '[1, 2, 4]'
See this tutorial for further details
Related
Is there any way of chaining queries dynamically?
For example, given the following GET request
/collection?field1=value1&field2=value2&sort=field3 asc
It is easy without the sort query
/collection?field1=value1&field2=value2
var query = {}
for (var key in query) {
query[key] = req.query[key]
}
Collection.find(query)
But how do I build the GET request if there are optional query keys such as sort, expand, and select which map to Collection.sort, Collection.populate, Collection.select respectively?
In other words, suppose you have a dynamic array of Query methods:
queries = [populate, select, sort]
Would the solution be the following:
var query = Collection.find()
for (var q in queries)
query = query.q
You just iterate through the query parameters and separate out the ones that are operations versus actual query criteria. Using your examples:
// sample data for req.query
const req = {
query: {
sort: "field9",
field1: "someValue",
field2: "otherValue",
field3: "highValue"
}
};
const queries = new Map();
const operations = new Map([
["populate", false],
["sort", false],
["select", false]
]);
for (const [key, value] of Object.entries(req.query)) {
if (operations.has(key)) {
operations.set(key, value);
} else {
queries.set(key, value);
}
}
// here:
// queries contain the non-operation pairs
// operations (if not false) contain the operation value such
// as sort => "field9"
console.log("operations:");
for (let [key, value] of operations.entries()) {
console.log(`${key} => ${value}`);
}
console.log("queries:");
for (let [key, value] of queries.entries()) {
console.log(`${key} => ${value}`);
}
To run the operation, you'd then have to check which operations are present and branch your code and query based on which operations are present.
Below iam calling addUpdateDailyLeads with an array like
[{
"yyyymmdd": "20191124",
"admin_login":"rasheed.s",
"category":"PO",
"amount":10,
"office_id":10000,
"new_leads_attempted":10
},
{
"yyyymmdd": "20191124",
"admin_login":"rasheed.s",
"category":"PO",
"amount":10,
"office_id":10000,
"new_leads_attempted":10
},
{
"yyyymmdd": "20191125",
"admin_login":"prajeed.av",
"category":"FHL",
"amount":10,
"office_id":10000,
"new_leads_attempted":10
}
]
So,key 0 should insert,
key 1 should update because duplicate key constratint,
key 2 will insert,
but im getting duplicate key constraint error on key 1,because array map not waiting for the query to be executed .
const addUpdateDailyLeads = async (req, res) => {
let admin_login,category,office_id,new_leads_attempted,yyyymmdd,where,values;
let data = req.body;
req.body.map(async function(item,i){
admin_login = item.admin_login,
category = item.category,
office_id = item.office_id,
new_leads_attempted = item.new_leads_attempted,
yyyymmdd = item.yyyymmdd;
where = {yyyymmdd:yyyymmdd, admin_login:admin_login, category:category};
values = {yyyymmdd:yyyymmdd, admin_login:admin_login, category:category,office_id:office_id,new_leads_attempted:new_leads_attempted,update_date:moment().format('YYYYMMDDHHmmss')};
console.log("calling ",i);
let chck = await addUpdateDailyLeadsCollection({where:where,values:values})
console.log("")
console.log("called")
})
res.json({ code: '200', message: `Advisor Daily Leads Updated ${admin_login}` });
}
const addUpdateDailyLeadsCollection = async data => {
let transaction;
let where = data.where
let values = data.values
var Sequelize = require("sequelize");
console.log("startef 1");
await AdvisorLeads.findOne({ where: where }, { useMaster: true }).then( async(data)=>{
console.log("waited");
if(data){
await data.update({new_leads_attempted: Sequelize.literal('new_leads_attempted + '+values.new_leads_attempted)}).then(data=>{
console.log("updated")
return Promise.resolve(1);
})
}else{
AdvisorLeads.create(values).then(data=>{
console.log("inserted")
return Promise.resolve(1);
})
}
})
};
final output on console
calling 0
startef 1
waiting 1
calling 1
startef 1
waiting 1
calling 2
startef 1
waiting 1
waited
waited
waited
called
called
called
inserted
inserted
My expected output like
calling 0
startef 1
waiting 1
waited
inserted
called
calling 1
startef 1
waiting 1
waited
updated
called
calling 2
startef 1
waiting 1
waited
inserted
called
Finally whait i need is to wait for each item ,execute all queries and then process next item
I think you can solve by using await on the update and create statements....
But also take a look at the UPSERT method, which could simplify your code quite a bit. From
The Sequelize API Reference: "Insert or update a single row. An update will be executed if a row which matches the supplied values on either the primary key or a unique key is found."
Addendum: for synchronizing async/await, there are many ways to do this, as detailed in this post. Here's some code I set up following the ES7 method:
let params = [{id : 1, sal : 10}, {id : 44, sal: 30}, {id : 1, sal : 20}];
async function doUpsertArrayInSequence(myParams) {
let results = [];
for (let i = 0; i < myParams.length; i++) {
let x = await User.findByPk(myParams[i].id).then(async (u) => {
if (u != null) {
await u.update({ sal : u.sal + myParams[i].sal});
} else {
await User.create({id: myParams[i].id, sal: myParams[i].sal});
}
});
results.push(x);
}
return results;
}
await doUpsertArrayInSequence(params).then(function(result) {
User.findAll().then(proj => {
res.send(proj);
next();
});
})
From the log, I can see
a) a SELECT, followed by an UPDATE or INSERT for each row (in sequence).
b) the 2nd occurrence of id=1 reflects the update of the 1st occurrence.
c) the final findAll reflects all inserts and updates.
HTH
I have three queries on Firestore based on a time range. (24, 12 and 6 hour). I am using Promise.all and it works. As you can see from the code, I am accessing the result of each query by using an index to the returned snapshot. I have read that the Returned values will be in the order of the Promises passed, regardless of completion order.
Now, I want to be able to pass an object to the Promise.all because my number of queries will be unpredictable for what I want to do, basically, I will be looping to a number of vehicles and building the same 3 queries for each and I will pass it all on a Promise.all. And when Promise.all returns, I want to be able to know which vehicle and time range that snapshot is for.
So instead of an array, I want to pass this argument to Promise.all instead.
{"vehicle1_24":query, "vehicle1_12":query, "vehicle1_6":query,
"vehicle2_24":query, "vehicle2_12":query, "vehicle2_6":query}
code
var queries = [
vehicleRef.collection('telemetry').where('time_stamp', '<', today).where('time_stamp', '>', yesterday).get(),
vehicleRef.collection('telemetry').where('time_stamp', '<', today).where('time_stamp', '>', twelveHours).get(),
vehicleRef.collection('telemetry').where('time_stamp', '<', today).where('time_stamp', '>', sixHours).get()
]
for (var i = 0; i < queries.length; i++) {
queryResults.push(
queries[i]
)
}
Promise.all(queryResults)
.then(snapShot=> {
const yesterdayResult = result => getEnergy(result);
const twelveHourResult = result => getEnergy(result);
const sixHourResult = result => getEnergy(result);
allYesterdayResult += yesterdayResult(snapShot[0])
allTwelveHourResult += twelveHourResult(snapShot[1])
allSixHourResult +=sixHourResult(snapShot[2])
console.log("Done updating vehicle ", vehicle)
// return res.send({"Result" : "Successful!"})
}).catch(reason => {
console.log(reason)
// return res.send({"Result" : "Error!"})
This feature does not exist natively, but should be fairly easy to write, something along the lines of
async function promiseAllObject(obj) {
// Convert the object into an array of Promise<{ key: ..., value: ... }>
const keyValuePromisePairs = Object.entries(obj).map(([key, valuePromise]) =>
valuePromise.then(value => ({ key, value }))
);
// Awaits on all the promises, getting an array of { key: ..., value: ... }
const keyValuePairs = await Promise.all(keyValuePromisePairs);
// Turn it back into an object.
return keyValuePairs.reduce(
(result, { key, value }) => ({ ...result, [key]: value }),
{}
);
}
promiseAllObject({ foo: Promise.resolve(42), bar: Promise.resolve(true) })
.then(console.log); // { foo: 42, bar: true }
You can use the following code to transform your object into an array that you will pass to Promise.all()
var queriesObject = {"vehicle1_24":query, "vehicle1_12":query, "vehicle1_6":query, "vehicle2_24":query, "vehicle2_12":query, "vehicle2_6":query};
//Of course, queriesObject can be an oject with any number of elements
var queries = [];
for (var key in queriesObject) {
if (queriesObject.hasOwnProperty(key)) {
queries.push(queriesObject[key]);
}
}
Promise.all(queries);
You will receive the results of Promise.all in an array corresponding to the fulfillment values in the same order than the queries array, see: Promise.all: Order of resolved values and https://www.w3.org/2001/tag/doc/promises-guide#aggregating-promises
I have two collections, one contains my static items and other collection contains reverse geocode results for that item. They are matched by id property.
I am writing a script that would fill reverse geocode collection with missing items.
This is my current solution which is super slow, it does:
Get total count of static items
Create read stream from static items collection
Uses find one on reverse geocode collection for each item that comes from the read stream
If items exists, increase counter by 1 and ignore it
If item doesn't exist, fetch it from API, save it to collection and increase counter by 1
When counter is equal total count, it means all items are fetched,
therefore resolve function
function fetchMissingData(){
return new Promise((resolve, reject) => {
const staticData = Global.state.db.collection('static_data')
const googleData = Global.state.db.collection('google_reverse_geocode')
staticData.count((countErr, count) => {
if (countErr) return reject(countErr)
let counter = 0
let fetched = 0
function finishIfReady(){
process.stdout.write(`Progress...(${counter}/${count}), (fetched total: ${fetched})\r`)
if (count === counter) {
resolve({ fetched, counter })
}
}
staticData.find()
.on('data', (hotel) => {
googleData.findOne({ id: hotel.id }, (findErr, geocodedItem) => {
if (findErr) return reject(findErr)
if (geocodedItem) {
counter++
finishIfReady()
} else {
GMClient.reverseGeocode({ latlng: hotel.geo }, (err, response) => {
if (err) return reject(err)
googleData.insertOne({
id: hotel.id,
payload: response,
}).then(() => {
fetched++
counter++
finishIfReady()
}).catch(e => reject(e))
})
}
})
})
.on('error', e => reject(e))
})
})
}
Is there more elegant solution using aggregation framework that would allow me same behavior without O(n^{2}) O(nlogn) complexity?
First, the actual complexity is O(nlogn) because findOne on id use binary search. Second, although there is no way to pass the theory complexity O(nlogn) in this case, there is way to help make your code faster in practice. This is what I would do:
function getIdOfAllGeoData() {
// return an array of existing Geo data IDs
return Global.state.db.collection('google_reverse_geocode')
.find().toArray().map(o => o.id);
}
function getStaticDataMissingGeo(existingGeoDataIds) {
const staticData = Global.state.db.collection('static_data');
return staticData.find({
id: {
$nin: existingGeoDataIds
}
}).toArray();
}
function fetchMissingData() {
const existingGeoDataIds = getIdOfAllGeoData();
const staticDataMissingGeo = getStaticDataMissingGeo(existingGeoDataIds);
// staticDataMissingGeo is all the static that need geo data
// you can loop through this array, get each items geo data and insert to database
// ...
}
Finally, you could use bulk operation to speed thing up, it will be much faster. Also, my mongo related code above may not be correct, consider it as an idea.
I'm writing a small utility to copy data from one sqlite database file to another. Both files have the same table structure - this is entirely about moving rows from one db to another.
My code right now:
let tables: Array<string> = [
"OneTable", "AnotherTable", "DataStoredHere", "Video"
]
tables.forEach((table) => {
console.log(`Copying ${table} table`);
sourceDB.each(`select * from ${table}`, (error, row) => {
console.log(row);
destDB.run(`insert into ${table} values (?)`, ...row) // this is the problem
})
})
row here is a js object, with all the keyed data from each table. I'm certain that there's a simple way to do this that doesn't involve escaping stringified data.
If your database driver has not blocked ATTACH, you can simply tell the database to copy everything:
ATTACH '/some/where/source.db' AS src;
INSERT INTO main.MyTable SELECT * FROM src.MyTable;
You could iterate over the row and setup the query with dynamically generated parameters and references.
let tables: Array<string> = [
"OneTable", "AnotherTable", "DataStoredHere", "Video"
]
tables.forEach((table) => {
console.log(`Copying ${table} table`);
sourceDB.each(`select * from ${table}`, (error, row) => {
console.log(row);
const keys = Object.keys(row); // ['column1', 'column2']
const columns = keys.toString(); // 'column1,column2'
let parameters = {};
let values = '';
// Generate values and named parameters
Object.keys(row).forEach((r) => {
var key = '$' + r;
// Generates '$column1,$column2'
values = values.concat(',', key);
// Generates { $column1: 'foo', $column2: 'bar' }
parameters[key] = row[r];
});
// SQL: insert into OneTable (column1,column2) values ($column1,$column2)
// Parameters: { $column1: 'foo', $column2: 'bar' }
destDB.run(`insert into ${table} (${columns}) values (${values})`, parameters);
})
})
Tried editing the answer by #Cl., but was rejected. So, adding on to the answer, here's the JS code to achieve the same:
let sqlite3 = require('sqlite3-promise').verbose();
let sourceDBPath = '/source/db/path/logic.db';
let tables = ["OneTable", "AnotherTable", "DataStoredHere", "Video"];
let destDB = new sqlite3.Database('/your/dest/logic.db');
await destDB.runAsync(`ATTACH '${sourceDBPath}' AS sourceDB`);
await Promise.all(tables.map(table => {
return new Promise(async (res, rej) => {
await destDB.runAsync(`
CREATE TABLE ${table} AS
SELECT * FROM sourceDB.${table}`
).catch(e=>{
console.error(e);
rej(e);
});
res('');
})
}));