I have three queries on Firestore based on a time range. (24, 12 and 6 hour). I am using Promise.all and it works. As you can see from the code, I am accessing the result of each query by using an index to the returned snapshot. I have read that the Returned values will be in the order of the Promises passed, regardless of completion order.
Now, I want to be able to pass an object to the Promise.all because my number of queries will be unpredictable for what I want to do, basically, I will be looping to a number of vehicles and building the same 3 queries for each and I will pass it all on a Promise.all. And when Promise.all returns, I want to be able to know which vehicle and time range that snapshot is for.
So instead of an array, I want to pass this argument to Promise.all instead.
{"vehicle1_24":query, "vehicle1_12":query, "vehicle1_6":query,
"vehicle2_24":query, "vehicle2_12":query, "vehicle2_6":query}
code
var queries = [
vehicleRef.collection('telemetry').where('time_stamp', '<', today).where('time_stamp', '>', yesterday).get(),
vehicleRef.collection('telemetry').where('time_stamp', '<', today).where('time_stamp', '>', twelveHours).get(),
vehicleRef.collection('telemetry').where('time_stamp', '<', today).where('time_stamp', '>', sixHours).get()
]
for (var i = 0; i < queries.length; i++) {
queryResults.push(
queries[i]
)
}
Promise.all(queryResults)
.then(snapShot=> {
const yesterdayResult = result => getEnergy(result);
const twelveHourResult = result => getEnergy(result);
const sixHourResult = result => getEnergy(result);
allYesterdayResult += yesterdayResult(snapShot[0])
allTwelveHourResult += twelveHourResult(snapShot[1])
allSixHourResult +=sixHourResult(snapShot[2])
console.log("Done updating vehicle ", vehicle)
// return res.send({"Result" : "Successful!"})
}).catch(reason => {
console.log(reason)
// return res.send({"Result" : "Error!"})
This feature does not exist natively, but should be fairly easy to write, something along the lines of
async function promiseAllObject(obj) {
// Convert the object into an array of Promise<{ key: ..., value: ... }>
const keyValuePromisePairs = Object.entries(obj).map(([key, valuePromise]) =>
valuePromise.then(value => ({ key, value }))
);
// Awaits on all the promises, getting an array of { key: ..., value: ... }
const keyValuePairs = await Promise.all(keyValuePromisePairs);
// Turn it back into an object.
return keyValuePairs.reduce(
(result, { key, value }) => ({ ...result, [key]: value }),
{}
);
}
promiseAllObject({ foo: Promise.resolve(42), bar: Promise.resolve(true) })
.then(console.log); // { foo: 42, bar: true }
You can use the following code to transform your object into an array that you will pass to Promise.all()
var queriesObject = {"vehicle1_24":query, "vehicle1_12":query, "vehicle1_6":query, "vehicle2_24":query, "vehicle2_12":query, "vehicle2_6":query};
//Of course, queriesObject can be an oject with any number of elements
var queries = [];
for (var key in queriesObject) {
if (queriesObject.hasOwnProperty(key)) {
queries.push(queriesObject[key]);
}
}
Promise.all(queries);
You will receive the results of Promise.all in an array corresponding to the fulfillment values in the same order than the queries array, see: Promise.all: Order of resolved values and https://www.w3.org/2001/tag/doc/promises-guide#aggregating-promises
Related
I'd like to write a wrapper function function select(db: any, ids: number[]): Cat[] that returns an array of Cat rows fetched from the DB by ID. The function should return the entire array of rows.
Below is one approach I've written. Instead of calling db.each on every ID in a for-loop as I do below, is it possible to pass my entire ids: number[] array as a parameter to a db.all / db.each query function?
// dbmethods.ts
async function select(db: any, ids: number[]): Promise<Cat[]> {
let query = "SELECT * FROM cats_table WHERE id = ?;";
let cats_back: Cat[] = [];
for (let i = 0; i < ids.length; i++) {
let cat: Promise<Cat> = new Promise(async function (resolve, reject) {
await db.get(query, ids[i], (err: Error, row: any) => {
if (err) {
reject(err);
} else {
let cat: Cat = {
index: row.id,
cat_type: row.cat_type,
health: row.health,
num_paws: row.num_paws
};
resolve(cat);
}
});
});
cats_back.push(await cat);
}
return cats_back;
}
and
// index.ts
let ids = create_many_ids(10_000); // returns an array of unique ordered ints between 0 and 10K
let res = await select(db, ids);
console.log(res); // successfully prints my cats
Benchmarks on my select function above suggest that it takes 300ms to select 10_000 rows by ID. It seems to me that's a little long; 10K rows shouldn't take that long for sqlite's select by id functionality... How can I be more efficient?
SELECT * FROM cats_table WHERE id IN (SELECT value FROM json_each(?));
The query parameter is a string representing a JSON array of ids, e.g., '[1, 2, 4]'
See this tutorial for further details
Is there any way of chaining queries dynamically?
For example, given the following GET request
/collection?field1=value1&field2=value2&sort=field3 asc
It is easy without the sort query
/collection?field1=value1&field2=value2
var query = {}
for (var key in query) {
query[key] = req.query[key]
}
Collection.find(query)
But how do I build the GET request if there are optional query keys such as sort, expand, and select which map to Collection.sort, Collection.populate, Collection.select respectively?
In other words, suppose you have a dynamic array of Query methods:
queries = [populate, select, sort]
Would the solution be the following:
var query = Collection.find()
for (var q in queries)
query = query.q
You just iterate through the query parameters and separate out the ones that are operations versus actual query criteria. Using your examples:
// sample data for req.query
const req = {
query: {
sort: "field9",
field1: "someValue",
field2: "otherValue",
field3: "highValue"
}
};
const queries = new Map();
const operations = new Map([
["populate", false],
["sort", false],
["select", false]
]);
for (const [key, value] of Object.entries(req.query)) {
if (operations.has(key)) {
operations.set(key, value);
} else {
queries.set(key, value);
}
}
// here:
// queries contain the non-operation pairs
// operations (if not false) contain the operation value such
// as sort => "field9"
console.log("operations:");
for (let [key, value] of operations.entries()) {
console.log(`${key} => ${value}`);
}
console.log("queries:");
for (let [key, value] of queries.entries()) {
console.log(`${key} => ${value}`);
}
To run the operation, you'd then have to check which operations are present and branch your code and query based on which operations are present.
I was initially just running one query in node.js but I now need two sets of data so I ran two queries and used Promise.all like this:
Promise.all([products, subcats]);
res.status(200).json({
products,
subcats
});
In React I have:
class ProductList extends Component {
state = {
products: []
};
async componentDidMount() {
const catslug = this.props.match.params.catslug;
const { data: products } = await getCatProducts(catslug);
this.setState({ products: products });
}
When I was only running the one query I was running this without any issue:
this.state.products.map(product => (
Now because I have the 2 queries I need to change it to:
this.state.products.products.map(product => (
But as soon as I do that I get the error:
Cannot read property 'map' of undefined
So, I changed it to this and now it works with no errors:
{this.state.products.products &&
this.state.products.products.map(product => (
My question is why did it work before without having to put in the ... && bit but now I have to use it with the Promise.all in node.js?
It's because of the initial shape of your state
state = {
products: []
};
You see this.state.products is already defined as an array, so it can be mapped over (despite being empty). However this.products.products is NOT an array, it is undefined, so trying to map will return the error you are seeing. The line you have entered
{this.state.products.products &&
this.state.products.products.map(product => (
checks that the array exists before attempting to map it (which wont evaluate to true until your async code finishes, and then the array is defined).
An alternative fix would be to set your initial state shape to match your final state shape. i.e
state ={
products:{
products:[]
}
};
Or if you don't want the nested products property you can change
async componentDidMount() {
const catslug = this.props.match.params.catslug;
const { data } = await getCatProducts(catslug);
this.setState({ products: data.products });
}
and return to your old this.state.products.map()
The issue is in the following line:
const { data: products } = await getCatProducts(catslug);
If I understand correctly, when you were sending one value, you were sending it like:
res.status(200).json(products); // an array
But now you are sending an object, which further contains 2 arrays products and subcats.
What you have 2 do is add below changes to make it work:
const obj = await getCatProducts(catslug);
const products = obj.products
const subcats = obj.subcats
I have two collections, one contains my static items and other collection contains reverse geocode results for that item. They are matched by id property.
I am writing a script that would fill reverse geocode collection with missing items.
This is my current solution which is super slow, it does:
Get total count of static items
Create read stream from static items collection
Uses find one on reverse geocode collection for each item that comes from the read stream
If items exists, increase counter by 1 and ignore it
If item doesn't exist, fetch it from API, save it to collection and increase counter by 1
When counter is equal total count, it means all items are fetched,
therefore resolve function
function fetchMissingData(){
return new Promise((resolve, reject) => {
const staticData = Global.state.db.collection('static_data')
const googleData = Global.state.db.collection('google_reverse_geocode')
staticData.count((countErr, count) => {
if (countErr) return reject(countErr)
let counter = 0
let fetched = 0
function finishIfReady(){
process.stdout.write(`Progress...(${counter}/${count}), (fetched total: ${fetched})\r`)
if (count === counter) {
resolve({ fetched, counter })
}
}
staticData.find()
.on('data', (hotel) => {
googleData.findOne({ id: hotel.id }, (findErr, geocodedItem) => {
if (findErr) return reject(findErr)
if (geocodedItem) {
counter++
finishIfReady()
} else {
GMClient.reverseGeocode({ latlng: hotel.geo }, (err, response) => {
if (err) return reject(err)
googleData.insertOne({
id: hotel.id,
payload: response,
}).then(() => {
fetched++
counter++
finishIfReady()
}).catch(e => reject(e))
})
}
})
})
.on('error', e => reject(e))
})
})
}
Is there more elegant solution using aggregation framework that would allow me same behavior without O(n^{2}) O(nlogn) complexity?
First, the actual complexity is O(nlogn) because findOne on id use binary search. Second, although there is no way to pass the theory complexity O(nlogn) in this case, there is way to help make your code faster in practice. This is what I would do:
function getIdOfAllGeoData() {
// return an array of existing Geo data IDs
return Global.state.db.collection('google_reverse_geocode')
.find().toArray().map(o => o.id);
}
function getStaticDataMissingGeo(existingGeoDataIds) {
const staticData = Global.state.db.collection('static_data');
return staticData.find({
id: {
$nin: existingGeoDataIds
}
}).toArray();
}
function fetchMissingData() {
const existingGeoDataIds = getIdOfAllGeoData();
const staticDataMissingGeo = getStaticDataMissingGeo(existingGeoDataIds);
// staticDataMissingGeo is all the static that need geo data
// you can loop through this array, get each items geo data and insert to database
// ...
}
Finally, you could use bulk operation to speed thing up, it will be much faster. Also, my mongo related code above may not be correct, consider it as an idea.
I'm writing a small utility to copy data from one sqlite database file to another. Both files have the same table structure - this is entirely about moving rows from one db to another.
My code right now:
let tables: Array<string> = [
"OneTable", "AnotherTable", "DataStoredHere", "Video"
]
tables.forEach((table) => {
console.log(`Copying ${table} table`);
sourceDB.each(`select * from ${table}`, (error, row) => {
console.log(row);
destDB.run(`insert into ${table} values (?)`, ...row) // this is the problem
})
})
row here is a js object, with all the keyed data from each table. I'm certain that there's a simple way to do this that doesn't involve escaping stringified data.
If your database driver has not blocked ATTACH, you can simply tell the database to copy everything:
ATTACH '/some/where/source.db' AS src;
INSERT INTO main.MyTable SELECT * FROM src.MyTable;
You could iterate over the row and setup the query with dynamically generated parameters and references.
let tables: Array<string> = [
"OneTable", "AnotherTable", "DataStoredHere", "Video"
]
tables.forEach((table) => {
console.log(`Copying ${table} table`);
sourceDB.each(`select * from ${table}`, (error, row) => {
console.log(row);
const keys = Object.keys(row); // ['column1', 'column2']
const columns = keys.toString(); // 'column1,column2'
let parameters = {};
let values = '';
// Generate values and named parameters
Object.keys(row).forEach((r) => {
var key = '$' + r;
// Generates '$column1,$column2'
values = values.concat(',', key);
// Generates { $column1: 'foo', $column2: 'bar' }
parameters[key] = row[r];
});
// SQL: insert into OneTable (column1,column2) values ($column1,$column2)
// Parameters: { $column1: 'foo', $column2: 'bar' }
destDB.run(`insert into ${table} (${columns}) values (${values})`, parameters);
})
})
Tried editing the answer by #Cl., but was rejected. So, adding on to the answer, here's the JS code to achieve the same:
let sqlite3 = require('sqlite3-promise').verbose();
let sourceDBPath = '/source/db/path/logic.db';
let tables = ["OneTable", "AnotherTable", "DataStoredHere", "Video"];
let destDB = new sqlite3.Database('/your/dest/logic.db');
await destDB.runAsync(`ATTACH '${sourceDBPath}' AS sourceDB`);
await Promise.all(tables.map(table => {
return new Promise(async (res, rej) => {
await destDB.runAsync(`
CREATE TABLE ${table} AS
SELECT * FROM sourceDB.${table}`
).catch(e=>{
console.error(e);
rej(e);
});
res('');
})
}));