I'm having trouble adjusting to the async-first nature of node / js / typescript. This intent of this little function should be pretty clear: it takes a database and returns an array of courses that are listed in that database.
The problem is that the return statement gets run before any of the database operations have run, and I get an empty list. When I set a breakpoint inside the database each loop, I can see that the rows are being found and that courses are being put into ret one by one, but these courses never become visible in the scope where courseList() was called.
const courseList = (database: sqlite3.Database): Course[] => {
let ret = new Array<Course>();
database.serialize();
database.each("select ID, Title from Course", (err: Error, row: Object) => {
ret.push(new Course(
row.ID,
row.Title
))
})
return ret;
}
Suggestions?
The calling code just wants to print information about courses. For example:
let courses = courseList(db);
console.log(courses.length); // logs 0, even though the db contains courses
database.each takes a complete callback. Use that to resume e.g.
const courseList = (database: sqlite3.Database, complete): Course[] => {
let ret = new Array<Course>();
database.serialize();
database.each("select ID, Title from Course", (err: Error, row: Object) => {
ret.push(new Course(
row.ID,
row.Title
))
}, complete);
return ret;
}
let courses = courseList(db, () => {
console.log(courses.length);
});
More
There are better ways to write this. Use promises https://basarat.gitbooks.io/typescript/content/docs/promise.html
The documentation is horrible : https://github.com/mapbox/node-sqlite3/wiki I would be tempted to look elsewhere (TS First) for a database solution. Its not worth the pain for me personally. YMMV.
Related
There may be more than one correct answer to this question, but here's my issue: I have a user document in firebase with many fields that can be updated and which interact in different ways. If one field is updated, it may require a change to another field on the backend. Is it better to have a whole bunch of if statements each with their own write action if the condition is met or, or do single write at the end of the function for all the fields that might change. If a field does not change, I would have to write its original value back to itself. That seems clunky, but so does the other option. Am I missing a third option? What is the best practice here?
Here's an example of what I'm talking about. Updating fields one at a time is what I have now, which looks like this:
export const userUpdate = functions.firestore
.document("users/{userID}")
.onUpdate(async (change) => {
const beforeData = change.before.data();
const afterData = change.after.data();
// user levels up and gets more HP
if(beforeData.userLevel != afterData.userLevel){
const newMaxHP = 15 + 5 * afterData.userLevel;
change.after.ref.update({
maxHp: newMaxHP
})
}
//update user rating
if (beforeData.numberOfRatings != afterData.numberOfRatings) {
const newRating = placerRating(beforeData.userRating, beforeData.numberOfRatings, afterData.latestRating);
change.after.ref.update({
userRating: newRating
})
}
//replenish user funds from zero
if (afterData.money == 0){
change.after.ref.update({
money: 20
})
}
If I did it all in a single write, the if statements would assign a value to a variable, but not update the firestore document. Each if statement would include an else statement assigning the variable to the field's original value. There would be a single write at the end like this:
change.after.ref.update({
maxHp: newMaxHP,
userRating: newRating,
money: 20
})
I hope that helps.
[edit to add follow-up question about updating a map value]
#Dharmaraj's answer works great, but I'm struggling to apply it when updating a map value. BTW - I'm using Typescript.
Before using #Dharmaraj's solution, I was doing this:
admin.firestore().collection("users").doc(lastPlayerAttacker).update({
"equipped.weapon.usesLeft": admin.firestore.FieldValue.increment(-1)
});
Using the update object, I'm trying it like this, but I get the error "Object is of type 'unknown'"
const lastPlayerUpdates:{[key:string]:unknown} = {};
lastPlayerUpdates.equipped.weapon.usesLeft = admin.firestore.FieldValue.increment(-1);
admin.firestore().collection("users").doc(lastPlayerAttacker).update(lastPlayerUpdates);
Any advice on how to fix it?
Every time you call update(), you are being charged for 1 write operation. It'll be best to accumulate all updated fields in an object and then update the document only once as it'll be more efficient too. Try refactoring the code as shown below:
export const userUpdate = functions.firestore
.document("users/{userID}")
.onUpdate(async (change) => {
const beforeData = change.before.data();
const afterData = change.after.data();
const updatedData = {};
// user levels up and gets more HP
if (beforeData.userLevel != afterData.userLevel) {
const newMaxHP = 15 + 5 * afterData.userLevel;
updatedData.maxHp = newMaxHP;
}
//update user rating
if (beforeData.numberOfRatings != afterData.numberOfRatings) {
const newRating = placerRating(beforeData.userRating, beforeData.numberOfRatings, afterData.latestRating);
updatedData.userRating = newRating;
}
//replenish user funds from zero
if (afterData.money == 0) {
updatedData.money = 20;
}
await change.after.ref.update(updatedData);
console.log("Data updated");
return null;
})
I'm trying to build a db with bluebird and sqlite3 to manage a lot of "ingredients".
So far I've managed to parse a file and extrapolate some data from it using regex.
Every time a line matches the regex I want to search in the database if an element with the same name has already been inserted, if so the element is skipped, otherwise it must be inserted.
The problem is that some elements get inserted more than one time.
The code partially works, and I'm saying it partially works because if I remove the line of code where I check the existence of an element with the same name, the duplicate rows are much more.
Here's the piece of code:
lineReader.eachLine(FILE_NAME, (line, last, cb) => {
let match = regex.exec(line);
if (match != null) {
let matchedName = match[1].trim();
//This function return a Promise for all the rows with corresponding name
ingredientsRepo.getByName(matchedName)
.then((entries) => {
if (entries.length > 0) {
console.log("ALREADY INSERTED INGREDIENT: " + matchedName)
} else {
console.log("ADDING " + matchedName)
ingredientsRepo.create(matchedName)
}
})
}
});
I know I'm missing something about Promises but I can't understand what I'm doing wrong.
Here's the code of Both getByName(name) and create(name):
create(name) {
return this.dao.run(
`INSERT INTO ingredients (name) VALUES (?)`,
[name]
)
}
getByName(name) {
return this.dao.all(
'SELECT * FROM ingredients WHERE name == ?',
[name]
)
}
ingredientsRepo.getByName(matchedName) returns a promise, this means it's asynchronous. I am also guessing that ingredientsRepo.create(matchedName) is asynchronous, because you are inserting something into a DB.
Your loop doesn't wait for these async functions to complete. Hence the loop could already be on the 50th iteration, when the .then(...) is called for the first iteration.
So let's say the first 10 elements have the same name. Well, since the asynchronous functions take some time to complete, the name will not be inserted into the DB until say maybe the 20th iteration of the loop. So for the first 10 elements, the name does not exist within the DB.
It's a timing issue.
I have two queries:
a) select id from ingredietns where name = my_param;
b) select word_id from synonyms where name = my_param;
Both return 0 or 1 row. I can also add limit 1 if needed (or in knex first()).
I can translate each into knex like this:
knex("ingredients").select('id').where('name', my_param) //do we need first()?
knex("synonyms").select('word_id').where('name', my_param) //do we need first()?
I need function called "ingredientGetOrCreate(my_param)". This function would
a) check if any of above queries return result
b) if any of these return, then return ingredients.id or synonyms.word_id - only one could be returned
c) if record doesn't eixst in any of tables, I need to do knex inesrt aand return newly added id from function
d) later I am not sure I also understand how to call this newly create function.
Function ingredientGetOrCreate would be used later as seperate function or in the following scenario (like "loop") that doesn't work for me either:
knex("products") // for each product
.select("id", "name")
.map(function (row) {
var descriptionSplitByCommas = row.desc.split(",");
Promise.all(descriptionSplitByCommas
.map(function (my_param) {
// here it comes - call method for each param and do insert
ingredientGetOrCreate(my_param)
.then(function (id_of_ingredient) {
knex('ingredients_products').insert({ id_of_ingredient });
});
...
I am stuck with knex and Promise queries because of asynchronouse part. Any clues, please?
I though I can somehow use Promise.all or Promise.some to call both queries.
P.S. This is my first day with nodejs, Promise and knex.
As far as I decode your question, it consists of two parts:
(1) You need to implement upsert logic (get-or-create logic).
(2) Your get part requires to query not a single table, but a pair of tables in specific order. Table names imply that this is some sort of aliasing engine inside of your application.
Let's start with (2). This could definitely be solved with two queries, just like you sense it.
function pick_name (rows)
{
if (! rows.length) return null
return rows[0].name
}
// you can sequence queries
function ingredient_get (name)
{
return knex('ingredients')
.select('id').where('name', name)
.then(pick_name)
.then(name =>
{
if (name) return name
return knex('synonyms')
.select('word_id').where('name', name)
.then(pick_name)
})
}
// or run em parallel
function ingredient_get (name)
{
var q_ingredients = knex('ingredients')
.select('id').where('name', name)
.then(pick_name)
var q_synonyms = knex('synonyms')
.select('word_id').where('name', name)
.then(pick_name)
return Promise.all([ q_ingredients, q_synonyms ])
.then(([name1, name2]) =>
{
return name1 || name2
})
}
Important notions here:
Both forms works well and return first occurence or JS' null.
First form optimizes count of queries to DB.
Second form optimizes answer time.
However, you can go deeper and use more SQL. There's a special tool for such task called COALESCE. You can consult your SQL documentation, here's COLASCE of PostgreSQL 9. The main idea of COALESCE is to return first non-NULL argument or NULL otherwise. So, you can leverage this to optimize both queries and answer time.
function ingredient_get (name)
{
// preparing but not executing both queries
var q_ingredients = knex('ingredients')
.select('id').where('name', name)
var q_synonyms = knex('synonyms')
.select('word_id').where('name', name)
// put them in COALESCE
return knex.raw('SELECT COALESCE(?, ?) AS name', [ q_ingredients, q_synonyms ])
.then(pick_name)
This solution guarantees single query and furthermore DB engine can optimize execution in any way it sees appropriate.
Now let's solve (1): We now got ingredient_get(name) which returns Promise<string | null>. We can use its output to activate create logic or return our value.
function ingredient_get_or_create (name, data)
{
return ingredient_get(name)
.then(name =>
{
if (name) return name
// …do your insert logic here
return knex('ingredients').insert({ name, ...data })
// guarantee homohenic output across get/create calls:
.then(() => name)
})
}
Now ingredient_get_or_create do your desired upsert logic.
UPD1:
We already got ingredient_get_or_create which returns Promise<name> in any scenario (both get or create).
a) If you need to do any specific logic after that you can just use then:
ingredient_get_or_create(…)
.then(() => knex('another_table').insert(…))
.then(/* another logic after all */)
In promise language that means «do that action (then) if previous was OK (ingredient_get_or_create)». In most of the cases that is what you need.
b) To implement for-loop in promises you got multiple different idioms:
// use some form of parallelism
var qs = [ 'name1', 'name2', 'name3' ]
.map(name =>
{
return ingredient_get_or_create(name, data)
})
var q = Promise.all(qs)
Please, note, that this is an agressive parallelism and you'll get maximum of parallel queries as your input array provides.
If it's not desired, you need to limit parallelism or even run tasks sequentially. Bluebird's Promise.map is a way to run map which analogous to example above but with concurrency option available. Consider the docs for details.
There's also Bluebird's Promise.mapSeries which conceptually is an analogue to for-loop but with promises. It's like map which runs sequentially. Look the docs for details.
Promise.mapSeries([ 'name1', 'name2', 'name3' ],
(name) => ingredient_get_or_create(name, data))
.then(/* logic after all mapSeries are OK */)
I believe the last is what you need.
First: This is the first project in which I am using RxJs, I thought I will learn best by using it.
I found this answer: Turning paginated requests into an Observable stream with RxJs
But it says in the comments:
You're exceeding the maximum call stack still. At around 430 pages returned. I think recursion might not be the best solution here
I want to query the Youtube Data API, the results come back in pages and I need to paginate through them.
I imagined a work flow like this could work:
1)Initiate a call
2)Check if the response has a 'nextPageToken'
3)If it has, do another request to the Youtube API
4)If not, finish
So to do this I could Imagine the following Observables / streams:
FirstRequestStream -A-X--------------->
ResponseStream -A-A-A-A--X-------->
RequestStream -I-A-I-A----------->
A = Action
I = Info from upper stream
X = Termination
(Not sure if this diagram is correct the way I made it)
So the ResponseStream depends on FirstRequestStream and RequestStream(using the merge function). The RequestStream depends on the ResponseStream( is this called a circulating observable ?)
-Is this the right approach ?
-Are 'circulating observables' a good thing, are they even possible ?(I had problems creating one).
-Any other way I should try first?
-Is it possible to create interdependent observable streams ?
Thank you for your help.
You are overcomplicating this problem, it can be solved a lot easier using defer operator.
Idea is that you are creating deferred observable (so it will be created and start fetching data only after subscription) and concatenate it with the same observable but for the next page, which will be also concatenated with the next page, and so on ... . And all of that can be done without recursion.
Here is how the code looks:
const { defer, from, concat, EMPTY, timer } = rxjs; // = require("rxjs")
const { mergeMap, take, mapTo, tap } = rxjs.operators; // = require("rxjs/operators")
// simulate network request
function fetchPage(page=0) {
return timer(100).pipe(
tap(() => console.log(`-> fetched page ${page}`)),
mapTo({
items: Array.from({ length: 10 }).map((_, i) => page * 10 + i),
nextPage: page + 1,
})
);
}
const getItems = page => defer(() => fetchPage(page)).pipe(
mergeMap(({ items, nextPage }) => {
const items$ = from(items);
const next$ = nextPage ? getItems(nextPage) : EMPTY;
return concat(items$, next$);
})
);
// process only first 30 items, without fetching all of the data
getItems()
.pipe(take(30))
.subscribe(e => console.log(e));
<script src="https://unpkg.com/rxjs#6.2.2/bundles/rxjs.umd.min.js"></script>
Here is my solution using the rxjs operators expand, reduce and empty using the HttpClient module:
Suppose your API response is an object containing shaped like the following
interface Response {
data: items[]; // array of result items
next: string|null; // url of next page, or null if there are no more items
}
You could use expand and reduce like so
getAllResults(url) {
return this.http.get(url).pipe(
expand((res) => res.next ? this.http.get(res.next) : EMPTY),
reduce((acc, res) => acc.concat(res.data), [])
);
}
I shamelessly reuse the code snippet from Oles Savluk, with its good fetchPage function, and I apply the ideas explained in the blog article linked to by Picci (in the comments), using expand.
Article on expand by Nicholas Jamieson
It gives a slightly simpler code, with recursion hidden in the expand call (and comments of the article show how to linearize it, if needed).
const { timer, EMPTY } = rxjs; // = require("rxjs")
const { concatMap, expand, mapTo, tap, toArray } = rxjs.operators; // = require("rxjs/operators")
// simulate network request
const pageNumber = 3;
function fetchPage(page = 0) {
return timer(1000).pipe(
tap(() => console.log(`-> fetched page ${page}`)),
mapTo({
items: Array.from({ length: 10 }).map((_, i) => page * 10 + i),
nextPage: ++page === pageNumber ? undefined : page,
}),
);
}
const list = fetchPage().pipe(
expand(({ nextPage }) => nextPage ? fetchPage(nextPage) : EMPTY),
concatMap(({ items }) => items),
// Transforms the stream of numbers (Observable<number>)
// to a stream with only an array of numbers (Observable<number[]>).
// Remove if you want a stream of numbers, not waiting for all requests to complete.
toArray(),
);
list.subscribe(console.log);
<script src="https://unpkg.com/rxjs#6.2.2/bundles/rxjs.umd.min.js"></script>
LuJaks is definitively the simplest approach !
For a one line example, suppose you have a function that make a http request for a given page and that returns a (partial) array of data. We call that function until server returns empty array :
import { Observable, EMPTY, of } from "rxjs";
import { expand, reduce } from "rxjs/operators";
// Mock a request that returns only 5 pages...
function httpGet(p): Observable<number[]> {
if (p > 5) { return of([]); }
return of(new Array(10).fill(0).map((_, i) => p * 10 + i));
}
httpGet(0).pipe( // get the fist page
expand((value, index) => (value.length > 0 ? httpGet(index + 1) : EMPTY)), // other pages
reduce((a, v) => [...a, ...v], []), // optional if you want only one emit
).subscribe((x) => console.log(x));
There are two collections products and stock in my database.Each product have multipple stock with different supplier and attributes of products.
I have selected the products from products collection and run a loop for each product. I need to append price and offer price from the stock collection to products i have selected already.
I think the for loop compleated its execution before executiong the find method on stock collection.I need to execute everything in the loop in serial manner(not asynchronous). Please check the code below and help me to solve this. I'm new in node.js and mongodb
collection = db.collection('products');
collection.find().toArray(function(err, abc) {
var finalout = [];
for( var listProducts in abc){
var subfinal = {
'_id' :abc[listProducts]['_id'],
'min_price' :abc[listProducts]['value']['price'],
'stock' :abc[listProducts]['value']['stock'],
'name' :abc[listProducts]['value']['name'],
'price' :'',
'offer_price' :'',
};
collection = db.collection('stock');
collection.find({"product":abc[listProducts]['_id'] ,"supplier":abc[listProducts]['value']['def_supplier']}).toArray(function(err, newprod) {
for( var row in newprod){
subfinal['price'] = newprod[row]['price'];
subfinal.offer_price = newprod[row]['offer_price'];
}
finalout.push(subfinal);
});
}
console.log(finalout);
});
Yes, the loop is running and starting the get requests on the database all at the same time. It is possible, like you said, to run them all sequentially, however it's probably not what you're looking for. Doing it this way will take more time, since each request to Mongo would need to wait for the previous one to finish.
Considering that each iteration of the loop doesn't depend on the previous ones, all you're really looking for is a way to know once ALL of the operations are finished. Keep in mind that these can end in any order, not necessarily the order in which they were initiated in the loop.
There's was also an issue in the creation of the subfinal variable. Since the same variable name is being used for all iterations, when they all come back they'll be using the final assignment of the variable (all results will be written on the same subfinal, which will have been pushed multiple times into the result). To fix that, I've wrapped the entire item processing iteration into another function to give each subfinal variable it's own scope.
There are plenty of modules to ease this kind of management, however here is one way to run some code only after all the Mongo calls are finished using a counter and callback that doesn't require any additional installs:
var finished = 0; // incremented every time an iteration is done
var total = Object.keys(abc).length; // number of keys in the hash table
collection = db.collection('products');
collection.find().toArray(function(err, abc) {
var finalout = [];
for( var listProducts in abc){
processItem(listProducts);
}
});
function processItem(listProducts) {
var subfinal = {
'_id' :abc[listProducts]['_id'],
'min_price' :abc[listProducts]['value']['price'],
'stock' :abc[listProducts]['value']['stock'],
'name' :abc[listProducts]['value']['name'],
'price' :'',
'offer_price' :'',
};
collection = db.collection('stock');
collection.find({"product":abc[listProducts]['_id'] ,"supplier":abc[listProducts]['value']['def_supplier']}).toArray(function(err, newprod) {
for( var row in newprod){
subfinal['price'] = newprod[row]['price'];
subfinal.offer_price = newprod[row]['offer_price'];
}
finalout.push(subfinal);
finished++;
if (finished === total) { // if this is the last one to finish.
allFinished(finalout);
}
});
}
function allFinished(finalout) {
console.log(finalout);
}
It could be written more concisely with less variables, but it should be easier to understand this way.