Node Synchronous loops - node.js

I've got a function.
function async extractTars (tarList) {
try {
for (let i = 0; i < tarList.length; i ++) {
// I need this loop to be sync but it isn't hitting the next iteration
return new Promise((resolve, reject) => {
fs.createReadStream(`${PATHS.TAR}/${tarList[i]}`)
.pipe(tar.extract(PATHS.GZ))
.on('error', err => reject(err))
.on('finish', () => resolve())
})
}
// What is the correct way to resolve this async fn? Should i just return or will it resolve anyway after the loop?
} catch (e) {
// handle error
}
}
For some reason it never hits the next iteration of the loop. What am i missing here? How can i make these type of loops synchronous. In this particular project i've got many many loops that need to complete before the next. I've tried several methods that i've see here on SO but i'm obviously missing something.
Any help would be much appreciated!

You're using return, that's why you're not hitting the next iteration, use await instead to wait until the promise is done before going to the next iteration.
return inside a for will stop the loop & end the function returning the new Promise
function async extractTars (tarList) {
try {
for (let i = 0; i < tarList.length; i ++) {
// I need this loop to be sync but it isn't hitting the next iteration
await new Promise((resolve, reject) => {
fs.createReadStream(`${PATHS.TAR}/${tarList[i]}`)
.pipe(tar.extract(PATHS.GZ))
.on('error', err => reject(err))
.on('finish', () => resolve())
})
}
// What is the correct way to resolve this async fn? Should i just return or will it resolve anyway after the loop?
} catch (e) {
// handle error
}
}
And just to clear things up:
I need this loop to be sync but it isn't hitting the next iteration
That loop won't be sync, it will look like it's sync, when using async/await but you can't make asynchronous code be synchronous. And createReadStream is asnychronous.
And regarding:
What is the correct way to resolve this async fn? Should i just
return or will it resolve anyway after the loop?
When all iterations are done or an error is thrown (since you're catching it and not rejecting), meaning that you read all the files in tarList, the function will be resolved. Since you are not returning anything, undefined will be the resolved value.
From MDN:
When a return statement is used in a function body, the execution of the function is stopped. If specified, a given value is returned to the function caller. For example, the following function returns the square of its argument, x, where x is a number.

Related

Node js issue with loop order

Hello i really need help with this issue, my last console.log is execute BEFORE the for loop and i dont know how to fix it. I really need to have access at my array nbfilm after the for loop
Can someone help me?
What the console print : lien
client.db.query("SELECT name,id FROM film", function (err, result) {
if (err) throw err;
const catalog = new MessageEmbed()
.setTitle("Catalogue")
.setColor("#fcfe80")
.setFooter({text:"🍿 ・ PopFlix"})
let testresult =[]
let nbfilm =[]
for (let compteur of result){
testresult.push(compteur.id)
testresult.push(compteur.name)
}
console.log(testresult)
for (let compteur2 = 0; compteur2 < testresult.length; compteur2+=2){
client.db.query(`SELECT link FROM lien WHERE fid=${testresult[compteur2]}`, function (err,result) {
nbfilm.push(testresult[compteur2+1])
nbfilm.push(result.length)
console.log("nbfilm in for loop",nbfilm)
});
}
console.log("nbfilmAFTER",nbfilm)
});
The body of the loop delays execution. Due to the fact that javascript is an asynchronous i/o type operation language, it is common to return a promise. In other words, the code is executed as expected, but the result of the actions will be visible only after all pending tasks have been completed. In your case, adding async/await in code might help. Use it node docs
It looks like client.db.query is asynchroneous. JS here works as expected because it doesnt wait for the query to be finished before moving to the next line.
Its now a better practice to use async/await instead of callbacks. If you provide the package that you are using we can provide a code example.
client.db.query() is an asynchronous function and won't execute until after the console.log(nbfilm) is executed regardless of how fast that query actually runs.
I'd recommend using Promise.all(). You will also have to "promisify" the query() function. Pass everything you want to resolve(), and then concatenate them in the final line as below.
let promises = [];
for(....) {
promises.push(new Promise((resolve, reject) => {
client.db.query("select ... ", () => {
// all the same stuffs
resolve([testresult[compteru2+1], result.length]);
});
}))
}
Promise.all(promises)
.then((results) => results.reduce((p, c) => p.concat(c), []))
.then(data => console.log(data)); // do whatever you want with "data"
Here's a simplified demo:
https://codesandbox.io/s/promise-all-example-78r347

Trouble with Promise and Async functions

I am trying to use promises and async functions to send chunks of an array to an api call that inserts them into a DB. I am trying to get the top function to chunk the array and await for the backend to finsih then it will move on to the next chunk. It errors out after the first iteration. Any ideas??
async chunkArray(arr) {
let len = arr.length
let update_arr
let i = 1
for(i; i<=len; i++) {
if((i%125) === 0) {
update_arr = arr.slice(i-125,i)
await this.updateChuckArray(update_arr)
} else if(i === len) {
let div = (Math.floor(len/125) * 125)
update_arr = arr.slice(div, len)
await this.updateChuckArray(update_arr)
}
}
},
updateChuckArray(update) {
return new Promise(resolve => {
this.$http.put(`/route`, update).then(res => {
res.data.error ? this.$root.updateError(res.data.error) : this.$root.updateSuccess(res.data.message)
}).catch(error => {
this.$root.updateError(res.data.error)
})
})
}
First off your updateChuckArray() never resolves the promise it returns (you never call resolve()).
Instead of manually wrapping a new promise around your function call (that is a promise anti-pattern), you can just return the promise you already have and write it like this:
updateChuckArray(update) {
return this.$http.put(`/route`, update).then(res => {
res.data.error ? this.$root.updateError(res.data.error) : this.$root.updateSuccess(res.data.message);
}).catch(error => {
this.$root.updateError(error);
})
}
FYI, it's unclear what your error handling strategy is. The way you wrote the code (which is followed above), you catch an error from this.$http.put() and handle it and let the loop continue. If that's what you want, this will work. If you want the for loop to abort on error, then you need to rethrow the error in the .catch() handler so the error gets back to the await.
Also, not that in your .catch() handler, you were doing this:
this.$root.updateError(res.data.error)
but there is no res defined there. The error is in error. You would need to use that in order to report the error. I'm not sure what the structure of the error object is here or what exactly you pass to $.root.updateError(), but it must be something that comes from the error object, not an object named res.

Waiting in a while loop on an async function (Node.js/ES6)

I'm writing a Windows Node.js server app (using ES6 btw).
The first thing I want to do - in the top-level code - is sit in a while loop, calling an async function which searches for a particular registry key/value. This function is 'proven' - it returns the value data if found, or else throws:
async GetRegValue(): Promise<string> { ... }
I need to sit in a while loop until the registry item exists, and then grab the value data. (With a delay between retries).
I think I know how to wait for an async call to complete (one way or the other) before progressing with the rest of the start-up, but I can't figure out how to sit in a loop waiting for it to succeed.
Any advice please on how to achieve this?
(I'm fairly new to typescript, and still struggling to get my head round all async/await scenarios!)
Thanks
EDIT
Thanks guys. I know I was 'vague' about my code - I didn't want to put my real/psuedo code attempts, since they have all probably overlooked the points you can hopefully help me understand.
So I just kept it as a textual description... I'll try though:
async GetRegValue(): Promise<string> {
const val: RegistryItem = await this.GetKeyValue(this.KEY_SW, this.VAL_CONN);
return val.value
}
private async GetKeyValue(key: string, name: string): Promise<RegistryItem> {
return await new Promise((resolve, reject) => {
new this.Registry({
hive: this.Hive, key
}).get(name, (err, items) => {
if (err) {
reject(new Error('Registry get failed'));
}
else {
resolve( items );
}
});
})
.catch(err => { throw err });
}
So I want to do something like:
let keyObtained = false
let val
while (keyObtained == false)
{
// Call GetRegValue until val returned, in which case break from loop
// If exception then pause (e.g. ~100ms), then loop again
}
}
// Don't execute here till while loop has exited
// Then use 'val' for the subsequent statements
As I say, GetRegValue() works fine in other places I use it, but here I'm trying to pause further execution (and retry) until it does come back with a value
You can probably just use recursion. Here is an example on how you can keep calling the GetRegValue function until is resolves using the retryReg function below.
If the catch case is hit, it will just call GetRegValue over and over until it resolves successfully.
you should add a counter in the catch() where if you tried x amount of times you give up.
Keep in mind I mocked the whole GetRegValue function, but given what you stated this would still work for you.
let test = 0;
function GetRegValue() {
return new Promise((resolve, reject) => {
setTimeout(function() {
test++;
if (test === 4) {
return resolve({
reg: "reg value"
});
}
reject({
msg: "not ready"
});
}, 1000);
});
}
function retryReg() {
GetRegValue()
.then(registryObj => {
console.log(`got registry obj: ${JSON.stringify(registryObj)}`)
})
.catch(fail => {
console.log(`registry object is not ready: ${JSON.stringify(fail)}`);
retryReg();
});
}
retryReg();
I don't see why you need this line:
.catch(err => { throw err });
The loop condition of while isn't much use in this case, as you don't really need a state variable or expression to determine if the loop should continue:
let val;
while (true)
{
try {
val = await GetRegValue(/* args */);
break;
} catch (x) {
console.log(x); // or something better
}
await delay(100);
}
If the assignment to val succeeds, we make it to the break; statement and so we leave the loop successfully. Otherwise we jump to the catch block and log the error, wait 100 ms and try again.
It might be better to use a for loop and so set a sensible limit on how many times to retry.
Note that delay is available in an npm package of the same name. It's roughly the same as:
await new Promise(res => setTimeout(res, 100));

Promise map (node.js) with large list seems to stop iterating after certain index

I'm using the bluebird Promise library. From this library I'm using Promise.mapSeries() to perform a mapping operation on a very large list (size=747357).
The code looks as follows (psuedo):
function myFunc(data) {
return Promise.mapSeries(data, handler)
.then((data) => {
console.log('Success!');
return Promise.resolve(data);
})
.catch(console.log);
}
In the handler, I'm doing a couple of things:
Running other Promise functions (recursive)
Returning a new data structure to be stored in the resulting array
In the handler function, I add a console.log('Iter: ', i);.
So then this returns the iter # for each item that's been mapped. It slows and eventually stops at #288. Does this reflect some sort of limit I'm hitting?
I don't understand what the problem is - logic says perhaps this list is too large to handle with Promise.mapSeries.
Any advice or solutions would be appreciated.
Thanks in advance.
UPDATE:
Here's a snippet of the handler function:
function handler(v, i, len) {
return new Promise((resolve, reject) => {
promise
.then((data) => {
return recursivePromiseFn(data)
})
.then((data) => {
let _data = transformData(data);
statusLogger(i);
resolve(_data);
})
})
}
The problem was not with Promise.map or Promise.mapSeries. The problem was an infinite loop in one of my promise functions. This loop seemed to prevent the promise loop from continuing.
A noteworthy comment was that you should avoid using the Promise anti-pattern. This is true but was not the cause of the problem.

Node promises: use first/any result (Q library)

I understand using the Q library it's easy to wait on a number of promises to complete, and then work with the list of values corresponding to those promise results:
Q.all([
promise1,
promise2,
.
.
.
promiseN,
]).then(results) {
// results is a list of all the values from 1 to n
});
What happens, however, if I am only interested in the single, fastest-to-complete result? To give a use case: Say I am interested in examining a big list of files, and I'm content as soon as I find ANY file which contains the word "zimbabwe".
I can do it like this:
Q.all(fileNames.map(function(fileName) {
return readFilePromise(fileName).then(function(fileContents) {
return fileContents.contains('zimbabwe') ? fileContents : null;
}));
})).then(function(results) {
var zimbabweFile = results.filter(function(r) { return r !== null; })[0];
});
But I need to finish processing every file even if I've already found "zimbabwe". If I have a 2kb file containing "zimbabwe", and a 30tb file not containing "zimbabwe" (and suppose I'm reading files asynchronously) - that's dumb!
What I want to be able to do is get a value the moment any promise is satisfied:
Q.any(fileNames.map(function(fileName) {
return readFilePromise(fileName).then(function(fileContents) {
if (fileContents.contains('zimbabwe')) return fileContents;
/*
Indicate failure
-Return "null" or "undefined"?
-Throw error?
*/
}));
})).then(function(result) {
// Only one result!
var zimbabweFile = result;
}).fail(function() { /* no "zimbabwe" found */ });
With this approach I won't be waiting on my 30tb file if "zimbabwe" is discovered in my 2kb file early on.
But there is no such thing as Q.any!
My question: How do I get this behaviour?
Important note: This should return without errors even if an error occurs in one of the inner promises.
Note: I know I could hack Q.all by throwing an error when I find the 1st valid value, but I'd prefer to avoid this.
Note: I know that Q.any-like behavior could be incorrect, or inappropriate in many cases. Please trust that I have a valid use-case!
You are mixing two separate issues: racing, and cancelling.
Racing is easy, either using Promise.race, or the equivalent in your favorite promise library. If you prefer, you could write it yourself in about two lines:
function race(promises) {
return new Promise((resolve, reject) =>
promises.forEach(promise => promise.then(resolve, reject)));
}
That will reject if any promise rejects. If instead you want to skip rejects, and only reject if all promises reject, then
function race(promises) {
let rejected = 0;
return new Promise((resolve, reject) =>
promises.forEach(promise => promise.then(resolve,
() => { if (++rejected === promises.length) reject(); }
);
}
Or, you could use the promise inversion trick with Promise.all, which I won't go into here.
Your real problem is different--you apparently want to "cancel" the other promises when some other one resolves. For that, you will need additional, specialized machinery. The object that represents each segment of processing will need some way to ask it to terminate. Here's some pseudo-code:
class Processor {
promise() { ... }
terminate() { ... }
}
Now you can write your version of race as
function race(processors) {
let rejected = 0;
return new Promise((resolve, reject) =>
processors.forEach(processor => processor.promise().then(
() => {
resolve();
processors.forEach(processor => processor.terminate());
},
() => { if (++rejected === processors.length) reject(); }
);
);
}
There are various proposals to handle promise cancellation which might make this easier when they are implemented in a few years.

Resources