I'm writing an HTTP API with expressjs in Node.js and here is what I'm trying to achieve:
I have a regular task that I would like to run regularly, approx every minute. This task is implemented with an async function named task.
In reaction to a call in my API I would like to have that task called immediately as well
Two executions of the task function must not be concurrent. Each execution should run to completion before another execution is started.
The code looks like this:
// only a single execution of this function is allowed at a time
// which is not the case with the current code
async function task(reason: string) {
console.log("do thing because %s...", reason);
await sleep(1000);
console.log("done");
}
// call task regularly
setIntervalAsync(async () => {
await task("ticker");
}, 5000) // normally 1min
// call task immediately
app.get("/task", async (req, res) => {
await task("trigger");
res.send("ok");
});
I've put a full working sample project at https://github.com/piec/question.js
If I were in go I would do it like this and it would be easy, but I don't know how to do that with Node.js.
Ideas I have considered or tried:
I could apparently put task in a critical section using a mutex from the async-mutex library. But I'm not too fond of adding mutexes in js code.
Many people seem to be using message queue libraries with worker processes (bee-queue, bullmq, ...) but this adds a dependency to an external service like redis usually. Also if I'm correct the code would be a bit more complex because I need a main entrypoint and an entrypoint for worker processes. Also you can't share objects with the workers as easily as in a "normal" single process situation.
I have tried RxJs subject in order to make a producer consumer channel. But I was not able to limit the execution of task to one at a time (task is async).
Thank you!
You can make your own serialized asynchronous queue and run the tasks through that.
This queue uses a flag to keep track of whether it's in the middle of running an asynchronous operation already. If so, it just adds the task to the queue and will run it when the current operation is done. If not, it runs it now. Adding it to the queue returns a promise so the caller can know when the task finally got to run.
If the tasks are asynchronous, they are required to return a promise that is linked to the asynchronous activity. You can mix in non-asynchronous tasks too and they will also be serialized.
class SerializedAsyncQueue {
constructor() {
this.tasks = [];
this.inProcess = false;
}
// adds a promise-returning function and its args to the queue
// returns a promise that resolves when the function finally gets to run
add(fn, ...args) {
let d = new Deferred();
this.tasks.push({ fn, args: ...args, deferred: d });
this.check();
return d.promise;
}
check() {
if (!this.inProcess && this.tasks.length) {
// run next task
this.inProcess = true;
const nextTask = this.tasks.shift();
Promise.resolve(nextTask.fn(...nextTask.args)).then(val => {
this.inProcess = false;
nextTask.deferred.resolve(val);
this.check();
}).catch(err => {
console.log(err);
this.inProcess = false;
nextTask.deferred.reject(err);
this.check();
});
}
}
}
const Deferred = function() {
if (!(this instanceof Deferred)) {
return new Deferred();
}
const p = this.promise = new Promise((resolve, reject) => {
this.resolve = resolve;
this.reject = reject;
});
this.then = p.then.bind(p);
this.catch = p.catch.bind(p);
if (p.finally) {
this.finally = p.finally.bind(p);
}
}
let queue = new SerializedAsyncQueue();
// utility function
const sleep = function(t) {
return new Promise(resolve => {
setTimeout(resolve, t);
});
}
// only a single execution of this function is allowed at a time
// so it is run only via the queue that makes sure it is serialized
async function task(reason: string) {
function runIt() {
console.log("do thing because %s...", reason);
await sleep(1000);
console.log("done");
}
return queue.add(runIt);
}
// call task regularly
setIntervalAsync(async () => {
await task("ticker");
}, 5000) // normally 1min
// call task immediately
app.get("/task", async (req, res) => {
await task("trigger");
res.send("ok");
});
Here's a version using RxJS#Subject that is almost working. How to finish it depends on your use-case.
async function task(reason: string) {
console.log("do thing because %s...", reason);
await sleep(1000);
console.log("done");
}
const run = new Subject<string>();
const effect$ = run.pipe(
// Limit one task at a time
concatMap(task),
share()
);
const effectSub = effect$.subscribe();
interval(5000).subscribe(_ =>
run.next("ticker")
);
// call task immediately
app.get("/task", async (req, res) => {
effect$.pipe(
take(1)
).subscribe(_ =>
res.send("ok")
);
run.next("trigger");
});
The issue here is that res.send("ok") is linked to the effect$ streams next emission. This may not be the one generated by the run.next you're about to call.
There are many ways to fix this. For example, you can tag each emission with an ID and then wait for the corresponding emission before using res.send("ok").
There are better ways too if calls distinguish themselves naturally.
A Clunky ID Version
Generating an ID randomly is a bad idea, but it gets the general thrust across. You can generate unique IDs however you like. They can be integrated directly into the task somehow or can be kept 100% separate the way they are here (task itself has no knowledge that it's been assigned an ID before being run).
interface IdTask {
taskId: number,
reason: string
}
interface IdResponse {
taskId: number,
response: any
}
async function task(reason: string) {
console.log("do thing because %s...", reason);
await sleep(1000);
console.log("done");
}
const run = new Subject<IdTask>();
const effect$: Observable<IdResponse> = run.pipe(
// concatMap only allows one observable at a time to run
concatMap((eTask: IdTask) => from(task(eTask.reason)).pipe(
map((response:any) => ({
taskId: eTask.taskId,
response
})as IdResponse)
)),
share()
);
const effectSub = effect$.subscribe({
next: v => console.log("This is a shared task emission: ", v)
});
interval(5000).subscribe(num =>
run.next({
taskId: num,
reason: "ticker"
})
);
// call task immediately
app.get("/task", async (req, res) => {
const randomId = Math.random();
effect$.pipe(
filter(({taskId}) => taskId == randomId),
take(1)
).subscribe(_ =>
res.send("ok")
);
run.next({
taskId: randomId,
reason: "trigger"
});
});
I'm currently setting up a CI environment to automate e2e tests our team runs in a test harness. I am setting this up on Gitlab and currently using Puppeteer. I have an event that fires from our test harness that designates when the test is complete. Now I am trying to "pool" the execution so I don't use up all resources or run out of listeners. I decided to try out "puppeteer-cluster" for this task. I am close to having things working, however I can't seem to get it to wait for the event on page before closing the browser. Prior to using puppeteer-cluster, I was passing in a callback to my function and when the custom event was fired (injected via exposeFunction), I would go about calling it. That callback function is now being passed in data though now and therefore not waiting. I can't seem to find a way to get the execution to wait and was hoping someone might have an idea here. If anyone has any recommendations, I'd love to hear them.
test('Should launch the browser and run e2e tests', async (done) => {
try {
const cluster = await Cluster.launch({
concurrency: Cluster.CONCURRENCY_CONTEXT,
maxConcurrency: 10,
monitor: false,
timeout: 1200000,
puppeteerOptions: browserConfig
});
// Print errors to console
cluster.on("taskerror", (err, data) => {
console.log(`Error crawling ${data}: ${err.message}`);
});
//Setup our task to be run
await cluster.task( async ({page, data: {testUrl, isLastIndex, cb}, worker}) => {
console.log(`Test starting at url: ${testUrl} - isLastIndex: ${isLastIndex}`);
await page.goto(testUrl);
await page.waitForSelector('#testHarness');
await page.exposeFunction('onCustomEvent', async (e) => {
if (isLastIndex === true){ ;
//Make a call to our callback, finalizing tests are complete
cb();
}
console.log(`Completed test at url: ${testUrl}`);
});
await page.evaluate(() => {
document.addEventListener('TEST_COMPLETE', (e) => {
window.onCustomEvent('TEST_COMPLETE');
console.log("TEST COMPLETE");
});
});
});
//Perform the assignment of all of our xml tests to an array
let arrOfTests = await buildTestArray();
const arrOfTestsLen = arrOfTests.length;
for( let i=0; i < arrOfTestsLen; ++i){
//push our tests on task queue
await cluster.queue( {testUrl: arrOfTests[i], isLastIndex: (i === arrOfTestsLen - 1), cb: done });
};
await cluster.idle();
await cluster.close();
} catch (error) {
console.log('ERROR:',error);
done();
throw error;
}
});
So I got something working, but it really feels hacky to me and I'm not really sure it is the right approach. So should anyone have the proper way of doing this or a more recommended way, don't hesitate to respond. I am posting here shoudl anyone else deal with something similar. I was able to get this working with a bool and setInterval. I have pasted working result below.
await cluster.task( async ({page, data: {testUrl, isLastIndex, cb}, worker}) => {
let complete = false;
console.log(`Test starting at url: ${testUrl} - isLastIndex: ${isLastIndex}`);
await page.goto(testUrl)
await page.waitForSelector('#testHarness');
await page.focus('#testHarness');
await page.exposeFunction('onCustomEvent', async (e) => {
console.log("Custom event fired");
if (isLastIndex === true){ ;
//Make a call to our callback, finalizing tests are complete
cb();
complete = true;
//console.log(`VAL IS ${complete}`);
}
console.log(`Completed test at url: ${testUrl}`);
});
//This will run on the actual page itself. So setup an event listener for
//the TEST_COMPLETE event sent from the test harness itself
await page.evaluate(() => {
document.addEventListener('TEST_COMPLETE', (e) => {
window.onCustomEvent('TEST_COMPLETE');
});
});
await new Promise(resolve => {
try {
let timerId = setInterval(()=>{
if (complete === true){
resolve();
clearInterval(timerId);
}
}, 1000);
} catch (e) {
console.log('ERROR ', e);
}
});
});
I have a small web scraping application that downloads multiple files from a web application where the URLs require visting the page.
It works fine if I keep the browser instance alive in between runs, but I want to close the instance in between runs. When I call browser.close() my downloads are stopped because the chrome instance is closed before the downloads have finished.
Does puppeteer provide a way to check if downloads are still active, and wait for them to complete? I've tried page.waitForNavigation({ waitUntil: "networkidle0" }) and "networkidle2", but those seem to wait indefinitely.
node.js 8.10
puppeteer 1.10.0
Update:
It's 2022. Use Playwright to get away from this mass. manage downloads
It also has 'smarter' locator, which examine selectors every time before click()
old version for puppeteer:
My solution is to use chrome's own chrome://downloads/ page to managing download files. This solution can be very easily to auto restart a failed download using chrome's own feature
This example is 'single thread' currently, because it's only monitoring the first item appear in the download manager page. But you can easily adapt it to 'infinite threads' by iterating through all download items (#frb0~#frbn) in that page, well, take care of your network:)
dmPage = await browser.newPage()
await dmPage.goto('chrome://downloads/')
await your_download_button.click() // start download
await dmPage.bringToFront() // this is necessary
await dmPage.waitForFunction(
() => {
// monitoring the state of the first download item
// if finish than return true; if fail click
const dm = document.querySelector('downloads-manager').shadowRoot
const firstItem = dm.querySelector('#frb0')
if (firstItem) {
const thatArea = firstItem.shadowRoot.querySelector('.controls')
const atag = thatArea.querySelector('a')
if (atag && atag.textContent === '在文件夹中显示') { // may be 'show in file explorer...'? you can try some ids, classess and do a better job than me lol
return true
}
const btn = thatArea.querySelector('cr-button')
if (btn && btn.textContent === '重试') { // may be 'try again'
btn.click()
}
}
},
{ polling: 'raf', timeout: 0 }, // polling? yes. there is a 'polling: "mutation"' which kind of async
)
console.log('finish')
An alternative if you have the file name or a suggestion for other ways to check.
async function waitFile (filename) {
return new Promise(async (resolve, reject) => {
if (!fs.existsSync(filename)) {
await delay(3000);
await waitFile(filename);
resolve();
}else{
resolve();
}
})
}
function delay(time) {
return new Promise(function(resolve) {
setTimeout(resolve, time)
});
}
Implementation:
var filename = `${yyyy}${mm}_TAC.csv`;
var pathWithFilename = `${config.path}\\${filename}`;
await waitFile(pathWithFilename);
You need check request response.
await page.on('response', (response)=>{ console.log(response, response._url)}
You should check what is coming from response then find status, it comes with status 200
Using puppeteer and chrome I have one more solution which might help you.
If you are downloading the file from chrome it will always have ".crdownload" extension. And when file is completely downloaded that extension will vanish.
So, I am using recurring function and maximum number of times it can iterate, If it doesn't download the file in that time.. I am deleting it. And I am constantly checking a folder for that extention.
async checkFileDownloaded(path, timer) {
return new Promise(async (resolve, reject) => {
let noOfFile;
try {
noOfFile = await fs.readdirSync(path);
} catch (err) {
return resolve("null");
}
for (let i in noOfFile) {
if (noOfFile[i].includes('.crdownload')) {
await this.delay(20000);
if (timer == 0) {
fs.unlink(path + '/' + noOfFile[i], (err) => {
});
return resolve("Success");
} else {
timer = timer - 1;
await this.checkFileDownloaded(path, timer);
}
}
}
return resolve("Success");
});
}
Here is another function, its just wait for the pause button to disappear:
async function waitForDownload(browser: Browser) {
const dmPage = await browser.newPage();
await dmPage.goto("chrome://downloads/");
await dmPage.bringToFront();
await dmPage.waitForFunction(() => {
try {
const donePath = document.querySelector("downloads-manager")!.shadowRoot!
.querySelector(
"#frb0",
)!.shadowRoot!.querySelector("#pauseOrResume")!;
if ((donePath as HTMLButtonElement).innerText != "Pause") {
return true;
}
} catch {
//
}
}, { timeout: 0 });
console.log("Download finished");
}
I didn't like solutions that were checking DOM or file system for the file.
From Chrome DevTools Protocol documentation](https://chromedevtools.github.io/) I found two events,
Page.downloadProgress and Browser.downloadProgress. (Though Page.downloadProgress is marked as deprecated, that's the one that worked for me.)
This event has a property called state which tells you about the state of the download. state could be inProgress, completed and canceled.
You can wrap this event in a Promise to await it till the status changes to completed
async function waitUntilDownload(page, fileName = '') {
return new Promise((resolve, reject) => {
page._client().on('Page.downloadProgress', e => {
if (e.state === 'completed') {
resolve(fileName);
} else if (e.state === 'canceled') {
reject();
}
});
});
}
and await it as follows,
await waitUntilDownload(page, fileName);
Created simple await function that will check for file rapidly or timeout in 10 seconds
import fs from "fs";
awaitFileDownloaded: async (filePath) => {
let timeout = 10000
const delay = 200
return new Promise(async (resolve, reject) => {
while (timeout > 0) {
if (fs.existsSync(filePath)) {
resolve(true);
return
} else {
await HelperUI.delay(delay)
timeout -= delay
}
}
reject("awaitFileDownloaded timed out")
});
},
You can use node-watch to report the updates to the target directory. When the file upload is complete you will receive an update event with the name of the new file that has been downloaded.
Run npm to install node-watch:
npm install node-watch
Sample code:
const puppeteer = require('puppeteer');
const watch = require('node-watch');
const path = require('path');
// Add code to initiate the download ...
const watchDir = '/Users/home/Downloads'
const filepath = path.join(watchDir, "download_file");
(async() => {
watch(watchDir, function(event, name) {
if (event == "update") {
if (name === filepath)) {
browser.close(); // use case specific
process.exit(); // use case specific
}
}
})
Tried doing an await page.waitFor(50000); with a time as long as the download should take.
Or look at watching for file changes on complete file transfer
you could search in the download location for the extension the files have when still downloading 'crdownload' and when the download is completed the file is renamed with the original extension: from this 'video_audio_file.mp4.crdownload' turns into 'video_audio_file.mp4' without the 'crdownload' at the end
const fs = require('fs');
const myPath = path.resolve('/your/file/download/folder');
let siNo = 0;
function stillWorking(myPath) {
siNo = 0;
filenames = fs.readdirSync(myPath);
filenames.forEach(file => {
if (file.includes('crdownload')) {
siNo = 1;
}
});
return siNo;
}
Then you use is in an infinite loop like this and check very a certain period of time, here I check every 3 seconds and when it returns 0 which means there is no pending files to be fully downloaded.
while (true) {
execSync('sleep 3');
if (stillWorking(myPath) == 0) {
await browser.close();
break;
}
}
As far as I understand, in ES7/ES2016 putting multiple await's in code will work similar to chaining .then() with promises, meaning that they will execute one after the other rather than in parallel. So, for example, we have this code:
await someCall();
await anotherCall();
Do I understand it correctly that anotherCall() will be called only when someCall() is completed? What is the most elegant way of calling them in parallel?
I want to use it in Node, so maybe there's a solution with async library?
EDIT: I'm not satisfied with the solution provided in this question: Slowdown due to non-parallel awaiting of promises in async generators, because it uses generators and I'm asking about a more general use case.
You can await on Promise.all():
await Promise.all([someCall(), anotherCall()]);
To store the results:
let [someResult, anotherResult] = await Promise.all([someCall(), anotherCall()]);
Note that Promise.all fails fast, which means that as soon as one of the promises supplied to it rejects, then the entire thing rejects.
const happy = (v, ms) => new Promise((resolve) => setTimeout(() => resolve(v), ms))
const sad = (v, ms) => new Promise((_, reject) => setTimeout(() => reject(v), ms))
Promise.all([happy('happy', 100), sad('sad', 50)])
.then(console.log).catch(console.log) // 'sad'
If, instead, you want to wait for all the promises to either fulfill or reject, then you can use Promise.allSettled. Note that Internet Explorer does not natively support this method.
const happy = (v, ms) => new Promise((resolve) => setTimeout(() => resolve(v), ms))
const sad = (v, ms) => new Promise((_, reject) => setTimeout(() => reject(v), ms))
Promise.allSettled([happy('happy', 100), sad('sad', 50)])
.then(console.log) // [{ "status":"fulfilled", "value":"happy" }, { "status":"rejected", "reason":"sad" }]
Note: If you use Promise.all actions that managed to finish before rejection happen are not rolled back, so you may need to take care of such situation. For example
if you have 5 actions, 4 quick, 1 slow and slow rejects. Those 4
actions may be already executed so you may need to roll back. In such situation consider using Promise.allSettled while it will provide exact detail which action failed and which not.
TL;DR
Use Promise.all for the parallel function calls, the answer behaviors not correctly when the error occurs.
First, execute all the asynchronous calls at once and obtain all the Promise objects. Second, use await on the Promise objects. This way, while you wait for the first Promise to resolve the other asynchronous calls are still progressing. Overall, you will only wait for as long as the slowest asynchronous call. For example:
// Begin first call and store promise without waiting
const someResult = someCall();
// Begin second call and store promise without waiting
const anotherResult = anotherCall();
// Now we await for both results, whose async processes have already been started
const finalResult = [await someResult, await anotherResult];
// At this point all calls have been resolved
// Now when accessing someResult| anotherResult,
// you will have a value instead of a promise
JSbin example: http://jsbin.com/xerifanima/edit?js,console
Caveat: It doesn't matter if the await calls are on the same line or on different lines, so long as the first await call happens after all of the asynchronous calls. See JohnnyHK's comment.
Update: this answer has a different timing in error handling according to the #bergi's answer, it does NOT throw out the error as the error occurs but after all the promises are executed.
I compare the result with #jonny's tip: [result1, result2] = Promise.all([async1(), async2()]), check the following code snippet
const correctAsync500ms = () => {
return new Promise(resolve => {
setTimeout(resolve, 500, 'correct500msResult');
});
};
const correctAsync100ms = () => {
return new Promise(resolve => {
setTimeout(resolve, 100, 'correct100msResult');
});
};
const rejectAsync100ms = () => {
return new Promise((resolve, reject) => {
setTimeout(reject, 100, 'reject100msError');
});
};
const asyncInArray = async (fun1, fun2) => {
const label = 'test async functions in array';
try {
console.time(label);
const p1 = fun1();
const p2 = fun2();
const result = [await p1, await p2];
console.timeEnd(label);
} catch (e) {
console.error('error is', e);
console.timeEnd(label);
}
};
const asyncInPromiseAll = async (fun1, fun2) => {
const label = 'test async functions with Promise.all';
try {
console.time(label);
let [value1, value2] = await Promise.all([fun1(), fun2()]);
console.timeEnd(label);
} catch (e) {
console.error('error is', e);
console.timeEnd(label);
}
};
(async () => {
console.group('async functions without error');
console.log('async functions without error: start')
await asyncInArray(correctAsync500ms, correctAsync100ms);
await asyncInPromiseAll(correctAsync500ms, correctAsync100ms);
console.groupEnd();
console.group('async functions with error');
console.log('async functions with error: start')
await asyncInArray(correctAsync500ms, rejectAsync100ms);
await asyncInPromiseAll(correctAsync500ms, rejectAsync100ms);
console.groupEnd();
})();
Update:
The original answer makes it difficult (and in some cases impossible) to correctly handle promise rejections. The correct solution is to use Promise.all:
const [someResult, anotherResult] = await Promise.all([someCall(), anotherCall()]);
Original answer:
Just make sure you call both functions before you await either one:
// Call both functions
const somePromise = someCall();
const anotherPromise = anotherCall();
// Await both promises
const someResult = await somePromise;
const anotherResult = await anotherPromise;
There is another way without Promise.all() to do it in parallel:
First, we have 2 functions to print numbers:
function printNumber1() {
return new Promise((resolve,reject) => {
setTimeout(() => {
console.log("Number1 is done");
resolve(10);
},1000);
});
}
function printNumber2() {
return new Promise((resolve,reject) => {
setTimeout(() => {
console.log("Number2 is done");
resolve(20);
},500);
});
}
This is sequential:
async function oneByOne() {
const number1 = await printNumber1();
const number2 = await printNumber2();
}
//Output: Number1 is done, Number2 is done
This is parallel:
async function inParallel() {
const promise1 = printNumber1();
const promise2 = printNumber2();
const number1 = await promise1;
const number2 = await promise2;
}
//Output: Number2 is done, Number1 is done
I've created a gist testing some different ways of resolving promises, with results. It may be helpful to see the options that work.
Edit: Gist content as per Jin Lee's comment
// Simple gist to test parallel promise resolution when using async / await
function promiseWait(time) {
return new Promise((resolve, reject) => {
setTimeout(() => {
resolve(true);
}, time);
});
}
async function test() {
return [
await promiseWait(1000),
await promiseWait(5000),
await promiseWait(9000),
await promiseWait(3000),
]
}
async function test2() {
return {
'aa': await promiseWait(1000),
'bb': await promiseWait(5000),
'cc': await promiseWait(9000),
'dd': await promiseWait(3000),
}
}
async function test3() {
return await {
'aa': promiseWait(1000),
'bb': promiseWait(5000),
'cc': promiseWait(9000),
'dd': promiseWait(3000),
}
}
async function test4() {
const p1 = promiseWait(1000);
const p2 = promiseWait(5000);
const p3 = promiseWait(9000);
const p4 = promiseWait(3000);
return {
'aa': await p1,
'bb': await p2,
'cc': await p3,
'dd': await p4,
};
}
async function test5() {
return await Promise.all([
await promiseWait(1000),
await promiseWait(5000),
await promiseWait(9000),
await promiseWait(3000),
]);
}
async function test6() {
return await Promise.all([
promiseWait(1000),
promiseWait(5000),
promiseWait(9000),
promiseWait(3000),
]);
}
async function test7() {
const p1 = promiseWait(1000);
const p2 = promiseWait(5000);
const p3 = promiseWait(9000);
return {
'aa': await p1,
'bb': await p2,
'cc': await p3,
'dd': await promiseWait(3000),
};
}
let start = Date.now();
test().then((res) => {
console.log('Test Done, elapsed', (Date.now() - start) / 1000, res);
start = Date.now();
test2().then((res) => {
console.log('Test2 Done, elapsed', (Date.now() - start) / 1000, res);
start = Date.now();
test3().then((res) => {
console.log('Test3 Done, elapsed', (Date.now() - start) / 1000, res);
start = Date.now();
test4().then((res) => {
console.log('Test4 Done, elapsed', (Date.now() - start) / 1000, res);
start = Date.now();
test5().then((res) => {
console.log('Test5 Done, elapsed', (Date.now() - start) / 1000, res);
start = Date.now();
test6().then((res) => {
console.log('Test6 Done, elapsed', (Date.now() - start) / 1000, res);
});
start = Date.now();
test7().then((res) => {
console.log('Test7 Done, elapsed', (Date.now() - start) / 1000, res);
});
});
});
});
});
});
/*
Test Done, elapsed 18.006 [ true, true, true, true ]
Test2 Done, elapsed 18.009 { aa: true, bb: true, cc: true, dd: true }
Test3 Done, elapsed 0 { aa: Promise { <pending> },
bb: Promise { <pending> },
cc: Promise { <pending> },
dd: Promise { <pending> } }
Test4 Done, elapsed 9 { aa: true, bb: true, cc: true, dd: true }
Test5 Done, elapsed 18.008 [ true, true, true, true ]
Test6 Done, elapsed 9.003 [ true, true, true, true ]
Test7 Done, elapsed 12.007 { aa: true, bb: true, cc: true, dd: true }
*/
In my case, I have several tasks I want to execute in parallel, but I need to do something different with the result of those tasks.
function wait(ms, data) {
console.log('Starting task:', data, ms);
return new Promise(resolve => setTimeout(resolve, ms, data));
}
var tasks = [
async () => {
var result = await wait(1000, 'moose');
// do something with result
console.log(result);
},
async () => {
var result = await wait(500, 'taco');
// do something with result
console.log(result);
},
async () => {
var result = await wait(5000, 'burp');
// do something with result
console.log(result);
}
]
await Promise.all(tasks.map(p => p()));
console.log('done');
And the output:
Starting task: moose 1000
Starting task: taco 500
Starting task: burp 5000
taco
moose
burp
done
(async function(){
function wait(ms, data) {
console.log('Starting task:', data, ms);
return new Promise(resolve => setTimeout(resolve, ms, data));
}
var tasks = [
async () => {
var result = await wait(1000, 'moose');
// do something with result
console.log(result);
},
async () => {
var result = await wait(500, 'taco');
// do something with result
console.log(result);
},
async () => {
var result = await wait(5000, 'burp');
// do something with result
console.log(result);
}
]
await Promise.all(tasks.map(p => p()));
console.log('done');
})();
await Promise.all([someCall(), anotherCall()]); as already mention will act as a thread fence (very common in parallel code as CUDA), hence it will allow all the promises in it to run without blocking each other, but will prevent the execution to continue until ALL are resolved.
another approach that is worth to share is the Node.js async that will also allow you to easily control the amount of concurrency that is usually desirable if the task is directly linked to the use of limited resources as API call, I/O operations, etc.
// create a queue object with concurrency 2
var q = async.queue(function(task, callback) {
console.log('Hello ' + task.name);
callback();
}, 2);
// assign a callback
q.drain = function() {
console.log('All items have been processed');
};
// add some items to the queue
q.push({name: 'foo'}, function(err) {
console.log('Finished processing foo');
});
q.push({name: 'bar'}, function (err) {
console.log('Finished processing bar');
});
// add some items to the queue (batch-wise)
q.push([{name: 'baz'},{name: 'bay'},{name: 'bax'}], function(err) {
console.log('Finished processing item');
});
// add some items to the front of the queue
q.unshift({name: 'bar'}, function (err) {
console.log('Finished processing bar');
});
Credits to the Medium article autor (read more)
You can call multiple asynchronous functions without awaiting them. This will execute them in parallel. While doing so, save the returned promises in variables, and await them at some point either individually or using Promise.all() and process the results.
You can also wrap the function calls with try...catch to handle failures of individual asynchronous actions and provide fallback logic.
Here's an example:
Observe the logs, the logs printed at the beginning of execution of the individual asynchronous functions get printed immediately even though the first function takes 5 seconds to resolve.
function someLongFunc () {
return new Promise((resolve, reject)=> {
console.log('Executing function 1')
setTimeout(resolve, 5000)
})
}
function anotherLongFunc () {
return new Promise((resolve, reject)=> {
console.log('Executing function 2')
setTimeout(resolve, 5000)
})
}
async function main () {
let someLongFuncPromise, anotherLongFuncPromise
const start = Date.now()
try {
someLongFuncPromise = someLongFunc()
}
catch (ex) {
console.error('something went wrong during func 1')
}
try {
anotherLongFuncPromise = anotherLongFunc()
}
catch (ex) {
console.error('something went wrong during func 2')
}
await someLongFuncPromise
await anotherLongFuncPromise
const totalTime = Date.now() - start
console.log('Execution completed in ', totalTime)
}
main()
// A generic test function that can be configured
// with an arbitrary delay and to either resolve or reject
const test = (delay, resolveSuccessfully) => new Promise((resolve, reject) => setTimeout(() => {
console.log(`Done ${ delay }`);
resolveSuccessfully ? resolve(`Resolved ${ delay }`) : reject(`Reject ${ delay }`)
}, delay));
// Our async handler function
const handler = async () => {
// Promise 1 runs first, but resolves last
const p1 = test(10000, true);
// Promise 2 run second, and also resolves
const p2 = test(5000, true);
// Promise 3 runs last, but completes first (with a rejection)
// Note the catch to trap the error immediately
const p3 = test(1000, false).catch(e => console.log(e));
// Await all in parallel
const r = await Promise.all([p1, p2, p3]);
// Display the results
console.log(r);
};
// Run the handler
handler();
/*
Done 1000
Reject 1000
Done 5000
Done 10000
*/
Whilst setting p1, p2 and p3 is not strictly running them in parallel, they do not hold up any execution and you can trap contextual errors with a catch.
This can be accomplished with Promise.allSettled(), which is similar to Promise.all() but without the fail-fast behavior.
async function Promise1() {
throw "Failure!";
}
async function Promise2() {
return "Success!";
}
const [Promise1Result, Promise2Result] = await Promise.allSettled([Promise1(), Promise2()]);
console.log(Promise1Result); // {status: "rejected", reason: "Failure!"}
console.log(Promise2Result); // {status: "fulfilled", value: "Success!"}
Note: This is a bleeding edge feature with limited browser support, so I strongly recommend including a polyfill for this function.
I create a helper function waitAll, may be it can make it sweeter.
It only works in nodejs for now, not in browser chrome.
//const parallel = async (...items) => {
const waitAll = async (...items) => {
//this function does start execution the functions
//the execution has been started before running this code here
//instead it collects of the result of execution of the functions
const temp = [];
for (const item of items) {
//this is not
//temp.push(await item())
//it does wait for the result in series (not in parallel), but
//it doesn't affect the parallel execution of those functions
//because they haven started earlier
temp.push(await item);
}
return temp;
};
//the async functions are executed in parallel before passed
//in the waitAll function
//const finalResult = await waitAll(someResult(), anotherResult());
//const finalResult = await parallel(someResult(), anotherResult());
//or
const [result1, result2] = await waitAll(someResult(), anotherResult());
//const [result1, result2] = await parallel(someResult(), anotherResult());
I vote for:
await Promise.all([someCall(), anotherCall()]);
Be aware of the moment you call functions, it may cause unexpected result:
// Supposing anotherCall() will trigger a request to create a new User
if (callFirst) {
await someCall();
} else {
await Promise.all([someCall(), anotherCall()]); // --> create new User here
}
But following always triggers request to create new User
// Supposing anotherCall() will trigger a request to create a new User
const someResult = someCall();
const anotherResult = anotherCall(); // ->> This always creates new User
if (callFirst) {
await someCall();
} else {
const finalResult = [await someResult, await anotherResult]
}