I have a function tick that I wish to perform repeatedly. Each time tick completes, I want to trigger it again in 3000 ms time. If tick fails, I want to pause for an extra 1000 ms and then try again. I cannot use setInterval because I do not know how long tick will take to complete.
Here is my code for achieving this:
const loop = async () => {
try {
console.log('Starting operation... ');
await tick();
} catch (error) {
console.error(error);
await sleep(1000);
}
setTimeout(loop, 3000);
};
loop();
Unfortunately, this stops working after several days of operation. I think that I am doing something wrong with the stack.
What is the recommended way of running an async operation like this in Node.js?
Your current function runs the loop every 3 seconds no matter if it fails or not.
I rewrote it a bit. This should work
const loop = async () => {
try {
console.log('Starting operation... ');
await tick();
setTimeout(loop, 3000);
} catch (error) {
console.error(error);
setTimeout(loop, 1000);
}
}
loop()
The above code would have the following order:
first run tick()
if tick() succeeds, run loop again in 3 seconds
if tick() fails, run loop again in 1 second
Related
I am trying to stop for loop on certain condition but loop continuous even if use break on certain condition.
Following is the code,
let statusKey = 0;
const terminateLoop = async function () {
statusKey = 1;
mainFn()
}
const doSomething = async function () {
for (let i of temAnotherArr) {
await new Promise((resolve, reject) => {
execFile(`/temp/abc.sh`, (error, stdout, stderr) => {
console.log(error)
if (error) {
reject();
} else {
resolve()
}
});
});
}
}
const mainFn = async function () {
for (let i of tempArr) {
if (statusKey === 1) {
doSomething();
return;
} else {
await new Promise((resolve, reject) => {
execFile(`/temp/xyz.sh`,(error, stdout, stderr) => {
console.log(error)
if (error) {
reject();
} else {
resolve()
}
});
});
}
}
}
mainFn();
Initially mainFn() gets called and it starts looping tempArr, after certain time from UI user clicks on button which trigger terminateLoop() function and statusKey gets change to 1 then same mainFu() gets called. It enters into if condition and calls function doSomething(). But once doSomething() completes, else condition still gets executes. I am not sure why loop is not breaking. I guess it has something to do with async await and Promise. I am not sure how to break this and not sure what I am making mistake.
But once doSomething() completes, else condition still gets executes.
The else will not execute from scratch, but do realise that there can be a pending callback call from that execFile in the else block that is still to happen after statusKey has been set to 1. When eventually that callback is called, your code will still execute that console.log without having regard of statusKey's value.
Furthermore, your code then resolves the promise that is being awaited in the else block and then the execution context of mainFn is resumed in that else block. So yes, it is possible that some code in the else block still executes after statusKey has been set to 1, but it is not possible that the else block is executed from scratch. You can be sure that this execFile function is not called anymore from that particular spot in the code when statusKey is 1. Only its callback could still be called and any statements that follow the await in the mainFn context (your example code has no such statements).
So the easiest "fix" is to also add an if statement inside that callback, and have it test the value of statusKey.
Unrelated, but I would avoid code repetition and promisify the execFile function.
Im trying to get fs.createReadStream working as a promise, so after the entire file has been read, it will be resolved.
In the case bellow, im pausing the stream, executing the awaitable method and resuming.
How to make .on('end'... to be be executed in the end.
if 1. is not possible, why the `.on('wont be fired', maybe i can use it to resolve the promise.
function parseFile<T>(filePath: string, row: (x: T) => void, err: (x) => void, end: (x) => void) {
return new Promise((resolve, reject) => {
const stream = fs.createReadStream(filePath);
stream.on('data', async data => {
try {
stream.pause();
await row(data);
} finally {
stream.resume();
}
})
.on('end', (rowCount: number) => {
resolve();// NOT REALLY THE END row(data) is still being called after this
})
.on('close', () => {
resolve();// NEVER BEING CALLED
})
.on('error', (rowCount: number) => {
reject();// NEVER GETS HERE, AS EXPECTED
})
})
}
UPDATE
Here you can actually test it: https://stackblitz.com/edit/node-czktjh?file=index.js
run node index.js
The output should be 1000 and not 1
Thanks
Something to be aware of. You've removed the line processing from the current version of the question so the stream is being read in large chunks. It appears to be reading the entire file in just two chunks, thus just two data events so the expected count here is 2, not 1000.
I think the problem with this code occurs because stream.pause() does not pause the generation of the end event - it only pauses future data events. If the last data event has been fired and you then await inside the processing of that data event (which causes your data event handler to immediately return a promise, the stream will think it's done and the end event will still fire before you're done awaiting the function inside the processing of that last data event. Remember, the data event handler is NOT promise-aware. And, it appears that stream.pause() only affects data events, not the end event.
I can imagine a work-around with a flag that keeps track of whether you're still processing a data event and postpones processing the end event until you're done with that last data event. I will add code for that in a second that illustrates how to use the flag.
FYI, the missing close event is another stream weirdness. Your nodejs program actually terminates before the close event gets to fire. If you put this at the start of your program:
setTimeout(() => { console.log('done with timer');}, 5000);
Then, you will see the close event because the timer will prevent your nodejs program from exiting before the close event gets to fire. I'm not suggesting this as a solution to any problem, just to illustrate that the close event is still there and wants to fire if your program doesn't exit before it gets a chance.
Here's code that demonstrated the use of flags to work-around the pause issue. When you run this code, you will only see 2 data events, not 1000 because this code is not reading lines, it's reading much larger chunks that that. So, the expected result of this is not 1000.
// run `node index.js` in the terminal
const fs = require('fs');
const parseFile = row => {
let paused = true;
let ended = false;
let dataCntr = 0;
return new Promise((resolve, reject) => {
const stream = fs.createReadStream('./generated.data.csv');
stream
.on('data', async data => {
++dataCntr;
try {
stream.pause();
paused = true;
await row(data);
} finally {
paused = false;
stream.resume();
if (ended) {
console.log(`received ${dataCntr} data events`);
resolve();
}
}
})
.on('end', rowCount => {
ended = true;
if (!paused) {
console.log(`received ${dataCntr} data events`);
resolve();
}
})
.on('close', () => {
//resolve();
})
.on('error', rowCount => {
reject();
});
});
};
(async () => {
let count = 0;
await parseFile(async row => {
await new Promise(resolve => setTimeout(resolve, 50)); //sleep
count++;
});
console.log(`lines executed: ${count}, the expected is more than 1`);
})();
FYI, I still think your original version of the question had the problem I mentioned in my first comment - that you weren't pausing the right stream. What is documented here is yet another problem (where you can get end before your await in the last data event is done).
I'm writing an HTTP API with expressjs in Node.js and here is what I'm trying to achieve:
I have a regular task that I would like to run regularly, approx every minute. This task is implemented with an async function named task.
In reaction to a call in my API I would like to have that task called immediately as well
Two executions of the task function must not be concurrent. Each execution should run to completion before another execution is started.
The code looks like this:
// only a single execution of this function is allowed at a time
// which is not the case with the current code
async function task(reason: string) {
console.log("do thing because %s...", reason);
await sleep(1000);
console.log("done");
}
// call task regularly
setIntervalAsync(async () => {
await task("ticker");
}, 5000) // normally 1min
// call task immediately
app.get("/task", async (req, res) => {
await task("trigger");
res.send("ok");
});
I've put a full working sample project at https://github.com/piec/question.js
If I were in go I would do it like this and it would be easy, but I don't know how to do that with Node.js.
Ideas I have considered or tried:
I could apparently put task in a critical section using a mutex from the async-mutex library. But I'm not too fond of adding mutexes in js code.
Many people seem to be using message queue libraries with worker processes (bee-queue, bullmq, ...) but this adds a dependency to an external service like redis usually. Also if I'm correct the code would be a bit more complex because I need a main entrypoint and an entrypoint for worker processes. Also you can't share objects with the workers as easily as in a "normal" single process situation.
I have tried RxJs subject in order to make a producer consumer channel. But I was not able to limit the execution of task to one at a time (task is async).
Thank you!
You can make your own serialized asynchronous queue and run the tasks through that.
This queue uses a flag to keep track of whether it's in the middle of running an asynchronous operation already. If so, it just adds the task to the queue and will run it when the current operation is done. If not, it runs it now. Adding it to the queue returns a promise so the caller can know when the task finally got to run.
If the tasks are asynchronous, they are required to return a promise that is linked to the asynchronous activity. You can mix in non-asynchronous tasks too and they will also be serialized.
class SerializedAsyncQueue {
constructor() {
this.tasks = [];
this.inProcess = false;
}
// adds a promise-returning function and its args to the queue
// returns a promise that resolves when the function finally gets to run
add(fn, ...args) {
let d = new Deferred();
this.tasks.push({ fn, args: ...args, deferred: d });
this.check();
return d.promise;
}
check() {
if (!this.inProcess && this.tasks.length) {
// run next task
this.inProcess = true;
const nextTask = this.tasks.shift();
Promise.resolve(nextTask.fn(...nextTask.args)).then(val => {
this.inProcess = false;
nextTask.deferred.resolve(val);
this.check();
}).catch(err => {
console.log(err);
this.inProcess = false;
nextTask.deferred.reject(err);
this.check();
});
}
}
}
const Deferred = function() {
if (!(this instanceof Deferred)) {
return new Deferred();
}
const p = this.promise = new Promise((resolve, reject) => {
this.resolve = resolve;
this.reject = reject;
});
this.then = p.then.bind(p);
this.catch = p.catch.bind(p);
if (p.finally) {
this.finally = p.finally.bind(p);
}
}
let queue = new SerializedAsyncQueue();
// utility function
const sleep = function(t) {
return new Promise(resolve => {
setTimeout(resolve, t);
});
}
// only a single execution of this function is allowed at a time
// so it is run only via the queue that makes sure it is serialized
async function task(reason: string) {
function runIt() {
console.log("do thing because %s...", reason);
await sleep(1000);
console.log("done");
}
return queue.add(runIt);
}
// call task regularly
setIntervalAsync(async () => {
await task("ticker");
}, 5000) // normally 1min
// call task immediately
app.get("/task", async (req, res) => {
await task("trigger");
res.send("ok");
});
Here's a version using RxJS#Subject that is almost working. How to finish it depends on your use-case.
async function task(reason: string) {
console.log("do thing because %s...", reason);
await sleep(1000);
console.log("done");
}
const run = new Subject<string>();
const effect$ = run.pipe(
// Limit one task at a time
concatMap(task),
share()
);
const effectSub = effect$.subscribe();
interval(5000).subscribe(_ =>
run.next("ticker")
);
// call task immediately
app.get("/task", async (req, res) => {
effect$.pipe(
take(1)
).subscribe(_ =>
res.send("ok")
);
run.next("trigger");
});
The issue here is that res.send("ok") is linked to the effect$ streams next emission. This may not be the one generated by the run.next you're about to call.
There are many ways to fix this. For example, you can tag each emission with an ID and then wait for the corresponding emission before using res.send("ok").
There are better ways too if calls distinguish themselves naturally.
A Clunky ID Version
Generating an ID randomly is a bad idea, but it gets the general thrust across. You can generate unique IDs however you like. They can be integrated directly into the task somehow or can be kept 100% separate the way they are here (task itself has no knowledge that it's been assigned an ID before being run).
interface IdTask {
taskId: number,
reason: string
}
interface IdResponse {
taskId: number,
response: any
}
async function task(reason: string) {
console.log("do thing because %s...", reason);
await sleep(1000);
console.log("done");
}
const run = new Subject<IdTask>();
const effect$: Observable<IdResponse> = run.pipe(
// concatMap only allows one observable at a time to run
concatMap((eTask: IdTask) => from(task(eTask.reason)).pipe(
map((response:any) => ({
taskId: eTask.taskId,
response
})as IdResponse)
)),
share()
);
const effectSub = effect$.subscribe({
next: v => console.log("This is a shared task emission: ", v)
});
interval(5000).subscribe(num =>
run.next({
taskId: num,
reason: "ticker"
})
);
// call task immediately
app.get("/task", async (req, res) => {
const randomId = Math.random();
effect$.pipe(
filter(({taskId}) => taskId == randomId),
take(1)
).subscribe(_ =>
res.send("ok")
);
run.next({
taskId: randomId,
reason: "trigger"
});
});
function first(){
console.log('first')
}
function second(){
console.log('second')
}
let interval = async ()=>{
await setInterval(first,2000)
await setInterval(second,2000)
}
interval();
Imagine that I have this code above.
When I run it, first() and second() will be called at the same time; how do I call second() after first)() returns some data, for example, if first() is done, only then call second()?
Because first() in my code will be working with a big amount of data and if this 2 functions will be calling at the same time, it will be hard for the server.
How do I call second() each time when first() will return some data?
As mentioned above setInterval does not play well with promises if you do not stop it. In case you clear the interval you can use it like:
async function waitUntil(condition) {
return await new Promise(resolve => {
const interval = setInterval(() => {
if (condition) {
resolve('foo');
clearInterval(interval);
};
}, 1000);
});
}
Later you can use it like
const bar = waitUntil(someConditionHere)
You have a few problems:
Promises may only ever resolve once, setInterval() is meant to call the callback multiple times, Promises do not support this case well.
Neither setInterval(), nor the more appropriate setTimeout() return Promises, therefore, awaiting on them is pointless in this context.
You're looking for a function that returns a Promise which resolves after some times (using setTimeout(), probably, not setInterval()).
Luckily, creating such a function is rather trivial:
async function delay(ms) {
// return await for better async stack trace support in case of errors.
return await new Promise(resolve => setTimeout(resolve, ms));
}
With this new delay function, you can implement your desired flow:
function first(){
console.log('first')
}
function second(){
console.log('second')
}
let run = async ()=>{
await delay(2000);
first();
await delay(2000)
second();
}
run();
setInterval doesn't play well with promises because it triggers a callback multiple times, while promise resolves once.
It seems that it's setTimeout that fits the case. It should be promisified in order to be used with async..await:
async () => {
await new Promise(resolve => setTimeout(() => resolve(first()), 2000));
await new Promise(resolve => setTimeout(() => resolve(second()), 2000));
}
await expression causes async to pause until a Promise is settled
so you can directly get the promise's result without await
for me, I want to initiate Http request every 1s
let intervalid
async function testFunction() {
intervalid = setInterval(() => {
// I use axios like: axios.get('/user?ID=12345').then
new Promise(function(resolve, reject){
resolve('something')
}).then(res => {
if (condition) {
// do something
} else {
clearInterval(intervalid)
}
})
}, 1000)
}
// you can use this function like
testFunction()
// or stop the setInterval in any place by
clearInterval(intervalid)
You could use an IFFE. This way you could escape the issue of myInterval not accepting Promise as a return type.
There are cases where you need setInterval, because you want to call some function unknown amount of times with some interval in between.
When I faced this problem this turned out to be the most straight-forward solution for me. I hope it help someone :)
For me the use case was that I wanted to send logs to CloudWatch but try not to face the Throttle exception for sending more than 5 logs per second. So I needed to keep my logs and send them as a batch in an interval of 1 second. The solution I'm posting here is what I ended up using.
async function myAsyncFunc(): Promise<string> {
return new Promise<string>((resolve) => {
resolve("hello world");
});
}
function myInterval(): void {
setInterval(() => {
void (async () => {
await myAsyncFunc();
})();
}, 5_000);
}
// then call like so
myInterval();
Looked through all the answers but still didn't find the correct one that would work exactly how the OP is asked. This is what I used for the same purpose:
async function waitInterval(callback, ms) {
return new Promise(resolve => {
let iteration = 0;
const interval = setInterval(async () => {
if (await callback(iteration, interval)) {
resolve();
clearInterval(interval);
}
iteration++;
}, ms);
});
}
function first(i) {
console.log(`first: ${i}`);
// If the condition below is true the timer finishes
return i === 5;
}
function second(i) {
console.log(`second: ${i}`);
// If the condition below is true the timer finishes
return i === 5;
}
(async () => {
console.log('start');
await waitInterval(first, 1000);
await waitInterval(second, 1000);
console.log('finish');
})()
In my example, I also put interval iteration count and the timer itself, just in case the caller would need to do something with it. However, it's not necessary
In my case, I needed to iterate through a list of images, pausing in between each, and then a longer pause at the end before re-looping through.
I accomplished this by combining several techniques from above, calling my function recursively and awaiting a timeout.
If at any point another trigger changes my animationPaused:boolean, my recursive function will exit.
const loopThroughImages = async() => {
for (let i=0; i<numberOfImages; i++){
if (animationPaused) {
return;
}
this.updateImage(i);
await timeout(700);
}
await timeout(1000);
loopThroughImages();
}
loopThroughImages();
Async/await do not make the promises synchronous.
To my knowledge, it's just a different syntax for return Promise and .then().
Here i rewrote the async function and left both versions, so you can see what it really does and compare.
It's in fact a cascade of Promises.
// by the way no need for async there. the callback does not return a promise, so no need for await.
function waitInterval(callback, ms) {
return new Promise(resolve => {
let iteration = 0;
const interval = setInterval(async () => {
if (callback(iteration, interval)) {
resolve();
clearInterval(interval);
}
iteration++;
}, ms);
});
}
function first(i) {
console.log(`first: ${i}`);
// If the condition below is true the timer finishes
return i === 5;
}
function second(i) {
console.log(`second: ${i}`);
// If the condition below is true the timer finishes
return i === 5;
}
// async function with async/await, this code ...
(async () => {
console.log('start');
await waitInterval(first, 1000);
await waitInterval(second, 1000);
console.log('finish');
})() //... returns a pending Promise and ...
console.log('i do not wait');
// ... is kinda identical to this code.
// still asynchronous but return Promise statements with then cascade.
(() => {
console.log('start again');
return waitInterval(first, 1000).then(() => {
return waitInterval(second, 1000).then(() => {
console.log('finish again');
});
});
})(); // returns a pending Promise...
console.log('i do not wait either');
You can see the two async functions both execute at the same time.
So using promises around intervals here is not very useful, as it's still just intervals, and promises changes nothing, and make things confusing...
As the code is calling callbacks repeatedly into an interval, this is, i think, a cleaner way:
function first(i) {
console.log(`first: ${i}`);
// If the condition below is true the timer finishes
return i === 5;
}
function second(i) {
console.log(`second: ${i}`);
// If the condition below is true the timer finishes
return i === 5;
}
function executeThroughTime(...callbacks){
console.log('start');
let callbackIndex = 0; // to track current callback.
let timerIndex = 0; // index given to callbacks
let interval = setInterval(() =>{
if (callbacks[callbackIndex](timerIndex++)){ // callback return true when it finishes.
timerIndex = 0; // resets for next callback
if (++callbackIndex>=callbacks.length){ // if no next callback finish.
clearInterval(interval);
console.log('finish');
}
}
},1000)
}
executeThroughTime(first,second);
console.log('and i still do not wait ;)');
Also, this solution execute a callback every secondes.
if the callbacks are async requests that takes more than one sec to resolve, and i can't afford for them to overlap, then, instead of doing iterative call with repetitive interval, i would get the request resolution to call the next request (through a timer if i don't want to harass the server).
Here the "recursive" task is called lTask, does pretty much the same as before, except that, as i do not have an interval anymore, i need a new timer each iteration.
// slow internet request simulation. with a Promise, could be a callback.
function simulateAsync1(i) {
console.log(`first pending: ${i}`);
return new Promise((resolve) =>{
setTimeout(() => resolve('got that first big data'), Math.floor(Math.random()*1000)+ 1000);//simulate request that last between 1 and 2 sec.
}).then((result) =>{
console.log(`first solved: ${i} ->`, result);
return i==2;
});
}
// slow internet request simulation. with a Promise, could be a callback.
function simulateAsync2(i) {
console.log(`second pending: ${i}`);
return new Promise((resolve) =>{
setTimeout(() => resolve('got that second big data'), Math.floor(Math.random()*1000) + 1000);//simulate request that last between 1 and 2 sec.
}).then((result) =>{ // promise is resolved
console.log(`second solved: ${i} ->`,result);
return i==4; // return a promise
});
}
function executeThroughTime(...asyncCallbacks){
console.log('start');
let callbackIndex = 0;
let timerIndex = 0;
let lPreviousTime = Date.now();
let lTask = () => { // timeout callback.
asyncCallbacks[callbackIndex](timerIndex++).then((result) => { // the setTimeout for the next task is set when the promise is solved.
console.log('result',result)
if (result) { // current callback is done.
timerIndex = 0;
if (++callbackIndex>=asyncCallbacks.length){//are all callbacks done ?
console.log('finish');
return;// its over
}
}
console.log('time elapsed since previous call',Date.now() - lPreviousTime);
lPreviousTime = Date.now();
//console.log('"wait" 1 sec (but not realy)');
setTimeout(lTask,1000);//redo task after 1 sec.
//console.log('i do not wait');
});
}
lTask();// no need to set a timer for first call.
}
executeThroughTime(simulateAsync1,simulateAsync2);
console.log('i do not wait');
Next step would be to empty a fifo with the interval, and fill it with web request promises...
I have a function that does some work. When it is finished it calls the callback. After that I get the 'END' in console. But i need to execute again the same function. How can I do that? Thx.
function start(callback){
//... some work ...
if(work_finished){
callback();
}
}
start(function(){
console.log('END');
});
If your function is asynchronous, you can create a wrapper function and literally just call it again from the callback. Note, without any terminating conditions, this will run forever.
function start(callback){
//... some work ...
if(work_finished){
callback();
}
}
function run() {
start(function(){
console.log('END');
run();
});
}
If your function is not asynchronous, the above operation will eventually cause a stack overflow (due to infinite recursion) so you'd have to have the caller of start() trigger it again:
function start(callback){
//... some work ...
if(work_finished){
callback();
}
// then return true or false depending upon whether you want
// it to keep getting called again
return true;
}
function run() {
var more;
do {
more = start(function(){
console.log('END');
});
} while (more === true);
}
And, if you just want your function called on some regular time interval, you can use setInterval() like this:
// call my function every 5 seconds
var timer = setInterval(function() {
start(function(){
console.log('END');
});
}, 5000);
There are quite a few ways to repeatedly execute functions in node.
Recursive calls
code:
function start(callback){
//... some work ...
if(work_finished){
callback();
// call function again
// should be OK if some work is async so it doesn't
// block the thread
start(callback);
// or schedule to be called at end of event loop
// setImmediate(start, callback);
}
}
start(function(){
console.log('END');
});
Execute function any number of additional times which would require number of calls to be tracked
set the function up on an interval to be called ~n milliseconds
setInterval(start, 1000, callback);