Got a class, it extends EventEmitter.
Got function that fires a bunch of events on that emitter. These events trigger async tasks and they all have a done() callback.
What's the proper way to wait for all tasks to finish? I just want the process to sit there and wait for events until a certain event is fired (ALL_DONE), in which case it should exit.
I mean I know this can be done in multiple ways probably, but what I'm asking is can I do it without any packages, plugins etc, using just nodeJS APIs?
I want to wait without blocking the main thread.
EDIT:
Thx for the responses! I'm not sure these apply to my case. I should have provided more details. This is what I have:
class FoobarEmitter extends EventEmitter {
protected checkEventStatus() {
// this has some logic to check if all done() callbacks have been called or not.
if(allDone) {
this.emit('ALL_DONE')
}
}
protected fireEvents() {
for() {
this.emit('SOME_EVENT', () => {})
this.checkEventStatuses();
}
}
protected registerHandlers() {
this.on('SOME_EVENT', async (done) => {
// does async stuff
// might also call this.emit('OTHER_EVENT', () => {})
done();
})
this.on('ALL_DONE', () => { process.exit() })
}
constrcutor() {
this.registerHandlers();
this.fireEvents()
}
}
new FoobarEmitter()
So this will not wait for all events. The ones fired from callbacks won't finish. Some of them runs, then the process just stopes and ALL_DONE is never fired.
Doesn't seem like it ought to be any more complex than something like this:
const { once, EventEmitter } = require("events");
class FooBarEmitter extends EventEmitter {
}
async function doSomethingUseful( emitter ) {
// does something useful, emitter emits events in the process
}
async function main() {
const emitter = new FooBarEmitter();
const promise = doSomethingUseful(emitter) ;
await once(emitter, "ALL_DONE");
await promise;
}
let cc;
main()
.then( () => {
console.log(
cc = 0;
})
.catch( err => {
console.error(err.stack);
cc = 1;
})
.finally( () => {
process.exit(cc);
});
Or, another approach:
const { once, EventEmitter } = require("events");
class FooBarEmitter extends EventEmitter {
}
async function doSomethingUseful( emitter ) {
// does something useful, emitter emits events in the process
}
function main() {
let promise;
const emitter = new FooBarEmitter().on('ALL_DONE', async () => {
await promise;
});
promise = doSomethingUseful(emitter) ;
}
main();
Related
I'm writing an HTTP API with expressjs in Node.js and here is what I'm trying to achieve:
I have a regular task that I would like to run regularly, approx every minute. This task is implemented with an async function named task.
In reaction to a call in my API I would like to have that task called immediately as well
Two executions of the task function must not be concurrent. Each execution should run to completion before another execution is started.
The code looks like this:
// only a single execution of this function is allowed at a time
// which is not the case with the current code
async function task(reason: string) {
console.log("do thing because %s...", reason);
await sleep(1000);
console.log("done");
}
// call task regularly
setIntervalAsync(async () => {
await task("ticker");
}, 5000) // normally 1min
// call task immediately
app.get("/task", async (req, res) => {
await task("trigger");
res.send("ok");
});
I've put a full working sample project at https://github.com/piec/question.js
If I were in go I would do it like this and it would be easy, but I don't know how to do that with Node.js.
Ideas I have considered or tried:
I could apparently put task in a critical section using a mutex from the async-mutex library. But I'm not too fond of adding mutexes in js code.
Many people seem to be using message queue libraries with worker processes (bee-queue, bullmq, ...) but this adds a dependency to an external service like redis usually. Also if I'm correct the code would be a bit more complex because I need a main entrypoint and an entrypoint for worker processes. Also you can't share objects with the workers as easily as in a "normal" single process situation.
I have tried RxJs subject in order to make a producer consumer channel. But I was not able to limit the execution of task to one at a time (task is async).
Thank you!
You can make your own serialized asynchronous queue and run the tasks through that.
This queue uses a flag to keep track of whether it's in the middle of running an asynchronous operation already. If so, it just adds the task to the queue and will run it when the current operation is done. If not, it runs it now. Adding it to the queue returns a promise so the caller can know when the task finally got to run.
If the tasks are asynchronous, they are required to return a promise that is linked to the asynchronous activity. You can mix in non-asynchronous tasks too and they will also be serialized.
class SerializedAsyncQueue {
constructor() {
this.tasks = [];
this.inProcess = false;
}
// adds a promise-returning function and its args to the queue
// returns a promise that resolves when the function finally gets to run
add(fn, ...args) {
let d = new Deferred();
this.tasks.push({ fn, args: ...args, deferred: d });
this.check();
return d.promise;
}
check() {
if (!this.inProcess && this.tasks.length) {
// run next task
this.inProcess = true;
const nextTask = this.tasks.shift();
Promise.resolve(nextTask.fn(...nextTask.args)).then(val => {
this.inProcess = false;
nextTask.deferred.resolve(val);
this.check();
}).catch(err => {
console.log(err);
this.inProcess = false;
nextTask.deferred.reject(err);
this.check();
});
}
}
}
const Deferred = function() {
if (!(this instanceof Deferred)) {
return new Deferred();
}
const p = this.promise = new Promise((resolve, reject) => {
this.resolve = resolve;
this.reject = reject;
});
this.then = p.then.bind(p);
this.catch = p.catch.bind(p);
if (p.finally) {
this.finally = p.finally.bind(p);
}
}
let queue = new SerializedAsyncQueue();
// utility function
const sleep = function(t) {
return new Promise(resolve => {
setTimeout(resolve, t);
});
}
// only a single execution of this function is allowed at a time
// so it is run only via the queue that makes sure it is serialized
async function task(reason: string) {
function runIt() {
console.log("do thing because %s...", reason);
await sleep(1000);
console.log("done");
}
return queue.add(runIt);
}
// call task regularly
setIntervalAsync(async () => {
await task("ticker");
}, 5000) // normally 1min
// call task immediately
app.get("/task", async (req, res) => {
await task("trigger");
res.send("ok");
});
Here's a version using RxJS#Subject that is almost working. How to finish it depends on your use-case.
async function task(reason: string) {
console.log("do thing because %s...", reason);
await sleep(1000);
console.log("done");
}
const run = new Subject<string>();
const effect$ = run.pipe(
// Limit one task at a time
concatMap(task),
share()
);
const effectSub = effect$.subscribe();
interval(5000).subscribe(_ =>
run.next("ticker")
);
// call task immediately
app.get("/task", async (req, res) => {
effect$.pipe(
take(1)
).subscribe(_ =>
res.send("ok")
);
run.next("trigger");
});
The issue here is that res.send("ok") is linked to the effect$ streams next emission. This may not be the one generated by the run.next you're about to call.
There are many ways to fix this. For example, you can tag each emission with an ID and then wait for the corresponding emission before using res.send("ok").
There are better ways too if calls distinguish themselves naturally.
A Clunky ID Version
Generating an ID randomly is a bad idea, but it gets the general thrust across. You can generate unique IDs however you like. They can be integrated directly into the task somehow or can be kept 100% separate the way they are here (task itself has no knowledge that it's been assigned an ID before being run).
interface IdTask {
taskId: number,
reason: string
}
interface IdResponse {
taskId: number,
response: any
}
async function task(reason: string) {
console.log("do thing because %s...", reason);
await sleep(1000);
console.log("done");
}
const run = new Subject<IdTask>();
const effect$: Observable<IdResponse> = run.pipe(
// concatMap only allows one observable at a time to run
concatMap((eTask: IdTask) => from(task(eTask.reason)).pipe(
map((response:any) => ({
taskId: eTask.taskId,
response
})as IdResponse)
)),
share()
);
const effectSub = effect$.subscribe({
next: v => console.log("This is a shared task emission: ", v)
});
interval(5000).subscribe(num =>
run.next({
taskId: num,
reason: "ticker"
})
);
// call task immediately
app.get("/task", async (req, res) => {
const randomId = Math.random();
effect$.pipe(
filter(({taskId}) => taskId == randomId),
take(1)
).subscribe(_ =>
res.send("ok")
);
run.next({
taskId: randomId,
reason: "trigger"
});
});
I have a typescript method that is trying to call a method, on(), that takes in a callback method. I'd like to have myConnect() wait until the callback method is executed. I assume this is a promise, but I'm not sure how to write this method so it waits until the callback is called.
myConnect(): void {
this.innerProducer.connect();
this.innerProducer.on("ready", () => {
Logger.info("producer is ready to produce for topic {}", this.topic);
this.isReadyToProduce = true;
});
}
I attempted to use util.promisify, but it's not quite right:
async connect2() {
this.innerProducer.connect();
const util = require("util");
const readyCallbackFunc = util.promisify(this.innerProducer.on);
await readyCallbackFunc("ready", () => {
Logger.info("producer is ready to produce for topic {}", this.topic);
this.isReadyToProduce = true;
});
}
error: (node:23144) UnhandledPromiseRejectionWarning: TypeError: Cannot read property '_events' of undefined
API is defined here:
https://github.com/Blizzard/node-rdkafka/blob/129cb733f5b3271523fb27cd38c08de0f20e0515/index.d.ts#L196
on<E extends Events>(event: E, listener: EventListener<E>): this;
You need to wrap on function into a Promise.
async myConnect(): Promise<void> {
this.innerProducer.connect();
await new Promise<void>((resolve, reject) => {
// TODO: Add reject call on some error.
this.innerProducer.on("ready", () => {
Logger.info("producer is ready to produce for topic {}", this.topic);
this.isReadyToProduce = true;
resolve();
});
});
}
Then when you need to use use use await myConnect() or myConnect().then(() => { some code here! });
I trying to make a puppeteer.js bot to be able to pause and resume its work.
In general, i have a class with a dozen of async methods, event emitter and a property called 'state' with setter to change it. When I have event 'stop', I want some async functions to be aborted. How can I achieve this?
I thought i need to observe when this.state becomes 'stop', and run return; but hadn't found any solution.
Then I decided to try to set a handler on an event which changes state to 'stop', but I cannot abort async functions from the handler on the stop event.
constructor() {
this.state = 'none';
this.emiter = new events.EventEmitter();
this.setHandler('stop', () => this.stop());
this.setHandler('resume', () => this.resume());
this.setHandler('onLoginPage', () => this.passAuth());
// ...
// And dozen of other states with its handlers
}
stop= () => this.setState('stoped', true);
resume = () => this.setState(this.getPreviousState());
getPreviousState = () => ...
// Just an example of a state handler. It has async calls as well
// I want to abort this function when event 'stop' is emitted
#errorCatcher()
async passAuth() {
const { credentials } = Setup.Instance;
await this.page.waitForSelector(LOGIN);
await typeToInput(this.page, EMAIL_INPUT, credentials.login);
await typeToInput(this.page, PWD_INPUT, credentials.pass);
await Promise.all([
await this.page.click(LOGIN),
this.page.waitForNavigation({ timeout: 600000 }),
]);
await this.page.waitFor(500);
await DomMutations.setDomMutationObserver(this.page, this.socketEmitter);
// ...
// And dozen of handlers on corresponding state
setState(nextState, resume) {
// Avoiding to change state if we on pause.
// But resume() can force setstate with argument resume = true;
if (this.state === 'stoped' && !resume) return false;
console.log(`\nEmmited FSM#${nextState}`);
this.emiter.emit(`FSM#${nextState}`);
}
setHandler(state, handler) {
this.emiter.on(`FSM#${state}`, async () => {
this.state = state;
console.log(`State has been changed: ${this.getPreviousState()} ==> ${this.state}. Runnig handler.\n`);
//
// On the next line, we run a corresponding handler func,
// like passAuth() for state 'onLoginPage'. It has has to be aborted
// if emiter gets 'FSM#stoped' event.
//
await handler();
});
}
}```
I expect the async functions to be aborted when event emitter emits 'stop';
It is impossible to do it natively.
Alternatively, there are two other way to do it.
check your state after any call of await, for example:
class Stated {
async run() {
await foo()
if(this.stopped) return
await bar()
if(this.stopped) return
await done()
}
}
const s = new Stated()
s.run()
use generator with custom wrapper rather than async/await.
// the wrapper
function co(gen, isStopped = () => false) {
return new Promise((resolve, reject) => {
if (!gen || typeof gen.next !== 'function') return resolve(gen)
onFulfilled()
function onFulfilled(res) {
let ret
try {
ret = gen.next(res)
} catch (e) {
return reject(e)
}
next(ret)
}
function onRejected(err) {
let ret
try {
ret = gen.throw(err)
} catch (e) {
return reject(e)
}
next(ret)
}
function next(ret) {
if (ret.done || isStopped()) return resolve(ret.value)
Promise.resolve(ret.value).then(onFulfilled, onRejected)
}
});
}
// the following is your code:
class Stated {
* run() {
yield foo()
yield bar()
yield done()
}
}
const s = new Stated()
co(s.run(), () => s.stopped)
That's how my consumer is initialised:
const client = new kafka.Client(config.ZK_HOST)
const consumer = new kafka.Consumer(client, [{ topic: config.KAFKA_TOPIC, offset: 0}],
{
autoCommit: false
})
Now the consumer consumer.on('message', message => applyMessage(message))
The thing is applyMessage talks to the database using knex, the code looks something like:
async function applyMessage(message: kafka.Message) {
const usersCount = await db('users').count()
// just assume we ABSOLUTELY need to calculate a number of users,
// so we need previous state
await db('users').insert(inferUserFromMessage(message))
}
The code above makes applyMessage to execute in parallel for all the messages in kafka, so in the code above given that there are no users in the database yet, usersCount will ALWAYS be 0 even for the second message from kafka where it should be 1 already since first call to applyMessage inserts a user.
How do I "synchronise" the code in a way that all the applyMessage functions run sequentially?
You'll need to implement some sort of Mutex. Basically a class which queues up things to execute synchronously. Example
var Mutex = function() {
this.queue = [];
this.locked = false;
};
Mutex.prototype.enqueue = function(task) {
this.queue.push(task);
if (!this.locked) {
this.dequeue();
}
};
Mutex.prototype.dequeue = function() {
this.locked = true;
const task = this.queue.shift();
if (task) {
this.execute(task);
} else {
this.locked = false;
}
};
Mutex.prototype.execute = async function(task) {
try { await task(); } catch (err) { }
this.dequeue();
}
In order for this to work, your applyMessage function (whichever handles Kafka messages) needs to return a Promise - notice also the async has moved from the parent function to the returned Promise function:
function applyMessage(message: kafka.Message) {
return new Promise(async function(resolve,reject) {
try {
const usersCount = await db('users').count()
// just assume we ABSOLUTELY need to calculate a number of users,
// so we need previous state
await db('users').insert(inferUserFromMessage(message))
resolve();
} catch (err) {
reject(err);
}
});
}
Finally, each invocation of applyMessage needs to be added to the Mutex queue instead of called directly:
var mutex = new Mutex();
consumer.on('message', message => mutex.enqueue(function() { return applyMessage(message); }))
I have a function that takes a transaction object as an argument. Can this function subscribe to an event that fires when the transaction is commited?
function createUser (data, trx) {
trx.on('success', .. )
return User.create(data, { transacting: trx })
}
I don't see anything like that in the source, if not inner/outer transaction can be used somehow.
https://github.com/tgriesser/knex/blob/master/src/transaction.js
I never found better solution. But transaction is event emmiter so you can override default knex functions to emit your custom event.
Override commit to fire event.
knex = require('knex')({...});
const _transaction = knex.transaction;
knex.transaction = (cb) => {
return _transaction(trx => {
const _commit = trx.commit;
trx.commit =async (conn, value) => {
const out = await _commit(conn, value);
trx.emit('commit');
return out;
}
return cb(trx);
})
};
Listen to commit event anywhere in code
knex.transaction(async trx => {
trx.on('commit', async () => {
// fired after commit is done
});
await trx.select().from('users').update({...});
})