How to implement a queue using RxJS? - node.js

Question

I'm designing an API (in NodeJS) which receives a list of user ids, for instance, and access Google Firebase's API to send a notification to those users.
Considering Firebase having a quota limit on how many requests you can concurrently send to, I buffered the Ids and send the requests one by one to Firebase with a delay of two seconds. Here's some simplified sample code for explanation:
app.post('/send-request', (req, res, next) =>{
const userIds = req.body.userIds;
‎const streams = userIds.map((userId) => {
‎ return Observable
‎ .fromPromise(fetch(`firebase url`, {
‎ method: 'POST',
‎ headers: ...,
body: ...
‎ }))
‎ .delay(2000)
‎ .retryWhen(attempts => {
‎ return attempts.zip(Rx.Observable.range(1, 4))
.mergeMap(([error, i]) => {
if (i > 3) {
return Rx.Observable.throw(error);
}
console.log(`Wait ${i} seconds, then retry!`);
return Rx.Observable.timer(i * 1000);
});
‎ });
});
const stream = Observable.merge(...streams);
‎ stream.subscribe();
});
This can handle the case when a single request consists of a lot of users. However, if my API receives similar requests concurrently, it's bound to fail.
Therefore, I want to buffer all these user ids in a queue where this queue can keep receiving more and more user ids an buffer them, and at the same time 'top' the queue by sending requests to Firebase at a steady rate. However, I don't know how to do it with RxJS. Do I have to use an Scheduler? Or is there actually a better solution than using Rx?
Note: I understand Javascript is single-threaded so it's not exactly concurrency, I only used this word so you can get the idea better.

Answer 1

I think I managed to come up with something, the key part is to use Subject to publish values, then use the zip operator to emit value on a regular interval. It'd be even better if I can take the values on demand, but the current solution is already much better than my original method.
const subject = new Rx.Subject();
const stream = subject
.zip(Rx.Observable.interval(3000), function(a, b) { return a; });
stream.subscribe(
(x) => { console.log(`onNext: ${val}`); },
(e) => { console.log(`onError: ${e}`); },
() => { console.log('onCompleted'); });
I used VueJS to build a simple web page for a little demo.
const app = new Vue({
el: '#app',
data: {
subject: undefined,
stream: undefined,
count: 0,
emitHistory: [],
disableBtn: true
},
created() {
console.log('created');
this.subject = new Rx.Subject();
this.stream = this.subject
.zip(Rx.Observable.interval(3000), function(a, b) { return a; })
// .observeOn(Rx.Scheduler.queue); // not working
this.stream.subscribe(
(val) => {
console.log(`onNext: ${val}`);
this.emitHistory.push(val);
if (val === this.count) {
this.disableBtn = false;
} else {
this.disableBtn = true;
}
},
(e) => { console.log(`onError: ${e}`); },
() => { console.log('onCompleted'); });
},
methods: {
clickHandler() {
this.count++;
this.subject.onNext(this.count);
},
clear() {
this.count = 0;
this.emitHistory = [];
}
}
});
/*
Rx.Observable
.fromArray([1,2,3])
.zip(Rx.Observable.interval(500), function(a, b) { return a; })
.subscribe(
function(x) { document.write(x + '<br \>'); },
null,
function() { document.write("complete"); });
*/
<script src="https://cdnjs.cloudflare.com/ajax/libs/vue/2.5.13/vue.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/4.0.7/rx.all.js"></script>
<div id="app">
<button #click="clickHandler()">Click me</button>
<button #click="clear()" v-bind:disabled="disableBtn">Clear</button>
<div>
<h5>Count: {{count}}</h5>
</div>
<div>
<ul>
<li v-for="(item, idx) in emitHistory" v-bind:key="idx">{{item}}</li>
</ul>
</div>
</div>