Rephrased at the end
NodeJS communicates with other APIs through GRPC.
Each external API has its own dedicated GRPC connection with Node and every dedicated GRPC connection has an upper bound of concurrent clients that it can serve simultaneously (e.g. External API 1 has an upper bound of 30 users).
Every request to the Express API, may need to communicate with External API 1, External API 2, or External API 3 (from now on, EAP1, EAP2 etc) and the Express API also has an upper bound of concurrent clients (e.g. 100 clients) that can feed the EAPs with.
So, how I am thinking of solving the issue:
A Client makes a new request to the Express API.
A middleware, queueManager, creates a Ticket for the client (think of it as a Ticket that approves access to the System - it has basic data of the Client (e.g. name))
The Client gets the Ticket, creates an Event Listener that listens
to an event with their Ticket ID as the event name (when the System
is ready to accept a Ticket, it yields the Ticket's ID as an event)
and enters a "Lobby" where, the Client, just waits till their ticket
ID is accepted/announced (event).
My issue is that I can't really think of how to implement the way that the system will keep track of the tickets and how to have a queue based on the concurrent clients of the system.
Before the client is granted access to the System, the System itself should:
Check if the Express API has reached its upper-bound of concurrent clients -> If that's true, it should just wait till a new Ticket position is available
If a new position is available, it should check the Ticket and find out which API it needs to contact. If, for example, it needs to contact EAP1, it should check how many current clients use the GRPC connection. This is already implemented (Every External API is under a Class that has all the information that is needed). If the EAP1 has reached its upper-bound, then NodeJS should try again later (But, how much later? Should I emit a system event after the System has completed another request to EAP1?)
I'm aware of Bull, but I am not really sure if it fits my requirements.
What I really need to do is to have the Clients in a queue, and:
Check if Express API has reached its upper-bound of concurrent users
If a position is free, pop() a Ticket from the Ticket's array
Check if the EAPx has reached its upper-bound limit of concurrent users
If true, try another ticket (if available) that needs to communicate
with a different EAP
If false, grant access
Edit: One more idea could be to have two Bull Queues. One for the Express API (where the option "concurrency" could be set as the upper bound of the Express API) and one for the EAPs. Each EAP Queue will have a distinct worker (in order to set the upper bound limits).
REPHRASED
In order to be more descriptive about the issue, I'll try to rephrase the needs.
A simple view of the System could be:
I have used Clem's suggestion (RabbitMQ), but again, I can't achieve concurrency with limits (upper-bounds).
So,
Client asks for a Ticket from the TicketHandler. In order for the TicketHandler to construct a new Ticket, the client, along with other information, provides a callback:
TicketHandler.getTicket(variousInfo, function () {
next();
})
The callback will be used by the system to allow a Client to connect with an EAP.
TickerHandler gets the ticket:
i) Adds it to the queue
ii) When the ticket can be accessed (upper-bound is not reached), it asks the appropriate EAP Handler if the client can make use of the GRPC connection. If yes, then asks the EAP Handler to lock a position and then it calls the ticket's available callback (from Step 1)
If no, TicketHandler checks the next available Ticket that needs to contact a different EAP. This should go on until the EAP Handler that first informed TicketHandler that "No position is available", sends a message to TicketHandler in order to inform it that "Now there are X available positions" (or "1 available position"). Then TicketHandler, should check the ticket that couldn't access EAPx before and ask again EAPx if it can access the GRPC connection.
From your description I understand what follows:
You have a Node.js front-tier. Each Node.js box needs to be limited to up to 100 clients
You have an undefined back-tier that has GRPC connections with the boxes in the front-tier (let's call them EAPs). Each EAP <-> Node.js GRPS link is limited to N concurrent connections.
What I see here are only server-level and connection-level limits thus I see no reason to have any distributed system (like Bull) to manage the queue (if the Node.js box dies there is no one able to recover the HTTP request context to offer a response to that specific request - therefore when a Node.js box dies responses to its requests are not more useful).
This being considered I would simply create a local queue (as simple as an array) to manage your queuing.
Disclaimer: this has to be considered pseudo-code what follows is simplified and untested
This may be a Queue implementation:
interface SimpleQueueObject<Req, Res> {
req: Req;
then: (Res) => void;
catch: (any) => void;
}
class SimpleQueue<Req = any, Res = any> {
constructor(
protected size: number = 100,
/** async function to be executed when a request is de-queued */
protected execute: (req: Req) => Promise<Res>,
/** an optional function that may ba used to indicate a request is
not yet ready to be de-queued. In such case nex request will be attempted */
protected ready?: (req: Req) => boolean,
) { }
_queue: SimpleQueueObject<Req, Res>[] = [];
_running: number = 0;
private _dispatch() {
// Queues all available
while (this._running < this.size && this._queue.length > 0) {
// Accept
let obj;
if (this.ready) {
const ix = this._queue.findIndex(o => this.ready(o.req));
// todo : this may cause queue to stall (for now we throw error)
if (ix === -1) return;
obj = this._queue.splice(ix, 1)[0];
} else {
obj = this._queue.pop();
}
// Execute
this.execute(obj.req)
// Resolves the main request
.then(obj.then)
.catch(obj.catch)
// Attempts to queue something else after an outcome from EAP
.finally(() => {
this._running --;
this._dispatch();
});
obj.running = true;
this._running ++;
}
}
/** Queue a request, fail if queue is busy */
queue(req: Req): Promise<Res> {
if (this._running >= this.size) {
throw "Queue is busy";
}
// Queue up
return new Promise<Res>((resolve, reject) => {
this._queue.push({ req, then: resolve, catch: reject });
this._dispatch();
});
}
/** Queue a request (even if busy), but wait a maximum time
* for the request to be de-queued */
queueTimeout(req: Req, maxWait: number): Promise<Res> {
return new Promise<Res>((resolve, reject) => {
const obj: SimpleQueueObject<Req, Res> = { req, then: resolve, catch: reject };
// Expire if not started after maxWait
const _t = setTimeout(() => {
const ix = this._queue.indexOf(obj);
if (ix !== -1) {
this._queue.splice(ix, 1);
reject("Request expired");
}
}, maxWait);
// todo : clear timeout
// Queue up
this._queue.push(obj);
this._dispatch();
})
}
isBusy(): boolean {
return this._running >= this.size;
}
}
And then your Node.js business logic may do something like:
const EAP1: SimpleQueue = /* ... */;
const EAP2: SimpleQueue = /* ... */;
const INGRESS: SimpleQueue = new SimpleQueue<any, any>(
100,
// Forward request to EAP
async req => {
if (req.forEap1) {
// Example 1: this will fail if EAP1 is busy
return EAP1.queue(req);
} else if (req.forEap2) {
// Example 2: this will fail if EAP2 is busy and the request can not
// be queued within 200ms
return EAP2.queueTimeout(req, 200);
}
}
)
app.get('/', function (req, res) {
// Forward request to ingress queue
INGRESS.queue(req)
.then(r => res.status(200).send(r))
.catch(e => res.status(400).send(e));
})
Or this solution will allow you (as requested) to also accept requests for busy EAPs (up to a max of 100 in total) and dispatch them when they become ready:
const INGRESS: SimpleQueue = new SimpleQueue<any, any>(
100,
// Forward request to EAP
async req => {
if (req.forEap1) {
return EAP1.queue(req);
} else if (req.forEap2) {
return EAP2.queue(req);
}
},
// Delay queue for busy consumers
req => {
if (req.forEap1) {
return !EAP1.isBusy();
} else if (req.forEap2) {
return !EAP2.isBusy();
} else {
return true;
}
}
)
Please note that:
in this example, Node.js will start throwing when more than 100 concurrent requests are received (it is not unusual to throw a 503 while throttling)
Be careful when you have more throttling limits (Node.js and GRPC in your case) as the first may cause the seconds starvation (think about receiving 100 requests for EAP1 and then 10 for EAP2, Node.js will be full with EAP1 requests and will refuse EAP2 ones all do EAP2 is doing nothing)
Related
I want to make a progress bar kind of telling where the user where in process of fetching the API my backend is. But it seems like every time I send a response it stops the request, how can I avoid this and what should I google to learn more since I didn't find anything online.
React:
const {data, error, isError, isLoading } = useQuery('posts', fetchPosts)
if(isLoading){<p>Loadinng..</p>}
return({data&&<p>{data}</p>})
Express:
app.get("api/v1/testData", async (req, res) => {
try {
const info = req.query.info
const sortByThis = req.query.sortBy;
if (info) {
let yourMessage = "Getting Data";
res.status(200).send(yourMessage);
const valueArray = await fetchData(info);
yourMessage = "Data retrived, now sorting";
res.status(200).send(yourMessage);
const sortedArray = valueArray.filter((item) => item.value === sortByThis);
yourMessage = "Sorting Done now creating geojson";
res.status(200).send(yourMessage);
createGeoJson(sortedArray)
res.status(200).send(geojson);
}
else { res.status(400) }
} catch (err) { console.log(err) res.status(500).send }
}
You can only send one response to a request in HTTP.
In case you want to have status updates using HTTP, the client needs to poll the server i.e. request status updates from the server. Keep in mind though that every request needs to be processed on the server side and will take resources away which are then not available for other (more important) requests from other clients. So don't poll too frequently.
If you want to support long running operations using HTTP have a look at the following API design pattern.
Alternatively you could also use a WebSockets connection to push updates from the server to the client. I assume your computation on the backend will not be minutes long and you want to update the client in real-time, so probably WebSockets will be the best option for you. A WebSocket connection has, once established, considerably less overhead than sending huge HTTP requests/ responses between client and server.
Have a look at this thread which dicusses abovementioned and other possibilites.
I'm sure this kind of problem have been resolved here many time but I can't find how those question was formulated.
I have a micro-services that handle the communication between my infrastructure and a MQTT Broker. Every time a HTTP request is received I send a "Who is alive in the room XXX ?" message on the MQTT Broker, and every client registered on the "XXX/alive" topic have to answer and I wait Y milliseconds before closing the request by sending back the responses received to the client.
It works well when I'm handling one request. But it screws up when more than one request is asked at a time.
Here is the Express route handling the HTTP requests :
app.get('/espPassports', (req, res) => {
mqttHelper.getESPPassports(req.query.model_Name).then((passports) => {
res.send(passports).end();
}).catch(err => {
res.send(err).end();
})
})
Here is how the getESPPassports works :
getESPPassports: async (model_Name) => {
return new Promise((resolve, reject) => {
// Say there is a request performed
ongoing_request.isOpen = true;
ongoing_request.model_Name = model_Name;
// Ask who is alive
con.publish(topic, "ASK");
setTimeout(() => {
// If no answer after given timeout
if (ongoing_request.passports.length == 0) {
reject({ error: "No MQTT passports found" });
// Else send a deep clone of the answers (else it's empty)
} else {
resolve(JSON.parse(JSON.stringify(ongoing_request.passports)));
}
// Delete the current request object and 'close it'
ongoing_request.passports.length = 0;
ongoing_request.isOpen = false;
ongoing_request.model_Name = ""
}, process.env.mqtt_timeout || 2000)
})
}
};
And here is the MQTT listener :
con.on("message", (topic, message) => {
// If a passport is received check the topic and if there is a request opened
if (_checkTopic(topic) && ongoing_request.isOpen) {
try {
ongoing_request.passports.push(JSON.parse(message));
} catch (error) {
// do stuff if error
}
}
}
})
I know the problem come from the boolean i'm using to specify if there is a request ongoing, I was thinking to create an object for each new request and identify them by a unique id (like a timetamp) but I have no way to make the MQTT listneners to know this unique id.
I have some other solution in mind but I'm not sure they'll work and I feel like there is a way to handle that nicely that I don't know about.
Have a good day.
You need to generate a unique id for each request and include it in the MQTT message, you can then cache the Express response object keyed by the unique id.
The devices need to include the unique id in their responses so they can be paired up with the right response.
The other approach is just to cache responses from the devices and assign the cache a Time to Live so you don't need to ask the devices every time.
let's say I have an application that returns exchange rates for today.
The service should read data via REST then save in cache and give clients from this cache. I want this request to 3rd party API to happen upon first attempt to get today's rate (kind of lazy init for every day).
Something like this:
(1) HTTP Request to get rate (form my app's client)
(2) if rate for today is available then return it
else
(3) read it from 3rd party service (via REST request)
(4) save in cache
(5) return from cache
The whole logic is written by mean of promises but the is a problem if i have millions of requests simultaneously at the very beginning of the day. In this case if one of the requests is on operations (3), (4) or (5) ( which are organized as a promise chain) operation (1) and (2) for other request can be handled by node in between.
E.g. while first requests is still waiting for the 3rd party API to response and the cache is empty other million of requests can also fire the same request to the same 3rd party API.
My thought is to chain operation (3) to some kind of an object A with the promise ( A.promise) inside that exposes resolve function to A. All other requests would wait (not synchronously wait of course) till the first request updates the cache and calls A.resolve() which will resolve A.promise.
But it looks a bit ugly, any idea of a better approach?
Update
I've got one solution, not sure whether it's node.js style:
function Deferred(){;
this.promise = false
this.markInProgress = ()=>{
this.promise = new Promise((res, rej)=>{
this.resolve = res;
this.reject = rej;
})
}
this.markDone = ()=>{
this.resolve()
this.promise = false
}
this.isInProgress = this.promise
}
let state = new Deferred();
function updateCurrencyRate(){
return db.any(`select name from t_currency group by name`)
.then((currencies) => {
return getRateFromCbr()
.then(res => Promise.all(
currencies.map((currency, i, currencies) =>
saveCurrency(
currency.name,
parseRate(res, currency.name)))));
})
}
function loadCurrencyRateFroDate(date){
if (state.isInProgress){
return state.promise
} else {
state.markInProgress();
return updateCurrencyRate()
.then(()=> {
state.markDone();
})
}
}
function getCurrencyRateForDate(date){
return getCurrencytRateFromDb(date)
.then((rate) => {
if (rate[0]) {
return Promise.resolve(rate)
} else {
loadCurrencyRateFroDate(date)
.then(()=>getCurrencytRateFromDb(date))
}
})
}
I would take a very simple queue, flush and fallback approach to this.
Implement a queuing mechanism (maybe with RabbitMQ) and route all your requests to the queue. This way you can hold off responding to requests when cache expires.
Create an expirable cache layer (maybe a redis cache) and expire your cache everyday.
By default route your requests from the queue to get data from cache. On receiving the data from cache, if the cache has expired, hold the queue and get data directly from 3rd party and update your cache and its expiry.
flush your cache every day
With queues, you have better control over the traffic. You can also add 3rd party API call as a fallback way to get data when your cache fails or anything goes wrong.
I'm using simply like below:
class Bot {
constructor(token) {
let _baseApiURL = `https://api.telegram.org`;
//code here
}
getAPI(apiName) {
return axios.get(`${this.getApiURL()}/${apiName}`);
}
getApiURL() {
return `${this.getBaseApiUrl()}/bot${this.getToken()}`;
}
getUpdates(fn) {
this.getAPI('getUpdates')
.then(res => {
this.storeUpdates(res.data);
fn(res.data);
setTimeout(() => {
this.getUpdates(fn);
}, 1000);
})
.catch(err => {
console.log('::: ERROR :::', err);
});
}
}
const bot = new Bot('mytoken');
bot.start();
I'd like to know whether there is a better way to listen for Telegram's updates, instead of using a timeout and redo an Ajax call to 'getUpdates' API
Telegram supports polling or webhooks, so you can use the latter to avoid polling the getUpdates API
Getting updates
There are two mutually exclusive ways of receiving updates for your
bot — the getUpdates method on one hand and Webhooks on the other.
Incoming updates are stored on the server until the bot receives them
either way, but they will not be kept longer than 24 hours.
Regardless of which option you choose, you will receive JSON-serialized Update objects as a result.
More info on: https://core.telegram.org/bots/api#getting-updates
You can use telegraf to easily setup a webhook or to handle the polling for you with a great API
I built a simple API endpoint with NodeJS using Sails.js.
When someone access my API endpoint, the server starts to wait for data and whenever a new data appears, he broadcasts it using sockets. Each client should receive his own stream of data based on his user input.
var Cap = require('cap').Cap;
collect: function (req, res) {
var iface = req.param("ip");
var c = new Cap(),
device = Cap.findDevice(ip);
c.on('data', function(myData) {
sails.sockets.blast('message', {"host": myData});
});
});
The response do not complete (I never send a res.json() - what actually happens is that the browser keep loading - but the above functionality works).
2 Problems:
I'm trying to subscribe and unsubscribe to to this API endpoint from my client (using RxJS). When I subscribe, I start to receive data via sockets - but I can't unsubscribe to the API endpoint (the browser expect the request to be completed).
Each client should subscribe to his own socket room based on the request IP parameter ( see updated code ). Currently it blasts the message to everyone.
How I can create a stream/service-like API endpoint with Sails.js that will emit new data to each user based on his input?
My goal is to be able to subscribe / unsubscribe to this API endpoint from each client.
Revised Answer
Let's assume your API endpoint is defined in config/routes.js like this:
...
'get /collect': 'SomeController.collectSubscribe',
'delete /collect': 'SomeController.collectUnsubscribe',
Since each Cap instance is tied to one device, we need one instance for each subscription. Instead of using the sails join/leave methods, we keep track of Cap instances in memory and just broadcast to the request socket's id. This works because Sails sockets are subscribed to their own ids by default.
In api/controllers/SomeController.js:
// In order for the `Cap` instances to persist after `collectSubscribe` finishes, we store them all in an Object, associated with which socket the were created for.
var caps = {/* req.socket.id: <instance of Cap>, */};
module.exports = {
...
collectSubscribe: function(req, res) {
if (!res.isSocket) return res.badRequest("I need a websocket! Help!");
if (!!caps[req.socket.id]) return res.badRequest("Dude, you are already subscribed.");
caps[req.socket.id] = new Cap();
var c = caps[req.socket.id]; // remember that `c` is a reference to our new `Cap`, not a copy.
var device = c.findDevice(req.param('ip'));
c.open(device, ...);
c.on('data', function(myData) {
sails.sockets.broadcast(req.socket.id, 'message', {host: myData});
});
return res.ok();
},
collectUnsubscribe: function(req, res) {
if (!res.isSocket) return res.badRequest("I need a websocket! Help!");
if (!caps[req.socket.id]) return res.badRequest("I can't unsubscribe you unless you actually subscribe first.");
caps[req.socket.id].removeAllListeners('data');
delete caps[req.socket.id];
return res.ok();
}
}
Basically, it goes like this: when a browser request triggers collectSubscribe, a new Cap instance listens to the provided IP. When the browser triggers collectUnsubscribe, the server retreives that Cap instance, tells it to stop listening, and then deletes it.
Production Considerations: please be aware that the list of Caps is NOT PERSISTENT (since it is stored in memory and not a DB)! So if your server is turned off and rebooted (due to lightning storm, etc), the list will be cleared, but considering that all websocket connections will be dropped anyway, I don't see any need to worry about this.
Old Answer, Kept for Reference
You can use sails.sockets.join(req, room) and sails.sockets.leave(req, room) to manage socket rooms. Essentially you have a room called "collect", and only sockets joined in that room will receive a sails.sockets.broadcast(room, eventName, data).
More info on how to user sails.sockets here.
In api/controllers/SomeController.js:
collectSubscribe: function(req, res) {
if (!res.isSocket) return res.badRequest();
sails.sockets.join(req, 'collect');
return res.ok();
},
collectUnsubscribe: function(req, res) {
if (!res.isSocket) return res.badRequest();
sails.sockets.leave(req, 'collect');
return res.ok();
}
Finally, we need to tell the server to broadcast messages to our 'collect' room.
Note that this only need to happen once, so you can do this in a file under the config/ directory.
For this example, I'll put this in config/sockets.js
module.exports = {
// ...
};
c.on('data', function(myData) {
var eventName = 'message';
var data = {host: myData};
sails.sockets.broadcast('collect', eventName, data);
});
I am assuming that c is accessible here; If not, you could define it as sails.c = ... to make it globally accessible.