i have a question that is keeping me busy and i wondered if anyone might have the answer.
Lets assume that i have an express app that listens for a post request,
This post request triggers a function in the program that calls over 1000 times with axios to a service (for that matter an sms provider that with the api call sends an sms message).
Now, assuming that one request takes 200 ms for the SMS provider, 1000 of them will take over 2 minutes.
Of course i will send a response from the server earlier saying that the request have been recieved.
My question is, lets say that there are 20 or even 100 requests of it at the same time, how can i get the app to handle that traffic? Even if it would it will take very long to perform all of those actions.
How can it be performed? Is there a preferd language to do so?
Is it possible with node?
If you are broadcasting same sms to many users, you should search for a broadcasting command in the sms api. This would decrease number of operations/bandwidth required on your server.
If broadcasting support doesn't exist, you should let other requests be fetched while processing those sms tasks. Something like:
function smsTask(n)
{
doSmsWork(function(response){
if(n)
{
setTimeout(function(){
smsTask(n-1);
},0);
}
})
}
smsTask(num);
Between if(n) and smsTask(n-1), other asynchronous tasks can find time to work, including fetching/intercepting new smsTasks and RAM requirement depends only on number of requests in-flight, not number of sms tasks. If you need to cap the bandwidth for all sms tasks, you can use a dynamical waiting value (instead of 0) for the setTimeout function and this can dedicate more bandwidth to other tasks (like serving web pages, etc) instead of being fully consumed by sms spam.
If you don't ask for asynchronously letting new requests be handled, then you can complete whole work per request much quicker than 2 minutes, if its I/O bound:
function smsTask(n)
{
for(let i=0;i<n;i++)
doSmsWork(function(response){
})
}
smsTask(num);
in perfect I/O conditions, this can complete within 200ms-300ms even for n=1000. This still doesn't stop new tasks/requests but puts too much pressure on the queue and probably consumes more memory while other version keeps stead memory consumption depending on number of requests in flight only.
If you need even less I/O contention, you can put the tasks into a queue and have a dedicated function for processing the queue:
let work = [];
function smsTask(n){ work.push(some_task(n)); }
setInterval(function(){
if(work.length > 0)
{
let task = work.shift();
task?.compute(); // does sms work
}
},1000);
This is steady-state processing and still doesn't stop because of sudden request spikes as long as work queue does not overflow memory. Maybe setTimeout version of this is better for CPU.
Related
2 types of triggers for Azure Functions are message queues and http triggers. I'm guessing that one difference is that with http triggers a request could be denied if there are not enough instances to service the request, whereas with message queues it will check if there is an instance available and if not spin one up before trying to process the message. Is that a correct understanding?
Not quite..! I think you're getting the information from here:
Whenever possible, refactor large functions into smaller function sets that work together and return responses fast. For example, a webhook or HTTP trigger function might require an acknowledgment response within a certain time limit; it is common for webhooks to require an immediate response.
The acknowledgement within a certain time limit is all about how long the client is willing to wait for a response. So if you were to execute some task that takes a long time (on the order of minutes), you may not receive an informative response because the client would just call the connection dead. However, your function would still execute just fine as long as it stayed within the functionTimeout limits (one a consumption plan, the default is 5 minutes, max being 10 minutes).
I have a question that nobody seems to help with. How will this be handled in a production mode with thousands of requests at the same time?
I did a simple test case:
module.exports = {
index: function (req, res){
if (req.param('foo') == 'bar'){
async.series([
function(callback){
for (k=0; k <= 50000; k++){
console.log('did something stupid a few times');
}
callback();
}
], function(){
return res.json(null);
});
}else{
return res.view('homepage', {
});
}
}
};
Now if I go to http://localhost:1337/?foo=bar it will obviously wait a while before it responds. So if I now open a different session (other browser or incognito, and go to http://localhost:1337/ I am expecting a result immediately. Instead it is waiting for the other request to finish and only then it will let this request go through.
Therefore it is not asynchronous and it is a huge problem if I have as much as 2 ppl at the same time operating this app. I mean this app will have drop downs coming from databases, html files being served etc...
My question is this: how does one handle such an issue??? I hear the word "promises vs callbacks" - is this some sort of a solution to this?
I know about clustering, but that only separates the requests over the amount of cpu's, ultimately you will fix it by at most allowing 8 people at the same time without being blocked. It won;t handle 100 requests at the same time...
P.S. That test was to simplify the example, but think of someone uploading a file, a web service that goes to a different server, a point of sales payment terminal waiting for a user to input the pin, someone downloading a file from the app, etc...
nodejs is event driven and runs your Javascript as single threaded. So, as long as your code from the first request is sitting in that for loop, nodejs can't do anything else and won't get to the next event in the event queue so thus your second request has to wait for the first one to finish.
Now, if you used a true async operation such as setTimeout() instead of your big for loop, then nodejs could service other events while the first request was waiting for the setTimeout().
The general rule in nodejs is to avoid doing anything that takes a ton of CPU in your main nodejs app. If you are stuck with something CPU-intensive, then you're best to either run clusters (as many as CPUs you have) or move the CPU-intensive work to some sort of worker queue that is served by different processes and let the OS time slice those other processes while the main nodejs process stays free and ready to service new incoming requests.
My question is this: how does one handle such an issue??? I hear the word "promises vs callbacks" - is this some sort of a solution to this?
I know about clustering, but that only separates the requests over the amount of cpu's, ultimately you will fix it by at most allowing 8 people at the same time without being blocked. It won;t handle 100 requests at the same time...
Most of the time, a server process spends most of the time of a request doing things that are asynchronous in nodejs (reading files, talking to other servers, doing database operations, etc...) where the actual work is done outside the nodejs process. When that is the case, nodejs does not block and is free to work on other requests while the async operations from other requests are underway. The little bit of CPU time coordinating these operations can be helped further by clustering though it's probably worth testing a single process first to see if clustering is really needed.
P.S. That test was to simplify the example, but think of someone uploading a file, a web service that goes to a different server, a point of sales payment terminal waiting for a user to input the pin, someone downloading a file from the app, etc...
All the operations you mentioned here can be done truly asynchronously so they won't block your nodejs app like your for loop does so basically the for loop isn't a good simulation of any of this. You need to use a real async operation to simulate it. Real async operations do their work outside of the main nodejs thread and then just post an event to the event queue when they are done, allowing nodejs to do other things while the async operations are doing their work. That's the key.
I want this kind of structure;
express backend gets a request and runs a function, this function will get data from different apis and saves it to db. Because this could takes minutes i want it to run parallel while my web server continues to processing requests.
i want this because of this scenario:
user has dashboard, after logs in app starts to collect data from apis and preparing the dashboard for user, at that time user can navigate through site even can close the browser but the function has to run until it finishes fetching data.Once it finishes, all data will be saved db and the dashboard is ready for user.
how can i do this using child_process or any kind of structure in nodejs?
Since what you're describing is all async I/O (networking or disk) and is not CPU intensive, you don't need multiple child processes in order to effectively serve multiple requests. This is the beauty of node.js. With async I/O, node.js can be working on many different requests over the same period of time.
Let's supposed part of your process is downloading an image. Your node.js code sends a request to fetch an image. That request is sent off via TCP. Immediately, there is nothing else to do on that request. It's winging it's way to the destination server and the destination server is preparing the response. While all that is going on, your node.js server is completely free to pull other events from it's event queue and start working on other requests. Those other requests do something similar (they start async operations and then wait for events to happen sometime later).
Your server might get 10 different async operations started and "in flight" before the first one actually starts getting a response. When a response starts coming in, the system puts an event into the node.js event queue. When node.js has a moment between other requests, it pulls the next event out of the event queue and processes it. If the processing has further async operations (like saving it to disk), the whole async and event-driven process starts over again as node.js requests a write to disk and node.js is again free to serve other events. In this manner, events are pulled from the event queue one at a time as they become available and lots of different operations can all get worked on in the idle time between async operations (of which there is a lot).
The only thing that upsets the apple cart and ruins the ability of node.js to juggle lots of different things all at once is an operation that takes a lot of CPU cycles (like say some unusually heavy duty crypto). If you had something like that, it would "hog" too much of the CPU and the CPU couldn't be effectively shared among lots of other operations. If that were the case, then you would want to move the CPU-intensive operations to a group of child processes. But, just doing async I/O (disk, networking, other hardware ports, etc...) does not hog the CPU - in fact it barely uses much node.js CPU.
So, the next question is often "how do I know if I have too much stuff that uses the CPU". The only way to really know is to just code your server properly using async I/O and then measure its performance under load and see how things go. If you're doing async things appropriately and the CPU still spikes to 100%, then you have too much CPU load and you'll want to either use generic clustering or move specific CPU-heavy operations to a group of child processes.
We need to implement a Async web service.
Behaviour of web service:
We send the request for an account to server and it sends back the sync response with an acknowledgement ID. After that we get multiple Callback requests which contains that acknowldegment ID. The last callback request for an acknowledgement ID will contain a text(completed:true) in the response which will tell us that this is the last callback request for that account and acknowledgement ID. This will help us to know that async call for a particular account is completed and we can mark its final status. We need to execute this web service for multiple accounts. So, we will be getting callback requests for many accounts.
Question:
What is the optimal way to process these multiple callback requests coming for multiple accounts.
Solutions that we thought of:
ExecutorService Fixed Thread Pool: This will parallely process our callback requests but the concern is that it does not maintain the sequence. So it will be difficult for us to determine that the last callback request for an acknowledgment ID(account) has come. Hence, we will not be able to mark the final status of that account as completed with surity.
ExecutorService Single Thread Executor: Here, only one thread is there in the pool with an unbouded queue. If we use this then processing will be pretty slow as only one thread will be actually processing.
Please suggest an optimal way to implement requirement both memory and performance wise.
Let's be clear about one thing: HTTP is a blocking, synchronous protocol. Request/response pairs aren't asynch. What you're doing is spawning asynch requests and returning to the caller to let them know the request is being processed (HTTP 200) or not (HTTP 500).
I'm not sure that I know optimal for this situation, but there are other considerations:
Use an ExecutorServiceThreadPool that you can configure. Make sure you have a prefix that lets you distinguish these threads from others.
Add request task to a blocking dequeue and have a pool of consumer threads process them. You can tune the dequeue and the consumer thread pool sizes.
If processing is intensive, send request messages to a queue running on another server. Have a pool of queue listeners process the requests.
You cannot assume that the callbacks will return in a certain order. Don't depend on "last" being "true". You'll have to join all those threads together to know when they're finished.
It sounds like the web service should have a URL that lets users query for status.
I have the following piece of code:
var schedules = io.on('connection', function(client) {
var schJSON = JSON.parse(fs.readFileSync(__dirname +'huge_file.json', 'utf8'));
client.json.send(schJSON);
});
readFileSync being a blocking call, I figured that while one client's request is being processed by the server, other clients would get queued up. So, if reading the file takes around 10 seconds and I fire off three different connections to the server, the third connection would take around 30 seconds to get a response.
In practice, all three clients get the response almost at the same time (after 10 seconds have elapsed). The three requests were fired from three different machines (with the same external IP address).
How is this possible?
I would expect the flow to go something like this:
First request comes in and does a blocking read of the file.
Two more requests come in and their 'connection' callbacks get queued up.
The initial file read completes and the async client.json.send call starts sending the response.
The two other requests' callbacks get called, with each one's readFileSync call completing very quickly since the file is in the disk cache and then client.json.send is called so that all three requests' responses are being sent in parallel.
All three requests complete, potentially close to the same time because step 4 is so quick.