I have the following piece of code:
var schedules = io.on('connection', function(client) {
var schJSON = JSON.parse(fs.readFileSync(__dirname +'huge_file.json', 'utf8'));
client.json.send(schJSON);
});
readFileSync being a blocking call, I figured that while one client's request is being processed by the server, other clients would get queued up. So, if reading the file takes around 10 seconds and I fire off three different connections to the server, the third connection would take around 30 seconds to get a response.
In practice, all three clients get the response almost at the same time (after 10 seconds have elapsed). The three requests were fired from three different machines (with the same external IP address).
How is this possible?
I would expect the flow to go something like this:
First request comes in and does a blocking read of the file.
Two more requests come in and their 'connection' callbacks get queued up.
The initial file read completes and the async client.json.send call starts sending the response.
The two other requests' callbacks get called, with each one's readFileSync call completing very quickly since the file is in the disk cache and then client.json.send is called so that all three requests' responses are being sent in parallel.
All three requests complete, potentially close to the same time because step 4 is so quick.
Related
i have a question that is keeping me busy and i wondered if anyone might have the answer.
Lets assume that i have an express app that listens for a post request,
This post request triggers a function in the program that calls over 1000 times with axios to a service (for that matter an sms provider that with the api call sends an sms message).
Now, assuming that one request takes 200 ms for the SMS provider, 1000 of them will take over 2 minutes.
Of course i will send a response from the server earlier saying that the request have been recieved.
My question is, lets say that there are 20 or even 100 requests of it at the same time, how can i get the app to handle that traffic? Even if it would it will take very long to perform all of those actions.
How can it be performed? Is there a preferd language to do so?
Is it possible with node?
If you are broadcasting same sms to many users, you should search for a broadcasting command in the sms api. This would decrease number of operations/bandwidth required on your server.
If broadcasting support doesn't exist, you should let other requests be fetched while processing those sms tasks. Something like:
function smsTask(n)
{
doSmsWork(function(response){
if(n)
{
setTimeout(function(){
smsTask(n-1);
},0);
}
})
}
smsTask(num);
Between if(n) and smsTask(n-1), other asynchronous tasks can find time to work, including fetching/intercepting new smsTasks and RAM requirement depends only on number of requests in-flight, not number of sms tasks. If you need to cap the bandwidth for all sms tasks, you can use a dynamical waiting value (instead of 0) for the setTimeout function and this can dedicate more bandwidth to other tasks (like serving web pages, etc) instead of being fully consumed by sms spam.
If you don't ask for asynchronously letting new requests be handled, then you can complete whole work per request much quicker than 2 minutes, if its I/O bound:
function smsTask(n)
{
for(let i=0;i<n;i++)
doSmsWork(function(response){
})
}
smsTask(num);
in perfect I/O conditions, this can complete within 200ms-300ms even for n=1000. This still doesn't stop new tasks/requests but puts too much pressure on the queue and probably consumes more memory while other version keeps stead memory consumption depending on number of requests in flight only.
If you need even less I/O contention, you can put the tasks into a queue and have a dedicated function for processing the queue:
let work = [];
function smsTask(n){ work.push(some_task(n)); }
setInterval(function(){
if(work.length > 0)
{
let task = work.shift();
task?.compute(); // does sms work
}
},1000);
This is steady-state processing and still doesn't stop because of sudden request spikes as long as work queue does not overflow memory. Maybe setTimeout version of this is better for CPU.
2 types of triggers for Azure Functions are message queues and http triggers. I'm guessing that one difference is that with http triggers a request could be denied if there are not enough instances to service the request, whereas with message queues it will check if there is an instance available and if not spin one up before trying to process the message. Is that a correct understanding?
Not quite..! I think you're getting the information from here:
Whenever possible, refactor large functions into smaller function sets that work together and return responses fast. For example, a webhook or HTTP trigger function might require an acknowledgment response within a certain time limit; it is common for webhooks to require an immediate response.
The acknowledgement within a certain time limit is all about how long the client is willing to wait for a response. So if you were to execute some task that takes a long time (on the order of minutes), you may not receive an informative response because the client would just call the connection dead. However, your function would still execute just fine as long as it stayed within the functionTimeout limits (one a consumption plan, the default is 5 minutes, max being 10 minutes).
I want to get API server status in nodejs. I'm using nodejs to open an interface: "api/request?connId=50&timeout=90". This API will keep the request running for provided time on the server side. After the successful completion of the provided time it should return status/OK. And when we have multiple connection ids & timeout, we want the API return all the running requests on the server with their time left for completion, something like below, where 4 and 8 are the connId and 25 and 15 is the time remaining for the requests to complete (in seconds):
{"4":"25","8":"15"}
please help.
Node.js server uses async model in one single thread, which means at any time, only one request (connId) is under execution by Node (except... you have multiple node.js instance, but let's keep the scenario simple and ignore this case).
When one request is processed (running its handler code), it may start an async task such as read a file, and continue execution. The request itself's handler code would be executed without waiting for async task, and when this handler code is finished running, from Node.js point of view, the request handling itself is done -- the handling of async task's result is another thing in another time, node does not care about the progress of it.
Thus, in order to return remaining time of all requests -- I guess this is the remaining time of other request's async task, because remaining time of other request's handler code execution does not make any sense, there must be some place to store the information of all requests, including:
request's connId and startTime (the time when request is received).
request's timeout value, which is passed as parameter in URL.
request's estimated remaining time, this information is mission specific and must be retrieved from other async task related services (you can pull time by time using setInterval or make other services push the latest remaining time). Node.js doesn't know the remaining time information of any async task.
In this way, you can track all running requests and their remaining time. Before one request is returned, you can check the above "some place" to calculate all requests' remaining time. This "some place" could be global variable, memory database such as Redis, or even a plain database such as MySQL.
Please note: the calculated remaining time would not be accurate, as the read&calculation itself would cost time and introduce error.
I have a question that nobody seems to help with. How will this be handled in a production mode with thousands of requests at the same time?
I did a simple test case:
module.exports = {
index: function (req, res){
if (req.param('foo') == 'bar'){
async.series([
function(callback){
for (k=0; k <= 50000; k++){
console.log('did something stupid a few times');
}
callback();
}
], function(){
return res.json(null);
});
}else{
return res.view('homepage', {
});
}
}
};
Now if I go to http://localhost:1337/?foo=bar it will obviously wait a while before it responds. So if I now open a different session (other browser or incognito, and go to http://localhost:1337/ I am expecting a result immediately. Instead it is waiting for the other request to finish and only then it will let this request go through.
Therefore it is not asynchronous and it is a huge problem if I have as much as 2 ppl at the same time operating this app. I mean this app will have drop downs coming from databases, html files being served etc...
My question is this: how does one handle such an issue??? I hear the word "promises vs callbacks" - is this some sort of a solution to this?
I know about clustering, but that only separates the requests over the amount of cpu's, ultimately you will fix it by at most allowing 8 people at the same time without being blocked. It won;t handle 100 requests at the same time...
P.S. That test was to simplify the example, but think of someone uploading a file, a web service that goes to a different server, a point of sales payment terminal waiting for a user to input the pin, someone downloading a file from the app, etc...
nodejs is event driven and runs your Javascript as single threaded. So, as long as your code from the first request is sitting in that for loop, nodejs can't do anything else and won't get to the next event in the event queue so thus your second request has to wait for the first one to finish.
Now, if you used a true async operation such as setTimeout() instead of your big for loop, then nodejs could service other events while the first request was waiting for the setTimeout().
The general rule in nodejs is to avoid doing anything that takes a ton of CPU in your main nodejs app. If you are stuck with something CPU-intensive, then you're best to either run clusters (as many as CPUs you have) or move the CPU-intensive work to some sort of worker queue that is served by different processes and let the OS time slice those other processes while the main nodejs process stays free and ready to service new incoming requests.
My question is this: how does one handle such an issue??? I hear the word "promises vs callbacks" - is this some sort of a solution to this?
I know about clustering, but that only separates the requests over the amount of cpu's, ultimately you will fix it by at most allowing 8 people at the same time without being blocked. It won;t handle 100 requests at the same time...
Most of the time, a server process spends most of the time of a request doing things that are asynchronous in nodejs (reading files, talking to other servers, doing database operations, etc...) where the actual work is done outside the nodejs process. When that is the case, nodejs does not block and is free to work on other requests while the async operations from other requests are underway. The little bit of CPU time coordinating these operations can be helped further by clustering though it's probably worth testing a single process first to see if clustering is really needed.
P.S. That test was to simplify the example, but think of someone uploading a file, a web service that goes to a different server, a point of sales payment terminal waiting for a user to input the pin, someone downloading a file from the app, etc...
All the operations you mentioned here can be done truly asynchronously so they won't block your nodejs app like your for loop does so basically the for loop isn't a good simulation of any of this. You need to use a real async operation to simulate it. Real async operations do their work outside of the main nodejs thread and then just post an event to the event queue when they are done, allowing nodejs to do other things while the async operations are doing their work. That's the key.
Yesterday I was giving some presentation on nodeJS.
Some one asked me following question:
As we know nodeJS is a single threaded server, several request is
coming to the server and it pushes all request to event loop. What if
two request is coming to server at exact same time, how server will
handle this situation?
I guessed a thought and replied back following:
I guess No two http request can come to the server at exact same
time, all request come through a single socket so they will be in queue. Http
request has following format:
timestamp of request is contained in it's header and they may be pushed to the event loop depending upon their timestamp in header.
but I'm not sure whether I gave him right or wrong answer.
I guess No two http request can come to the server at exact same time,
all request come through pipe so they will be in queue.
This part is basically correct. Incoming connections go into the event queue and one of them has to be placed in the queue first.
What if two request is coming two server at exact same time, how
server will handle this situation?
Since the server is listening for incoming TCP connections on a single socket in a single process, there cannot be two incoming connections at exactly the same time. One will be processed by the underlying operating system slightly before the other one. Think of it this way. An incoming connection is a set of packets over a network connection. One of the incoming connections will have its packets before the other one.
Even if you had multiple network cards and multiple network links so two incoming connections could literally arrive at the server at the exact same moment, the node.js queue will be guarded for concurrency by something like a mutex and one of the incoming connections will grab the mutex before the other and get put in the event queue before the other.
The one that is processed by the OS first will be put into the node.js event queue before the other one. When node.js is available to process the next item in the event queue, then whichever incoming request was first in the event queue will start processing first.
Because node.js JS execution is single threaded, the code processing that request will run its synchronous code to completion. If it has an async operation, then it will start that async operation and return. That will then allow the next item in the event queue to be processed and the code for the second request will start running. It will run its synchronous to completion. Like with the first request, if it has an async operation, then it will start that async operation and return.
At that point after both requests have started their async operations and then returned, then its just up to the event queue. When one of those async operations finishes, it will post another event to the event queue and when the single thread of node.js is free, it will again process the next item in the event queue. If both requests have lots of async operations, their progress could interleave and both could be "in-flight" at the same as they fire an async operation and then return until that async operation completes where their processing picks up again when node.js is free to process that next event.
timestamp of request is contained in it's header and they may be
pushed to the event loop depending upon their timestamp in header.
This part is not really right. Incoming events of the same type are added to the queue as they arrive. The first ones to arrive go into the queue first - there's isn't any step that examines some timestamp.
Multiple concurrent HTTP requests are not a problem. Node.js is asynchronous, and will handle multiple requests in the event-loop without having to wait for each one to finish.
For reference, example: https://stackoverflow.com/a/34857298/1157037