I'm quite new to nodejs and I'm doing some experiments.
What I get from them (and I hope I'm wrong!) is that nodejs couldn't serve many concurrent requests for the same resource without putting them in sequence.
Consider following code (I use Express framework in the following example):
var express = require('express');
var app = express();
app.get('/otherURL', function (req, res) {
res.send('otherURL!');
});
app.get('/slowfasturl', function (req, res)
{
var test = Math.round(Math.random());
if(test == "0")
{
var i;
setTimeout
(
function()
{
res.send('slow!');
}, 10000
);
}
else
{
res.send('fast!');
}
});
app.listen(3000, function () {
console.log('app listening on port 3000!');
});
The piece of code above exposes two endpoints:
http://127.0.0.1:3000/otherurl , that just reply with "otherURL!" simple text
http://127.0.0.1:3000/slowfasturl , that randomly follow one of the two behaviors below:
scenario 1 : reply immediately with "fast!" simple text
or
scenario 2 : reply after 10 seconds with "slow!" simple text
My test:
I've opened several windows of chrome calling at the same time the slowfasturl URL and I've noticed that the first request that falls in the "scenario 2", causes the blocking of all the other requests fired subsequentely (with other windows of chrome), indipendently of the fact that these ones are fallen into "scenario 1" (and so return "slow!") or "scenario 2" (and so return "fast!"). The requests are blocked at least until the first one (the one falling in the "scenario 2") is not completed.
How do you explain this behavior? Are all the requests made to the same resource served in sequence?
I experience a different behavior if while the request fallen in the "scenario 2" is waiting for the response, a second request is done to another resource (e.g. the otherurl URL explained above). In this case the second request is completed immediately without waiting for the first one
thank you
Davide
As far as I remember, the requests are blocked browser side.
Your browser is preventing those parallel requests but your server can process them. Try in different browsers or using curl and it should work.
The behavior you observe can only be explained through any sequencing which browser does. Node does not service requests in sequence, instead it works on an event driven model, leveraging the libuv framework
I have ran your test case with non-browser client, and confirmed that requests do not influence each other.
To gain further evidence, I suggest the following:
Isolate the problem scope. Remove express (http abstraction) and use either http (base http impl), or even net (TCP) module.
Use non-browser client. I suggest ab (if you are in Linux) - apache benchmarking tool, specifically for web server performance measurement.
I used
ab -t 60 -c 100 http://127.0.0.1:3000/slowfasturl
collect data for 60 seconds, for 100 concurrent clients.
Make it more deterministic by replacing Math.random with a counter, and toggling between a huge timeout and a small timeout.
Check result to see the rate and pattern of slow and fast responses.
Hope this helps.
Davide: This question needs an elaboration, so adding as another answer rather than comment, which has space constraints.
If you are hinting at the current node model as a problem:
Traditional languages (and runtimes) caused code to be run in sequence. Threads were used to scale this but has side effects such as:
i) shared data access need sequencing, ii) I/O operations block. Node is the result of a careful integration between three entities
libuv(multiplexer), v8 (executor), and node (orchestrator) to address those issues. This model ensures improved performance and scalability under web and cloud deployments. So there is no problem with this approach.
If you are hinting at further improvements to manage stressful CPU bound operations in node where there will be waiting period yes, leveraging the multi-core and introducing more threads to share the CPU intensive workload would be the right way.
Hope this helps.
Related
For over a year we've seen interesting patterns that don't always rear themselves but on occasion repeat and we've never been able to figure out why and I'm hoping someone can make sense of it. It may be our approach, it may be the environment (node 8.x & koa), it may be a number of things.
We make two async calls in parallel to our dependencies using the request-promise module.
Simplified code of a single api dependency:
const httpRequest = require("request-promise");
module.exports = function (url) {
const requestOptions = {
uri: ...,
json: true,
resolveWithFullResponse: true
}
return httpRequest(requestOptions)
.then(response => {
status = response.statusCode;
tmDiff = moment().diff(tmStart);
return createResponseObject({
status,
payload: response.body,
})
})
.catch(err => { .... };
});
};
Parallel calls:
const apiResponses = yield {
api1: foo(ctx, route),
api2: bar(ctx)
};
Yet we've seen situations in our response time charts where if 1 is slow, latency seems to follow the other separate service. It doesn't matter what services they are, the pattern has been noticed across > 5 services that may be called in parallel. Does anyone have any ideas what could be causing the supposed latency?
If the latency is caused by a temporarily slowed network connection, then it would be logical to expect both parallel requests to feel that same effect. ping or tracert during the slowdown might give you useful diagnostics to see if it's a general transport issue. If your node.js server (which runs Javascript single threaded) was momentarily busy doing something else with the CPU (serving another request, garbage collecting, etc...), then that would affect the apparent responsiveness of API calls just because it took a moment for node.js to get around to processing the waiting responses.
There are tools that monitor the responsiveness of your own http server on a continual basis (you can set whatever monitoring interval you want). If you have a CPU-hog somewhere, those tools would show a similar lag in responsiveness. There are also tools that monitor the health of your network connection which would also show a connectivity glitch. These are the types of monitoring tools that people whose job it is to maintain a healthy server farm might use. I don't have a name handy for either one, but you can presumably find such tools by searching.
I have a web app that accepts api requests from an ios app. My web app is hosted on Heroku using their free dyno which is able to process 512 mb of data per request. Because node is a single threaded application this will be a problem once we start getting higher levels of traffic from the ios end to the web server. I'm also not the richest person in the world so i'm wondering if it would be smart to create another free heroku app and use a round robin approach to balance the load received from the ios app?
I just need to be pointed into the right direction. Vertical scaling is not really an option financially.
I'm the Node.js platform owner at Heroku.
You may be doing some premature optimization. Node.js, on our smallest 1X size (512MB RAM), can handle hundreds of simultaneous connections and thousands of requests per minute.
If your iOS app is consistently maxing that out, it may be time to consider monetization!
As mentioned by Daniel it's against Heroku rules. Having said that there are probably other services that would allow you to do that.
One way to approach this problem is to use cluster module with ZeroMQ (you need to have ZeroMQ installed before using the module - see module description).
var cluster = require('cluster');
var zmq = require('zmq');
var ROUTER_SOCKET = 'tcp://127.0.0.1:5555';
var DEALER_SOCKET = 'tcp://127.0.0.1:7777';
if (cluster.isMaster) {
// this is the main process - create Router and Dealer sockets
var router = zmq.socket('router').bind(ROUTER_SOCKET);
var dealer = zmq.socket('dealer').bind(DEALER_SOCKET);
// forward messages between router and dealer
router.on('message', function() {
var frames = Array.prototype.slice.cal(arguments);
dealer.send(frames);
});
dealer.on('message', function() {
var frames = Array.prototype.slice.cal(arguments);
router.send(frames);
});
// listen for workers processes to come online
cluster.on('online', function() {
// do something with a new worker, maybe keep an array of workers
});
// fork worker processes
for (var i = 0, i < 100; i++) {
cluster.fork();
}
} else {
// worker process - connect to Dealer
let responder = zmq.socket('rep').connect(DEALER_SOCKET);
responder.on('message', function(data) {
// do something with incomming data
})
}
This is just to point you in the right direction. If you think about it you can create a script with a parameter that will tell it if it's a master or a worker process. Then on the main server run it as is, and on additional servers run it using worker flag which will force it to connect to the main dealer.
Now your main app needs to send the requests to the router, which will be later forwarded to the worker processes:
var zmq = require('zmq');
var requester = zmq.socket('req');
var ROUTER_SOCKET = 'tcp://127.0.0.1:5555';
// handle replies - for example completion status from the worker processes
requester.on('message', function(data) {
// do something with the replay
});
requester.connect(ROUTER_SOCKET);
// send requests to the router
requester.send({
// some object describing the task
});
So first off, as the other replies have pointed out, running two copies of your app to avoid Heroku's limits violates their ToS, which may not be a great idea.
There is, however, some good news. For starters (from Heroku's docs):
The dyno manager will restart your dyno and log an R15 error if the memory usage of a:
free, hobby or standard-1x dyno reaches 2.5GB, five times its quota.
As I understand it, despite the fact that your dyno has 512mb of actual RAM, it'll swap out to 5x that before it actually restarts. So you can go beyond 512mb (as long as you're willing to pay the performance penalty for swapping to disk, which can be severe).
Further to that, Heroku bills by the second and allows you to scale your dyno formation up and down as needed. This is fairly easy to do within your own app by hitting the Heroku API – I see that you've tagged this with NodeJS so you might want to check out:
Heroku's node client
the very-barebones-but-still-functional toots/node-heroku module
Both of these modules allow you to scale up and down your formation of dynos — with a simple heuristic (say, always have a spare 1X dyno running), you could add capacity while you're processing a request, and get rid of the spare capacity when api requests aren't running. Given that you're billed by the second, this can end up being very inexpensive; 1X dynos work out to something like 5¢ an hour to run. If you end up running extra dynos for even a few hours a day, it's a very, very small cost to you.
Finally: there are also 3rd party services such as Adept and Hirefire (two random examples from Google, I'm sure there are more) that allow you to automate this to some degree, but I don't have any experience with them.
You certainly could, I mean, programatically - but that would bypass Heroku's TOS:
4.4 You may not develop multiple Applications to simulate or act as a single Application or otherwise access the Heroku Services in a manner intended to avoid incurring fees.
Now, I'm not sure about this:
Because node is a single threaded application this will be a problem once we start getting higher levels of traffic from the ios end to the web server.
There are some threads discussing that, with some interesting answers:
Clustering Node JS in Heavy Traffic Production Environment
How to decide when to use Node.js?
Also, they link to this video, introducing Node.js, which talks a bit about benchmarks:
Introduction of Node JS by Ryan Dahl
In an app that I was working, I encountered "headers sent already error" if I test using concurrency and parallel request methods.
ultimately I resolved the problem using !response.headersSent but my question is why am I forced to use it? is node caching similar requests and reuses them for the next repeated call.
if(request.headers.accept == "application/json") {
if(!response.headersSent) {response.writeHead(200, {'Content-Type': 'application/json'})}
response.end(JSON.stringify({result:{authToken:data.authToken}}));
}
Edit
var express = require('express');
var app = express();
var server = app.listen(process.env.PORT || 3000, function () {
console.log('Example app listening at http://%s:%s', server.address().address, server.address().port);
});
Edit 2:
Another problem is while testing using mocha, super agent and while the tests in progress if I just send another request through postman on the side, one of the tests in mocha end with a timeout error. These steps I'm taking to ensure the code is production ready for simultaneous, parallel requests? please advise on what measures I can take to ensure node/code works under stress.
Edit 3:
app.use(function(request, response, next){
request.id = Math.random();
next();
});
OK, in an attempt to capture what solved this for you via all our conversation in comments, I will attempt to summarize here:
The message "headers sent already error" is nearly always caused by improper async handling which causes the code to call methods on the response object in a wrong sequence. The most common case is non-async code that ends the request and then an async operation that ends some time later that then tries to use the request (but there are other ways to misuse it too).
Each request and response object is uniquely created at the time each individual HTTP request arrives at the node/express server. They are not cached or reused.
Because of asynchronous operations in the processing of a request, there may be more than one request/response object in use at any given time. Code that is processing these must not store these objects in any sort of single global variable because multiple ones can be in the state of processing at once. Because node is single threaded, code will only be running on any given request at any given moment, but as soon as that code hits an async operation (and thus has nothing to do until the async operation is done), another request could start running. So multiple requests can easily be "in flight" at the same time.
If you have a system where you need to keep track of multiple requests at once, you can coin a request id and attach it to each new request. One way to do that is with a few lines of express middleware that is early in the middleware stack that just adds a unique id property to each new request.
One simple way of coining a unique id is to just use a monotonically increasing counter.
I'm writing proxy in Node.js + Express 2. Proxy should:
decrypt POST payload and issue HTTP request to server based on result;
encrypt reply from server and send it back to client.
Encryption-related part works fine. The problem I'm facing is timeouts. Proxy should process requests in less than 15 secs. And most of them are under 500ms, actually.
Problem appears when I increase number of parallel requests. Most requests are completed ok, but some are failed after 15 secs + couple of millis. ab -n5000 -c300 works fine, but with concurrency of 500 it fails for some requests with timeout.
I could only speculate, but it seems thant problem is an order of callbacks exectuion. Is it possible that requests that comes first are hanging until ETIMEDOUT because of node's focus in latest ones which are still being processed in time under 500ms.
P.S.: There is no problem with remote server. I'm using request for interactions with it.
upd
The way things works with some code:
function queryRemote(req, res) {
var options = {}; // built based on req object (URI, body, authorization, etc.)
request(options, function(err, httpResponse, body) {
return err ? send500(req, res)
: res.end(encrypt(body));
});
}
app.use(myBodyParser); // reads hex string in payload
// and calls next() on 'end' event
app.post('/', [checkHeaders, // check Content-Type and Authorization headers
authUser, // query DB and call next()
parseRequest], // decrypt payload, parse JSON, call next()
function(req, res) {
req.socket.setTimeout(TIMEOUT);
queryRemote(req, res);
});
My problem is following: when ab issuing, let's say, 20 POSTs to /, express route handler gets called like thousands of times. That's not always happening, sometimes 20 and only 20 requests are processed in timely fashion.
Of course, ab is not a problem. I'm 100% sure that only 20 requests sent by ab. But route handler gets called multiple times.
I can't find reasons for such behaviour, any advice?
Timeouts were caused by using http.globalAgent which by default can process up to 5 concurrent requests to one host:port (which isn't enough in my case).
Thouthands of requests (instead of tens) were sent by ab (Wireshark approved fact under OS X; I can not reproduce this under Ubuntu inside Parallels).
You can have a look at node-http-proxy module and how it handles the connections. Make sure you don't buffer any data and everything works by streaming. And you should try to see where is the time spent for those long requests. Try instrumenting parts of your code with conosle.time and console.timeEnd and see where is taking the most time. If the time is mostly spent in javascript you should try to profile it. Basically you can use v8 profiler, by adding --prof option to your node command. Which makes a v8.log and can be processed via a v8 tool found in node-source-dir/deps/v8/tools. It only works if you have installed d8 shell via scons(scons d8). You can have a look at this article to help you further to make this working.
You can also use node-webkit-agent which uses webkit developer tools to show the profiler result. You can also have a look at my fork with a bit of sugar.
If that didn't work, you can try profiling with dtrace(only works in illumos-based systems like SmartOS).
This is the first time I've used Node.js and Mongo, so please excuse any ignorance. I come from a PHP background. It was my understanding that Node.js scaled well because of the event-driven nature of it. As such, I built my API in node and have been testing it on a localhost. Today, I deployed it to my cloud server and everything works great, except...
As the requests start to pile up, they start to take a long time to fulfill. With just 2 clients connecting to the API, already I'm seeing 30sec+ page load times when both clients are trying to make several requests at once (which does sometimes happen).
Most of the work done by the API is either (a) reading/writing to MongoDB, which resides on a 2nd server on the cloud (b) making requests to other APIs, websites, etc. and returning the results. Both of these operations should not be blocking, but I can imagine the problem being something to do with a bottleneck either on the Mongo DB server (a) or to the external APIs (b).
Of course, I will have multiple application servers in the end, but I would expect each one to handle more than a couple concurrent clients without choking.
Some considerations:
1) I have some console.logs that I left in my node code, and I have a SSH client open to monitor the cloud server. I suspect that this could cause slowdown
2) I use express, mongoose, Q, request, and a handful of other modules
Thanks for taking the time to help a node newb ;)
Edit: added some pics of performance graphs after some responses below...
EDIT: here's a typical callback -- it is called by the express router, and it uses the Q module and OAuth to make a Post API call to Facebook:
post: function(req, links, images, callback)
{
// removed some code that calculates the target (string) and params (obj) variables
// the this.request function is a simple wrapper around the oauth.getProtectedResource function
Q.ncall(this.request, this, target, 'POST', params)
.then(function(res){
callback(null, res);
})
.fail(callback).end();
},
EDIT: some "upsert" code
upsert: function(query, callback)
{
var id = this.data._id,
upsertData = this.data.toObject(),
query = query || {'_id': id};
delete upsertData._id;
this.model.update(query, upsertData, {'upsert': true}, function(err, res, out){
if(err)
{
if(callback) callback(new Errors.Database({'message':'the data could not be upserted','error':err, 'search': query}));
return;
}
if(callback) callback(null);
});
},
Admittedly, my knowledge of Q/promises is weak. But, I think I have consistently implemented them in a way that does not block...
Your question has provided half of the relevant data: the technology stack. However, when debugging performance issues, you also need the other half of the data: performance metrics.
You're running some "cloud servers", but it's not clear what these servers are actually doing. Are they spiked on CPU? on Memory? on IO?
There are lots of potential issues. Are you running Express in production mode? Are you taking up too much IO on your MongoDB server? Are you legitimately downloading too much data? Did you get caught in an infinite Node.JS loop? (it happens)
I would like to provide better advice, but without knowing the status of the servers involved it's really impossible to start picking at any specific underlying technology. You may be a "Node newb", but basic server monitoring is pretty standard across programming languages.
Thank you for the extra details, I will re-iterate the most important part of my comments above: Where are these servers blocked?
CPU? (clearly not from your graph)
Memory? (doesn't seem likely here)
IO? (where are the IO graphs, what is your DB server doing?)