For over a year we've seen interesting patterns that don't always rear themselves but on occasion repeat and we've never been able to figure out why and I'm hoping someone can make sense of it. It may be our approach, it may be the environment (node 8.x & koa), it may be a number of things.
We make two async calls in parallel to our dependencies using the request-promise module.
Simplified code of a single api dependency:
const httpRequest = require("request-promise");
module.exports = function (url) {
const requestOptions = {
uri: ...,
json: true,
resolveWithFullResponse: true
}
return httpRequest(requestOptions)
.then(response => {
status = response.statusCode;
tmDiff = moment().diff(tmStart);
return createResponseObject({
status,
payload: response.body,
})
})
.catch(err => { .... };
});
};
Parallel calls:
const apiResponses = yield {
api1: foo(ctx, route),
api2: bar(ctx)
};
Yet we've seen situations in our response time charts where if 1 is slow, latency seems to follow the other separate service. It doesn't matter what services they are, the pattern has been noticed across > 5 services that may be called in parallel. Does anyone have any ideas what could be causing the supposed latency?
If the latency is caused by a temporarily slowed network connection, then it would be logical to expect both parallel requests to feel that same effect. ping or tracert during the slowdown might give you useful diagnostics to see if it's a general transport issue. If your node.js server (which runs Javascript single threaded) was momentarily busy doing something else with the CPU (serving another request, garbage collecting, etc...), then that would affect the apparent responsiveness of API calls just because it took a moment for node.js to get around to processing the waiting responses.
There are tools that monitor the responsiveness of your own http server on a continual basis (you can set whatever monitoring interval you want). If you have a CPU-hog somewhere, those tools would show a similar lag in responsiveness. There are also tools that monitor the health of your network connection which would also show a connectivity glitch. These are the types of monitoring tools that people whose job it is to maintain a healthy server farm might use. I don't have a name handy for either one, but you can presumably find such tools by searching.
Related
I recently coded a discord bot which can track the user engagement on a server once the user has opted in. I coded it primarily using the Discord.js framework It accepts events emitted from the Discord API when users of a server are active under certain conditions (message send, message reaction, etc.)
How would I be able to tell when, if ever, the single threaded nature of Node.js becomes the bottleneck for this application? Alternatively, how could I test the speed of my application response time to see if my actual code is the inefficient section? While the event-based nature of the application seems like a perfect fit for Node, I want to understand a bit more about the theory behind it and when it'd be a good idea to use another framework.
As you have said, nodejs scales up really well for this kind of application. So, as long as your code runs on a machine with a decent amount of CPU power it will be a long time until you hit a bottleneck. It will run fine with one or two vCPUs, but if your cloud provider throttles the cpu allocated to the virtual machine you're renting, your bot might not keep up with a heavy workload.
If your event handlers use a database, that is just as likely to become a bottleneck as discord.js. Nodejs itself can handle hundreds of concurrent event processors without breaking a sweat (as long as your vendor doesn't throttle your CPU).
How to know you're not keeping up with events? Try this. Each event is processed by a function looking something like this (your code style will certainly be different, this is just the general idea).
execute(client) {
try {
await someOperation(client)
await someOtherOperation(client)
}
catch (err) {
/* handle the error */
}
}
You can time these event handlers like so
execute(client) {
const startTime = Date.now()
try {
await someOperation(client)
await someOtherOperation(client)
}
catch (err) {
/* handle the error */
}
finally {
const elapsedTime = Date.now() - startTime
if (elapsedTime > 1000){ /*one second, you may want another value */
/* report it somehow */
}
}
This will keep track of event handlers taking more than a second.
If your bot does get busy enough to require refactoring, I suspect your best approach will be to shard your application; direct some clients to nodejs instance and others to another.
I'm using ioredis.
In order to prevent a huge buffer that can crash my app, I want to ignore requests when redis is down, and to catch those requests.
Is there any way to achieve that?
You may use circuit breaker design pattern.
Circuit breaker is a design pattern used in modern software development. It is used to detect failures and encapsulates the logic of preventing a failure from constantly recurring, during maintenance, temporary external system failure or unexpected system difficulties.
Generally Circuit Breaker can be used to check the availability of an external service. An external service can be a database server or a web service used by the application.
Martin fowler's blog post has a good explanation and a basic implementation about how to do it.
For other people finding this who want a bit more help. This intercepts commends to ioredis and just returns false if it's not in a ready state.
// Proxify commands to redis client.
// If redis is unavailable then return false.
// This stops everything dying if redis is down.
module.exports.redis = new Proxy(client, {
get(target, property, receiver) {
if (client.status !== 'ready') {
return (() => false);
}
return Reflect.get(target, property, receiver);
},
});
"Request sent" is taking nearly 3+ seconds.
client code:
JQ.ajax({
url: '/test',
type: 'POST',
contentType: 'application/json',
data: JSONdata,
success: function(response) {
console.log('done');
}
});
server code:
app.post('/test', function(req, res){
res.send(req.body)
});
dataLength is 865853 ( as seen from headers )
server is running Nginx behind which NodeJS server is running.
Want to understand why this is happening and ways to reduce it.
Its a post request. adding the headers image:
Request sent is taking too long. why would that be?
It could be anything.
Want to understand why this is happening and ways to reduce it.
There may be lots of reasons. Without seeing even a single line of your code it's impossible to tell you what's wrong with it. You may be using blocking functions, you may be calling external services, you may do too much work in the main process, you may be blocking the event loop, you may forget to handle errors, you may be doing a number of things wrong in your application and you may have misconfigured your reverse proxy, you may be using wrong ports, wrong hosts, the network may be slow, routers may be overloaded, packets may be lost, you can have memory leaks, swap trashing, CPU maxed out etc.
I'm quite new to nodejs and I'm doing some experiments.
What I get from them (and I hope I'm wrong!) is that nodejs couldn't serve many concurrent requests for the same resource without putting them in sequence.
Consider following code (I use Express framework in the following example):
var express = require('express');
var app = express();
app.get('/otherURL', function (req, res) {
res.send('otherURL!');
});
app.get('/slowfasturl', function (req, res)
{
var test = Math.round(Math.random());
if(test == "0")
{
var i;
setTimeout
(
function()
{
res.send('slow!');
}, 10000
);
}
else
{
res.send('fast!');
}
});
app.listen(3000, function () {
console.log('app listening on port 3000!');
});
The piece of code above exposes two endpoints:
http://127.0.0.1:3000/otherurl , that just reply with "otherURL!" simple text
http://127.0.0.1:3000/slowfasturl , that randomly follow one of the two behaviors below:
scenario 1 : reply immediately with "fast!" simple text
or
scenario 2 : reply after 10 seconds with "slow!" simple text
My test:
I've opened several windows of chrome calling at the same time the slowfasturl URL and I've noticed that the first request that falls in the "scenario 2", causes the blocking of all the other requests fired subsequentely (with other windows of chrome), indipendently of the fact that these ones are fallen into "scenario 1" (and so return "slow!") or "scenario 2" (and so return "fast!"). The requests are blocked at least until the first one (the one falling in the "scenario 2") is not completed.
How do you explain this behavior? Are all the requests made to the same resource served in sequence?
I experience a different behavior if while the request fallen in the "scenario 2" is waiting for the response, a second request is done to another resource (e.g. the otherurl URL explained above). In this case the second request is completed immediately without waiting for the first one
thank you
Davide
As far as I remember, the requests are blocked browser side.
Your browser is preventing those parallel requests but your server can process them. Try in different browsers or using curl and it should work.
The behavior you observe can only be explained through any sequencing which browser does. Node does not service requests in sequence, instead it works on an event driven model, leveraging the libuv framework
I have ran your test case with non-browser client, and confirmed that requests do not influence each other.
To gain further evidence, I suggest the following:
Isolate the problem scope. Remove express (http abstraction) and use either http (base http impl), or even net (TCP) module.
Use non-browser client. I suggest ab (if you are in Linux) - apache benchmarking tool, specifically for web server performance measurement.
I used
ab -t 60 -c 100 http://127.0.0.1:3000/slowfasturl
collect data for 60 seconds, for 100 concurrent clients.
Make it more deterministic by replacing Math.random with a counter, and toggling between a huge timeout and a small timeout.
Check result to see the rate and pattern of slow and fast responses.
Hope this helps.
Davide: This question needs an elaboration, so adding as another answer rather than comment, which has space constraints.
If you are hinting at the current node model as a problem:
Traditional languages (and runtimes) caused code to be run in sequence. Threads were used to scale this but has side effects such as:
i) shared data access need sequencing, ii) I/O operations block. Node is the result of a careful integration between three entities
libuv(multiplexer), v8 (executor), and node (orchestrator) to address those issues. This model ensures improved performance and scalability under web and cloud deployments. So there is no problem with this approach.
If you are hinting at further improvements to manage stressful CPU bound operations in node where there will be waiting period yes, leveraging the multi-core and introducing more threads to share the CPU intensive workload would be the right way.
Hope this helps.
This is the first time I've used Node.js and Mongo, so please excuse any ignorance. I come from a PHP background. It was my understanding that Node.js scaled well because of the event-driven nature of it. As such, I built my API in node and have been testing it on a localhost. Today, I deployed it to my cloud server and everything works great, except...
As the requests start to pile up, they start to take a long time to fulfill. With just 2 clients connecting to the API, already I'm seeing 30sec+ page load times when both clients are trying to make several requests at once (which does sometimes happen).
Most of the work done by the API is either (a) reading/writing to MongoDB, which resides on a 2nd server on the cloud (b) making requests to other APIs, websites, etc. and returning the results. Both of these operations should not be blocking, but I can imagine the problem being something to do with a bottleneck either on the Mongo DB server (a) or to the external APIs (b).
Of course, I will have multiple application servers in the end, but I would expect each one to handle more than a couple concurrent clients without choking.
Some considerations:
1) I have some console.logs that I left in my node code, and I have a SSH client open to monitor the cloud server. I suspect that this could cause slowdown
2) I use express, mongoose, Q, request, and a handful of other modules
Thanks for taking the time to help a node newb ;)
Edit: added some pics of performance graphs after some responses below...
EDIT: here's a typical callback -- it is called by the express router, and it uses the Q module and OAuth to make a Post API call to Facebook:
post: function(req, links, images, callback)
{
// removed some code that calculates the target (string) and params (obj) variables
// the this.request function is a simple wrapper around the oauth.getProtectedResource function
Q.ncall(this.request, this, target, 'POST', params)
.then(function(res){
callback(null, res);
})
.fail(callback).end();
},
EDIT: some "upsert" code
upsert: function(query, callback)
{
var id = this.data._id,
upsertData = this.data.toObject(),
query = query || {'_id': id};
delete upsertData._id;
this.model.update(query, upsertData, {'upsert': true}, function(err, res, out){
if(err)
{
if(callback) callback(new Errors.Database({'message':'the data could not be upserted','error':err, 'search': query}));
return;
}
if(callback) callback(null);
});
},
Admittedly, my knowledge of Q/promises is weak. But, I think I have consistently implemented them in a way that does not block...
Your question has provided half of the relevant data: the technology stack. However, when debugging performance issues, you also need the other half of the data: performance metrics.
You're running some "cloud servers", but it's not clear what these servers are actually doing. Are they spiked on CPU? on Memory? on IO?
There are lots of potential issues. Are you running Express in production mode? Are you taking up too much IO on your MongoDB server? Are you legitimately downloading too much data? Did you get caught in an infinite Node.JS loop? (it happens)
I would like to provide better advice, but without knowing the status of the servers involved it's really impossible to start picking at any specific underlying technology. You may be a "Node newb", but basic server monitoring is pretty standard across programming languages.
Thank you for the extra details, I will re-iterate the most important part of my comments above: Where are these servers blocked?
CPU? (clearly not from your graph)
Memory? (doesn't seem likely here)
IO? (where are the IO graphs, what is your DB server doing?)