I'm running a clustered node app, with 8 worker processes. I'm giving output when serving requests, and the output includes the ID of the process which handled the request:
app.get('/some-url', function(req, res) {
console.log('Request being handled by process #' + process.pid);
res.status(200).text('yayyy');
});
When I furiously refresh /some-url, I see in the output that the same process is handling the request every time.
I used node load-test to query my app. Again, even with 8 workers available, only one of them handles every single request. This is obviously undesirable as I wish to load-test the clustered app to see the overall performance of all processes working together.
Here's how I'm initializing the app:
var cluster = require('cluster');
if (cluster.isMaster) {
for (var i = 0; i < 8; i++) cluster.fork();
} else {
var app = require('express')();
// ... do all setup on `app`...
var server = require('http').createServer(app);
server.listen(8000);
}
How do I get all my workers working?
Your request does not use any ressources. I suspect that the same worker is always called, because it just finishes to handle the request before the next one comes in.
What happens if you do some calculation inside that takes more time than the time needed to handle a request ? As it stands, the worker is never busy between accepting a request and answering it.
Related
I have a single-threaded, web-based, CPU intensive workload implemented as a Node.js server (express) and being deployed on Kubernetes with no CPU requests/limits (best effort). This workload, on average, takes ~700-800ms to execute on a quad-core physical machine. The server is behind an Nginx load-balancer (all with default configuration). In short, the workload is simply as follows:
for (let $i = 0; $i < 100; $i++) {
const prime_length = 100;
const diffHell = crypto.createDiffieHellman(prime_length);
const key = diffHell.generateKeys('base64');
checksum(key);
}
I have an event handler in my Express app.js that logs a timestamp in console when it recieves or send a request, as follows:
app.use((req, res, next) => {
const start = process.hrtime();
console.log(`Received ${req.method} ${req.originalUrl} from ${req.headers['referer']} at {${now()}} [RECEIVED]`)
res.on('close', () => {
const durationInMilliseconds = helper.getDurationInMilliseconds(start);
console.log(`Closed received ${req.method} ${req.originalUrl} from ${req.headers['referer']} {${now()}} [CLOSED] ${durationInMilliseconds.toLocaleString()} ms`)
});
next();
})
I'm sending 3 parallel requests to this service from 3 different physical machines at the same time. All these servers, plus all kubernetes nodes has an NTP enabled and their local times are synced together.
To run the traffic, I ssh into all these 3 servers in separate screen (using linux's screen), prepare the curl command in the command line, and then send and Enter using the following command to run the traffic all at the same time:
screen -S 12818.3 -X stuff "
" & screen -S 12783.2 -X stuff "
" & screen -S 12713.1 -X stuff "
"
From the logs, I can see all 3 requests is being sent at the same time: at 17:26:37.888
Interestingly, the server receives each request right after finishing the next one:
Request 1 is received at 17:26:37.922040382 and takes 740.128ms to
process
Request 2 is received at 17:26:38.663390107 and takes 724.524ms to
process
Request 3 is received at 17:26:39.388508923 and takes 695.894ms to
process
Here is the generated logs in the container (extracted using kubectl logs -l name=s1 --tail=9999999999 --timestamps):
2020-10-11T17:26:37.922040382Z Received GET /s1/cpu/100 from undefined at {1602429997921393902} [RECEIVED]
2020-10-11T17:26:38.662193765Z Closed received GET /s1/cpu/100 from undefined {1602429998661523611} [CLOSED] 740.128 ms
2020-10-11T17:26:38.663390107Z Received GET /s1/cpu/100 from undefined at {1602429998662810195} [RECEIVED]
2020-10-11T17:26:39.387987847Z Closed received GET /s1/cpu/100 from undefined {1602429999387339320} [CLOSED] 724.524 ms
2020-10-11T17:26:39.388508923Z Received GET /s1/cpu/100 from undefined at {1602429999387912718} [RECEIVED]
2020-10-11T17:26:40.084479697Z Closed received GET /s1/cpu/100 from undefined {1602430000083806321} [CLOSED] 695.894 ms
I checked the CPU usage, using both htop and pidstat, and strangely only 1 core is being utilized all the time...
I was expecting that the node.js server receives all requests at the same time and handles them in different threads (by generating new threads), but it seems it's not the case. How can I make node.js handle requests in parallel, and utilize all the cores it has?
Here is my full code:
var express = require('express');
const crypto = require('crypto');
const now = require('nano-time');
var app = express();
function checksum(str, algorithm, encoding) {
return crypto
.createHash(algorithm || 'md5')
.update(str, 'utf8')
.digest(encoding || 'hex');
}
function getDurationInMilliseconds (start) {
const NS_PER_SEC = 1e9;
const NS_TO_MS = 1e6;
const diff = process.hrtime(start);
return (diff[0] * NS_PER_SEC + diff[1]) / NS_TO_MS;
}
app.use((req, res, next) => {
const start = process.hrtime();
console.log(`Received ${req.method} ${req.originalUrl} from ${req.headers['referer']} at {${now()}} [RECEIVED]`)
res.on('close', () => {
const durationInMilliseconds = getDurationInMilliseconds(start);
console.log(`Closed received ${req.method} ${req.originalUrl} from ${req.headers['referer']} {${now()}} [CLOSED] ${durationInMilliseconds.toLocaleString()} ms`)
});
next();
})
app.all('*/cpu', (req, res) => {
for (let $i = 0; $i < 100; $i++) {
const prime_length = 100;
const diffHell = crypto.createDiffieHellman(prime_length);
const key = diffHell.generateKeys('base64');
checksum(key);
}
res.send("Executed 100 Diffie-Hellman checksums in 1 thread(s)!");
});
module.exports = app;
app.listen(9121)
I was expecting that the node.js server receives all requests at the same time and handles them in different threads (by generating new threads), but it seems it's not the case. How can I make node.js handle requests in parallel, and utilize all the cores it has?
By design, node.js runs your Javascript only in a single thread. It my use other threads for certain things such as built-in crypto operations that have an asynchronous calling interface or disk I/O, but anything you've written in pure Javascript, it will only run in a single thread. For CPU-intensive Javascript code, you will need to specifically change the design of your code in order to use multiple CPUs for processing that CPU intensive code.
Your options are to use child processes or WorkerThreads. Probably what you want to do is to set up a set of worker threads (probably one per CPU core) and then create a queue for jobs that should be processed by a worker thread. Then, as a job is inserted into the queue, you see if there is an available worker thread and if so, you immediately send the job off to the worker thread. If not, you wait until a worker thread notifies you that it is finished and is available for the next job.
In node.js WorkerThreads are entirely separate instances of the V8 Javascript engine and you can use messages between worker threads and your main process.
I used the cluster module of node.js and followed the approach mentioned here: https://medium.com/tech-tajawal/clustering-in-nodejs-utilizing-multiple-processor-cores-75d78aeb0f4f
I am trying to run a Node.js cluster within my Express app, but only for one specific function.
My app is a standard Express app generated with the express app generator.
My app initially scrapes an eCommerce website to get a list of categories in an array. I want to be able to then scrape each category's products, concurrently, using child processes.
I do not want to have the whole Express app inside the child processes. When the app starts up I want only one process to scrape for the initial categories. Once that is done I only want the function that scrapes the products to be run concurrently in the cluster.
I have tried the following:
delegation-controller.js
var {em} = require('./entry-controller');
const cluster = require('cluster');
const numCPUs = require('os').cpus().length;
class DelegationController {
links = [];
constructor() {
em.on('PageLinks', links => {
this.links = links;
this.startCategoryCrawl();
});
}
startCategoryCrawl() {
if (cluster.isMaster) {
console.log(`Master ${process.pid} is running`);
for (let i = 0; i < numCPUs; i++) {
cluster.fork();
}
cluster.on('exit', (worker, code, signal) => {
console.log(`worker ${worker.process.pid} died`);
});
} else {
console.log(`Worker ${process.pid} started`);
process.exit();
}
}
}
module.exports = DelegationController;
But then I got an error:
/ecommerce-scraper/bin/www:58
throw error;
^
Error: bind EADDRINUSE null:3000
Which I am guessing is because it is trying to start the express server again, but it is in use.
Am I able to do what I am trying to do, or am I misunderstanding how Node.js clusters work?
I believe this is not the case where you make use of cluster module. Instead you need the child_process module. This module lets you create a separate process. Here is the documentation.
I typically create my own Worker bootstrap that sits on top of my application. For things that need to run once, I have a convenient runonce function that is given a name and callback. The function checks the primary process for an open (non-busy process) and sends back the PID. If the PID matches (because all processes will claim ownership) the callback executes. If not, the function returns.
Example:
https://gist.github.com/jonshipman/abe627c687a46e7f5ea4b36bb919666c
NodeJS clustering creates identical copies of your application (through the cluster.fork(). It's up to your application to ensure that multiple actions aren't run twice (when they aren't expected to).
I believe, when using Express or https.createServer, it's setup in a way so that it doesn't listen to the same port multiple times. Instead each the prime process will distribute the load internally.
I'd like to fork a long running express request in node and send an express response with the child, allowing the parent to serve other requests. I'm already using cluster but I'd like to fork another process in addition to the cluster for specific long running requests. What I'd like to prevent is all the processes in the cluster being consumed by a specific long running processes, while most of the other requests are fast.
Thanks
var express = require('express');
var webserver = express();
webserver.get("/test", function(request, response) {
// long running HTTP request
response.send(...);
});
What I'm thinking of is something like following, although I'm not sure this works:
var cp = require('child_process');
var express = require('express');
var webserver = express();
webserver.get("/test", function(request, response) {
var child = cp.fork('do_nothing.js');
child.on("message", function(message) {
if(message == "start") {
response.send(...);
process.exit();
}
});
child.send("start");
});
Let me know if anyone knows how to do this.
Edit: So, the idea is that the child could take a long time. There are a limited number of processes in the cluster serving express responses and I don't want to consume them all on a specific long-running request type. In the code below, the entire cluster would be consumed by the long running express requests.
while(1) {
if(rand() % 100 == 0) {
if(fork() == 0) {
sleep(hour(1));
exit(0);
}
} else {
sleep(second(1));
}
waitpid(WAIT_ANY, &status, WNOHANG);
}
Edit: I am going to mark the self-answer as solved. I'm sure there's a way to pass a socket to a child but it's not really necessary because the cluster master can manage all child processes. Thanks for your help.
Your second code block is confusing because it appears that you're killing the parent process with process.exit() rather than the child.
In any case, if we assume the problem is this:
You have a cluster of "regular processes".
Occasionally, you want to take an incoming request that was assigned to one of the cluster processes and pass it off to a long running child that will eventually send the response.
After sending the response, the long running child process should exit.
You have a couple options.
You can have the clustered process that was assigned the request, start up a child, send it some initial data and listen for a message back from the child. When it gets the message back from the child, it can send the response and kill the child. This appears to be what you're attempting to do in your second code block.
You can have the clustered process that was assigned the request, start up a child and reassign the request socket to the child process and the child can then own that socket from then on. When it finally sends the response, it can then exit itself.
The first is simpler because no socket assignment from one process to another is required. To implement the second, you'd have to write or find the code to do socket reassignment and then reconstituted as an express request within the child. The cluster module does something like this so the code is there to be found and learned from, but I'm not aware of a trivial way to do it.
Personally, I don't see any particular downside to the first. I suppose if the clustered process were to die for some , you'd lose the long running request socket, but hopefully you can just code your clustered processes not to die unnecessarily.
You can read this article on sending a socket to a new node.js process:
Sending a socket to a forked process
And, this node.js doc on sending a socket:
Example: sending a socket object
So, I've verified that this is not necessary for my use case, but I was able to get it working using the code below. It's not exactly what the OP asks for, but it works.
What it's doing is sending an instruction to the cluster master, which forks the additional process upon receipt of the slow express request.
Since the express request doesn't need to know the status of the newly forked cluster worker, it just handles the slow request as normal and then exits.
The instruction to the cluster master informs the master not to replace the dying slow express request process, so the number of workers reverts to the original number after the slow request finishes.
The pool will increase in size when there are slow requests, but revert to normal. This will prevent like 20 simultaneous slow requests from bringing down the cluster.
var numberOfWorkers = 10;
var workerCount = 0;
var slowRequestPids = { };
if (cluster.isMaster) {
for(var i = 0; i < numberOfWorkers; i++) {
workerCount++;
cluster.fork();
}
cluster.on('exit', function(worker) {
workerCount--;
var pidString = String(worker.process.pid);
if(pidString in slowRequestPids) {
delete slowRequestPids[pidString];
if(workerCount >= numberOfWorkers) {
logger.info('not forking replacement for slow process');
return;
}
}
logger.info('forking replacement for a process that died unexpectedly');
workerCount++;
cluster.fork();
}
cluster.on("message", function(msg) {
if(typeof msg.fork != "undefined" && workerCount < 100) {
logger.info("forking additional process upon slow request");
slowRequestPids[msg.fork] = 1;
workerCount++;
cluster.fork();
}
});
return;
}
webserver.use("/slow", function(req, res) {
process.send({fork: String(process.pid) });
sleep.sleep(300);
res.send({ response_from: "virtual child" });
res.on("finish", function() {
logger.info('process exits, restoring cluster to original size');
process.exit();
});
});
we're diving deeper in Node.js architecture, to achieve fully understanding, how to scale our application.
Clear solution is cluster usage https://nodejs.org/api/cluster.html. Everything seems to be fine, apart of workers management description:
Node.js does not automatically manage the number of workers for you, however. It is your responsibility to manage the worker pool for your application's needs.
I was searching, how to really manage the workers, but most solutions, says:
Start so many workers as you've got cores.
But I would like to dynamically scale up or down my workers count, depending on current load on server. So if there is load on server and queue is getting longer, I would like to start next worker. In another way, when there isn't so much load, I would like to shut down workers (and leave f.e. minimum 2 of them).
The ideal place, will be for me Master Process queue, and event when new Request is coming to Master Process. On this place we can decide if we need next worker.
Do you have any solution or experience with managing workers from Master Thread in Cluster? Starting and killing them dynamically?
Regards,
Radek
following code will help you to understand to create cluster on request basis.
this program will genrate new cluster in every 10 request.
Note: you need to open http://localhost:8000/ and refresh the page for increasing request.
var cluster = require('cluster');
var http = require('http');
var numCPUs = require('os').cpus().length;
var numReqs = 0;
var initialRequest = 10;
var maxcluster = 10;
var totalcluster = 2;
if (cluster.isMaster) {
// Fork workers.
for (var i = 0; i < 2; i++) {
var worker = cluster.fork();
console.log('cluster master');
worker.on('message', function(msg) {
if (msg.cmd && msg.cmd == 'notifyRequest') {
numReqs++;
}
});
}
setInterval(function() {
console.log("numReqs =", numReqs);
isNeedWorker(numReqs) && cluster.fork();
}, 1000);
} else {
console.log('cluster one initilize');
// Worker processes have a http server.
http.Server(function(req, res) {
res.writeHead(200);
res.end("hello world\n");
// Send message to master process
process.send({ cmd: 'notifyRequest' });
}).listen(8000);
}
function isNeedWorker(numReqs) {
if( numReqs >= initialRequest && totalcluster < numCPUs ) {
initialRequest = initialRequest + 10;
totalcluster = totalcluster + 1;
return true;
} else {
return false;
}
}
To manually manage your workers, you need a messaging layer to facilitate inter process communication. With IPC master and worker can communicate effectively, by default and architecture stand point this behavior is already implemented in the process module native. However i find the native implementation not flexible or robust enough to handle horizontal scaling due to network requests.
One obvious solution Redis as a message broker to facilitate this method of master and slave communication. However this solution also as its faults , which is context latency, directly linked to command and reply.
Further research led me to RabbitMQ,great fit for distributing time-consuming tasks among multiple workers.The main idea behind Work Queues (aka: Task Queues) is to avoid doing a resource-intensive task immediately and having to wait for it to complete. Instead we schedule the task to be done later. We encapsulate a task as a message and send it to the queue. A worker process running in the background will pop the tasks and eventually execute the job. When you run many workers the tasks will be shared between them.
To implement a robust server , read this link , it may give some insights. Link
I know that Node.js uses a single-thread and an event loop to process requests only processing one at a time (which is non-blocking). But i am unable to determine Event loop capacity to run 100k request per second.
Here i want to capacity planning for nodejs server to handle the 100k request per second.
Please let me know how can i determine the capacity of event loop to increase capacity.
A single instance of Node.js runs in a single thread. To take advantage of multi-core systems the user will sometimes want to launch a cluster of Node.js processes to handle the load.
More info here and here
For the reference check following code for simple implementation of cluster in node.js
var cluster = require('cluster');
var express = require('express');
var numCPUs = require('os').cpus().length;
if (cluster.isMaster) {
for (var i = 0; i < numCPUs; i++) {
// Create a worker
cluster.fork();
}
} else {
// Workers share the TCP connection in this server
var app = express();
app.get('/', function (req, res) {
res.send('Hello World!');
});
// All workers use this port
app.listen(8080);
}
Cluster is an extensible multi-core server manager for node.js for more source check here.