I am making a card game on node.js and I am thinking about spawning a new process per game room. I plan on doing the connection using fork(). After quite some research, I found that this isn't the best approach, because I should have a node process per physical core. But isn't my approach better for scalability and modularity? If let's say a game room crashes, it wouldn't crash the rest of them. Can someone help me analyse a bit better the situation? I plan on running the game on aws EC2 instances and expect a maximum of 1500 concurrent users, playing on rooms of 4 people, and communicating with socket.io messages.
A single Node instance can handle that type of load, as one of NodeJS's strongpoints is real time communication and many concurrent connections.
Regarding crashes, you need to plan for those. Some initial tips:
Catch errors and log error messages so that your Node instance does not completely fail. Often errors may stop that particular function chain from finishing correctly, but will not kill your process.
Persist your game state to another service, like a database, so things (like connections) can recover. Use Case example: "User loses connection and logs back in, they are re-connected to the room and can see the game in its current state"
You can auto-recover your Node process by running something like forever or PM2 (there are others). These will monitor and restart on process failure (though that shouldn't happen too often).
I have a question that nobody seems to help with. How will this be handled in a production mode with thousands of requests at the same time?
I did a simple test case:
module.exports = {
index: function (req, res){
if (req.param('foo') == 'bar'){
async.series([
function(callback){
for (k=0; k <= 50000; k++){
console.log('did something stupid a few times');
}
callback();
}
], function(){
return res.json(null);
});
}else{
return res.view('homepage', {
});
}
}
};
Now if I go to http://localhost:1337/?foo=bar it will obviously wait a while before it responds. So if I now open a different session (other browser or incognito, and go to http://localhost:1337/ I am expecting a result immediately. Instead it is waiting for the other request to finish and only then it will let this request go through.
Therefore it is not asynchronous and it is a huge problem if I have as much as 2 ppl at the same time operating this app. I mean this app will have drop downs coming from databases, html files being served etc...
My question is this: how does one handle such an issue??? I hear the word "promises vs callbacks" - is this some sort of a solution to this?
I know about clustering, but that only separates the requests over the amount of cpu's, ultimately you will fix it by at most allowing 8 people at the same time without being blocked. It won;t handle 100 requests at the same time...
P.S. That test was to simplify the example, but think of someone uploading a file, a web service that goes to a different server, a point of sales payment terminal waiting for a user to input the pin, someone downloading a file from the app, etc...
nodejs is event driven and runs your Javascript as single threaded. So, as long as your code from the first request is sitting in that for loop, nodejs can't do anything else and won't get to the next event in the event queue so thus your second request has to wait for the first one to finish.
Now, if you used a true async operation such as setTimeout() instead of your big for loop, then nodejs could service other events while the first request was waiting for the setTimeout().
The general rule in nodejs is to avoid doing anything that takes a ton of CPU in your main nodejs app. If you are stuck with something CPU-intensive, then you're best to either run clusters (as many as CPUs you have) or move the CPU-intensive work to some sort of worker queue that is served by different processes and let the OS time slice those other processes while the main nodejs process stays free and ready to service new incoming requests.
My question is this: how does one handle such an issue??? I hear the word "promises vs callbacks" - is this some sort of a solution to this?
I know about clustering, but that only separates the requests over the amount of cpu's, ultimately you will fix it by at most allowing 8 people at the same time without being blocked. It won;t handle 100 requests at the same time...
Most of the time, a server process spends most of the time of a request doing things that are asynchronous in nodejs (reading files, talking to other servers, doing database operations, etc...) where the actual work is done outside the nodejs process. When that is the case, nodejs does not block and is free to work on other requests while the async operations from other requests are underway. The little bit of CPU time coordinating these operations can be helped further by clustering though it's probably worth testing a single process first to see if clustering is really needed.
P.S. That test was to simplify the example, but think of someone uploading a file, a web service that goes to a different server, a point of sales payment terminal waiting for a user to input the pin, someone downloading a file from the app, etc...
All the operations you mentioned here can be done truly asynchronously so they won't block your nodejs app like your for loop does so basically the for loop isn't a good simulation of any of this. You need to use a real async operation to simulate it. Real async operations do their work outside of the main nodejs thread and then just post an event to the event queue when they are done, allowing nodejs to do other things while the async operations are doing their work. That's the key.
I'm working on a Twitter project, using their streaming API, built on Heroku with Node.js.
I have a collection of topics that my app needs to process, which are pulled from MongoDB. I need to track each of these topics via the API, however it needs to be done such that each topic is tracked only once. As each worker process expires after approximately 1 hour, when a worker receives SIGTERM it needs to untrack each topic assigned, and release it back to the pool again.
I've been using RabbitMQ to communicate between app and worker processes, however with this I'm a little stuck. Are there any good examples, or advice you can offer on the correct way to do this?
Couldn't the worker just send a message via the messagequeue to the application when it receives a SIGTERM? According to the heroku docs on shutdown the process is allowed a couple of seconds (10) before it will be forecefully killed.
So you can do something like this:
// listen for SIGTERM sent by heroku
process.on('SIGTERM', function () {
// - notify app that this worker is shutting down
messageQueue.sendSomeMessageAboutShuttingDown();
// - shutdown process (might need to wait for async completion
// of message delivery to not prevent it from being delivered)
process.exit()
});
Alternatively you could break up your work in much smaller chunks and have workers only 'take' work that will run for a couple of minutes or even seconds max. Your main application should be the bookkeeper and if a process doesn't complete its task within a specified time assume it has gone missing and make the task available for another process to handle. You can probably also implement this behavior using confirms in rabbitmq.
RabbitMQ won't do this for you.
It will allow you to distribute the work to another process and/or computer, but it won't provide the kind of mechanism you need to prevent more than one process / computer from working on a particular topic.
What you want is a semaphore - a way to control access to a particular "resource" from multiple processes... a way to ensure only one process is working on a particular resource at a given time. In your case the "resource" will be the topic... but it will still be the resource that you want to control access to.
FWIW, there has been discussion of using RabbitMQ to implement a distributed semaphore in the past:
https://www.rabbitmq.com/blog/2014/02/19/distributed-semaphores-with-rabbitmq/
https://aphyr.com/posts/315-call-me-maybe-rabbitmq
but the general consensus is that this is a bad idea. there are too many edge cases and scenarios in which RabbitMQ will fail to work as proper semaphore.
There are some node.js semaphore libraries available. I would recommend looking at them, and using one of them. Have a single process manage the semaphore and decide which other process can / cannot work on which topic.
We are using socketIO on a large chat application.
At some points we want to dispatch "presence" (user availability) to all other users.
io.in('room1').emit('availability:update', {userid='xxx', isAvailable: false});
room1 may contains a lot of users (500 max). We observe a significant raise in our NodeJS load when many availability updates are triggered.
The idea was to use something similar to redis store with Socket IO. Have web browser clients to connect to different NodeJS servers.
When we want to emit to a room we dispatch the "emit to room1" payload to all other NodeJS processes using Redis PubSub ZeroMQ or even RabbitMQ for persistence. Each process will itself call his own io.in('room1').emit to target his subset of connected users.
One of the concern with this setup is that the inter-process communication may become quite busy and I was wondering if it may become a problem in the future.
Here is the architecture I have in mind.
Could you batch changes and only distribute them every 5 seconds or so? In other words, on each node server, simply take a 'snapshot' every X seconds of the current state of all users (e.g. 'connected', 'idle', etc.) and then send that to the other relevant servers in your cluster.
Each server then does the same, every 5 seconds or so it sends the same message - of only the changes in user state - as one batch object array to all connected clients.
Right now, I'm rather surprised you are attempting to send information about each user as a packet. Batching seems like it would solve your problem quite well, as it would also make better use of standard packet sizes that are normally transmitted via routers and switches.
You are looking for this library:
https://github.com/automattic/socket.io-redis
Which can be used with this emitter:
https://github.com/Automattic/socket.io-emitter
About available users function, I think there are two alternatives,you can create a "queue Users" where will contents "public data" from connected users or you can use exchanges binding information for show users connected. If you use an "user's queue", this will be the same for each "room" and you could update it when an user go out, "popping" its state message from queue (Although you will have to "reorganize" all queue message for it).
Nevertheless, I think that RabbitMQ is designed for asynchronous communication and it is not very useful approximation have a register for presence or not from users. I think it's better for applications where you don't know when the user will receive the message and its "real availability" ("fire and forget architectures"). ZeroMQ require more work from zero but you could implement something more specific for your situation with a better performance.
An publish/subscribe example from RabbitMQ site could be a good point to begin a new design like yours where a message it's sent to several users at same time. At summary, I will create two queues for user (receive and send queue messages) and I'll use specific exchanges for each "room chat" controlling that users are in each room using exchange binding's information. Always you have two queues for user and you create exchanges to binding it to one or more "chat rooms".
I hope this answer could be useful for you ,sorry for my bad English.
This is the common approach for sharing data across several Socket.io processes. You have done well, so far, with a single process and a single thread. I could lamely assume that you could pick any of the mentioned technologies for communicating shared data without hitting any performance issues.
If all you need is IPC, you could perhaps have a look at Faye. If, however, you need to have some data persisted, you could start a Redis cluster with as many Redis masters as you have CPUs, though this will add minor networking noise for Pub/Sub.
Node.js looks interesting, BUT I must miss something - isn't Node.js tuned only to run on a single process and thread?
Then how does it scale for multi-core CPUs and multi-CPU servers? After all, it is all great to make fast as possible single-thread server, but for high loads I would want to use several CPUs. And the same goes for making applications faster - seems today the way is use multiple CPUs and parallelize the tasks.
How does Node.js fit into this picture? Is its idea to somehow distribute multiple instances or what?
[This post is up-to-date as of 2012-09-02 (newer than above).]
Node.js absolutely does scale on multi-core machines.
Yes, Node.js is one-thread-per-process. This is a very deliberate design decision and eliminates the need to deal with locking semantics. If you don't agree with this, you probably don't yet realize just how insanely hard it is to debug multi-threaded code. For a deeper explanation of the Node.js process model and why it works this way (and why it will NEVER support multiple threads), read my other post.
So how do I take advantage of my 16 core box?
Two ways:
For big heavy compute tasks like image encoding, Node.js can fire up child processes or send messages to additional worker processes. In this design, you'd have one thread managing the flow of events and N processes doing heavy compute tasks and chewing up the other 15 CPUs.
For scaling throughput on a webservice, you should run multiple Node.js servers on one box, one per core and split request traffic between them. This provides excellent CPU-affinity and will scale throughput nearly linearly with core count.
Scaling throughput on a webservice
Since v6.0.X Node.js has included the cluster module straight out of the box, which makes it easy to set up multiple node workers that can listen on a single port. Note that this is NOT the same as the older learnboost "cluster" module available through npm.
if (cluster.isMaster) {
// Fork workers.
for (var i = 0; i < numCPUs; i++) {
cluster.fork();
}
} else {
http.Server(function(req, res) { ... }).listen(8000);
}
Workers will compete to accept new connections, and the least loaded process is most likely to win. It works pretty well and can scale up throughput quite well on a multi-core box.
If you have enough load to care about multiple cores, then you are going to want to do a few more things too:
Run your Node.js service behind a web-proxy like Nginx or Apache - something that can do connection throttling (unless you want overload conditions to bring the box down completely), rewrite URLs, serve static content, and proxy other sub-services.
Periodically recycle your worker processes. For a long-running process, even a small memory leak will eventually add up.
Setup log collection / monitoring
PS: There's a discussion between Aaron and Christopher in the comments of another post (as of this writing, its the top post). A few comments on that:
A shared socket model is very convenient for allowing multiple processes to listen on a single port and compete to accept new connections. Conceptually, you could think of preforked Apache doing this with the significant caveat that each process will only accept a single connection and then die. The efficiency loss for Apache is in the overhead of forking new processes and has nothing to do with the socket operations.
For Node.js, having N workers compete on a single socket is an extremely reasonable solution. The alternative is to set up an on-box front-end like Nginx and have that proxy traffic to the individual workers, alternating between workers for assigning new connections. The two solutions have very similar performance characteristics. And since, as I mentioned above, you will likely want to have Nginx (or an alternative) fronting your node service anyways, the choice here is really between:
Shared Ports: nginx (port 80) --> Node_workers x N (sharing port 3000 w/ Cluster)
vs
Individual Ports: nginx (port 80) --> {Node_worker (port 3000), Node_worker (port 3001), Node_worker (port 3002), Node_worker (port 3003) ...}
There are arguably some benefits to the individual ports setup (potential to have less coupling between processes, have more sophisticated load-balancing decisions, etc.), but it is definitely more work to set up and the built-in cluster module is a low-complexity alternative that works for most people.
One method would be to run multiple instances of node.js on the server and then put a load balancer (preferably a non-blocking one like nginx) in front of them.
Ryan Dahl answers this question in the tech talk he gave at Google last summer. To paraphrase, "just run multiple node processes and use something sensible to allow them to communicate. e.g. sendmsg()-style IPC or traditional RPC".
If you want to get your hands dirty right away, check out the spark2 Forever module. It makes spawning multiple node processes trivially easy. It handles setting up port sharing, so they can each accept connections to the same port, and also auto-respawning if you want to make sure a process is restarted if/when it dies.
UPDATE - 10/11/11: Consensus in the node community seems to be that Cluster is now the preferred module for managing multiple node instances per machine. Forever is also worth a look.
You can use cluster module. Check this.
var cluster = require('cluster');
var http = require('http');
var numCPUs = require('os').cpus().length;
if (cluster.isMaster) {
// Fork workers.
for (var i = 0; i < numCPUs; i++) {
cluster.fork();
}
cluster.on('exit', function(worker, code, signal) {
console.log('worker ' + worker.process.pid + ' died');
});
} else {
// Workers can share any TCP connection
// In this case its a HTTP server
http.createServer(function(req, res) {
res.writeHead(200);
res.end("hello world\n");
}).listen(8000);
}
Node Js is supporting clustering to take full advantages of your cpu. If you are not not running it with cluster, then probably you are wasting your hardware capabilities.
Clustering in Node.js allows you to create separate processes which can share same server port. For example, if we run one HTTP server on Port 3000, it is one Server running on Single thread on single core of processor.
Code shown below allow you to cluster your application. This code is official code represented by Node.js.
var cluster = require('cluster');
var numCPUs = require('os').cpus().length;
if (cluster.isMaster) {
// Fork workers.
for (var i = 0; i < numCPUs; i++) {
cluster.fork();
}
Object.keys(cluster.workers).forEach(function(id) {
console.log("I am running with ID : " + cluster.workers[id].process.pid);
});
cluster.on('exit', function(worker, code, signal) {
console.log('worker ' + worker.process.pid + ' died');
});
} else {
//Do further processing.
}
check this article for the full tutorial
As mentioned above, Cluster will scale and load-balance your app across all cores.
adding something like
cluster.on('exit', function () {
cluster.fork();
});
Will restart any failing workers.
These days, a lot of people also prefer PM2, which handles the clustering for you and also provides some cool monitoring features.
Then, add Nginx or HAProxy in front of several machines running with clustering and you have multiple levels of failover and a much higher load capacity.
The cluster module allows you to utilise all cores of your machine. In fact you can take advantage of this in just 2 commands and without touching your code using a very popular process manager pm2.
npm i -g pm2
pm2 start app.js -i max
Multi-node harnesses all the cores that you may have.
Have a look at http://github.com/kriszyp/multi-node.
For simpler needs, you can start up multiple copies of node on different port numbers and put a load balancer in front of them.
Future version of node will allow you to fork a process and pass messages to it and Ryan has stated he wants to find some way to also share file handlers, so it won't be a straight forward Web Worker implementation.
At this time there is not an easy solution for this but it's still very early and node is one of the fastest moving open source projects I've ever seen so expect something awesome in the near future.
Spark2 is based on Spark which is now no longer maintained. Cluster is its successor, and it has some cool features, like spawning one worker process per CPU core and respawning dead workers.
You may run your node.js application on multiple cores by using the cluster module on combination with os module which may be used to detect how many CPUs you have.
For example let's imagine that you have a server module that runs simple http server on the backend and you want to run it for several CPUs:
// Dependencies.
const server = require('./lib/server'); // This is our custom server module.
const cluster = require('cluster');
const os = require('os');
// If we're on the master thread start the forks.
if (cluster.isMaster) {
// Fork the process.
for (let i = 0; i < os.cpus().length; i++) {
cluster.fork();
}
} else {
// If we're not on the master thread start the server.
server.init();
}
I'm using Node worker to run processes in a simple way from my main process. Seems to be working great while we wait for the official way to come around.
The new kid on the block here is LearnBoost's "Up".
It provides "Zero-downtime reloads" and additionally creates multiple workers (by default the number of CPUs, but it is configurable) to provide the best of all Worlds.
It is new, but seems to be pretty stable, and I'm using it happily in one of my current projects.
💢 IMPORTANT DIFFERENCE - ROLLING RESTART
I have to add an important difference between using node's build in cluster mode VS a process manager like PM2's cluster mode.
PM2 allows zero down time reloads when you are running.
pm2 start app.js -i 2 --wait-ready
In your codes add the following
process.send('ready');
When you call pm2 reload app after code updates, PM2 will reload
the first instance of the app, wait for the 'ready' call, then it move on
to reloads the next instance, ensuring you always have an app active to respond to requests.
While if you use nodejs' cluster, there will be down time when you restart and waiting for server to be ready since there is only one instance of the app and you are restarting all the cores together.
I searched for Clusterize an app for all CPU cores available and found my myself here. Where I found this keyword Is Pm2 command
pm2 examples
This is what I found
Clusterize an app to all CPU cores available:
$ pm2 start -i max
If you need to install pm2 use these commands
npm install -g pm2
yan add -g pm2
or
Use this link Here
It's also possible to design the web-service as several stand alone servers that listen to unix sockets, so that you can push functions like data processing into seperate processes.
This is similar to most scrpting/database web server architectures where a cgi process handles business logic and then pushes and pulls the data via a unix socket to a database.
the difference being that the data processing is written as a node webserver listening on a port.
it's more complex but ultimately its where multi-core development has to go. a multiprocess architecture using multiple components for each web request.
It's possible to scale NodeJS out to multiple boxes using a pure TCP load balancer (HAProxy) in front of multiple boxes running one NodeJS process each.
If you then have some common knowledge to share between all instances you could use a central Redis store or similar which can then be accessed from all process instances (e.g. from all boxes)