Let's say I am trying to scale my app for over 200k+ socket connection in a node js app, how do I go about this. I have been doing a lot of research and was thinking about:
-application layer
-socket.io layer
-load balancer
This was after I read a socket.io benchmark testing article but stumbled upon socketcluster afterwards! I have also thought about using socket.io + redis and increasing heroku dyno's, not sure how much this will increase socket.io scalability.
My question now is wouldn't cluster js already run multiple instances of the node js server with socket.io on all cores available? If so why would you need socketcluster?
Thanks!
Socket.io requires sticky load balancing which makes your app more vulnerable to DoS attacks (with sticky load balancing, a malicious user could target specific workers and crash them one at a time). Also, sticky load balancing can lead to uneven distribution of traffic across workers when you have to handle users who are behind a corporate proxy.
Unlike SocketCluster, Socket.io does not support pub/sub which makes it difficult to communicate between users who are connected to different workers/hosts.
Related
Currently I have two load-balanced single-process nodeJS instances on Amazon beanstalk. They each just use a simple in-memory object as a cache. Because they are on different servers, their cache is not shared.
This is where Redis comes in. I want to utilize Redis to create a shared cache, but only to prime the internal memory of NodeJs.
You see, currently I am caching 4KB-10KB objects, if I solely relied on Redis then I have not only the Redis latency to retrieve the obejct but the network latency as well. I rather use Redis as a persistent cache that will prime my nodeJS instances when they are booted up, and also keep both internal caches in sync periodically (every x minutes)
A pretty basic nodeJs memory cache is https://github.com/tcs-de/nodecache
To complicate things even more, I am looking to start using nodeJS cluster ability to fork multiple processes of the application under the same server. Therefore, it is important that all clusters share 1 in-memory local server cache.
The aforementioned nodejs lib has a wrapper that aids the use in a cluster environment : https://github.com/lvx3/cluster-cache
To recap,
I will have Server A and server B who will be load balanced evenly. Each server (A&B) will have say 4 nodeJs processes who need to share just 1 cache (That is, the 4 server A nodeJs processes should all use a Server A cache, same for B)
Then I want the Server A and Server B cache to periodically sync and "persist" onto Redis. In the event of a crash or redeployment the Server cache would be primed with what is in Redis.
What are my options? Is there any well established solutions or mix of solutions that would be suitable? Such as a plugin like nodecache (simple) that has a Redis plugin? I also use express so perhaps there are express middleware that would be well suited for this.
Is it even worth the complexity to use Redis to prime a local server memory cache or should I just rely solely on Redis and take the network latency hit?
An acceptable but somewhat disappointing time for me to get back a 10KB object is 20 ms. I'd much prefer around 1ms. Redis and the nodeJs servers will be on Amazon so will be pretty close together.
I understand that if I have a redis cache of say 50MB that the same 50MB would exist on Server A and Server B. I am more than willing to spend money on hardware/ram for the benefit of speed.
I recently started with node and I have been reading a lot about its limitation of it being single threaded and how it does not utilise your cores and then I read this
http://bit.ly/1n2YW68 (which talk about the new cluster module of nodejs for loadbalancing)
Now I'm not sure I completely agree to it :) because the first thing that I thought of before starting with node on how to make it utilise cores with proper load balancing is via web-server some like upstream module like nginx
like doing something like this
upstream domain1 {
server http://nodeapp1;
server http://nodeapp2;
server http://nodeapp3;
}
So my question is there an advantage to use such cluster module for load balancing to utilise the cores does it has any significant advantage over web server load balancing
or is blog post too far from real use.
Note: I'm ain't concerned about load balancing handle by various app server like passenger(passenger has nodejs support as well but something that I'm not looking for answer :)) which I already know since I'm mostly a ruby programmer
One other option you can use to cluster NodeJs applications is to deploy the app using PM2.
Clustering is just easy as this, You don't need to implement clustering by hand
pm2 start app.js -i max
PM2 is an expert to auto detect the number of available CPUs and run as many processes as possible
Read about PM2 cluster mode here
http://pm2.keymetrics.io/docs/usage/cluster-mode/
For controlling the load of IO operations, I wrote a library called QueueP using the memoization concept. You can even customize the memoization logic and gain speedup values of more than 10, sometimes
https://www.npmjs.com/package/queuep
As far as I know, the built in node cluster is not a good solution yet (load is not evenly distributed across cores). Until v0.12: http://strongloop.com/strongblog/whats-new-in-node-js-v0-12-cluster-round-robin-load-balancing/
So you should use nginx until then. After that we will see some benchmarks comparing both options and see if the built in cluster module is a good choice.
How to scale Node.js application in standard way locally and between servers?
Cluster API as of Node v0.10.18 is still experimental, that is API is very like to change that is risk for big project.
Are there other standard practices, tools to scale automatically (raise up number of running instances on the same server and on other servers)
This question is not about making inter-process communication like Node.js multi-server cluster: how to share object in several nodes cluster or How to make a node.js server with rooms scalable, but what infrastructure could automatically manage number of Node.js instances?
How to automatically start Node.js apps on other server? Are there ready-to-use tools or framework to automate "deploying" Node.js apps?
Setting up load balancers, security would be next steps.
Creating a node.js application is simple enough.
var app = require('express')();
app.get('/',function(req,res){
res.send("Hello world!");
});
But suppose people became obsessed with your Hello World! application and exhausted your resources. How could this example be scaled up on practice? I don't understand it, because yes, you could open several node.js instance in different computers - but when someone access http://your_site.com/ it aims directly that specific machine, that specific port, that specific node process. So how?
There are many many ways to deal with this, but it boils down to 2 things:
being able to use more cores per server
being able to scale beyond more than one server.
node-cluster
For the first option, you can user node-cluster or the same solution as for the seconde option. node-cluster (http://nodejs.org/api/cluster.html) essentially is a built in way to fork the node process into one master and multiple workers. Typically, you'd want 1 master and n-1 to n workers (n being your number of available cores).
load balancers
The second option is to use a load balancer that distributes the requests amongst multiple workers (on the same server, or across servers).
Here you have multiple options as well. Here are a few:
a node based option: Load balancing with node.js using http-proxy
nginx: Node.js + Nginx - What now? (using more than one upstream server)
apache: (no clearly helpful link I could use, but a valid option)
One more thing, once you start having multiple processes serving requests, you can no longer use memory to store state, you need an additional service to store shared states, Redis (http://redis.io) is a popular choice, but by no means the only one.
If you use services such as cloudfoundry, heroku, and others, they set it up for you so you only have to worry about your app's logic (and using a service to deal with shared state)
I've been working with node for quite some time but recently got the opportunity to try scaling my node apps and have been researching on the same topic for some time now and have come across following pre-requisites for scaling:
My app needs to be available on a distributed system each running multiple instances of node
Each system should have a load balancer that helps distribute traffic across the node instances.
There should be a master load balancer that should distribute traffic across the node instances on distributed systems.
The master balancer should always be running OR should have a dependable restart mechanism to keep the app stable.
For the above requisites I've come across the following:
Use modules like cluster to start multiple instances of node in a system.
Use nginx always. It's one of the most simplest mechanism for creating a load balancer i've came across so far
Use HAProxy to act as a master load balancer. A few pointers on how to use it and keep it forever running.
Useful resources:
Horizontal scaling node.js and websockets.
Using cluster to take advantages of multiple cores.
I'll keep updating this answer as I progress.
The basic way to use multiple machines is to put them behind a load balancer, and point all your traffic to the load balancer. That way, someone going to http://my_domain.com, and it will point at the load balancer machine. The sole purpose (for this example anyways; in theory more could be done) of the load balancer is to delegate the traffic to a given machine running your application. This means that you can have x number of machines running your application, however an external machine (in this case a browser) can go to the load balancer address and get to one of them. The client doesn't (and doesn't have to) know what machine is actually handling its request. If you are using AWS, it's pretty easy to set up and manage this. Note that Pascal's answer has more detail about your options here.
With Node specifically, you may want to look at the Node Cluster module. I don't really have alot of experience with this module, however it should allow you to spawn multiple process of your application on one machine all sharing the same port. Also node that it's still experimental and I'm not sure how reliably it will be.
I'd recommend to take a look to http://senecajs.org, a microservices toolkit for Node.js. That is a good start point for beginners and to start thinking in "services" instead of monolitic applications.
Having said that, building distributed applcations is hard, take time to learn, take LOT of time to master it, and usually you will face a lot trade-off between performance, reliability, manteinance, etc.
So the first app that people usually build with SocketIO and Node is usually a chatting app. This chatting app basically has 1 Node server that will broadcast to multiple clients. In the Node code, you would have something like.
//Psuedocode
for(client in clients){
if(client != messageSender){
user.send(message);
}
}
This is great for a low number of users, but I see a problem with this. First of all, there is a single point of failure which is the Node server. Second of all, the app will slow down as the number of clients grow. What is there to do then when we reach this bottleneck? Is there an architecture (horizontal/vertical scaling) that can be used to alleviate this problem?
For that "one day" when your chat app needs multiple, fault-tolerant node servers, and you want to use socket.io to cross communicate between the server and the client, there is a node.js module that fits the bill.
https://github.com/hookio/hook.io
It's basically an event emitting framework to cross communicate between multiple "things" -- such as multiple node servers.
It's relatively complicated to use, compared to most modules, which is understandable since this is a complex problem to solve.
That being said, you'd probably have to have a few thousand simultaneous users and lots of other problems before you begin to have problems with this.
Another thing you can do, is try to develop your application in a way so that if a connection is lost (which happens all the time anyway), eg. server goes down, client has network issues (eg. mobile user), etc, your application should be able to handle that and recover from such issues gracefully.
Since Node.js has a single event-loop thread, this single point of failure is written into its DNA. Even reloading a server after code changes require this thread to be stopped.
There are however a lot of tools available to handle such failures gracefully. You could use forever; a simple CLI tool for ensuring that a given script runs continuously. Other options include distribute and up. Distribute is a load balancing middleware for Node. Up builds on top of Distribute to offer zero downtime reloads using either a JavaScript API or command line interface:
Further reading I find you just need to use Redis Store with Socket.io to maintain connection references between two or more processes/ servers. These options have already been discussed extensively here and here.
There's also the option of using socket.io-clusterhub if you don't intend to use the Redis store.