I have 3 cpu cores on my machine, and I have 3 instances of node running, one for each core. When I access such server directly, that's a master process that always gets called. However, when I use a reversed nginx proxy, the process is random. Where does nginx chooses which node process to run?
http://domain.com:1000 -> proxy
http://domain.com:2000 -> node processes
Nginx config:
server {
listen 1000;
server_name node;
location / {
proxy_pass http://domain.com:2000/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
Nginx has no knowledge of your back-end's clustering the way you've configured it. As noted in a comment against your original question, Nginx load balancing only applies if you configure it with multiple backends. You are not doing this.
Therefore, if your backend itself is something like NodeJS using the Cluster module, the load balancing is happening there. Node doesn't need any "help" from Nginx to do this - it has its own mechanisms. They're described in https://nodejs.org/api/cluster.html#cluster_how_it_works:
The cluster module supports two methods of distributing incoming connections.
The first one (and the default one on all platforms except Windows), is the round-robin approach, where the master process listens on a port, accepts new connections and distributes them across the workers in a round-robin fashion, with some built-in smarts to avoid overloading a worker process.
The second approach is where the master process creates the listen socket and sends it to interested workers. The workers then accept incoming connections directly.
You can therefore choose how you want this to work in your Node back-end. At the moment, you are telling Nginx to always go to port 2000, therefore whatever you have running there is getting the traffic. If that is the master process for a Node Cluster, it will round-robin load balance. If that is one of the child workers, then that child worker will get ALL the traffic and the others will get none.
Related
We are using nodejs + socketIO with transport type as polling as we have to pass token in headers so that we can authenticate the client so i cannot avoid polling transport type.
Now we are using nginx and 4 socket application for this.
I am getting two problem because of this.
When polling call finishes and upgrade to websocket transport type I am getting 400 bad request. That i got to know is because the second request is landing on other socket server which is rejecting this transport type websocket.
these connection are getting triggers to rapidly even once the websocket connection is successful.
This problem#2 comes only when we are running multiple instance of socket server. with single server its works fine and connection doent terminates
When using NGINX as a load balancer to implement reverse proxy to a multi-istance websocket application you have to configure Nginx so that each time a connection is made to an istance, all consecutive requests from the same client should be proxied to the same istance, to avoid unwanted disconnections. Basically you want to implement sticky sessions.
This is well documented in the Socket.io official documentation.
http {
server {
listen 3000;
server_name io.yourhost.com;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_pass http://nodes;
# enable WebSockets
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
upstream nodes {
# enable sticky session with either "hash" (uses the complete IP address)
hash $remote_addr consistent;
# or "ip_hash" (uses the first three octets of the client IPv4 address, or the entire IPv6 address)
# ip_hash;
# or "sticky" (needs commercial subscription)
# sticky cookie srv_id expires=1h domain=.example.com path=/;
server app01:3000;
server app02:3000;
server app03:3000;
}
}
The key line is hash $remote_addr consistent;, declared inside the upstream block.
Note that here there are 3 different socket istances deployed on hosts app01, app02, and app03 (always port 3000). If you want to run all of your istances on the same host, you should run them on different ports (example: app01:3001, app02:3002, app03:3003).
Moreover, note that if you have multiple socket server istances with several clients connected, you want that client1 connected to ServerA should be able to "see" and communicate with client2 connected to ServerB. To do this, you want ServerA and ServerB to communicate or at least to share informations. Socket.io can handle this for you with a small effort by using a Redis istance and the redis-adapter module. Check this part of the socket.io documentation.
Final note: both links I shared are from the same socket.io doc page but they point to a different section of the page. I strongly suggest you to read the whole page to have a complete overview about the whole architecture.
I am trying to scale my Socket.io Node.js server horizontally using Cloud Foundry (on IBM Cloud).
As of now, my manifest.yml for cf looks like this:
applications:
- name: chat-app-server
memory: 512M
instances: 2
buildpacks:
- nginx_buildpack
This way the deployment goes through, but of course the socket connections between client and server fail because the connection is not sticky.
The official Socket.io documentation gives an example for using NginX for using multiple nodes.
When using a custom nginx.conf file using the Socket.io template I am missing some information (highlighted with ???).
events { worker_connections 1024; }
http {
server {
listen {{port}};
server_name ???;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_pass http://nodes;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
upstream nodes {
# enable sticky session based on IP
ip_hash;
server ???:???;
server ???:???;
}
}
I've tried to find out where cloud foundry runs the two instances specified in the manifest.yml file with no luck.
How do I get the required server addresses/ports from cloud foundry?
Is there a way to obtain this information dynamically from CF?
I am deploying my application using cf push.
I haven't used Socket.IO before, so I may be off base, but from a quick read of the docs, it seems like things should just work.
Two points from the docs:
a.) When using WebSockets, this is a non-issue. Cloud Foundry fully supports WebSockets. Hopefully, most of your clients can do that.
b.) When falling back to long polling, you need sticky sessions. Cloud Foundry supports sticky sessions out-of-the-box, so again, this should just work. There is one caveat though regarding CF's support of sticky sessions, it expects the session cookie name to be JSESSIONID.
Again, I'm not super familiar with Socket.IO, but I suspect it's probably using a different session cookie name by default (most things outside of Java do). You just need to change the session cookie name to JSESSIONID and sticky sessions should work.
TIP: you can check the session cookie name by looking at your cookies in your browser's dev tools.
Final note. You don't need Nginx here at all. Gorouter, which is Cloud Foundry's routing layer, will handle the sticky session support for you.
I am running a nodejs server on port 8080, so my server can only process one request at a time.
I can see that if i send multiple requests in one single shot, new requests are queued and executed sequentially one after another.
What I am trying to find is, how do i run multiple instances/threads of this process. Example like gunicorn for python servers. Is there something similar, instead of running the nodejs server on multiple ports for each instance.
I have placed nginx infront of the node process. Is that sufficient and recommended method.
worker_processes auto;
worker_rlimit_nofile 1100;
events {
worker_connections 1024;
multi_accept on;
use epoll;
}
pid /var/run/nginx.pid;
http {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
server {
listen 80;
server_name localhost;
access_log /dev/null;
error_log /dev/null;
location / {
proxy_pass http://localhost:8080;
}
}
}
First off, make sure your node.js process is ONLY using asynchronous I/O. If it's not compute intensive and using asynchronous I/O, it should be able to have many different requests "in-flight" at the same time. The design of node.js is particularly good at this if your code is designed properly. If you show us the crux of what it is doing on one of these requests, we can advise more specifically on whether your server code is designed properly for best throughput.
Second, instrument and measure, measure, measure. Understand where your bottlenecks are in your existing node.js server and what is causing the delay or sequencing you see. Sometimes there are ways to dramatically fix/improve your bottlenecks before you start adding lots more clusters or servers.
Third, use the node.js cluster module. This will create one master node.js process that automatically balances between several child processes. You generally want to creates a cluster child for each actual CPU you have in your server computer since that will get you the most use out of your CPU.
Fourth, if you need to scale to the point of multiple actual server computers, then you would use either a load balancer or reverse proxy such as nginx to share the load among multiple hosts. If you had a quad core CPUs in your server, you could run a cluster with four node.js processes on it on each server computer and then use nginx to balance among the several server boxes you had.
Note that adding multiple hosts that are load balanced by nginx is the last option here, not the first option.
Like #poke said, you would use a reverse proxy and/or a load balancer in front.
But if you want a software to run multiple instances of node, with balancing and other stuffs, you should check pm2
http://pm2.keymetrics.io/
Just a point to be added here over #sheplu, the pm2 module uses the node cluster module under the hood. But even then, pm2 is a very good choice, as it provides various other abstractions other than node cluster.
More info on it here: https://pm2.keymetrics.io/docs/usage/pm2-doc-single-page/
I'm very new to docker and productionizing nodejs web apps. However, after some reading I've determined that a good setup would be:
nginx container serving static files, ssl, proxying nodejs requests
nodejs container
postgesql container
However, I'm now trying to tackle scalability. Seeing as you can define multiple proxy_pass statements in an nginx config, could you not spin up a duplicate nodejs container (exactly the same but exposing a different port) and effectively "load balance" your web app? Is it a good architecture?
Also, how would this effect database writes? Are there race conditions I need to specifically architecture for? Any guidance would be appreciated.
Yes, it's possible to use Nginx to load balance requests between different instances of your Node.js services. Each Node.js instance could be running in a different Docker container. Increasing the scalability of your configuration is as easy as starting up another Docker container and ensure it's registered in the Nginx config. (Depending on how often you have to update the Nginx config, a variety of tools/frameworks are available to do this last step automatically.)
For example, below is an Nginx configuration to load balance incoming requests across different Node.js services. In our case, we have multiple Node.js services running on the same machine, but it's perfectly possible to use Docker containers instead.
File /etc/nginx/sites-enabled/apps:
upstream apps-cluster {
least_conn;
server localhost:8081;
server localhost:8082;
server localhost:8083;
keepalive 512;
}
server {
listen 8080;
location "/" {
proxy_next_upstream error timeout http_500 http_502 http_503 http_504;
proxy_set_header Connection "";
proxy_http_version 1.1;
proxy_pass http://apps-cluster;
}
access_log off;
}
Despite running multiple instances of your Node.js services, your database should not be negatively affected. The PostgreSQL database itself can perfectly handle multiple open connections and automatically resolves any race conditions. From a developer point of view, the code for running 1 Node.js service is the same as for running x Node.js services.
You can set "Function Level Concurrent Execution Limit" on the function you are using to connect to RDS. This will contain the number of RDS connections. The requests from Dynamo will be throttled though.
Another option is to stream them into Kinesis or SQS from this lambda and have another worker lambda to read it from there and pump the data into RDS. This is scalable and reliable with no throttling.
I have a couple of small production sites and a bunch of fun hobbyist/experimental apps and such. I'd like to run all of them on one EC2 instance.
Can I install node.js, npm, express and couchdb once, and then run each app on a different port, and adjust the dns settings at my domain registry to point to the appropriate place?
Update: Thanks Mike! For anyone else who's looking for multiple IP addresses on EC2: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-instance-addressing.html
There are multiple ways to go about it.
Different Ports
You could run each Node.js process on a different port and simply open the ports to the world. However, your URLs would need a port on the end of the hostname for each project. yoururl.com:8080/ would technically work, but probably not what you're looking for.
Multiple IP Addresses
You could use multiple IP addresses on one EC2 instance, however, they come with an additional cost of about $3.65 a month each. So if you have 10 different domains you want to host on once instance then that's over $30 a month extra in hosting fees.
On the flip side, any domain using SSL needs it's own IP address.
Also, there are limits to the number of IP addresses you can assign to an instance and the smaller the instance, the less IP addresses you get.
The number of IP addresses that you can assign varies by instance type. Small instances can accommodate up to 8 IP addresses (across 2 elastic network interfaces) whereas High-Memory Quadruple Extra Large and Cluster Computer Eight Extra Large instances can be assigned up to 240 IP addresses (across 8 elastic network interfaces). For more information about IP address and elastic network interface limits, go to Instance Families and Types in the Amazon EC2 User Guide.
Express Vhosts
Express comes with virtual host functionality. You can run multiple domains under one Node.js/Express server and setup routes based on domain name. vhost under Express enables this.
Reverse Proxy
You can setup Nginx in front of multiple application servers. This has the most flexibility. You can have one Node.js process per domain which allows you to do updates and restarts on one domain at a time. It also allows you to host applications servers such as Apache/PHP under the same EC2 instance along side your Node.js process.
With Nginx acting as a reverse proxy you could also host different application servers under the same domain, but serving different paths.
For instance, Node.js could serve the main root path of a domain but you could setup the /blog/ path to go to a Apache/PHP/Wordpress setup on the same EC2 instance.
Already answered # https://stackoverflow.com/a/50528580/3477353 but mentioning it again.
Adding to the #Daniel answer, I would like to mention the configuration of my nginx for those who are looking for the exact syntax in implementing it.
server {
listen 80;
server_name mysite.com;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_pass http://127.0.0.1:3000;
}
}
server {
listen 80;
server_name api.mysite.com;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_pass http://127.0.0.1:4500;
}
}
Just create two server objects with unique server name and the port address.
Mind proxy_pass in each object.
Thank you.