I've been rewriting my nodejs game app to read memory from redis so I could ultimately scale it if it were to ever grow large. But, I stopped because I feel like I am going about it the wrong way. For example:
Server 1 is on port 9300
Server 2 is on port 9301
Now, let's say a player from the Server 1 wants to send a private message to a player that is on Server 2.
What I currently do
Server 2 will send a publish signal to redis, and Server 1 will catch that signal as well, and if that user is on that server, it will send them a notification along with the message.
Some questions
1) Wouldn't it be more appropiate to just have Server 2 send a message to Server 1 without publishing to redis?
2) Server 2 doesn't keep a track of all the connected clients that Server 1 has, so it wouldn't be possible. Unless, I keep track of all connected clients on every server aswell? That would require the client to connect to multiple servers on each visit.
3) Let's say I have 10 servers. A user on Server 5 wants to send a private message to a user on Server 1. If I send a pub signal through redis, Servers 4,6,7,8,9,2,3 will all receive that signal as well... which is unneeded? Is that when Peer to Peer connections come into play? Or is that the extra bandwidth that is required for scaling and I'm overthinking everything?
Perhaps you could change concept of your app by adding RabbitMQ instead of redis pub/sub. RMQ would allow smarter message routing.
Basicly each user can listen its own messages:
User connects to server (1-x)
Server subscribes to RMQ exchange messages with user routing key
When user publishes private message it's sent to provided exchange with specific user routing key
Still that concept will hit the limit if you get large number of users. Connection count to RMQ server will grow intensivly. In that case you can scale RMQ or change connection concept:
Save all user connection info to redis
When user sends private message to some other user first find on which server instance user is
Send message only to specific server instance (fetched from user connection information)
Related
I'm trying to build a realtime (private) chat between users of a video game with 25K+ concurrent connections. We currently run 32 nodes where users can connect through a load balancer. The problem I'm trying to solve is how to route messages to each user?
Currently, we are using socket.io & socket.io-redis, where each websocket joins a room with its user ID, and we emit each message they should receive to that room. The problem with this design is that we are reaching the limits of Redis Pubsub, and Socket.io which doesn't scale well (socket.io emit messages to all nodes which check if the user is connected, this is not viable).
Our current stack is composed of Postgres, Redis & RabbitMQ. I have been thinking about this problem a lot and have come up with 3 different solutions :
Route all messages with RabbitMQ. When a user connects, we create an exchange with type fanout with the user ID and a queue per websocket connection (we have to handle multiple connections per user). When we want to emit to that user, we simply publish to that exchange. The problem with that approach is that we have to create a lot of queues, and I heard that this may not be very efficient.
Create a queue for each node in RabbitMQ. When a user connects, we save the node & socket ID in a Redis Set, so that when we have to send a message to that specific user, we first get the list of nodes, emit to each node queue, which then handle routing to specific client in the app. The problems with that approach is that in the case of a node failure, we may store that a user is connected when this is not the case. To fix that, we would need to expire the users's Redis entry but this is not a perfect fix. Also, if we later want to implement group chat, it would mean we have to send duplicates messages in Rabbit, this is not ideal.
Go all in with Firebase Cloud Messaging. We have a mobile app, and we plan to use it for push notifications when the user isn't connected, but would it be a good fit even if the user is connected?
What do you think is the best fit for our use case? Do you have any other idea?
I found a better solution : create a binding for each user but using only one queue on each node, then we route each messages to each user.
I have a node server accepting websocket connections from the clients. Each client can broadcast a message to all of the other clients.
UPDATE: I am using https://github.com/websockets/ws as my library of choice.
At the moment, the server has an array with all of the connections. Each connection has a tabId. When one of the client emits a message, I go through all of the connections and check: if the connection's tabId doesn't match, I send the message to the client.
For loading issues, I am facing the problem of having to have more than one server. So, there will be say two servers, each one with a number of clients.
How do I make sure that a message gets broadcast to all of the websocket clients, and not only the ones connected to the same server?
One possible solution I thought is to have the connections stored on a database, where each record has the tabId and the serverId. However, even a simple broadcast gets tricky as messages to "local" sockets are easy to broadcast (the socket is local and available) whereas messages to "remote" sockets are tricky, and would imply intra-server communication.
Is there a good pattern to solve this? Surely, this is something that people face every day.
You could use a messagequeue like RabbitMQ.
When a client logs in to your server, create a consumer which listens to a queue which will receive messages directed to that particular client. And when the clients are sending messages, just use a publisher to publish them to the recipients queue.
This way it doesn't matter and you don't need to know on which nodes the clients are on, or if they jump from a node to another.
When the socket.io client performs an (automatic) reconnection - as might happen if a mobile client went to sleep then woke up again - does the server get a reconnect event? Or does it just see a disconnection and fresh connection?
In either case is there a way to
identify that it's the same client e.g. by a unique client id that persists across connections
have the client automatically re-join any rooms it was in before
Or do I need to code that functionality manually e.g. by having the client supply the id or rooms itself on reconnection?
I had a read of the socket.io docs and can't see any list of events that the server might receive.
I'm using socket.io and nodejs,
I have a server and I use it as my nodeJS server. What I'm trying to do is moving clients according to messages sent as client -> server -> clients
For example; client1 sending a message "MOVE-RIGHT" to server. Server redirecting this message to all clients LIKE "MOVE-RIGHT-CLIENT1" and according to this message, all clients starting to move client1 to the right direction.
The problem is, all clients may have different latency according to their network status. For example, if server->client1 communication happens in 50 ms, server->client2 communication may happen in 250 ms. Therefore, client1 does this job nearly 200 ms earlier. So we can say that these two movements are not synchronized because one of them happens earlier than other ones.
As you know latency between clients and server may be different for each clients, and also it can be different for each message for the same client.
My question is, Which method should I use to synchronize these clients, to do their jobs at the same time. Is there any feature of socket.io or nodejs about this? What would you recommend for me?
I'm about to begin with socket.io and this is more of a theoretical question,
let's say that I want to send a message to a specific user with socket.io,
normally I would have to store the socketid with the relevant userid and when sending, get the socketid and send to.
but what if I have mutliple server processes running ? I'll have to make sure the correct server that the client is actually connected to does the sending. is it possible ?
For multiple server instances, you need to have a caching service (memcache, redis) for authentication and a central message queue service (stormMQ, rabbitMQ, AQ, java-based mq) where all your node instances bind to. Thus, a Node instance binds to the message queue for each client / channel / whatever, and all the other bound Node instances receive the messages and forward them to the client.
The problem is typically about how to play with a WebSocket cluster:
Several front-end servers which will be in charge of handling bidirectional connections with each client. They form the WebSocket cluster.
Several back-end servers which will be in charge of handling the business logic of your application.
Each time the back-end wants to inform the client, it will send a request to the WebSocket cluster which has the responsibility to communicate with the client.
A possible scenario:
Identify each WebSocket cluster's server with a unique id.
Identify each client with a unique id.
Each time a client will connect one of your WebSocket cluster's server, store its unique id along with the server's unique id in a a distributed key/value like database.
Thus you know which client is connected with which server.
The next time your back-end application wants to notify a client there are two possibilities:
The pair (clientId, serverId) is not present in the database and you cannot inform the client.
The pair (clientId, serverId) is present in the database, then you have to ask to the server identified by serverId to notify the client identified by clientId.
Notes:
Each WebSocket cluster's server can run a node.js instance supercharged with socket.io. It has to provide a route which will take the clientId as a parameter and will use socket.io to notify this client. Indeed, socket.io is aware of whcih client is using which socket on this server.
Every time your server will crash, you have to clean your database and remove all pairs which contain the server id.
Deploying a WebSocket cluster can be tedious, so you have commercial offers like Kaazing.
A good distributed key/value like database is Riak. It is better than Redis or Memcached for the above purpose because it can be easily distributed in a data-center and over several data-centers.