So I have a socket.io server which works well. It's very simple: kind of mimicking screen sharing as it broadcasts one clients position on the page to the other, who catches it and moves to said location etc. All of this works fine, but because of the way I'm catching movement, its possible (and quite common) for it to be sent too many messages at once, making it impossible for the other client to keep up.
I was wondering if there is a way to make socket.io 'sleep' or 'wait' for a certain interval, ignore the messages sent during this interval without returning an error, and then begin listening again?
It is feasible to implement this in each client (and this may be the better option), but I just wanted to know if this is possible on the server side too.
Use volatile messages. If there are too much messages, they will just be dropped to go again with real time messages.
socket.volatile.emit('msg', data);
From socket.io website :
Sending volatile messages.
Sometimes certain messages can be dropped. Let's say you have an app that shows realtime tweets for the keyword `bieber`.
If a certain client is not ready to receive messages (because of network slowness or other issues, or because he's connected through long polling and is in the middle of a request-response cycle), if he doesn't receive ALL the tweets related to bieber your application won't suffer.
In that case, you might want to send those messages as volatile messages.
Related
I am having serious problems to make messages delivery fail proof in a chat system.
Having several node.js and live communication via websocket to the clients, I use rabbit to callback the correct consumer at a specific node.
I declare my queues as {durable: true, prefetch:1, expires: 2*3600*1000, autoDelete: true}
consumerOption is {noAck: false, exclusive: false}
Once I receive a message from the server, I callback the server, get the message, and use message.ack(false)
Sometimes, a message appears with a pendent ACK in rabbit and as I would expect, the consumers stop being callbacked.
Here is my overall strategy:
1- when socket disconnects, I recover the queue using queue.recover() during the the reconnection/connection (more frequent).
2- When I send a message to the server and not receive it back, I send a message to the server to recover the queue.
3- I use the socket callback function to send the ack confirmation. On the server, I use message.ack(false) The server keeps a hashmap {[ackCode: string]: RabbitMessage} and I send the ackCode back to the server, so it can retrieve the correct message and ack it.
5- If client is not receiving any message for 2 minutes, I ask to the server to recover the queue.
The step 5 should not exist but even with this step, sometimes I send a recover queue request to the server, the server executes the command, but nothing happens and chat is freeze.
These are very difficult events to debug. I am using a Typescript library which is 3 year without any commit and this could be one of the causes.
Regarding the strategy, is it correct? Any idea on what I could be facing?
What I've learned and why I think that I couldn't use rabbit to solve the specific problem mentioned in the original post.
The domain: A "chat" where the message order is very important (some are chains) and we must be sure that the message will be delivery if/when the client is online.
The problem: We have several node.js servers, sockets are spread among them. Sockets falls all time, and it is common to a client connection that was in the first server be connected again in another. We don't use cookies, session affinity by IP won't handle the issue.
Limitations: That being said, I can't activate a consumer that is currently activated in another server, so if a customer Queue is tied to server 1 I can't activate it in server 2. And all the messages that need to be sent are tied to this specific queue.
Another limitation is that I don't have an easy way to consume queues, re-queue, to know in advance how much not ack messages I have in the queue, aggregate them and bulk send them via socket.
The solution: I am no longer using {noAck: false} and I am controlling the ack in a Redis queue. Thus, I am using Rabbit as a pub-sub, to callback the correct consumers to send the message using the socket. Rabbit wake me up, first thing I do is to put the message at the end of a redis queue. When I send a message via socket, I always start sending the messages from the beginning of the queue, regardless of the message that have just woke me up. I send the message, wait for the callback event, If it is not ok, I re-queue the messages,
After decoupling the pub-sub from the queue/ack control, I can now easily change my rabbit pub/sub from one server to another (declaring using socket.id and no more with the client queue), with no concern of loosing any message. Also, now I am capable of much more advanced operations on my queue.
As my use case don't allow me to use the full power of exchanges/binds (i have complex routing rules), I am evaluating the possibility of changing from rabbit to redis pub/sub, but in this case, I would continue to differentiate pub/sub from the queue.
After more than a month trying to make rabbit working in this scenery, I think that I was using a good technology to the wrong use case. It is much simpler now.
So i currently have a chat system running NodeJS that passes messages via rabbit and each connected user has their own unique queue that subscribed and only listening to messages (for only them). The backend can also use this chat pipeline to communicate other system messages like notifications/friend requests and other user event driven information.
Currently the backend would have to loop and publish each message 1 by 1 per user even if the payload of the message is the same for let's say 1000 users. I would like to get away from that and be able to send the same message to multiple different users but not EVERY user who's connected.
(example : notifying certain users their friend has come online).
I considered implementing a rabbit queue system where all messages are pooled into the same queue and instead of rabbit sending all user queues node takes these messages and emit's the message to the appropriate user via socket connections (to whoever is online).
Proposed - infrastructure
This way the backend does not need to loop for 100s and 1000s of users and can send a single payload containing all users this message should go to. I do plan to cluster the nodejs servers together.
I was also wondering since ive never done this in a production environment, will i need to track each socketID.
Potential pitfalls i've identified so far:
slower since 1000s of messages can pile up in a single queue.
manually storing socket IDs to manually trasmit to users.
offloading routing to NodeJS instead of RabbitMQ
Has anyone done anything like this before? If so, what are your recommendations. Is it better to scale with user unique queues, or pool all grouped messages for all users into smaller (but larger pools) of queues.
as a general rule, queue-per-user is an anti-pattern. there are some valid uses of this, but i've never seen it be a good idea for a chat app (in spite of all the demos that use this example)
RabbitMQ can be a great tool for facilitating the delivery of messages between systems, but it shouldn't be used to push messages to users.
I considered implementing a rabbit queue system where all messages are pooled into the same queue and instead of rabbit sending all user queues node takes these messages and emit's the message to the appropriate user via socket connections (to whoever is online).
this is heading down the right direction, but you have to remember that RabbitMQ is not a database (see previous link, again).
you can't randomly seek specific messages that are sitting in the queue and then leave them there. they are first in, first out.
in a chat app, i would have rabbitmq handling the message delivery between your systems, but not involved in delivery to the user.
your thoughts on using web sockets are going to be the direction you want to head for this. either that, or Server Sent Events.
if you need persistence of messages (history, search, last-viewed location, etc) then use a database for that. keep a timestamp or other marker of where the user left off, and push messages to them starting at that spot.
you're concerns about tracking sockets for the users are definitely something to think about.
if you have multiple instances of your node server running sockets with different users connected, you'll need a way to know which users are connected to which node server.
this may be a good use case for rabbitmq - but not in a queue-per-user manner. rather, in a binding-per-user. you could have each node server create a queue to receive messages from the exchange where messages are published. the node server would then create a binding between the exchange and queue based on the user id that is logged in to that particular node server
this could lead to an overwhelming number of bindings in rmq, though.
you may need a more intelligent method of tracking which server has which users connected, or just ignore that entirely and broadcast every message to every node server. in that case, each server would publish an event through the websocket based on the who the message should be delivered to.
if you're using a smart enough websocket library, it will only send the message to the people that need it. socket.io did this, i know, and i'm sure other websocket libraries are smart like this, as well.
...
I probably haven't given you a concrete answer to your situation, and I'm sure you have a lot more context to consider. hopefully this will get you down the right path, though.
How can one prevent Pusher from automatically pushing all the piled up messages to the client after the client eventually goes online after being offline, i.e. after the client re-establishes the connection?
After exchanging messages with a Pusher support enginner, the issue became more clear.
The connection may still be opened even when the laptop gets asleep (this behaviour varies among computers). Thus, after waking up, it may still be connected. (This is exactly what happened in my case so that everything looked like Pusher pushed the accumulated messages.)
However, the default activity timeout is 120s, and the time to wait for a pong response before closing the connection is 30s. So, allowing it around three minutes would make the client disconnect completely, and the behaviour I encountered would not take place.
Pusher doesn't presently buffer messages to be delivered upon reconnection. So the functionality described in the questions isn't something an application needs to consider right now.
Future releases may contains something called Event Buffer which will offer this functionality. Documentation will be released around that time to detail how to avoid receiving buffered events.
Before I dive into the code, can someone tell me if there is any documentation available for confirmed delivery in Socket.IO?
Here's what I've been able to glean so far:
A callback can be provided to be invoked when and if a message is acknowledged
There is a special mode "volatile" that does not guarantee delivery
There is a default mode that is not "volatile"
This leaves me with some questions:
If a message is not volatile, how is it handled? Will it be buffered indefinitely?
Is there any way to be notified if a message can't be delivered within a reasonable amount of time?
Is there any way to unbuffer a message if I want to give up?
I'm at a bit of a loss as to how Socket.IO can be used in a time sensitive application without falling back to volatile mode and using an external ACK layer that can provide failure events and some level of configurability. Or am I missing something?
TL;DR You can't have reliable confirmed delivery unless you're willing to wait until the universe dies.
The delivery confirmation you seek is related to the theoretical Two Generals Problem, which is also discussed in this SO answer.
TCP manages the reliability problem by guaranteeing delivery after infinite retries. We live in a finite universe, so the word "guarantee" is theoretically dubious :-)
Theory aside, consider this: engine.io, the underpinnings of socket.io 1.x, uses the following transports:
WebSocket
FlashSocket
XHR polling
JSONP polling
Each of those transports is based upon TCP, and TCP is reliable. So as long as connections stay connected and transports don't change, each individual socket.io message or event should be reliable. However, two things can happen on the fly:
engine.io can change transports
socket.io can reconnect in case the underlying transport disconnects
So what happens when a client or your server squirts off a few messages while the plumbing is being fiddled with like that? It doesn't say in either the engine.io protocol or the socket.io protocol (at versions 3 and 4, respectively, as of this writing).
As you suggest in your comments, there is some acknowledgement logic in the implementation. But even simple digital communications has notrivial behavior, so I do not trust an unsupervised socket.io connection for reliable delivery for mission- or safety-critical operations. That won't change until reliable delivery is part of their protocol and their methods have been independently and formally verified.
You're welcome to adopt my policies:
Number my messages
Ask for a resend when in doubt
Do not mutate my state - client or server - unless I know I'm ready
In Short:
Guaranteed message delivery acknowledgement is proven impossible, but TCP guarantees delivery and order given "infinite" retries. I'm less confident about socket.io messages, but they're really powerful and easy to use so I just use them with care.
I ensured delivery using different strategies
I send data using socket including nonce in the message to prevent repeated message errors
The other party sends a confirmation of recived meassage or i resend after x seconds
I used a REST call by the client every 30 seconds to request all new messages sent by server to catch any dropped messages during transport
I'm in process of making realtime multiplayer racing game. Now I need help writing game logics in Node.js TCP (net) server. I don't know if it's possible, I don't know if i'm doing that right, but I'm trying my best. I know it's hard to understand my broken english, so i made this "painting" :)
Thank you for your time
To elaborate on driushkin's answer, you should use remote procedure calls (RPC) and an event queue. This works like in the image you've posted, where each packet represents a 'command' or RPC with some arguments (i.e. movement direction). You'll also need an event queue to make sure RPCs are executed in order and on time. This will require a timestamp or framecount for each command to be executed on (at some point in the future, in a simple scheme), and synchronized watches (World War II style).
You might notice one critical weakness in this scheme: RPC messages can be late (arrive after the time they should be applied) due to network latency, malicious users, etc. In a simple scheme, late RPCs are dropped. This is fine since all clients (even the originator!) wait for the server to send an RPC before acting (if the originating client didn't wait for the server message, his game state would be out of sync with the server, and your game would be broken).
Consider the impact of lag on such a scheme. Let's say the lag for Client A to the server was 100ms, and the return trip was also 100ms. This means that client input goes like:
Client A presses key, and sends RPC to server, but doesn't add it locally (0ms)
Server receives and rebroadcasts RPC (100ms)
Client A receives his own event, and now finally adds it to his event queue for processing (200ms)
As you can see, the client reacts to his own event 1/5 of a second after he presses the key. This is with fairly nice 100ms lag. Transoceanic lag can easily be over 200ms each way, and dialup connections (rare, but still existent today) can have lag spikes > 500ms. None of this matters if you're playing on a LAN or something similar, but on the internet this unresponsiveness could be unbearable.
This is where the notion of client side prediction (CSP) comes in. CSP is made out to be big and scary, but implemented correctly and thoughtfully it's actually very simple. The interesting feature of CSP is that clients can process their input immediately (the client predicts what will happen). Of course, the client can (and often will) be wrong. This means that the client will need a way of applying corrections from the Server. Which means you'll need a way for the server to validate, reject, or amend RPC requests from clients, as well as a way to serialize the gamestate (so it can be restored as a base point to resimulate from).
There are lots of good resources about doing this. I like http://www.gabrielgambetta.com/?p=22 in particular, but you should really look for a good multiplayer game programming book.
I also have to suggest socket.io, even after reading your comments regarding Flex and AS3. The ease of use (and simple integration with node) make it one of the best (the best?) option(s) for network gaming over HTTP that I've ever used. I'd make whatever adjustments necessary to be able to use it. I believe that AIR/AS3 has at least one WebSockets library, even if socket.io itself isn't available.
This sounds like something socket.io would be great for. It's a library that gives you real time possibilities on the browser and on your server.
You can model this in commands in events: client sends command move to the server, then server validates this command and if everything is ok, he publishes event is moving.
In your case, there is probably no need for different responses to P1 (ok, you can move) and the rest (P1 is moving), the latter suffices in both cases. The is moving event should contain all necessary info (like current position, velocity, etc).
In this simplest form, the one issuing command would experience some lag until the event from server arrives, and to avoid that you could start moving immediately, and then apply some compensating actions if necessary when event arrives. But this can get complicated.