In my express/socket app, (which is running behind HAproxy server), I am using sticky session(cookie based) to route requests to same worker. I have total 16 processes running (8 /machine- 2 machines). Socket session data is being stored in Redis adapter.
The problem I have is, when an event is fired from server, client can't receive it. Inspite, it will keep throwing disconnection errors, after every few seconds (4-5) :
Update : It will only fire event if transport was opened when event was fired, which is getting closed instantly, and than restarting.
Can someone please suggest something on this..
Finally, I found the solution. It was timeout client which was set to too low in HAproxy config. Increasing it, fixed the issue.
Related
Background
We have a server that has socket.io 2.0.4. This server receives petitions from a stress script that simulates clients using socket.io-client 2.0.4.
The scrip simulates the creation of clients ( each client with its own socket ) that sends a petition and immediately dies after, using socket.disconnect();
Problem
During the first few of seconds all goes well. But every test reaches a point in which the script starts spitting out the following error:
connect_error: Error: websocket error
This means that the clients my script is creating are not connecting to the server because they are unable to connect.
This script creates 7 clients per second ( spaced evenly throughout the second ), each client makes 1 petition and then dies.
Research
At first I thought there was an issue with file descriptors and limits imposed by UNIX, since the server is in a Debian machine:
https://github.com/socketio/socket.io/issues/1393
After following these suggestions, the issue remained however.
Then I though maybe my test script was not connecting correctly, so I changed the connection options as in this discussion:
https://github.com/socketio/socket.io-client/issues/1097
Still, to no avail.
What could be wrong?
I see the machine's CPU's are constantly at 100% so I guess I am pounding the server with requests.
But if I am not mistaken, the server should simply accept more requests and process them when possible.
Questions
Is there a limit to the amount of connections a socket.io server can handle?
When making such stress tests one needs to be aware of protections and gate keepers.
In our case, our stack was deployed in AWS. So first, the AWS load balancers started blocking us because they thought the system was being DDOSed.
Then, the Debian system was getting flooded and it started refusing connections with SYN_FLOOD.
But after fixing that we were still having the error. Turns out we had to increase TCP connection's buffer and how TCP connections were being handled in the kernel.
Now it accepts all connections, but I wish no one the suffering we went through to find it out...
In Chrome, Socket IO seems to stop transmitting data. Is there an internal reason for this?
I've tried a very simple client and simple server side but consistently the server stops receiving any emits after 5 minute, will then reconnect and it's fine for another 5 minutes.
On top of the internal ping mechanism I have a polling mechanism which sends back session data every 20 seconds.
I don't use WebSocket with NodeJS or Socket.io but experienced the same behaviour with Jetty. It turns out that Jetty has an idle timeout default to 5 minutes (or 300 seconds) for all WebSocket's sessions. You could change the default idle timeout value to an appropriate value or ping/pong those connections before it timed out.
In my situation, I decided to use ping/pong as it also helps determine when the connection is no longer there. I observed that in some cases, connection was not closed even when the network is down.
According to engine.io (which is used by socket.io) docs, the server seems to have default pingInterval of 25 seconds. So unless you inadvertently disabled or changed default options, the ping/pong mechanism should be in place.
I have a production app that uses socket.io (node.js back-end)to distribute messages to all the logged in clients. Many of my users are experiencing disconnections from the socket.io server. The normal use case for a client is to keep the web app open the entire working day. Most of the time on the app in a work day time is spent idle, but the app is still open - until the socket.io connection is lost and then the app kicks them out.
Is there any way I can make the connection more reliable so my users are not constantly losing their connection to the socket.io server?
It appears that all we can do here is give you some debugging advice so that you might learn more about what is causing the problem. So, here's a list of things to look into.
Make sure that socket.io is configured for automatic reconnect. In the latest versions of socket.io, auto-reconnect defaults to on, but you may need to verify that no piece of code is turning it off.
Make sure the client is not going to sleep such that all network connections will become inactive get disconnected.
In a working client (before it has disconnected), use the Chrome debugger, Network tab, webSockets sub-tab to verify that you can see regular ping messages going between client and server. You will have to open the debug window, get to the network tab and then refresh your web page with that debug window open to start to see the network activity. You should see a funky looking URL that has ?EIO=3&transport=websocket&sid=xxxxxxxxxxxx in it. Click on that. Then click on the "Frames" sub-tag. At that point, you can watch individual websocket packets being sent. You should see tiny packets with length 1 every once in a while (these are the ping and pong keep-alive packets). There's a sample screen shot below that shows what you're looking for. If you aren't seeing these keep-alive packets, then you need to resolve why they aren't there (likely some socket.io configuration or version issue).
Since you mentioned that you can reproduce the situation, one thing you want to know is how is the socket getting closed (client-end initiated or server-end initiated). One way to gather info on this is to install a network analyzer on your client so you can literally watch every packet that goes over the network to/from your client. There are many different analyzers and many are free. I personally have used Fiddler, but I regularly hear people talking about WireShark. What you want to see is exactly what happens on the network when the client loses its connection. Does the client decide to send a close socket packet? Does the client receive a close socket packet from someone? What happens on the network at the time the connection is lost.
webSocket network view in Chrome Debugger
The most likely cause is one end closing a WebSocket due to inactivity. This is commonly done by load balancers, but there may be other culprits. The fix for this is to simply send a message every so often (I use 30 seconds, but depending on the issue you may be able to go higher) to every client. This will prevent it from appearing to be inactive and thus getting closed.
I have a web application where the user needs to be constantly connected. By default, socket.io will disconnect the connection after 60 seconds. I have 'reconnection' turned on though, so it is essentially closing and reopening the connection every minute. This can cause issues with feeds/notifications to my connected clients. Would it be safe to set this timeout to lets say, 10 minutes, or possibly higher? Is there a reason it is so low right now?
My guess is that you may be misinterpreting the 'close timeout' configuration. It does not cause the connection to be closed after 60 seconds. (Heartbeats would be pointless if clients constantly reconnected).
If a client disconnects, close timeout is the amount of time the server will wait before releasing resources associated with that connection. Essentially, this allows clients with intermittent connectivity issues to attempt to reconnect before the server has forgotten about them. Setting close timeout to ten minutes is probably a bad idea since it will tie up server resources.
If your clients are, in fact, disconnecting every 60 seconds, then, like samjm said, something else is wrong.
I don't believe your socket should disconnect after 60 seconds. I would investigate why that is actually happening. After handshaking correctly the socket should heartbeat and stay open indefinitely (barring network issues out of your control) until either the client or the server closes the connection, that is definitely my experience.
The fact that your connection is actually closing sounds like it may not be handshaking correctly, or heartbeats are not being received.
You might have already figured this out, but your socket might be disconnecting after 60 seconds because you're not sending a heartbeat ("2::") back to the server.
Here's some Python code that works with the websocket client module.
# on_message handles messages from the server
def on_message(ws, message):
if message[:3] == '2::':
ws.send('2::')
I have a socket.io client connected to a node.js server. If I kill node.js at the command line, the client immediately freezes (i.e., communication stops), but there is a ~20 second delay before the "disconnect" event is fired. Is this behavior by design? Is there a configuration option to reduce the delay in firing the disconnect event?
It appears that this behavior changed in a relatively recent (last 6 months) update of socket.io. Before the reconnect functionality was built in to socket.io itself, I implemented my own reconnect logic using a "disconnect" event handler and at that time the "disconnect" event fired almost instantly when server communication halted.
I think this is likely a design pattern. The client may be presuming the server is 'temporarily' unreachable (network trafic etc) and essentially will keep trying to reach it... until the client timeout kicks in.
I send a disconnect (socket.disconnect()) to the server directly from the client, and I don't get this issue.