I'm trying to reduce socket.io bandwidth when using websockets. I switched to binary data, but looking in the browser developer console, the packets are sent as:
[ 'type of the packet (first argument of .emit)', associated data ]
I'm using only one packet type, so this causes unnecessary overhead - useless bytes are sent and whole thing is json encoded for no reason.
How can I get rid of the packet type and just send raw data?
socket.io is an abstraction on top of webSocket. In order to support the features it provides, it adds some overhead to the messages. The message name is one such piece of that overhead since it is a messaging system, not just a packet delivery system.
If you want to squeeze all bytes out of the transport, then you probably need to get rid of socket.io and just use a plain webSocket where you control more of the contents of each packet (though you will have to reimplement some things that socket.io does for you).
With socket.io in node.js, you can send binary by sending an ArrayBuffer or Buffer. In the browser, you can send binary by sending an ArrayBuffer or Blob.
Related
I have a websocket server made with express in nodejs
Websocket documentation states that websocket.send() can pass string or array buffer. Now I want to send a big array of objects (200k lines when formatted) over websocket to clients. What is the best way to send up data such as this.
What I've tried: I've tried sending it directly by stringifying it, Now this works completely fine but delay is very long.
So is there any way to send such a massive array data to clients while keeping speed intact? I think array buffers might help but I couldn't find any suggested examples
Also if this issue is code specific then let me know in comment so that I can share code snippets as well.
Usually sockets are used to handle real time messaging, sacrificing common http features with the goal of to be light and fast. Check:
How websockets can be faster than a simple HTTP request?
Transfer large amount of data goes against this. Check :
Sending large files over socket
Advice 1
Use socket to detect the real time notification of some change in crypto ticker should use socket.
After tha when client knows (thank to the socket) that there are a new or and updated crypto ticker (which is large), download it using common http instead socket as #jfriend00 also recommends in comments.
Also if data is large, you should use an approach to split the data and send chunk by chunk using common http, not sockets.
Advice 2
As #jfriend00 said and also the major file host services do, implement and algorithm to split the data and send part by part to the client.
If the chunk or part is small, maybe you could use sockets but this is a common feature , so use the known way: common http
Is there a way to log or view the actual bytestream being sent to the server when using either the grpc or #grpc/grpc-js clients in NodeJS?
I'm working with an opaque GRPC server that accepts my bytes when I stream them, but doesn't do what it's supposed to do. I'd like to view the actual bytes being sent to the server, as we suspect it's a problem with how the GRPC libraries are serializing 64 bit integers.
The GRPC_VERBOSITY=debug GRPC_TRACE=tcp,http,api,http2_stream_state env variables for the native grpc module haven't been helpful in this specific case -- they show part of one byte stream, but not the full byte-stream.
Even a "here's the place in the code where the serialization happens" would be useful.
The GRPC_VERBOSITY setting there is correct. If you are using TLS, you can see all of the data that is sent and received with GRPC_TRACE=secure_endpoint. If you are using plaintext connections, you can instead see it with GRPC_TRACE=tcp. In both cases, you will need to pick the data you are looking for out of the HTTP/2 framing, and it may show compressed messages, which would be essentially impossible to interpret.
Alternatively, if your setup allows it, you may want to try Wireshark. It should be able to handle the HTTP/2 framing for you, and I believe it has plugins to handle gRPC traffic specifically.
I'm writing a application in NodeJs where a client sends udp messages to a server with udp. I'm trying to find out how people normally handle different message types in NodeJs but can only find tons of examples of echo servers where the kind of message is not relevant.
The only example I have found so far is https://github.com/vbo/node-webkit-mp-game-template/tree/networking_1/networking
Maybe the best way is to send the udp messages as json?
User Datagram Protocol (UDP) is a network protocol and mechanism for sending short messages from one host to another without any guarantee of delivery. What you put in the message is entirely up to you.
While JSON can be used to encode your message, it suffers from two problems: it is not secure and is self-describing.
The first problem means that bad actors can easily see the content of your message while in flight and the second implies a substantial overhead for any message above and beyond its intended purpose.
Depending on your needs, a better choice might be to define your own binary protocol specific to your purpose using a node Buffer.
Another might be to use a more compact interchange format like thrift.
I'm currently studying for a project involving a web-server and some raspberrys.
The challenge would basically be to watch raspberry "status" on a web interface.
Raspberrys are connected to the Internet through a GSM connection (mostly 3G).
I'm developping using Node.js on both the clients and the server, and I'd like to use websockets through socket.io in order to watch the raspberry connection status (actually, this is more like watching the raspberry capability to upload data through my application), dealing with "connected" and "disconnected" events.
Would an always-alive websocket connection be reliable for such a use-case?
Are websocket designed-to (or reliable-for) staying opened?
Since this is a hard-testable situation, does anyone know a data-consumption estimate for an always-alive websocket?
If I'm going in a wrong way, does anyone ever worked on such a use-case via another reliable way?
Would an always-alive WebSocket connection be reliable for such a use-case ? Are WebSocket designed-to (or reliable-for) staying opened ?
Yes, WebSocket was designed to stay open and yes it's reliable for your use-case, a WebSocket connection is just a TCP connection which transmits data in frames.
Since this is a hard-testable situation, does anyone know an data-consumption estimate for an always-alive websocket ?
As I wrote, data in WebSocket connections is transmitted using frames, each frame has a header and the payload. The data sent from the client to the server is always masked and like this adds 4 bytes (the masking key) to each frame. The length of the header depends on the payload length:
2 bytes header for <=125 bytes payload
4 bytes header for <=65535 bytes payload
10 bytes header for <=2^64-1 bytes payload (rarely used)
Base Framing Protocol: https://www.rfc-editor.org/rfc/rfc6455#section-5.2
To keep the connection open, the server sends at a specific timeout (depends on the implementation, usually ~30 seconds) ping frames which are 2-127 bytes long, usually 2 bytes (only the header, without payload) and the client responds with pong frames which are also 2-127 bytes long.
I'm working actually on a file transfer system using socket.io and HTML5 file API.
https://github.com/xblaster/Nodjawnloader (stable branch)
The main problem I have is for huge file. Socket.io send me all packets in one huge transfer chunk and the Google Chrome javascript VM just crash when it receive around 70MB of packets.
Can I limit socket.io chunks for xhr-poll or jsonp calls ?
There isn't a mechanism for limiting the packet size on XHR or JSONP, but there is nothing stopping you from splitting up the file yourself and sending chunks. Then you can reassemble on the client side.