Is there a way to log or view the actual bytestream being sent to the server when using either the grpc or #grpc/grpc-js clients in NodeJS?
I'm working with an opaque GRPC server that accepts my bytes when I stream them, but doesn't do what it's supposed to do. I'd like to view the actual bytes being sent to the server, as we suspect it's a problem with how the GRPC libraries are serializing 64 bit integers.
The GRPC_VERBOSITY=debug GRPC_TRACE=tcp,http,api,http2_stream_state env variables for the native grpc module haven't been helpful in this specific case -- they show part of one byte stream, but not the full byte-stream.
Even a "here's the place in the code where the serialization happens" would be useful.
The GRPC_VERBOSITY setting there is correct. If you are using TLS, you can see all of the data that is sent and received with GRPC_TRACE=secure_endpoint. If you are using plaintext connections, you can instead see it with GRPC_TRACE=tcp. In both cases, you will need to pick the data you are looking for out of the HTTP/2 framing, and it may show compressed messages, which would be essentially impossible to interpret.
Alternatively, if your setup allows it, you may want to try Wireshark. It should be able to handle the HTTP/2 framing for you, and I believe it has plugins to handle gRPC traffic specifically.
Related
I have a mini microservice project using NodeJS as backend and MQTT as the media for communicating between services. and I have a service that suppose to send an excel file to another service. Is there any way for MQTT to publish a file?
So far I only managed to send it as binary-data but had no idea what to do with that, or is there any way to recreate a file from binary-data in NodeJS?
This stackoverflow thread talks about MQTT byte limits.
The length of the actual topic string is at most 65536 bytes. This is a limit imposed by the mqtt spec, you can't change it. It is also worth noting that the topic is encoded with utf-8, so you may have less than 65536 characters available. The payload of the message is limited to 268,435,456 bytes. Again, this is defined by the spec.
If you exceed these limits, you need to break your file in chunks and use Base64 algorithm to encode them to ASCII. Make sure you send a hash of the whole file to check and guarantee that your file is consistent in the other side of the wire, after restoration.
This article does something similar using Python, in case you want to see some code. Hope it helps!
A file is just binary data and MQTT payloads are just binary data.
If you want to include meta data, e.g. a file name then you are going to have to come up with a data format to encode the filename along with the files content. That can be done any number of ways, be it in the topic you publish or by creating a data structure that includes the filename and the contents of the file.
since mqtt payload has limit, as #Fabio Manzano cited above, and it is impossible to publish binary data from even a small-sized file, i think i've managed to make this work by breaking it down into chunks (the binary data) and publish them separately. then merge them all back again when it finishes sending the last chunk.
and then do like what this thread suggests:
Writing image to local server
i've tried it and it succeed. thank you for the responses.
I'm writing a application in NodeJs where a client sends udp messages to a server with udp. I'm trying to find out how people normally handle different message types in NodeJs but can only find tons of examples of echo servers where the kind of message is not relevant.
The only example I have found so far is https://github.com/vbo/node-webkit-mp-game-template/tree/networking_1/networking
Maybe the best way is to send the udp messages as json?
User Datagram Protocol (UDP) is a network protocol and mechanism for sending short messages from one host to another without any guarantee of delivery. What you put in the message is entirely up to you.
While JSON can be used to encode your message, it suffers from two problems: it is not secure and is self-describing.
The first problem means that bad actors can easily see the content of your message while in flight and the second implies a substantial overhead for any message above and beyond its intended purpose.
Depending on your needs, a better choice might be to define your own binary protocol specific to your purpose using a node Buffer.
Another might be to use a more compact interchange format like thrift.
I'm trying to reduce socket.io bandwidth when using websockets. I switched to binary data, but looking in the browser developer console, the packets are sent as:
[ 'type of the packet (first argument of .emit)', associated data ]
I'm using only one packet type, so this causes unnecessary overhead - useless bytes are sent and whole thing is json encoded for no reason.
How can I get rid of the packet type and just send raw data?
socket.io is an abstraction on top of webSocket. In order to support the features it provides, it adds some overhead to the messages. The message name is one such piece of that overhead since it is a messaging system, not just a packet delivery system.
If you want to squeeze all bytes out of the transport, then you probably need to get rid of socket.io and just use a plain webSocket where you control more of the contents of each packet (though you will have to reimplement some things that socket.io does for you).
With socket.io in node.js, you can send binary by sending an ArrayBuffer or Buffer. In the browser, you can send binary by sending an ArrayBuffer or Blob.
in noVNC there is a way to send a string from client to server using the api 'send_string' which is implemented inside websock.js, one thing i noticed is sending string this way is it is not RFB encoded(correct me if i am wrong), so the advantage in this case is commands can be send to intermediate proxy which intern connect to VNC server...
Now my query is .. is there a way for this intermediate proxy to send back some string back to vnc client... means it is not RFB encoded, so will be handled differently # client
Thanks in advance
The websock.js library is the client-side part of the websockify project. The purpose of websockify is to bridge between WebSockets (which are message based) and normal TCP sockets (which are stream based).
The API that websock.js presents is a streaming API rather than a message based one. In addition, websockify/websock.js enables sending/receiving of binary data to the remote target even if the older WebSocket protocol (Hixie) is used which does not natively support binary data.
The send_string function is a convenience function so that you don't have to convert a string to an array form of the data before sending it. The data is still sent to the final target (it is not intercepted by websockify). The beginning of the RFB handshake is string based and so noVNC uses send_string in a couple of places (again, as a convenience).
If you want to have out-of-band communication between websock.js and websockify then you will need to modify both sides, perhaps by adding an initial byte to every message that indicates whether it is out-of-band signalling or part of the in-band stream. It is not builtin functionality.
Disclaimer: I made noVNC and websockify.
I have just started using node.js, I'm running a node server with sockets.io and i need to send a buffer of bytes to the client.
I understand that this can be done by first translating the byte buffer to base64 and sending that, then translating it back on the client side. but i was wondering if there is a more elegant way of getting the byte stream to the client.
Socket.IO 1.0 Now supports Binary data transfer. Please have a look here . You can use Blob, ArrayBuffer and File.
https://github.com/binaryjs/binaryjs can be a solution. base 64 have ~30% of overhead size, so if you need to transfer large amount of data it will become inefficient.
There is also socket.io-stream https://github.com/nkzawa/socket.io-stream
It is little difficult to use binaryjs with socket.io.
Try deliveryjs
https://github.com/liamks/Delivery.js
which provides the means of communication between clients and server via socket.io.
However this module also uses the base64 conversion, which is a drawback.