I have a mini microservice project using NodeJS as backend and MQTT as the media for communicating between services. and I have a service that suppose to send an excel file to another service. Is there any way for MQTT to publish a file?
So far I only managed to send it as binary-data but had no idea what to do with that, or is there any way to recreate a file from binary-data in NodeJS?
This stackoverflow thread talks about MQTT byte limits.
The length of the actual topic string is at most 65536 bytes. This is a limit imposed by the mqtt spec, you can't change it. It is also worth noting that the topic is encoded with utf-8, so you may have less than 65536 characters available. The payload of the message is limited to 268,435,456 bytes. Again, this is defined by the spec.
If you exceed these limits, you need to break your file in chunks and use Base64 algorithm to encode them to ASCII. Make sure you send a hash of the whole file to check and guarantee that your file is consistent in the other side of the wire, after restoration.
This article does something similar using Python, in case you want to see some code. Hope it helps!
A file is just binary data and MQTT payloads are just binary data.
If you want to include meta data, e.g. a file name then you are going to have to come up with a data format to encode the filename along with the files content. That can be done any number of ways, be it in the topic you publish or by creating a data structure that includes the filename and the contents of the file.
since mqtt payload has limit, as #Fabio Manzano cited above, and it is impossible to publish binary data from even a small-sized file, i think i've managed to make this work by breaking it down into chunks (the binary data) and publish them separately. then merge them all back again when it finishes sending the last chunk.
and then do like what this thread suggests:
Writing image to local server
i've tried it and it succeed. thank you for the responses.
Related
I have a websocket server made with express in nodejs
Websocket documentation states that websocket.send() can pass string or array buffer. Now I want to send a big array of objects (200k lines when formatted) over websocket to clients. What is the best way to send up data such as this.
What I've tried: I've tried sending it directly by stringifying it, Now this works completely fine but delay is very long.
So is there any way to send such a massive array data to clients while keeping speed intact? I think array buffers might help but I couldn't find any suggested examples
Also if this issue is code specific then let me know in comment so that I can share code snippets as well.
Usually sockets are used to handle real time messaging, sacrificing common http features with the goal of to be light and fast. Check:
How websockets can be faster than a simple HTTP request?
Transfer large amount of data goes against this. Check :
Sending large files over socket
Advice 1
Use socket to detect the real time notification of some change in crypto ticker should use socket.
After tha when client knows (thank to the socket) that there are a new or and updated crypto ticker (which is large), download it using common http instead socket as #jfriend00 also recommends in comments.
Also if data is large, you should use an approach to split the data and send chunk by chunk using common http, not sockets.
Advice 2
As #jfriend00 said and also the major file host services do, implement and algorithm to split the data and send part by part to the client.
If the chunk or part is small, maybe you could use sockets but this is a common feature , so use the known way: common http
Is there a way to log or view the actual bytestream being sent to the server when using either the grpc or #grpc/grpc-js clients in NodeJS?
I'm working with an opaque GRPC server that accepts my bytes when I stream them, but doesn't do what it's supposed to do. I'd like to view the actual bytes being sent to the server, as we suspect it's a problem with how the GRPC libraries are serializing 64 bit integers.
The GRPC_VERBOSITY=debug GRPC_TRACE=tcp,http,api,http2_stream_state env variables for the native grpc module haven't been helpful in this specific case -- they show part of one byte stream, but not the full byte-stream.
Even a "here's the place in the code where the serialization happens" would be useful.
The GRPC_VERBOSITY setting there is correct. If you are using TLS, you can see all of the data that is sent and received with GRPC_TRACE=secure_endpoint. If you are using plaintext connections, you can instead see it with GRPC_TRACE=tcp. In both cases, you will need to pick the data you are looking for out of the HTTP/2 framing, and it may show compressed messages, which would be essentially impossible to interpret.
Alternatively, if your setup allows it, you may want to try Wireshark. It should be able to handle the HTTP/2 framing for you, and I believe it has plugins to handle gRPC traffic specifically.
Javascript front end, servicestack back end.
I'm using the latest version of dropzone.js to upload large image files (up to 50GB). The file is broken into many chunks and the server receives them one by one. When I receive the last chunk I know I have the complete file and can begin processing. But what if the chunks don't arrive in order? Once the data leaves the client is it possible, due to internet routing, that the chunks could be received out of order?
The server side (service stack) has no persistence between calls (that I'm aware of) so I can't count chunks received (at least not without writing to a database or something).
Is this something I need to be concerned with and what is the best way to handle it?
First you need to know how the file chunks are sent in order to know how to handle them, e.g. whether they're using standard HTTP multipart/formdata File Uploads in which case they'll be available in ServiceStack's Request.Files collection or some other way like sending raw bytes, in which case your Request DTO will need to implement IRequiresStream to access the raw unserialized bytes.
The server can't guarantee how clients will send it, if it's guaranteed that clients only sends the chunks sequentially then the server can assume that's how it will always be sent, but for all the server knows the chunks can be sent concurrently, unordered and in parallel which it may need to support.
I'd personally avoid uploading files in chunks over independent HTTP API requests as it adds a tonne of complexity, but if the files can be up to 50GB then you're going to need to come up with a bespoke solution.
You would handle the chunks just as you would any chunked data (e.g. imagine if you had to stitch responses from several services together manually). Because the files can be so large storing them in memory (like a ConcurrentDictionary) is not an option. If you have access to a cloud storage service you may want to upload the temporary chunks in there, otherwise you'd need to store them on disk. Ideally your solution should take advantage of the final data storage solution where the file will persist.
Otherwise a naive solution would be that the server should generate a unique key like a Guid before the client uploads the file that the client would need to send along with the chunk index and total chunks that needs to be sent. Each Service would then be writing that chunk directly to disk, first at a temp file path (Path.GetTempFileName()) then after the file is written move it to a format like /uploads/{unique-id}/{chunk-index}.dat.
Either at the end of every chunk upload request, you can check that your /uploads/{unique-id}/ directory has all the chunks, if it does start the process of stitching it up and creating a single file. Although a more robust way would be for the client to initiate the file stitching after it's finished uploading all the chunks, that way if the stitch fails you can just manually call the service that stitches the files again, instead of needing to have the client re-upload the file.
So I want to transfer sound bytes over a websocket from a phone to a server. However according to http://crossbar.io/docs/Features crossbar seems to only implement json and msgpack. Can I stil transfer binary messages over crossbar using some other way?
Also multiple crossbar clients (for eg )seems to only provide json and webpack as de/serialization formats. Am I missing something?
WAMP is primarily intended for transmission of messages, not large (binary) payloads. For small chunks you can encode the audio so that it can be part of a regular WAMP payload. For an example of this for a webcam image, see the Tessel camera example - https://github.com/crossbario/crossbarexamples/tree/master/iotcookbook/device/tessel/camera. This works fine in principle, though there is, of course, the encoding/decoding overhead.
I have just started using node.js, I'm running a node server with sockets.io and i need to send a buffer of bytes to the client.
I understand that this can be done by first translating the byte buffer to base64 and sending that, then translating it back on the client side. but i was wondering if there is a more elegant way of getting the byte stream to the client.
Socket.IO 1.0 Now supports Binary data transfer. Please have a look here . You can use Blob, ArrayBuffer and File.
https://github.com/binaryjs/binaryjs can be a solution. base 64 have ~30% of overhead size, so if you need to transfer large amount of data it will become inefficient.
There is also socket.io-stream https://github.com/nkzawa/socket.io-stream
It is little difficult to use binaryjs with socket.io.
Try deliveryjs
https://github.com/liamks/Delivery.js
which provides the means of communication between clients and server via socket.io.
However this module also uses the base64 conversion, which is a drawback.