I'm testing out postgres binary abilities by storing some mp3 data in a table. I've read you're supposed to store them in an external filesystem like S3, but for various reasons I don't want to do that right now.
So, for now I'd like to test storing files in the db. The mp3 files are TTS mp3 files from a third-party and I've stored them in a postgres table. This is working ok. But how do I serve them to the client? In other words:
client http requests the files.
node requests (pg-promise) the records (one or many).
the data arrives from db to node in binary format.
??? Do I have to convert it to a mp3 file before sending? Can I send the binary file directly? Which would be better?
client receives file(s)
client queues files in order for playing audio.
My main question is whether I need to convert the binary record I received from postgres before sending, and how to do that?
Related
I would like to allow my users to upload large files <1GB to my database. I am using a database since storing raw files can be dangerous and I would like to have a single source of state in my system since its meant to be serverless.
Now the VPS am I am planning to run it on has limited ram. And multiple users should of course be able to upload simultaneously.
So in order to not exceed this ram, I would need to either
stream the image into the database as it is being uploaded from user
or I would need to first stream it into a file using something like multer and then stream it from the file into PostgreSQL as a BLOB
So is there a way to do this using pg-promise? Stream a file into the database without ever loading the whole thing into ram?
I can save the file to disk with formidable and then send the file bits to mongo with node, but how can I just handle streaming the file bits directly to mongo?
I don't need gridfs, these are small files. Just want to write them to the normal store.
Use options.fileWriteStreamHandler to setup your own stream. Then write to mongodb if the API accepts a stream
Javascript front end, servicestack back end.
I'm using the latest version of dropzone.js to upload large image files (up to 50GB). The file is broken into many chunks and the server receives them one by one. When I receive the last chunk I know I have the complete file and can begin processing. But what if the chunks don't arrive in order? Once the data leaves the client is it possible, due to internet routing, that the chunks could be received out of order?
The server side (service stack) has no persistence between calls (that I'm aware of) so I can't count chunks received (at least not without writing to a database or something).
Is this something I need to be concerned with and what is the best way to handle it?
First you need to know how the file chunks are sent in order to know how to handle them, e.g. whether they're using standard HTTP multipart/formdata File Uploads in which case they'll be available in ServiceStack's Request.Files collection or some other way like sending raw bytes, in which case your Request DTO will need to implement IRequiresStream to access the raw unserialized bytes.
The server can't guarantee how clients will send it, if it's guaranteed that clients only sends the chunks sequentially then the server can assume that's how it will always be sent, but for all the server knows the chunks can be sent concurrently, unordered and in parallel which it may need to support.
I'd personally avoid uploading files in chunks over independent HTTP API requests as it adds a tonne of complexity, but if the files can be up to 50GB then you're going to need to come up with a bespoke solution.
You would handle the chunks just as you would any chunked data (e.g. imagine if you had to stitch responses from several services together manually). Because the files can be so large storing them in memory (like a ConcurrentDictionary) is not an option. If you have access to a cloud storage service you may want to upload the temporary chunks in there, otherwise you'd need to store them on disk. Ideally your solution should take advantage of the final data storage solution where the file will persist.
Otherwise a naive solution would be that the server should generate a unique key like a Guid before the client uploads the file that the client would need to send along with the chunk index and total chunks that needs to be sent. Each Service would then be writing that chunk directly to disk, first at a temp file path (Path.GetTempFileName()) then after the file is written move it to a format like /uploads/{unique-id}/{chunk-index}.dat.
Either at the end of every chunk upload request, you can check that your /uploads/{unique-id}/ directory has all the chunks, if it does start the process of stitching it up and creating a single file. Although a more robust way would be for the client to initiate the file stitching after it's finished uploading all the chunks, that way if the stitch fails you can just manually call the service that stitches the files again, instead of needing to have the client re-upload the file.
In Node.js how can I stream multiple files in one response stream? I want to make a single api call from the browser application and in the response I should be able to send multiple files from the server. How can I do this in Node.js ? Please share some hints or code samples.
The files are stored as blob data in MongoDB and I use the module called gridfs-stream to read the files.
I have a mini microservice project using NodeJS as backend and MQTT as the media for communicating between services. and I have a service that suppose to send an excel file to another service. Is there any way for MQTT to publish a file?
So far I only managed to send it as binary-data but had no idea what to do with that, or is there any way to recreate a file from binary-data in NodeJS?
This stackoverflow thread talks about MQTT byte limits.
The length of the actual topic string is at most 65536 bytes. This is a limit imposed by the mqtt spec, you can't change it. It is also worth noting that the topic is encoded with utf-8, so you may have less than 65536 characters available. The payload of the message is limited to 268,435,456 bytes. Again, this is defined by the spec.
If you exceed these limits, you need to break your file in chunks and use Base64 algorithm to encode them to ASCII. Make sure you send a hash of the whole file to check and guarantee that your file is consistent in the other side of the wire, after restoration.
This article does something similar using Python, in case you want to see some code. Hope it helps!
A file is just binary data and MQTT payloads are just binary data.
If you want to include meta data, e.g. a file name then you are going to have to come up with a data format to encode the filename along with the files content. That can be done any number of ways, be it in the topic you publish or by creating a data structure that includes the filename and the contents of the file.
since mqtt payload has limit, as #Fabio Manzano cited above, and it is impossible to publish binary data from even a small-sized file, i think i've managed to make this work by breaking it down into chunks (the binary data) and publish them separately. then merge them all back again when it finishes sending the last chunk.
and then do like what this thread suggests:
Writing image to local server
i've tried it and it succeed. thank you for the responses.