I would like to ask question about Seaweedfs functionality.
In my case, I have a browser where is audio recorder.
Audio chunk files are transferred to NodeJS backend using socket.
When chunk is at backend I have to save it in storage and merge together.
Does Seaweedfs supports functionality where buffer data is received and merged in storage?
Or maybe there is a better solution how I can store audio chunks in live recording and merge them together getting full audio file at the end.
I added a filer REST API to append to a file. You can wait for the weekly release next Sunday.
Does that work for you?
//create or append the file
POST /path/to/file?op=append
PUT /path/to/file?op=append
Related
Is there an audio file format, where I can save all the individual chunks (recorded in javascript) while splitting them up at any point to save them to different files and have them still all playable?
Yes this is what WAV file does ... if you save the file to conform to WAV payload format you can play back the file you create as a WAV file even without file having its normal 44 byte header
I store raw audio data in arrays that can be sent to Web Audio API's AudioBuffer. The raw audio data arrays can be manipulated as you wish.
Specifics for obtaining the raw data are going to vary from language to language. I've not obtained raw data from within JavaScript. My experience comes from generating the data algorithmically or from reading .wav files with Java's AudioInputLine, and shipping the data to JavaScript via Thymefeaf.
I am trying to upload files to a s3 bucket using the node js aws-sdk V3.
I know I am supposed to be using the commmands: CreateMultipartUploadCommand,UploadPartCommandand so forth. But I can't find any working example of a full multipart upload.
Can anyone share any code samples?
Thanks in advance
I put together a bit of code for this: https://gist.github.com/kbanman/0aa36ffe415cdc6c44293bc3ddb6448e
The idea is to upload a part to S3 whenever we receive a chunk of data in the stream, and then finalize the upload when the stream is finished.
Complicating things is S3's minimum part size of 5MB on all but the last part in the series. This means that we need to buffer data until we can form those 5MB chunks. I accomplished this using a transformer that adds back-pressure on the content stream between each chunk upload.
Parallelization is also made difficult by the fact that S3 insists on receiving the parts in order (despite asking for the parts to be numbered).
I just started learning about Azure blob storage. I have come across various ways to upload and download the data. One thing that puzzles me to when to use what.
I am mainly interested in PutBlockAsync in conjunction with PutBlockListAsync and UploadFromStreamAsync.
As far as I understand when using PutBlockAsync it is up to the user to break the data into chunks and making sure each chunk is within the Azure block blob size limits. There is an id associated with each chunk that is uploaded. At the end, all the ids are committed.
When using UploadFromStreamAsync, how does this work? Who handles chunking the data and uploading it.
Why not convert the data into Stream and use UploadFromStreamAsync all the time and avoid two commits?
You can use fiddler, and observe what happens when use UploadFromStreamAsync.
If the file is larger(more than 256MB), such as 500MB, the Put Block and Put Block List api are called in the background(they are also called when use PutBlockAsync and PutBlockListAsync method)
If the file is small than 256MB, then it(UploadFromStreamAsync) will call the Put Blob api in the background.
I use UploadFromStreamAsync and uploading a file whose size is 600MB, then open the fidder.
Here are some findings from fidder:
1.The large file is broken into small size(4MB) one by one, and calls Put Block api in the background:
2.At the end, the Put Block List api will be called:
I need to let my users upload CSV data into my app. Data such as contacts, or products. Their are a number of web based client libraries that can handle the client side logic. What I am looking for is a fast reliable solution to get the data into a parse class.
I have not written any code. Right now I am trying to discover the best process to do this. I have played with parse batch save and know that is not reliable for 1000's of inserts. My thought is to upload the CSV store it in a parse class "uploads" and then have background job lift out say 100, or 1000 records at a time and insert them. Then send a notification when it is done.
Is this the best option, or has anybody found a simpler faster solution?
I have a node.js application using Highland stream. It consumes a data source and it needs to send data into multiple destinations. The trouble is, the destinations are discovered dynamically by reading the data and creating destination streams based on the data.
This means I cannot .fork the stream to add destinations (because a stream that has started consuming cannot be forked, right?)
Is there anything else I can do?
My current approach is to have a .consume where I create destination streams and write data into the destinations. The result is ugly and messy.