I am looking for a way to uploading files to Azure Blob Storage.
I found azure-storage npm package.
But I'm having a problem with 'createBlockBlobFromStream' method.
I dont know how create stream from Uint8Array.
xhr.onload = function (e) {
if (this.status == 200) {
// Note: .response instead of .responseText
const blob = new Blob([this.response]);
console.log(audios[i].file);
const reader = new FileReader();
reader.onload = function () {
const data = new Uint8Array(reader.result);
Meteor.call('uploadFilesViaSDK', data);
};
reader.readAsArrayBuffer(blob);
}
};
I`m trying to migarate files from S3 to Azure blob. Thats why I dowload files from S3, and than read it as ArrayBuffer and convert it to Uint8Array.
And now I am looking way how to upload this data to azure via azure.createBlockBlobFromStream meyhod.
Specifically, I need an example of creating a stream from Uint8Array.
I'll be grateful for any answer
In addition to an approach provided by Gaurav, once you created a stream from Uint8Array by using streamifier, you can use createWriteStreamToBlockBlob function to write to a block blob from a stream. With that you are able to transmit stream by calling .pipe():
streamifier.createReadStream(new Buffer(uint8)).pipe(blobSvc.createWriteStreamToBlockBlob('mycontainer', 'test.txt'));
Related
const blobServiceClient = await BlobServiceClient.fromConnectionString(connectionString);
const containerClient = await blobServiceClient.getContainerClient(container);
const blockBlobClient = containerClient.getBlockBlobClient(fileName);
const uploadBlobResponse = await blockBlobClient.upload(content, content.length);
console.log(uploadBlobResponse);
console.log(`FIle upload successfully on cloud ${uploadBlobResponse.requestId}`);
i am trying like this, but blockBlobClient.upload() needs content, i converted file in base64 and sent it into content, but i am having issue, file is uplaoded but corrupted. any help please.
Check the SDK, the upload method construct is upload(HttpRequestBody, number, BlockBlobUploadOptions), the content is HttpRequestBody, check the parameter it requires
Blob, string, ArrayBuffer, ArrayBufferView or a function which returns
a new Readable stream whose offset is from data source beginning.
So maybe you could try uploadFile, just use the file path to upload, I have tried this way it works.
Also, you could use uploadStream to upload the file readable stream.
I upload audio files to my azure blob, and I would like to know if it is possible to download only the parts I want of the audio ?
by the way, I'm using nodeJs
Thank you !
I upload audio files to my azure blob, and I would like to know if it
is possible to download only the parts I want of the audio?
Yes, it is certainly possible to download a portion of a blob. Azure Blobs support reading a range of bytes. For example, let's say you want to download only first 1KB of data from a file. This is how you would download that data:
import azure from 'azure-storage';
const ms = require('memory-streams');
const chunkStart = 0;
const chunkEnd = 1023;
const connectionString = 'your-azure-storage-connection-string';
const blobService = azure.createBlobService(connectionString);
const writableStream = new ms.WritableStream({
highWaterMark: (chunk.end - chunk.start) * 2,
writableHighWaterMark: (chunk.end - chunk.start) * 2,
});
const requestOptions = {
rangeStart: chunkStart,
rangeEnd: chunkEnd
};
blobService.getBlobToStream('container-name', 'blob-name', writableStream, requestOptions, (error, result, response) => {
if (error) {
console.log('Error occurred while downloading chunk!');
} else {
const dataBuffer = writableStream.toBuffer();
console.log('Blob chunk downloaded!');
}
});
Considering you mentioned that you're storing an audio file, please note that you can't instruct Azure Storage to download "x" duration of audio (e.g. download first 30 seconds of audio) as Azure Storage treats all blobs as a collection of bytes and has no idea if the file is an audio file or something else.
I have application that saves all uploaded files to S3. But sometimes I need save data to local file storage.
Is there some library that could do something like this:
var storage = new Storage(local);
storage.save(hello.mp3);
or
var storage = new Storage(s3);
storage.save(hello.mp3);
you are just looking at Factory pattern ?
var Storage = function (type) {
if (type === "local") return new LocalStorage();
if (type === "s3") return new S3Storage();
}
of course, LocalStorage and S3Storage must share the same method save()
What is the difference between OpenReadAsync and DownloadToStreamAsync functions of CloudBlockBlob in the Azure blob storage? Searched in google but could not find an answer.
Both OpenReadAsync and DownloadToStreamAsync could initiate an asynchronous operation for you to retrieve the blob stream.
Based on my testing, you could have a better understanding of them by the following sections:
Basic Concepts
DownloadToStreamAsync：Initiates an asynchronous operation to download the contents of a blob to a stream.
OpenReadAsync：Initiates an asynchronous operation to download the contents of a blob to a stream.
Usage
a) DownloadToStreamAsync
Sample Code:
using (var fs = new FileStream(<yourLocalFilePath>, FileMode.Create))
{
await blob.DownloadToStreamAsync(fs);
}
b) OpenReadAsync
Sample Code:
//Set buffer for reading from a blob stream, the default value is 4MB.
blob.StreamMinimumReadSizeInBytes=10*1024*1024; //10MB
using (var blobStream = await blob.OpenReadAsync())
{
using (var fs = new FileStream(localFile, FileMode.Create))
{
await blobStream.CopyToAsync(fs);
}
}
Capturing Network requests via Fiddler
a) DownloadToStreamAsync
b) OpenReadAsync
According to the above, DownloadToStreamAsync just sends one get request for retrieving blob stream, while OpenReadAsync sends more than one request to retrieving blob stream based on the “Blob.StreamMinimumReadSizeInBytes” you have set or by default value.
The difference between DownloadToStreamAsync and OpenReadAsync is that DownloadToStreamAsync will download the contents of the blob to the stream before returning, but OpenReadAsync will not trigger a download until the stream is consumed.
For example, if using this to return a file stream from an ASP.NET core service, you should use OpenReadAsync and not DownloadToStreamAsync:
Example with DownloadToStreamAsync (not recommended in this case):
Stream target = new MemoryStream(); // Could be FileStream
await blob.DownloadToStreamAsync(target); // Returns when streaming (downloading) is finished. This requires the whole blob to be kept in memory before returning!
_logger.Log(LogLevel.Debug, $"DownloadToStreamAsync: Length: {target.Length} Position: {target.Position}"); // Output: DownloadToStreamAsync: Length: 517000 Position: 517000
target.Position = 0; // Rewind before returning Stream:
return File(target, contentType: blob.Properties.ContentType, fileDownloadName: blob.Name, lastModified: blob.Properties.LastModified, entityTag: null);
Example with OpenReadAsync (recommended in this case):
// Do NOT put the stream in a using (or close it), as this will close the stream before ASP.NET finish consuming it.
Stream blobStream = await blob.OpenReadAsync(); // Returns when the stream has been opened
_logger.Log(LogLevel.Debug, $"OpenReadAsync: Length: {blobStream.Length} Position: {blobStream.Position}"); // Output: OpenReadAsync: Length: 517000 Position: 0
return File(blobStream, contentType: blob.Properties.ContentType, fileDownloadName: blob.Name, lastModified: blob.Properties.LastModified, entityTag: null);
Answer from a member of Microsoft Azure (here):
The difference between DownloadStreamingAsync and OpenReadAsync is
that the former gives you a network stream (wrapped with few layers
but effectively think about it as network stream) which holds on to
single connection, the later on the other hand fetches payload in
chunks and buffers issuing multiple requests to fetch content. Picking
one over the other one depends on the scenario, i.e. if the consuming
code is fast and you have good broad network link to storage account
then former might be better choice as you avoid multiple req-res
exchanges but if the consumer is slow then later might be a good idea
as it releases a connection back to the pool right after reading and
buffering next chunk. We recommend to perf test your app with both to
reveal which is best choice if it's not obvious.
OpenReadAsync returns a Task<Stream> and you use it with an await.
sample test method
CloudBlobContainer container = GetRandomContainerReference();
try
{
await container.CreateAsync();
CloudBlockBlob blob = container.GetBlockBlobReference("blob1");
using (MemoryStream wholeBlob = new MemoryStream(buffer))
{
await blob.UploadFromStreamAsync(wholeBlob);
}
using (MemoryStream wholeBlob = new MemoryStream(buffer))
{
using (var blobStream = await blob.OpenReadAsync())
{
await TestHelper.AssertStreamsAreEqualAsync(wholeBlob, blobStream);
}
}
}
DownloadToStreamAsync is a virtual (can be overridden) method returning a task and takes stream object as input.
sample usage.
await blog.DownloadToStreamAsync(memoryStream);
I am trying to rename blob in azure storage via .net API and it is I am unable to rename a blob file after a day : (
Here is how I am doing it, by creating new blob and copy from old one.
var newBlob = blobContainer.GetBlobReferenceFromServer(filename);
newBlob.StartCopyFromBlob(blob.Uri);
blob.Delete();
There is no new blob on server so I am getting http 404 Not Found exception.
Here is working example that i have found but it is for old .net Storage api.
CloudBlob blob = container.GetBlobReference(sourceBlobName);
CloudBlob newBlob = container.GetBlobReference(destBlobName);
newBlob.UploadByteArray(new byte[] { });
newBlob.CopyFromBlob(blob);
blob.Delete();
Currently I am using 2.0 API. Where I am I making a mistake?
I see that you're using GetBlobReferenceFromServer method to create an instance of new blob object. For this function to work, the blob must be present which will not be the case as you're trying to rename the blob.
What you could do is call GetBlobReferenceFromServer on the old blob, get it's type and then either create an instance of BlockBlob or PageBlob and perform copy operation on that. So your code would be something like:
CloudBlobContainer blobContainer = storageAccount.CreateCloudBlobClient().GetContainerReference("container");
var blob = blobContainer.GetBlobReferenceFromServer("oldblobname");
ICloudBlob newBlob = null;
if (blob is CloudBlockBlob)
{
newBlob = blobContainer.GetBlockBlobReference("newblobname");
}
else
{
newBlob = blobContainer.GetPageBlobReference("newblobname");
}
//Initiate blob copy
newBlob.StartCopyFromBlob(blob.Uri);
//Now wait in the loop for the copy operation to finish
while (true)
{
newBlob.FetchAttributes();
if (newBlob.CopyState.Status != CopyStatus.Pending)
{
break;
}
//Sleep for a second may be
System.Threading.Thread.Sleep(1000);
}
blob.Delete();
The code in OP was almost fine except that an async copy method was called. The simplest code in new API should be:
var oldBlob = cloudBlobClient.GetBlobReferenceFromServer(oldBlobUri);
var newBlob = container.GetBlobReference("newblobname");
newBlog.CopyFromBlob(oldBlob);
oldBlob.Delete();