When I call "await blockBlob.UploadFromStreamAsync(stream);", I find that stream has a content length of some value(500+ bytes).
But once the call is made, I see that there is no change in the blockblob.
I check it in fiddler and see the call made has a Entity content-length =0.
Would appreciate if someone could please guide how to debug a problem like this.
Thanks
The question was .. how to debug this?
I had same issue, found this helpful doc to get tracing enabled:
Client-side Logging with the .NET Storage Client Library
My issue was when uploading using UploadFromStreamAsync azure file storage while the destination resource already existed (an overwrite action), the fileStream.Seek(0, SeekOrigin.Begin) was needed. When uploading (and creating) new resources, the seek was not required.
Why are you converting the bytearray, instead of uploading it directly?
Like this:
blob.UploadFromByteArray(bytearray, 0, bytearray.Count());
Related
I am using Azure blob storage to storage all my images and videos. I have implemented the upload and fetch functionality and it's working quite good. I am facing 1 issue while loading the videos, because when I use the url which is generated after uploading that video on Azure blob storage, it's downloading all the content first before rendering it to the user. So if the video size is 100 mb, it'll download all the 100 mb and till than user won't able to see the video.
I have done a lot of R&D and came to know that while rendering the video, I need to fetch the partial content (status 206) rather than fetching the whole video at a time. After adding the request header "Range:bytes-500", I tried to hit the blog url, but it's still downloading the whole content. So I have checked with some open source video URLs and tried to hit the video URL along with the "Range" request header and it was successfully giving 206 response status, which means it was properly giving me the partial content instead of the full video.
I read some forum and they are saying Azure storage supports the partial content concept and need to enable it from the properties. But I have checked all the options under the Azure storage account but didn't find anything to enable this functionality.
Can anyone please help me out to resolve this or if there's anything on Azure portal that I need to enable? It's something that I have been doing the R&D for this since a week now. Any help would be really appreciated.
Thank you! Stay safe.
Suppose the Accept-Ranges is not enabled, from this blog I got it needs to set the default version of the service.
Below is a sample code to implement it.
var credentials = new StorageCredentials("account name", "account key");
var account = new CloudStorageAccount(credentials, true);
var client = account.CreateCloudBlobClient();
var properties = client.GetServiceProperties();
properties.DefaultServiceVersion = "2019-07-07";
client.SetServiceProperties(properties);
Below is a return header comparison after setting the property.
Before:
After:
Assuming the video content is MPEG-4 the issue may be the media itself needs to have the moov atom position changed from the end of the file to the beginning. The browser won't render the video until it finds the moov atom in the file therefore you want to make sure the atom is at the start of the file which can be accomplished using ffmpeg with the "FastStart". Here's a good article with more detail : HERE
You just need to update your Azure Storage version. It will work automatically after the update.
Using Azure CLI
Just run:
az storage account blob-service-properties update --default-service-version 2021-08-06 -n yourStorageAccountName -g yourStorageResourceGroupName
List of avaliable versions:
https://learn.microsoft.com/en-us/rest/api/storageservices/previous-azure-storage-service-versions
To see your current version, open a file and inspect the x-ms-version header
following is the SDK I used to download the contents:
var container = new BlobContainerClient("UseDevelopmentStorage=true", "sample-container");
await container.CreateIfNotExistsAsync();
BlobClient blobClient = container.GetBlobClient(fileName);
Stream stream = new MemoryStream();
var result = await blobClient.DownloadToAsync(stream, cancellationToken: ct);
which DOES download the whole file right away! Unfortunately the solution provided in other answers seems to be referencing another SDK? So for the SDK that I use - the solution is to use the method OpenReadAsync:
long kBytesToReadAtOnce = 300;
long bytesToReadAtOnce = kBytesToReadAtOnce * 1024;
//int mbBytesToReadAtOnce = 1;
var result = await blobClient.OpenReadAsync(0, bufferSize: (int)bytesToReadAtOnce, cancellationToken: ct);
By default - it fetches 4mb of data, so you have to override the value to smaller amount if you want your app to have smaller memory footprint.
I think that internally the SDK sends the requests with the byte range already set. So all you have to do is enable the partial content support in Web API like this:
return new FileStreamResult(result, contentType)
{
EnableRangeProcessing = true,
};
Update: This question used to ask about Google Cloud Storage, but I have since realized the issue actually is reproducable merely trying to save the download to local disk. Thus, I am rephrasing the question to be entirely about file downloads in Typescript and to no longer mention Google Cloud Storage.
When attempting to download and save a file in Typescript with WebRequests (though I experienced the same issue with requests and request-promises), all the code seems to execute correctly, but the resultant file is corrupted and cannot be viewed. For example, if I download an image, the file is not viewable in any applications.
// Seems to work correctly
const download = await WebRequest.get(imageUrl);
// `Buffer.from()` also takes an `encoding` parameter, but it's unclear how to determine the encoding of a download
const imageBuffer = Buffer.from(download.content);
// I *think* this line is straightforward
const imageByteArray = new Uint8Array(imageBuffer);
// Saves a corrupted file
const file = fs.writeFileSync("/path/to/file.png", imageByteArray);
I suspect the issue lies within the Buffer.from call not correctly interpreting the downloaded content, but I'm not sure how to do it right. Any help would be greatly appreciated.
Thanks so much!
From what I saw in the examples for web-request, download.content is just a string. If you want to upload a string to Cloud Storage using the node SDK, you can use File.save, passing that string directly.
Alternatively, you could use one the solutions seen here.
I have an Azure web app that returns a binary file using FileStreamResult. Works fine...
FileStreamResult fsr = File(blobStream, System.Net.Mime.MediaTypeNames.Application.Octet, "testfile.bin");
return fsr;
I copied this code to another web app, returning the exact same blob item, and it returns a 'corrupted' file that is almost 2x larger (65K vs 117K). Both apps are using the same version of .Net, the same Azure account, the same Azure storage account, the response headers are the same...but something must be different!
Update: The FileStreamResult has the correct data/size in the buffer. If I copy the data out of the FileStream (before returning), it's OK. So the FileStreamResult is correct; the problem happens when the response is generated. Something with how FileStreamResult is converted. I've also tried ActionResult, same problem.
Update 2: Still can't figure it out but...it's swapping out the non-UTF-8 chars with the EF BF BD unicode replacement characters. I'm not sure why. I thought the content type of application/octet-stream would indicate the data is not interpreted. Also can't find why one web app does this but the other does not.
You could update the package Swashbuckle.AspNetCore to the latest version 4.0.1.
Here is a similar issue you could refer to.
Finally success...turns out there was an ActionFilterAttribute on the base controller that was forcing the encoding.
I am trying to create a blob from a pdf I am creating from pdfmake so that I can send it to a remote api that only handles blobs.
This is how I get my PDF file:
var docDefinition = { content: 'This is an sample PDF printed with pdfMake' };
pdfDoc.pipe(fs.createWriteStream('./pdfs/test.pdf'));
pdfDoc.end();
The above lines of code do produce a readable pdf.
Now how can I get a blob from there? I have tried many options (creating the blob from the stream with the blob-stream module, creating from the file with fs, creating it from a base64 string with b64toBlob) but all of them require at some point to use the constructor Blob for which I always get an error even if I require the module blob:
TypeError: Blob is not a constructor
After some research I found that it seems that the Blob constructor is only supported client-side.
All the npm packages that I have found and which seem to deal with this issue seem to only work client-side: blob-stream, blob, blob-util, b64toBlob, etc.
So, how can I create a blob server-side on Node?
I don't understand why almost nobody also needs to create a blob server-side? The only thread I could find on the subject is this one.
According to that thread, apparently:
The Solution to this problem is to create a function which can convert between Array Buffers and Node Buffers. :)
Unfortunately this does not help me much as I clearly seem to lack some important knowledge here to be able to comprehend this.
use node-blob npm package
const Blob = require('node-blob');
let myBlob = new Blob(["something"], { type: 'text/plain' });
I'm moving a ClickOnce install from a regular web server to Azure Blob storage and have a problem with some of the files. The filenames contains [ ] and CloudBlob.UploadFile fails with an exception:
Microsoft.WindowsAzure.Storageclient.StorageException:
Error accessing blob storage: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
The code has been used for a while and only fails on files with [ ] in the name so I don't believe that it is an "authentication failure". In this particular case, this is the seventh file being uploaded in a loop. I found this link on MSDN about valid file names and this on stack overflow which both show problems with square brackets in URL's and reference UrlEncode. I added in a call to UrlEncode and that did not help. The container is created with public access since we use it to support customer downloads of our software. We have been hosting a "test" install in another container and have not had permission problems accessing that either.
I can upload the file with no name changes and then rename the file to add the "path" using newdesic's Azure Storage Explorer tool so what is that doing that I am not doing?
I see you're using the 1.7 SDK. This is a small encoding issue with the SDK which is also present in v2.0. Let's see what happens.
No encoding
account.CreateCloudBlobClient()
.GetContainerReference("temp")
.GetBlobReference("abc[]def.txt")
.UploadFile("myfile.txt");
If you don't encode the blob name, you'll end up with a request to the following URL which is causing the authentication exception:
http://account.blob.core.windows.net/temp/abc[]def.txt
The is because the SDK uses Uri.EscapeUriString internally to encode your string, but this doesn't take into account square brackets.
Encoding
Then you would expect the following to do the trick:
account.CreateCloudBlobClient()
.GetContainerReference("temp")
.GetBlobReference(HttpUtility.UrlEncode("abc[]def.txt"))
.UploadFile("myfile.txt");
The issue here is that you'll end up with this url:
http://account.blob.core.windows.net/temp/abc%255b%255ddef.txt
So what's happening here? Calling HttpUtility.UrlEncode turns abc[]def.txt to abc%5B%5Ddef.txt, which is correct. But internally, the SDK will encode this string again which results in abc%255b%255ddef.txt, which isn't what you want.
Workaround
The only way to apply encoding which takes square brackets into accounts is by using a small workaround. If you pass the full URL to the GetBlobReference method, the SDK assumes you did all the encoding yourself:
var container = account.CreateCloudBlobClient().GetContainerReference("temp");
var blob = container.GetBlobReference(String.Format("{0}/{1}",
container.Uri, System.Web.HttpUtility.UrlEncode("abc[]def.txt")));
blob.UploadFile("myfile.txt");
This results in a correctly encoded URL:
http://account.blob.core.windows.net/temp/abc%5b%5ddef.txt
And if you use a tool like CloudXplorer, you'll see the blob with the correct filename:
There are two known breaks in the Uri class in .Net 4.5
• ‘[‘,’]’ characters are no longer escaped
• ‘\’ character is now escaped as %5C
This is causing an authentication when the server attempts to validate the signature of the request as the canonicalized string is now different on the client and the server.
There are a few workarounds clients can use while this issue is present. The correct solution will depend on your specific application and requirements.
Avoid the ‘[‘,’]’, or ‘\’ characters in resource names
By simply avoiding these characters all together you will be able to avoid the issue described above.
Target .Net 4.0
Currently the recommendation is for clients to simply continue to target their applications to .Net 4.0 while a full solution is being investigated. Note, since .Net 4.5 is an in place upgrade clients can still take advantage of some performance improvements in the GC etc, without specifically targeting the .Net 4.5 profile. For Windows RT developers this is not an option and you will therefore require the workarounds detailed below.
Pre-Escape Data
If possible a client can pre- escape the data or replace the given characters with non-affected ones.
This is why the workaround above is working.