Azure.Storage.Blobs.BlobServiceClient CopyFromUri() doesn't seem to be finished copying before returning latest ETag - azure

I am trying to use StartCopyFromUri or StartCopyFromUriAsync to copy a blob from one storage account to another. Even though status.HasCompleted when I try to get the ETag either through
1. var etag = await _siteStorageClient.GetBlobETag(containerPath, asset.BlobName);
//this is the response from WaitForCompletionAsync
2. var etag = complete.GetRawResponse().Headers.Where(x => x.Name == "ETag").FirstOrDefault().Value;
I've tried both methods and both return an Etag that doesn't match what is shown in the properties of the blob when I log in through Azure Portal. It is almost as if the file wasn't done copying(or race condition) when the Etag check was executed. I couldn't find any usage samples on github for the SDK.
Any ideas what could be going awry?
This a similar question but using the older SDK. How to copy a blob from one container to another container using Azure Blob storage SDK
//Storage class
public async Task<CopyFromUriOperation> CopyFile(string containerName, string blobName, Uri sourceUri)
{
var container = _blobServiceClient.GetBlobContainerClient(containerName);
var blockBlobClient = container.GetBlockBlobClient(blobName);
//Made this the synchronous version try and block
//this is the target client
var status = await blockBlobClient.StartCopyFromUriAsync(sourceUri);
while(!status.HasCompleted)
{
//Per Documentation this calls UpdateStatusAsync() periodically
//until status.HasCompleted is true
await status.WaitForCompletionAsync();
}
return status;
}
//Calling Code
var status = await _siteStorageClient.CopyFile(container,BlobName, sasUri);
var etag = await _siteStorageClient.GetBlobETag(container, BlobName);

I was able to get this working after a few tries and troubleshooting. It would only happen in the Azure environment and not when running the web app locally.
Initially the status.WaitForCompletionAsync() was inside the loop and I started getting a socket error. I believe it was getting called too many times and was causing port exhaustion(just speculation at this point).
But this is what is working now.
public async Task<CopyFromUriOperation> CopyFile(string containerName, string blobName,Uri sourceUri)
{
var container = _blobServiceClient.GetBlobContainerClient(containerName);
var blockBlobClient = container.GetBlockBlobClient(blobName);
var status = await blockBlobClient.StartCopyFromUriAsync(sourceUri);
await status.WaitForCompletionAsync();
while(status.HasCompleted == false)
{
await Task.Delay(100);
}
return status;
}

Related

Authentication Failure when uploading to Azure Blob Storage using Azure.Storage.Blob v12.9.0

I get this error when trying to upload files to blob storage. The error is present both when I run on localhost and when I run in Azure Function.
My connection string looks like:
DefaultEndpointsProtocol=https;AccountName=xxx;AccountKey=xxx;EndpointSuffix=core.windows.net
Authentication information is not given in the correct format. Check the value of the Authorization header.
Time:2021-10-14T15:56:26.7659660Z
Status: 400 (Authentication information is not given in the correct format. Check the value of Authorization header.)
ErrorCode: InvalidAuthenticationInfo
But this used to work in the past but recently its started throwing this error for a new storage account I created. My code looks like below
public AzureStorageService(IOptions<AzureStorageSettings> options)
{
_connectionString = options.Value.ConnectionString;
_containerName = options.Value.ImageContainer;
_sasCredential = new StorageSharedKeyCredential(options.Value.AccountName, options.Value.Key);
_blobServiceClient = new BlobServiceClient(new BlobServiceClient(_connectionString).Uri, _sasCredential);
_containerClient = _blobServiceClient.GetBlobContainerClient(_containerName);
}
public async Task<string> UploadFileAsync(IFormFile file, string location, bool publicAccess = true)
{
try
{
await _containerClient.CreateIfNotExistsAsync(publicAccess
? PublicAccessType.Blob
: PublicAccessType.None);
var blobClient = _containerClient.GetBlobClient(location);
await using var fileStream = file.OpenReadStream();
// throws Exception here
await blobClient.UploadAsync(fileStream, true);
return blobClient.Uri.ToString();
}
catch (Exception e)
{
Console.WriteLine(e);
throw;
}
}
// To be able to do this, I have to create the container client via the blobService client which was created along with the SharedStorageKeyCredential
public Uri GetSasContainerUri()
{
if (_containerClient.CanGenerateSasUri)
{
// Create a SAS token that's valid for one hour.
var sasBuilder = new BlobSasBuilder()
{
BlobContainerName = _containerClient.Name,
Resource = "c"
};
sasBuilder.ExpiresOn = DateTimeOffset.UtcNow.AddHours(1);
sasBuilder.SetPermissions(BlobContainerSasPermissions.Write);
var sasUri = _containerClient.GenerateSasUri(sasBuilder);
Console.WriteLine("SAS URI for blob container is: {0}", sasUri);
Console.WriteLine();
return sasUri;
}
else
{
Console.WriteLine(#"BlobContainerClient must be authorized with Shared Key
credentials to create a service SAS.");
return null;
}
}
Please change the following line of code:
_blobServiceClient = new BlobServiceClient(new BlobServiceClient(_connectionString).Uri, _sasCredential);
to
_blobServiceClient = new BlobServiceClient(_connectionString);
Considering your connection string has all the necessary information, you don't really need to do all the things you're doing and you will be using BlobServiceClient(String) constructor which expects and accepts the connection string.
You can also delete the following line of code:
_sasCredential = new StorageSharedKeyCredential(options.Value.AccountName, options.Value.Key);
and can probably get rid of AccountName and Key from your configuration settings if they are not used elsewhere.

Stream media file using asp.net core and azure cloudblob

I want to create endpoint that will stream video stream that is stored in azure CloudBlob. Here is snippet of my code:
public async Task<IActionResult> GetVideo(string videoId)
{
var videoStream = await _contentStorage.Get(videoId);
var fileStreamResult = new FileStreamResult(videoStream, mimeType);
fileStreamResult.EnableRangeProcessing = true;
return fileStreamResult;
}
and in ContentStorage
public async Task<StoredContent> Get(string id)
{
var block = _blobContainer.GetBlobClient(id);
var ms = await block.OpenReadAsync();
return ms;
}
I had everything working fine except iPhones and safari, after some debugging it turned out that my endpoint is returning 200 http code, but it should 206 - partial content. So I made some changes into my code, here is some snippet:
public async Task<IActionResult> GetVideo(string videoId)
{
var videoStream = await _contentStorage.Get(videoId);
var ms = new MemoryStream();
await videoStream.CopyToAsync(ms);
var fileStreamResult = new FileStreamResult(ms, mimeType);
fileStreamResult.EnableRangeProcessing = true;
return fileStreamResult;
}
Now when I test it on iphone or by postman response is 206, and it's working fine. But I thing that copping the video stream into new memorystream is a valid approach.
Correct me if I'm wrong but I understand this code as for every partial of the video, I'm downloading whole video from blob storage, cut it and then return just the piece within range.
It's not sure for me how to handle this case, is there any out of the box solution for that, or do I need to read range header from request and use OpenReadAsync with parameters as position and buffer side? Or there is another way?
Solution for me was to update Azure.Storage.Blobs library. I had 12.6.0 and after update to 12.7.0 it started working as expected, since they added:
Added seekability to BaseBlobClient.OpenRead().

How to handle Optimistic Concurrency in Azure.Storage.Blobs v12.x.x of Azure dll

I am trying to Implement the example shared in Learn Path
https://github.com/MicrosoftDocs/mslearn-support-concurrency-blob-storage/blob/master/src/OptimisticNewsEditor/Program.cs
I am trying to use the v12 dll which is Azure.Storage.Blobs
this is the code I have.
public static async Task Main()
{
BlobContainerClient container;
try
{
container = new BlobServiceClient(connectionString).GetBlobContainerClient(containerName);
await container.CreateIfNotExistsAsync(PublicAccessType.None);
}
catch (Exception)
{
var msg = $"Storage account not found. Ensure that the environment variable " +
" is set to a valid Azure Storage connection string and that the storage account exists.";
Console.WriteLine(msg);
return;
}
// First, the newsroom chief writes the story notes to the blob
await SimulateChief();
Console.WriteLine();
await Task.Delay(TimeSpan.FromSeconds(2));
// Next, two reporters begin work on the story at the same time, one starting soon after the other
var reporterA = SimulateReporter("Reporter A", writingTime: TimeSpan.FromSeconds(12));
await Task.Delay(TimeSpan.FromSeconds(4));
var reporterB = SimulateReporter("Reporter B", writingTime: TimeSpan.FromSeconds(4));
await Task.WhenAll(reporterA, reporterB);
await Task.Delay(TimeSpan.FromSeconds(2));
Console.WriteLine();
Console.WriteLine("=============================================");
Console.WriteLine();
Console.WriteLine("Reporters have finished, here's the story saved to the blob:");
BlobDownloadInfo story = await container.GetBlobClient(blobName).DownloadAsync();
Console.WriteLine(new StreamReader(story.Content).ReadToEnd());
}
private static async Task SimulateReporter(string authorName, TimeSpan writingTime)
{
// First, the reporter retrieves the current contents
Console.WriteLine($"{authorName} begins work");
var blob = new BlobContainerClient(connectionString, containerName).GetBlobClient(blobName);
var contents = await blob.DownloadAsync();
Console.WriteLine($"{authorName} loads the file and sees the following content: \"{new StreamReader(contents.Value.Content).ReadToEnd()}\"");
// Store the current ETag
var properties = await blob.GetPropertiesAsync();
var currentETag = properties.Value.ETag;
Console.WriteLine($"\"{contents}\" has this ETag: {properties.Value.ETag}");
// Next, the author writes their story. This takes some time.
Console.WriteLine($"{authorName} begins writing their story...");
await Task.Delay(writingTime);
Console.WriteLine($"{authorName} has finished writing their story");
try
{
// Finally, they save their story back to the blob.
var story = $"[[{authorName.ToUpperInvariant()}'S STORY]]";
await uploadDatatoBlob(blob, story);
Console.WriteLine($"{authorName} has saved their story to Blob storage. New blob contents: \"{story}\"");
}
catch (RequestFailedException e)
{
// Catch error if the ETag has changed it's value since opening the file
Console.WriteLine($"{authorName} sorry cannot save the file as server returned an error: {e.Message}");
}
}
private static async Task SimulateChief()
{
var blob = new BlobContainerClient(connectionString, containerName).GetBlobClient(blobName);
var notes = "[[CHIEF'S STORY NOTES]]";
await uploadDatatoBlob(blob, notes);
Console.WriteLine($"The newsroom chief has saved story notes to the blob {containerName}/{blobName}");
}
private static async Task uploadDatatoBlob(BlobClient blob, string notes)
{
byte[] byteArray = Encoding.UTF8.GetBytes(notes);
MemoryStream stream = new MemoryStream(byteArray);
await blob.UploadAsync(stream, overwrite: true);
}
I need to modify the UploadAsync to check for ETag before Uploading.
In the old version of Azure .Net CLI we had Microsoft.Azure.Storage.Blob dll now this handled Optimistic Concurrency by
await blob.UploadTextAsync(story, null, accessCondition: AccessCondition.GenerateIfMatchCondition(currentETag), null, null);
How do i do this in v12 dll.
Any Help appreciated.
Please use the following override of UploadAsync method:
UploadAsync(Stream, BlobHttpHeaders, IDictionary<String,String>, BlobRequestConditions, IProgress<Int64>, Nullable<AccessTier>, StorageTransferOptions, CancellationToken)
You can define the access conditions as part of BlobRequestConditions parameter.

Azure encrypted blob download result is empty when downloading from different machine than the upload

I've uploaded a text blob to azure storage (using UploadTextASync). Everything went OK, even downloading it again on the same workstation.
The problem is when I try downloading it from other server (stage):
The call to blob.ExistsAsync reports the blob is there;
The call to blob.DownloadTextAsync returns an empty string
Setup:
the uploaded text is an html, UTF8 encoded;
upload uses BlobRequestOptions with an IKey obtained from a CachingKeyResolver which generates a secret key in the key vault;
download uses BlobRequestOptions with a CachingKeyResolver which obtaines the same secret key
Things checked so far:
blob connection strings are the same between local and the stage server;
key vault name, client id, client secret and blob url are all the same between local and the stage server;
Updated WindowsAzure.Storage to the latest version (7.2.1) even the old one used (6.2.0) was behaving the same;
encryption mechanism is being successfully used in the same project (to upload/download octet streams); the only difference is in the method used
Recap:
DownloadText (and its async method) returns empty string when called from a different machine
Code:
public async Task<bool> UploadTextAsync(UploadTextBlobParameter parameter)
{
var container = await EnsureBlobContainerAsync(parameter);
var blob = container.GetBlockBlobReference(parameter.Path);
var blobRequestOptions = await _blobRequestOptionsManager.GetUploadBlobRequestOptions().ConfigureAwait(false);
var exists = await blob.ExistsAsync(blobRequestOptions, null).ConfigureAwait(false);
if (!parameter.Overwrite && exists)
{
return false;
}
await blob.UploadTextAsync(parameter.InputStream, parameter.Encoding, null, blobRequestOptions, null).ConfigureAwait(false);
return true;
}
public async Task<string> DownloadTextAsync(DownloadTextBlobParameter parameter)
{
var container = await EnsureBlobContainerAsync(parameter);
var blob = container.GetBlockBlobReference(parameter.Path);
var blobRequestOptions = _blobRequestOptionsManager.GetDownloadBlobRequestOptions();
var exists = await blob.ExistsAsync(blobRequestOptions, null).ConfigureAwait(false);
if (exists)
{
return await blob.DownloadTextAsync(parameter.Encoding, null, blobRequestOptions, null).ConfigureAwait(false);
}
return null;
}
Thanks in advance!

ApproximateMessageCount always null after calling FetchAttributesAsync in a Universal windows App

I am making a small App that should list the number of items in my Azure queues.
When I use FetchAttributesAsync and ApproximateMessageCount in a Console App, I get the expected result in ApproximateMessageCount after a call to FetchAttributesAsync (or FetchAttributes).
When I use the same in a Universal Windows app, ApproximateMessageCount remains stuck at null after a call to FetchAttributesAsync (FetchAttributes is not available there).
Console code:
CloudStorageAccount _account;
if (CloudStorageAccount.TryParse(_connectionstring, out _account))
{
var queueClient = _account.CreateCloudQueueClient();
Console.WriteLine(" {0}", _account.QueueEndpoint);
Console.WriteLine(" ----------------------------------------------");
var queues = (await queueClient.ListQueuesSegmentedAsync(null)).Results;
foreach (CloudQueue q in queues)
{
await q.FetchAttributesAsync();
Console.WriteLine($" {q.Name,-40} {q.ApproximateMessageCount,5}");
}
}
Universal App code:
IEnumerable<CloudQueue> queues;
CloudStorageAccount _account;
CloudQueueClient queueClient;
CloudStorageAccount.TryParse(connectionstring, out _account);
queueClient = _account.CreateCloudQueueClient();
queues = (await queueClient.ListQueuesSegmentedAsync(null)).Results;
foreach (CloudQueue q in queues)
{
await q.FetchAttributesAsync();
var count = q.ApproximateMessageCount;
// count is always null here!!!
}
I have tried all kinds of alternatives, like Wait()'s and such on the awaitables. Whatever I try, the ApproximateMessageCount stays a null with dertermination :-(.
Am I missing something?
I think you have discovered a bug in the storage client library. I looked up the code on Github and essentially instead of reading the value for Approximate Message Count header, the code is reading the value for Lease Status header.
In QueueHttpResponseParsers.cs class:
public static string GetApproximateMessageCount(HttpResponseMessage response)
{
return response.Headers.GetHeaderSingleValueOrDefault(Constants.HeaderConstants.LeaseStatus);
}
This method should have been:
public static string GetApproximateMessageCount(HttpResponseMessage response)
{
return response.Headers.GetHeaderSingleValueOrDefault(Constants.HeaderConstants.ApproximateMessagesCount);
}
I have submitted a bug for this: https://github.com/Azure/azure-storage-net/issues/155.

Resources