Azure Storage Search Blobs by Metadata - azure

I have CloudBlockBlobs that have metadata.
CloudBlockBlob blockBlob = container.GetBlockBlobReference("myblob.jpg");
using (var fileStream = System.IO.File.OpenRead(filePath))
{
blockBlob.UploadFromStream(fileStream);
blockBlob.Properties.ContentType = "image/jpg";
blockBlob.Metadata.Add("Title", "Yellow Pear");
blockBlob.SetProperties();
}
I see the Metadata is there:
Debug.WriteLine(blockBlob.Metadata["Title"]);
Now later if I query from storage I see the blobs but the Metadata is missing:
(in the below I know blobItems[0] had Metadata when uploaded but now blobItems[0].Metadata.Count == 0)
var blobItems = container.ListBlobs(
null, false, BlobListingDetails.Metadata);
I also noticed the Metadata is not available when I obtain the blob by itself:
CloudBlockBlob a = container.GetBlockBlobReference("myblob.jpg");
//Below throws an exception
var b = a.Metadata["Title"];
Thank you!

There are some issues with your code :(.
The blob doesn't have any metadata set actually. After setting the metadata, you're calling blob.SetProperties() method which only sets the blob's properties (ContentType in your example). To set the metadata, you would actually need to call blob.SetMetadata() method.
Your upload code is currently making 2 calls to storage service: 1) upload blob and 2) set properties. If you call SetMetadata then it would be 3 calls. IMHO, these can be combined in just 1 call to storage service by doing something like below:
using (var fileStream = System.IO.File.OpenRead(filePath))
{
blockBlob.Properties.ContentType = "image/jpg";
blockBlob.Metadata.Add("Title", "Yellow Pear");
blockBlob.UploadFromStream(fileStream);
}
This will not only upload the blob but also set it's properties and metadata in a single call to storage service.
Regarding
I also noticed the Metadata is not available when I obtain the blob by
itself:
CloudBlockBlob a = container.GetBlockBlobReference("myblob.jpg");
//Below throws an exception
var b = a.Metadata["Title"];
Basically the code above is just creating an instance of the blob on the client side. It doesn't actually fetch the properties (and metadata) of the blob. To fetch details about the blob, you would need to call FetchAttributes method on the blob. Something like:
CloudBlockBlob a = container.GetBlockBlobReference("myblob.jpg");
a.FetchAttributes();
If after that you retrieve blob's metadata, you should be able to see it (provided metadata was created properly).

Related

Read the blob content on Azure Storage

I'm using the Microsoft.Azure.Storage.Blob nuget package trying to get the list of the blobs in a container and than reading the content.
With the ListBlobs() method I see all the blobs.
Every blob item has an URI but I cannot see the blob name that I need for the GetBlobReferenceFromServer().
For this reason the blob name is a constant in following sample code.
What is the right way? Do I have to split and parse the URI to find the blob name?
Do I have to use another method?
Microsoft.Azure.Storage.Blob.CloudBlobContainer container =
new Microsoft.Azure.Storage.Blob.CloudBlobContainer(new Uri("https://myaccount.blob.core.windows.net/containername"),
new Microsoft.Azure.Storage.Auth.StorageCredentials("myaccount", "**********=="));
IEnumerable<Microsoft.Azure.Storage.Blob.IListBlobItem> blobs = container.ListBlobs();
foreach (var blobItem in blobs)
{
//string blobUri = blobItem.Uri.ToString();
Microsoft.Azure.Storage.Blob.ICloudBlob blockBlob = container.GetBlobReferenceFromServer("blobname");
MemoryStream downloadStream = new MemoryStream();
blockBlob.DownloadToStream(downloadStream);
string blobContent = Encoding.UTF8.GetString(downloadStream.ToArray());
}
With the ListBlobs() method I see all the blobs. Every blob item has
an URI but I cannot see the blob name that I need for the
GetBlobReferenceFromServer().
The reason for this is that ListBlobs method returns an enumerable of type IListBlobItem which does not have the name property. In order to get the name of the blob, you can cast it to either CloudBlob or CloudBlockBlob which implement this interface and you will be able to get the name of the blob which you can use GetBlobReferenceFromServer method.
BTW, once you have listed the blob you don't really need to call GetBlobReferenceFromServer method as you already have all the information about the blob as part of listing. GetBlobReferenceFromServer makes another request to storage to fetch same set of properties that you already have as part of listing.
So your code can simply be:
foreach (var blobItem in blobs)
{
var blockBlob = (CloudBlockBlob) blobItem;
MemoryStream downloadStream = new MemoryStream();
blockBlob.DownloadToStream(downloadStream);
string blobContent = Encoding.UTF8.GetString(downloadStream.ToArray());
}
Or, if you don't go down casting route, you can simply create an instance of CloudBlockBlob using the URI you got as part of the listing.
Something like:
foreach (var blobItem in blobs)
{
var blockBlob = new CloudBlockBlob(blobItem.Uri, container.ServiceClient);
MemoryStream downloadStream = new MemoryStream();
blockBlob.DownloadToStream(downloadStream);
string blobContent = Encoding.UTF8.GetString(downloadStream.ToArray());
}

How to get MD5 of file stored in ADLS Gen2?

I receive daily files through sFTP to ADLS gen 2 storage account. I need to verify the file by checking the MD5 of the file stored in ADLS gen2.
I tried using the BLOB API , currently its not supporting ADLS gen2. I was able to get Content MD5 from blob properties if the file is stored in Blob storage.
Can someone help how to get the content MD5 from ADLS gen 2?
As of now, Blob api is not supported as you know, but you can take a look at Data Lake Storage Gen2 rest api -> Path - Get Properties, which can be used to fetch properties of files stored in ADLS Gen2.
Here is a sample code(Note that I use the sas token appended to the api url):
using System;
using System.Net;
namespace ConsoleApp3
{
class Program
{
static void Main(string[] args)
{
string sasToken = "?sv=2018-03-28&ss=b&srt=sco&sp=rwdl&st=2019-04-15T08%3A07%3A49Z&se=2019-04-16T08%3A07%3A49Z&sig=xxxx";
string url = "https://xxxx.dfs.core.windows.net/myfilesys1/app.JPG" + sasToken;
var req = (HttpWebRequest)WebRequest.CreateDefault(new Uri(url));
req.Method = "HEAD";
var res = (HttpWebResponse)req.GetResponse();
Console.WriteLine("the status code is: "+res.StatusCode);
var headers = res.Headers;
Console.WriteLine("the count of the headers is: "+headers.Count);
Console.WriteLine("*********");
Console.WriteLine();
//list all the properties if you don't know which correct format of property.
foreach (var h in headers.Keys)
{
Console.WriteLine(h.ToString());
}
Console.WriteLine("*********");
Console.WriteLine();
//take the Content-Type property for example.
var myheader = res.GetResponseHeader("Content-Type");
Console.WriteLine($"the header Content-Type is: {myheader}");
Console.ReadLine();
}
}
}
Result:
If you don't know how to generate sas token, you can nav to azure portal -> your storage account, then follow the screenshot below:

Delete "subpath" from Azure Storage

I know Azure doesn't have actual subpaths, but if I have for example container/projectID/iterationNumber/filename.jpg and I delete a project, how can I delete from ProjectID? Is it possible through coding?
I don't want to use the azure application as I am creating a web app.
Thanks in Advance
EDIT:
This is the code provided by Microsoft to target on specific item:
// Retrieve storage account from connection string.
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
CloudConfigurationManager.GetSetting("StorageConnectionString"));
// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Retrieve reference to a previously created container.
CloudBlobContainer container = blobClient.GetContainerReference("mycontainer");
// Retrieve reference to a blob named "myblob.txt".
CloudBlockBlob blockBlob = container.GetBlockBlobReference("myblob.txt");
// Delete the blob.
blockBlob.Delete();
SystemDesignModel
public static SystemDesign returnImageURL(IListBlobItem item)
{
if (item is CloudBlockBlob)
{
var blob = (CloudBlockBlob)item;
return new SystemDesign
{
URL = blob.Uri.ToString(),
};
}
return null;
}
}
As you know, blob storage does not have the concept of subfolders. It has just 2 level hierarchy - container & blobs. So in essence, a subfolder is just a prefix that you attach to blob name. In your example, the actual file you uploaded is filename.jpg but its name from blob storage perspective is projectID/iterationNumber/filename.jpg.
Since there is no concept of subfolder, you just can't delete it like we do on our local computer. However there's a way. Blob storage provides a way to search for blobs starting with a certain blob prefix. So what you have to do is first list all blobs that start with certain prefix (projectID in your case) and then delete the blobs one at a time returned as a result of listing operations.
Take a look at sample code below:
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
CloudConfigurationManager.GetSetting("StorageConnectionString"));
var container = storageAccount.CreateCloudBlobClient().GetContainerReference("container");
BlobContinuationToken token = null;
do
{
var listingResult = container.ListBlobsSegmented("blob-prefix (projectID in your case)", true, BlobListingDetails.None, 5000, token, null, null);
token = listingResult.ContinuationToken;
var blobs = listingResult.Results;
foreach (var blob in blobs)
{
(blob as ICloudBlob).DeleteIfExists();
Console.WriteLine(blob.Uri.AbsoluteUri + " deleted.");
}
}
while (token != null);

Windows Azure Blob

I've been trying to create a Windows Azure Blob containing an image file. I followed these tutorials: http://www.nickharris.net/2012/11/how-to-upload-an-image-to-windows-azure-storage-using-mobile-services/ and http://www.windowsazure.com/en-us/develop/mobile/tutorials/upload-images-to-storage-dotnet/. Finally the following code represents a merging of them. On the last line, however, an exception is raised:
An exception of type 'System.TypeLoadException' occurred in
mscorlib.ni.dll but was not handled in user code
Additional information: A binding for the specified type name was not
found. (Exception from HRESULT: 0x80132005)
Even the container is created the table, but It doesn't work properly.
private async void SendPicture()
{
StorageFile media = await StorageFile.GetFileFromPathAsync("fanny.jpg");
if (media != null)
{
//add todo item to trigger insert operation which returns item.SAS
var todoItem = new Imagem()
{
ContainerName = "mypics",
ResourceName = "Fanny",
ImageUri = "uri"
};
await imagemTable.InsertAsync(todoItem);
//Upload image direct to blob storage using SAS and the Storage Client library for Windows CTP
//Get a stream of the image just taken
using (var fileStream = await media.OpenStreamForReadAsync())
{
//Our credential for the upload is our SAS token
StorageCredentials cred = new StorageCredentials(todoItem.SasQueryString);
var imageUri = new Uri(todoItem.SasQueryString);
// Instantiate a Blob store container based on the info in the returned item.
CloudBlobContainer container = new CloudBlobContainer(
new Uri(string.Format("https://{0}/{1}",
imageUri.Host, todoItem.ContainerName)), cred);
// Upload the new image as a BLOB from the stream.
CloudBlockBlob blobFromSASCredential =
container.GetBlockBlobReference(todoItem.ResourceName);
await blobFromSASCredential.UploadFromStreamAsync(fileStream.AsInputStream());
}
}
}
Please use Assembly Binding Log Viewer to see which load is failing. As also mentioned in the article, the common language runtime's failure to locate an assembly typically shows up as a TypeLoadException in your application.

Can't rename blob file in Azure Storage

I am trying to rename blob in azure storage via .net API and it is I am unable to rename a blob file after a day : (
Here is how I am doing it, by creating new blob and copy from old one.
var newBlob = blobContainer.GetBlobReferenceFromServer(filename);
newBlob.StartCopyFromBlob(blob.Uri);
blob.Delete();
There is no new blob on server so I am getting http 404 Not Found exception.
Here is working example that i have found but it is for old .net Storage api.
CloudBlob blob = container.GetBlobReference(sourceBlobName);
CloudBlob newBlob = container.GetBlobReference(destBlobName);
newBlob.UploadByteArray(new byte[] { });
newBlob.CopyFromBlob(blob);
blob.Delete();
Currently I am using 2.0 API. Where I am I making a mistake?
I see that you're using GetBlobReferenceFromServer method to create an instance of new blob object. For this function to work, the blob must be present which will not be the case as you're trying to rename the blob.
What you could do is call GetBlobReferenceFromServer on the old blob, get it's type and then either create an instance of BlockBlob or PageBlob and perform copy operation on that. So your code would be something like:
CloudBlobContainer blobContainer = storageAccount.CreateCloudBlobClient().GetContainerReference("container");
var blob = blobContainer.GetBlobReferenceFromServer("oldblobname");
ICloudBlob newBlob = null;
if (blob is CloudBlockBlob)
{
newBlob = blobContainer.GetBlockBlobReference("newblobname");
}
else
{
newBlob = blobContainer.GetPageBlobReference("newblobname");
}
//Initiate blob copy
newBlob.StartCopyFromBlob(blob.Uri);
//Now wait in the loop for the copy operation to finish
while (true)
{
newBlob.FetchAttributes();
if (newBlob.CopyState.Status != CopyStatus.Pending)
{
break;
}
//Sleep for a second may be
System.Threading.Thread.Sleep(1000);
}
blob.Delete();
The code in OP was almost fine except that an async copy method was called. The simplest code in new API should be:
var oldBlob = cloudBlobClient.GetBlobReferenceFromServer(oldBlobUri);
var newBlob = container.GetBlobReference("newblobname");
newBlog.CopyFromBlob(oldBlob);
oldBlob.Delete();

Resources