azure CloudBlobDirectory.ListBlobs() returns "The specified resource does not exist.", but fetchAttributes() works using same data - azure

I am getting a "The specified resource does not exist" exception when I try to iterate the result of a ListBlobs() call. I can get the blob attributes when I access it directly, but I'm trying to get a list of all the blobs in the subdirectory.
I wrote this little test to see exactly where the problem is. I have a test driver and two methods here. The first method, "GetBlockBlobDateTime" runs fine and returns a date time of an existing blob. The second method "GetBlobDirFiles" uses the same inputs and throws the excpetion when I try to iterate the blobItems at.
foreach (IListBlobItem blobItem in blobItems)
Note that the same data is used for both methods. What am I missing?
public static void DoTest(string baseURL, string container, string directory, string fileName)
{
DateTime t = GetBlockBlobDateTime( baseURL, container, directory, fileName);
List<string> fileList = GetBlobDirFiles( baseURL, container, directory);
}
public static DateTime GetBlockBlobDateTime(string baseURL, string container, string directory, string fileName)
{
CloudBlobClient blobClient = new CloudBlobClient(baseURL);
CloudBlobDirectory blobDir = blobClient.GetBlobDirectoryReference(container);
CloudBlobDirectory subDirectory = blobDir.GetSubdirectory(directory);
CloudBlockBlob cloudBlockBlob = subDirectory.GetBlockBlobReference(fileName);
cloudBlockBlob.FetchAttributes();
DateTime cloudTimeStampUTC = cloudBlockBlob.Properties.LastModifiedUtc;
return cloudTimeStampUTC;
}
public static List<string> GetBlobDirFiles(string baseURL, string container, string directory)
{
CloudBlobClient blobClient = new CloudBlobClient(baseURL);
CloudBlobDirectory blobDir = blobClient.GetBlobDirectoryReference(container);
CloudBlobDirectory subDirectory = blobDir.GetSubdirectory(directory);
IEnumerable<IListBlobItem> blobItems = subDirectory.ListBlobs();
List<string> fileList = new List<string>();
foreach (IListBlobItem blobItem in blobItems)
{
fileList.Add(blobItem.Uri.ToString());
}
return fileList;
}

OK, I figured it out:
Apparently, you don't need permissions to get file attributes, but you do to list files in the directory.
CloudBlobClient blobClient = new CloudBlobClient(baseURL);
works when you are going to fetch attributes like this:
cloudBlockBlob.FetchAttributes();
But you need to provide credentials like this:
CloudBlobClient blobClient =
new CloudBlobClient(baseURL,
new StorageCredentialsAccountAndKey(myAccount, myKey));
when you are going to list the blobs like this:
var blobList = subDirectory.ListBlobs();
foreach (var blobInfo in blobList)

Related

How do I access blob storage sub containers in my ASP.NET Core web application?

I've created a solution to access my container contents within my asp.net core 3.1 application and return that contents as a list to my view. At the moment the application access data in the root container which is called upload, however, this container has many sub containers and I would like to list the blobs in a specific one called 1799.
So, instead of accessing upload and showing me the full contents of that container, I want to access upload/1799 and list all the blobs within that container.
I cannot see of anyway to add this sub container to my method and allow this to happen, can anyone help?
Here is my code so far:
CarController.cs
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
namespace MyProject.Controllers
{
public class HomeController : Controller
{
private readonly IConfiguration _configuration;
private readonly string accessKey = string.Empty;
public HomeController(IConfiguration configuration)
{
_configuration = configuration;
accessKey = _configuration.GetConnectionString("AzureStorage");
}
[HttpGet]
public IActionResult Edit(int id)
{
var car = _carService.GetVessel(id);
string strContainerName = "uploads";
string subdir = "1799";
var filelist = new List<BlobListViewModel>();
BlobServiceClient blobServiceClient = new BlobServiceClient(accessKey);
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(strContainerName);
var blobs = containerClient.GetBlobs();
foreach (var item in blobs)
{
filelist.Add(new BlobListViewModel
{
FileName = item.Name
});
}
return View(filelist);
}
}
I've hunted through all the documentation and I can't find anything related to this.
Thanks to David Browne for his comment, there is indeed a prefix option in GetBlob(). THey act like mini-filters in some ways allowing you to define what properties are returned and the blob state etc. Here is my code which has the options set to zero meaning Default for both Trait and State.
[HttpGet]
public IActionResult Edit(int id)
{
var car = _carService.GetVessel(id);
string strContainerName = "uploads";
string subdir = "1799";
var filelist = new List<BlobListViewModel>();
BlobServiceClient blobServiceClient = new BlobServiceClient(accessKey);
BlobContainerClient containerClient = blobServiceClient.GetBlobContainerClient(strContainerName);
//Traits, States, Prefix
var blobs = containerClient.GetBlobs(0, 0, subdir);
foreach (var item in blobs)
{
filelist.Add(new BlobListViewModel
{
FileName = item.Name
});
}
return View(filelist);
}

Tidier way of getting Directory Names only from Blob Container

I am trying to get the directory names only of any directories in a specific location withing Blob Storage
I have the helper class below
public static class BlobHelper
{
private static CloudBlobContainer _cloudBlobContainer;
private const string _containerName = "administrator";
public static void Setup(string connectionString)
{
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(connectionString);
CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();
_cloudBlobContainer = cloudBlobClient.GetContainerReference(_containerName);
}
public static List<string> GetDirectoryNames(string relativeAddress)
{
var result = new List<string>();
var directory = _cloudBlobContainer.GetDirectoryReference(relativeAddress);
var folders = directory.ListBlobs().OfType<CloudBlobDirectory>();
foreach (var folder in folders)
{
var name = folder.Uri.AbsolutePath;
name = name.Replace(folder.Parent.Prefix, string.Empty)
.Replace(#"/", string.Empty)
.Replace(_containerName, string.Empty);
result.Add(name);
}
}
}
The process to get the directory names only (i.e. not the full hierarchy) feels a bit hacky, although it does work
Is there a better way to do this?
I tried the approach below
var directory = _cloudBlobContainer.GetDirectoryReference(relativeAddress);
var blobs = directory.ListBlobs(true).OfType<CloudBlobDirectory>();;
var blobNames = blobs.OfType<CloudBlockBlob>().Select(b => b.Name).ToList();
return blobNames;
The main difference with the above is the use of UseFlatBlobListing as true
However, this approach results in no folders at all being returned, whereas my other logic does at least give me the 2 folders I expect to find
Any ideas what I am doing wrong?
Cheers
Paul
I suppose your code is OK, I don't understand what you mean "a bit hacky". I think you want to get the directory directly.
Cause no method directory get the directory, for now known method to do it with v11 sdk mostly use the blob uri to do it.
And the below is my way to do it.
CloudBlobContainer cloudBlobContainer = cloudBlobClient.GetContainerReference("test");
BlobContinuationToken blobContinuationToken = null;
var blobsSeg = cloudBlobContainer.ListBlobsSegmented(null, blobContinuationToken);
var directories= blobsSeg.Results.OfType<CloudBlobDirectory>().Select(b => b.Prefix).ToList();
foreach (string directory in directories) {
Console.WriteLine(directory);
}
The result it returns it will be like below pic. Hope this could help you.

reading content of blob from azure function

I'm trying to read the content of a blob inside an azure function.
Here's the code:
Note:
If I comment out the using block and return the blob i.e.
return new OkObjectResult(blob);
I get back the blob object.
However, if I use the using block, I get 500.
Any idea why I can't get the content?
string storageConnectionString = "myConnectionString";
CloudStorageAccount storageAccount;
CloudStorageAccount.TryParse(storageConnectionString, out storageAccount);
CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer container = cloudBlobClient.GetContainerReference("drawcontainer");
var blob = drawingsContainer.GetBlockBlobReference("notes.txt");
using (StreamReader reader = new StreamReader(blob.OpenRead()))
{
content = reader.ReadToEnd();
}
return new OkObjectResult(content);
HTTP 500 indicates that the code has error. The most probable reason for error is the variable 'content'. Define the variable 'content' outside the using block as the scope of the content variable defined inside it is limited to the block only. Declare it outside the using block, something like below:
try
{
string content = string.Empty;
using (StreamReader reader = new StreamReader(blob.OpenRead()))
{
content = reader.ReadToEnd();
}
}
catch (Exception ex)
{
// Log exception to get the details.
}
Always make use of try catch to get more details about errors in the code.
The OpenRead method didn't exist so I used the async one and it solved it.
I got to this solution after creating an azure function in VS and publishing it and it works.
Here's the code I used:
public static class Function1
{
[FunctionName("Function1")]
public static async Task<ActionResult> Run([HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", Route = null)]HttpRequest req, TraceWriter log)
{
log.Info("C# HTTP trigger function processed a request.");
string storageConnectionString = "DefaultEndpointsProtocol=https;AccountName=avitest19a1c;AccountKey=<AccessKey>";
CloudStorageAccount storageAccount = null;
CloudStorageAccount.TryParse(storageConnectionString, out storageAccount);
CloudBlobClient cloudBlobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer drawingsContainer = cloudBlobClient.GetContainerReference("drawcontainer");
var blob = drawingsContainer.GetBlockBlobReference("notes.txt");
string content = string.Empty;
**var contentStream = await blob.OpenReadAsync();**
using (StreamReader reader = new StreamReader(contentStream))
{
content = reader.ReadToEnd();
}
return new OkObjectResult(content);
}
}

How to upload a file to a storage location through URL using Azure function app

I want to upload the upload a file to a storage location through URL using Azure function app from Azure blob storage. I'm able to pull the file from Azure blob. But not able to upload the file through url.
Below I have attached the code which i have written. Could anyone help me on this?
#r "Newtonsoft.Json"
#r "Microsoft.WindowsAzure.Storage"
#r "System.IO"
using System;
using System.IO;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
using Microsoft.WindowsAzure.Storage.Auth;
using System.Xml;
using System.Collections.Generic;
using Newtonsoft.Json;
using System.Net;
public static void Run(string input, TraceWriter log)
{
log.Info($"C# manual trigger function processed\n");
const string StorageAccountName = "";
const string StorageAccountKey = "";
var storageAccount = new CloudStorageAccount(new StorageCredentials(StorageAccountName, StorageAccountKey), true);
var blobClient = storageAccount.CreateCloudBlobClient();
var container = blobClient.GetContainerReference("hannahtest");
var Destcontainer = blobClient.GetContainerReference("hannahtestoutput");
var blobs = container.ListBlobs();
log.Info($"Creating Client and Connecting");
foreach (IListBlobItem item in container.ListBlobs(null, false))
{
if (item is CloudBlockBlob blockBlob)
{
using (StreamReader reader = new StreamReader(blockBlob.OpenRead())
{
//old content string will read the blockblob (xml)till end
string oldContent1 = reader.ReadToEnd();
log.Info(oldContent1);
var content = new FormUrlEncodedContent(oldContent1);
var response = await client.PostAsync("http://www.example.com/recepticle.aspx", content);
var responseString = await response.Content.ReadAsStringAsync();
log.Info($"Success");
}
}
}
}
Have a look at Blob Output Binding - that's how the blobs are intended to be uploaded from Azure Functions, without messing with Azure Storage SDK.
Azure function to upload multiple image file to blob storage.
using Microsoft.WindowsAzure.Storage.Auth;
using Microsoft.WindowsAzure.Storage;
using Microsoft.WindowsAzure.Storage.Blob;
public static class ImageUploadFunction
{
[FunctionName("ImageUploadFunction")]
public static async Task<IActionResult> Run([HttpTrigger(AuthorizationLevel.Function, "post")]HttpRequestMessage req, ILogger log)
{
var provider = new MultipartMemoryStreamProvider();
await req.Content.ReadAsMultipartAsync(provider);
var files = provider.Contents;
List<string> uploadsurls = new List<string>();
foreach (var file in files) {
var fileInfo = file.Headers.ContentDisposition;
Guid guid = Guid.NewGuid();
string oldFileName = fileInfo.FileName;
string newFileName = guid.ToString();
var fileExtension = oldFileName.Split('.').Last().Replace("\"", "").Trim();
var fileData = await file.ReadAsByteArrayAsync();
try {
//Upload file to azure blob storage method
var upload = await UploadFileToStorage(fileData, newFileName + "." + fileExtension);
uploadsurls.Add(upload);
}
catch (Exception ex) {
log.LogError(ex.Message);
return new BadRequestObjectResult("Somthing went wrong.");
}
}
return uploadsurls != null
? (ActionResult)new OkObjectResult(uploadsurls)
: new BadRequestObjectResult("Somthing went wrong.");
}
private static async Task<string> UploadFileToStorage(byte[] fileStream, string fileName)
{
// Create storagecredentials object by reading the values from the configuration (appsettings.json)
StorageCredentials storageCredentials = new StorageCredentials("<AccountName>", "<KeyValue>");
// Create cloudstorage account by passing the storagecredentials
CloudStorageAccount storageAccount = new CloudStorageAccount(storageCredentials, true);
// Create the blob client.
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
// Get reference to the blob container by passing the name by reading the value from the configuration (appsettings.json)
CloudBlobContainer container = blobClient.GetContainerReference("digital-material-library-images");
// Get the reference to the block blob from the container
CloudBlockBlob blockBlob = container.GetBlockBlobReference(fileName);
// Upload the file
await blockBlob.UploadFromByteArrayAsync(fileStream,0, fileStream.Length);
blockBlob.Properties.ContentType = "image/jpg";
await blockBlob.SetPropertiesAsync();
return blockBlob.Uri.ToString();
//return await Task.FromResult(true);
}
}

Request.Content.ReadAsMultipartAsync Throws System.IO exception

I am uploading an image to an ASP WebAPI service and then uploading it to windows azure. everything was working great but suddenly I started getting the following exception:
System.IO.IOException: The process cannot access the file
'C:\DWASFiles\Sites\Tasteat\Temp\BodyPart_a5c79910-6077-4c24-b814-10fdc0e0b3d4'
because it is being used by another process.
This is the code that throws the exception:
var provider = new BlobStorageProvider(container);
Trace.TraceInformation("Uploading raw image to blob");
await Request.Content.ReadAsMultipartAsync(provider);
Trace.TraceInformation("Uploading finished");
I know its this line await Request.Content.ReadAsMultipartAsync(provider); because I see the line before it in the log but not the line after it.
Any ideas?
Everything was working great up until a few days
So as it seems the code I posted above actually saves a local file and only then uploaded it to the server, which causes the error but also is slow.
after a lot of trying I finally change to the following solution and everything started working and it was even faster!
first create a streamprovider:
public class BlobStorageMultipartStreamProvider : MultipartStreamProvider
{
private readonly string _containerName;
private readonly string _fileName;
public BlobStorageMultipartStreamProvider(string containerName, string fileName)
{
_containerName = containerName;
_fileName = fileName;
}
public override Stream GetStream(HttpContent parent, HttpContentHeaders headers)
{
Stream stream = null;
if (!String.IsNullOrWhiteSpace(_fileName))
{
string connectionString = ConfigurationManager.ConnectionStrings["BlobStorage"].ConnectionString;
CloudStorageAccount storageAccount = CloudStorageAccount.Parse(connectionString);
CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
CloudBlobContainer blobContainer = blobClient.GetContainerReference(_containerName);
CloudBlockBlob blob = blobContainer.GetBlockBlobReference(_fileName);
stream = blob.OpenWrite();
}
return stream;
}
}
The upload code:
string fileName = Guid.NewGuid()+".Png";
MultipartStreamProvider provider = new BlobStorageMultipartStreamProvider("container",fileName);
Trace.TraceInformation("Uploading raw image to blob");
await Request.Content.ReadAsMultipartAsync(provider);
Trace.TraceInformation("Uploading finished");

Resources