Azure Powershell: How Do I search for files in a BLOB storage quickly? - azure

Question

We store log files in an Azure storage account, sorted in directories, by date and customer, like this:
YYYY/MM/DD/customerNo/.../.../somestring.customerNo.applicatoinID.log
I need to parse some of these files automatically every day which works fine. However, all I know is the prefix mentioned above and the suffix, they might be in different subdirectories.
So this is how I did it:
$files = (Get-AzStorageBlob -Container logfiles -Context $context) | Where-Object { $_.Name -like "$customerId.$appID.txt" }
This was fast while there weren't any log files, but now after a year this search takes ages. I read somewhere that it would be faster to search by prefix than by suffix. Unfortunately, I have to use the suffix, but I now use the date as a prefix as well. I tried to improve it by doing this:
$date = Get-Date -UFormat "%Y/%m/%d"
$prefix = "$date/$customerId/"
$files = (Get-AzStorageBlob -Container logfiles -Context $context) | Where-Object { $_.Name -like "$prefix*$customerId.$appID.txt" }
However, there is no improvement whatsoever, it just takes as long as before. And it feels like the time the search takes is increasing exponentially with the amount of log files (A few hundred thousand in a very few tens of GBs)
I get a status message which stays there literally for half an hour:
From what I understand, Azure's BLOB storage does not have a hierarchical file system that supports folders, so the "/" are part of the BLOB name and are being interpreted as folders by client software.
However, that does not help me speeding up the search. Any suggestions on how to improve the situation?

Answer 1

Azure Blob Storage supports server-side filtering of blobs by prefix however your code is not taking advantage of that.
$files = (Get-AzStorageBlob -Container logfiles -Context $context) | Where-Object { $_.Name -like "$prefix*$customerId.$appID.txt" }
Essentially the code above is listing all blobs and then doing the filtering on the client side.
To speed up the search, please modify your code to something like:
$files = (Get-AzStorageBlob -Container logfiles -Prefix $prefix -Context $context) | Where-Object { $_.Name -like "$prefix*$customerId.$appID.txt" }
I simply passed the prefix in the Prefix parameter. Now you'll receive only the blobs names of which start with the prefix .