I need to read blobs from a azure container which are added after a particular date.
Basically, I have a windows service which runs once a day and gets the list of blobs added after the first run.
I do not see any such option in CloudBlobContainer.ListBlobsSegmentedAsync function or via the Get Blob REST API call.
I could think of only one option- have timestamp in the filename and filter by prefix but would like to know other better options to achieve this.
Unfortunately there's very limited server-side filtering available in Azure Blob Storage and only filtering allowed today is by blob name prefix.
One solution to your problem is list all blobs in a container. Each blob has a property called Created Date/Time which tells you when the blob was first created (there's another property called Last Modified as well).
When you have the list, you can filter on the client side by this Created Date/Time property to get the desired list of blobs.
Related
I have an azure storage account.
Inside the container, with a client-specific folder structure, every morning, some files get pushed.
I have a function app which processes and converts these files and calls some external service to work upon on these processed files.
I have got a file-share as well, which is basically mounted on a vm.
The external service, after processing the files (#3), generates the resultant success/failure files inside this file-share(#4).
Now the ask is:
Create a simple dashboard which will monitor the storage account(and in effect the container and the file-shares),it should capture & show basic information's, and should look like below table structure(with 3 simple variations of data):
FileName|ReceivedDateTime|NumberOfRecords
Original_file.csv20221011 5:21 AM|10
Original_file_Success.csv20221011 5:31 AM|9
Original_file_Failure.csv20221011 5:32 AM|1
In here the first record is captured from the Container and the second and third - both are generated in the file-share.
Also, whenever a new failure file is generated, i.e., Original file_Failure, it should send email with a predefines template adding the file name to a predefined recipient list.
Any guidance on the azure service to use?
I have seen Azure Monitor,workbook and other stuffs, but I feel that would be an overkill for such simple requirement.
Thanks in advance.
I have a scenario where I have to fetch the latest folder from the blob storage container and then process all files under that folder through Azure data factory, currently, all folder name based on timestamp and as we know CloudBlobDirectory don't hold LastModified Date so there is no way to extract metadata from Azure data factory activity like last modified time so that I can iterate with the timestamp and process the content.
Is there any other way to perform something like sort on the folder name and then pick it based on string sort (on folder name )?
I tried something similar using Azure functions.
Please have a look and tell if its of use.
https://www.youtube.com/watch?v=eUMjghIEsjw
I have searched google and MSDN and it's not clear if you can write a variable to blob storage? Searching the available steps/actions does not yield anything obvious either.
I have constructed an array variable of file names from an SFTP in per the following documentation, but I can't figure out if this can be stored or saved in any capacity.
Right now it seems these variables are essentially internal to the logic app and can't be made external or is there a way to export them?
https://learn.microsoft.com/en-us/azure/logic-apps/logic-apps-create-variables-store-values
If you just simply want to save the variable's value in a blob then you can do so with the Azure Blob Storage - Create Blob action:
I have an azure function, which is binded to blob storage. Once the blob is successfully processed I rename the file with a suffix '-Processed'.
But my azure function again picks up the same blob for processing. I tried putting {name}.csv filter in the BlobTrigger binding but that didn't help as the file will still be a csv even after the rename.
I know I can filter blobs to have a particular string in file name, for eg "original-{name}" will filter files starting with original.
But Is there a way in azure functions using which I can filter the blob names to not include a particular string, in my case '-Processed'?
Just use two different paths for processed and not processed blobs.
Put your new blobs with prefix ("notprocessed-" for example), when renaming remove prefix. Set "path": "input/notprocessed-{name}"
Actually, blob service only supports filtering by blob prefix and not by suffix. Your only option would be to list blobs and then do client side filtering.
Also, the list blobs operation has an additional delimiter parameter that enables the caller to traverse the blob namespace by using a user-configured delimiter.
You could refer to this article for more details.
I want to create following folder structure on Azure:
mycontainer
-images
--2007
---img001.jpg
---img002.jpg
Now, one way is to use PUT Blob request and upload img001.jpg specifying the whole path as
PUT "mycontainer/images/2007/img001.jpg"
But, I want to first create the folders images and 2007 and then in a different request upload the blob img001.jpg.
Right now when I tried to doing this using PUT BLOB request:
StringToSign:
PUT
x-ms-blob-type:BlockBlob
x-ms-date:Tue, 07 Feb 2017 23:35:12 GMT
x-ms-version:2016-05-31
/account/mycontainer/images/
HTTP URL
sun.net.www.protocol.http.HttpURLConnection:http://account.blob.core.windows.net/mycontainer/images/
It is creating a folder but its not empty. By, default its creating an
empty blob file without name.
Now, a lot of people say we can't create a empty folder. But, then how come, we can make it using the azure portal as the browser must be sending some type of rest request to create the folder.
I think it has to do something with Content-Type i.e. x-ms-blob-content-type, which should be specified in order to tell azure that its a folder not a blob.
But, I am confused.
I want to first create the folders images and 2007 and then in a different request upload the blob img001.jpg
I agree with Brendan Green, currently, Azure blob storage just enable us to create virtual directory structure by naming blobs with path information in their names.
I think it has to do something with Content-Type i.e. x-ms-blob-content-type, which should be specified in order to tell azure that its a folder not a blob. But, I am confused.
You could check the description of Request Headers that could be set for Put Blob operation and you will find it does not support creating an empty folder by specifying some request headers.
Besides, as Gaurav Mantri said, if you really want to create an empty folder structure without content, you could try to use Azure File storage and it also enables us to use REST API to access Azure File storage. And the Create Directory operation cloud be used to create a new directory under the specified share or parent directory.
PUT https://myaccount.file.core.windows.net/myshare/myparentdirectorypath/mydirectory?restype=directory
This is not possible - the folder structure is virtual only.
See Get started with Azure Blob storage using .NET. You can only create a container, and everything else held in that container is a blob.
Excerpt:
As shown above, you can name blobs with path information in their
names. This creates a virtual directory structure that you can
organize and traverse as you would a traditional file system. Note
that the directory structure is virtual only - the only resources
available in Blob storage are containers and blobs.