I want to remove containers and blobs except a specified container after 7 days in Azure Blob Storage. But now I only know the way how to apply filter in the specified prefix folders but don't know how to exclude specified container of blob. Is there any way to attrieve this?
So far, Lifecycle Management Policy only supports prefixMatch. As explained in this documentation : "The prefix match considers only positive (=) logical comparisons. Negative (!=) logical comparisons are ignored.".
For the moment, you would have to separate your blobs into different containers and/or find appropriate positively matching rules to select the blobs you want the lifecycle management policy to have effect on.
Related
So I have a quick question regarding filter prefixes related to lifecycle management for azure storage account V2.
So the scenario I'm faced with is that I have a blob directory/container which in turn contains sub directories created dynamically via a function that pushes/creates blobs depending on conditions, so the directories are created depending on that logic.
The problem I want to solve is that I want to delete the blobs after 7 days.
In the documentation for lifecycle management it says that I can set a filter prefix for which container I want to apply the "retention rule" for, so to speak.
So the question related to what I'm trying to do is the following:
When setting the filter prefix for a blob container to: "containerName/",
as it says to do in the documentation will it also look in the subfolders?
In the Microsoft documentation it says:
"A prefix match string like container1/ applies to all blobs in the
container named container1."
Does that also include all the blobs in all the subfolders automatically. or do I have to specify each subfolder after the slash as it says further down in the same part of the documentation?
I would like to include all blobs in that first container regardless if they are in subfolders or not as the subfolders are created dynamically as mentioned before.
Does that also include all the blobs in all the subfolders
automatically. or do I have to specify each subfolder after the slash
as it says further down in the same part of the documentation?
Yes, when you set the prefix as container name, all blobs (including those in subfolders) will be considered thus you need not specify subfolders specifically.
You would specify subfolder in prefix only when you want to lifecycle management to manage blobs inside a specific subfolder.
I am having set of folders in container named records (Azure storage account). In general what ever the blobs(folders) present in records container it will be deleted as per lifecycle management rule.
Rule: if blob exists more than 30days than it will delete the blob.
But As per my case, All blobs (folders) should delete except one blob (folder) where the blob(folder) name is Backup in the container.
Is there any way to add a rule for not deleting particular blob(In my case it is folder)?
So backup folder shouldn't delete when the existing rule run.
Create a lease for the particular blob using the azure portal for example. A lease prevents processes from doing anything with the blob. This includes lifecycle management rules.
You can also acquire or break a lease using the rest api or one of the many storage SKDs.
Another option would be to not use the lifecycle management rules but write a scheduled azure function that deletes blob older than 30 days except the ones having backup in their name.
Please do note: if you have enabled "Hierarchical namespace" then you have the concept of directories, but those cannot be leased. If you did not then you should realise that folders are a virtual construct and as such cannot be leased as they are actually blobs. See the docs. So in that case you have to individually take a lease on each blob or write a script that does it once.
I want to create an Azure Policy for storage accounts. During the creation of a storage account, it will count the number of storage accounts in the subscription which has a specific tag (like env:dev). If the number of this count exceeds 10, It should deny the creation of that storage account.
How I can count the number of storage account which has certain tag inside an Azure Policy declaration?
That is not something you can do using Azure Policy.
Even though the approach seems valid, the count operator is meant to be used with [*] aliases - see official documentation. One could think of scanning Microsoft.Subscription aliases, but running the command az provider show --namespace Microsoft.Subscription --expand "resourceTypes/aliases" --query "resourceTypes[].aliases[].name", what you get as an output is so far not extensive :
[
"Microsoft.Subscription/SubscriptionDefinitions/subscriptionId",
"Microsoft.Subscription/SubscriptionDefinitions/subscriptionDisplayName",
"Microsoft.Subscription/SubscriptionDefinitions/offerType",
"Microsoft.Subscription/SubscriptionDefinitions/etag",
"Microsoft.Subscription/aliases/subscriptionId"
]
Also, the way the policy engine works wouldn't allow you to count the number of instances meeting a particular rule : each ARM component is scanned individually and compared to the rules defined in the policy. You cannot see "all Storage Accounts in a given scope" - each one is scanned without the whole picture being seen at any point. That is why I started with looking if the Subscription object had something like a Resources [*] alias.
I have azure storage account on which I'm enabling life cycle management.
Is it possible to set filter at container level?
eg Container/folder1/x.txt
Container/folder2/y.txt
I want the life management filter set to be applied at container level instead of filtering by blob name prefix.
The only filter on container level is like below:
Suppose you have 3 containers, like container-1, container-2, container3, then you can set the filter as container-, then the policy is only applied to container-1, container-2.
So in your case, you can directly set the filter as the container name, like Container.
To apply filtering at the container level, simply specify the container name as blob prefix.
I'm currently trying to implement lifecycle management on a container where i have embeded folders as follows:
container1/1/test/done_filename.txt
container1/1/processed/complete_filename.txt
container1/1/wip_filename.txt
i tried creating a rule filter to delete files under the test folder that start with done however i waited 48 hours and it's still not removed.
Rule filter:
delete after 1 day
Prefix Match:
file with this prefix: container1/1/test/done_
All samples I see from microsoft and online only mention "myContainer/prefix", am i to understand that the lifecycle management feature doesn't support subfolders/full paths? do all files have to be within the the root of the container? or what am i doing wrong here.
Updated showing rule definition:
I expect the above to delete any file under "container1/1/test/" that starts with "done_" but the file is still there when i check after a few days even though the file has not been modified.
If you have enabled the rule, then your policy and prefix match are ok.
You need to check some settings which can block the blob delete operation:
1.Make sure you can manually delete these blobs. You can try to delete one of them manually. Why we need to do that? If you have leased these blobs or set immutability policies, the delete operation are not allowed.
2.Check the settings in Networking pane of your storage account. And make sure you select the option "Allow access from All networks":
I was able to find the solution here: https://learn.microsoft.com/answers/answers/154361/view.html
When you are using selected networks because of security restrictions like we do then correct solution is to enable “Allow trusted Microsoft services to access this storage account” under Networking.