Azure lifecycle policy for container is not working - azure

I have configured azure lifecycle policy for container as below,
{
"rules": [
{
"enabled": true,
"name": "name",
"type": "Lifecycle",
"definition": {
"actions": {
"baseBlob": {
"delete": {
"daysAfterModificationGreaterThan": 30
}
}
},
"filters": {
"blobTypes": [
"blockBlob"
],
"prefixMatch": [
"storageaccount/container"
]
}
}
}
]
}
So it will delete the blobs which was modified before 30 days. I am putting backups in the container so I want to delete the old backups which is 30 days older.
I configured this policy before 2 days before. Yet old backup files are not removed from the container
I analyzed and as per below links after the new policy configuration it will take up 24 hrs to take effect for new policy and policy update
ref:https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview?tabs=azure-portal#faq
Lifecycle management policy not working on Azure Data Lake Gen 2
I cant find many documents regarding this issue also
I configured Firewalls and virtual networks as 'all networks'. why i am mention this because according to below documents his solution works,
https://learn.microsoft.com/en-us/answers/questions/107954/lifecycle-mangement-isn39t-doing-anything.html
Yet there is no update in containers. So anyone know the reason and troubleshoots to resolve this?
thank you!

Please check if below points can help:
Things to check:
Please check if there any immutability policies configured for blob
versions .
If version-level immutability support is enabled for a container and
the container contains one or more blobs, then you must delete all
blobs in the container before you can delete the container, even if
there are no immutability policies in effect for the container or its
blobs.
Please check if last access time tracking is enabled, because
every time the blobs are accessed they move to hot tier and may
change its modified time. You need to make sure you are using a V2
storage account: optionally-enable-access-time-tracking
Work around :
Use az cli to add life cycle policy to delete the blobs in the container whose last modification time is more than 30 days.
Note: (Try simply specifying the container name as blob prefixcontainer name in prefixmatch instead of storageaccount/container)
Ex policy.json :
"rules": [
{
"name": "expirationRule1",
"enabled": true,
"type": "Lifecycle",
"definition": {
"filters": {
"blobTypes": [ "blockBlob" ],
"prefixMatch": [ "containername" ]
},
"actions": {
"baseBlob": {
"delete": { "daysAfterModificationGreaterThan": 30 }
}
}
}
}
]
}
Try to create this lifecycle management policy (policy.son) using the az cli command:
az storage account management-policy create --account-name youraccountname --policy #policy.json --resource-group myresourcegroup
References:
lifecycle - Unable to add new rule in Storage Management Policy on
Azure - Stack Overflow
How to automatically delete old Azure Blob Storage containers -
Stack Overflow

Related

Azure storage deletion policy does not work

I need to delete all blobs residing in a particular container in my storage account after 1 day since creation. To do that I have tried to set a policy but I cant make it work so far.
I have tried to create the policy both by using the azure portal and by using terraform. The policy code view shown in the portal is the following:
{
"rules": [
{
"enabled": true,
"name": "delete_old_records_csv_files",
"type": "Lifecycle",
"definition": {
"actions": {
"baseBlob": {
"delete": {
"daysAfterCreationGreaterThan": 1
}
}
},
"filters": {
"blobTypes": [
"blockBlob"
],
"prefixMatch": [
"queryresults"
]
}
}
}
]
}
I have waited 2 days for the files to get deleted but they did not. Is there something I am doing wrong?
Thanks!
I have waited 2 days for the files to get deleted but they did not. Is there something I am doing wrong?
If the blobs are created before the creation of the lifecycle management rule they will not be deleted.
To apply policy for existing blobs, you can move or copy them to a new container where the policy will be applied.
Make sure If you are updating or new policy it may take up to 48Hrs to complete.
Portal:
Ensure that the blobs you trying to delete match the prefix "queryresults" as same as in the specified policy.
Also check if any restriction is applied to a container or blob like Immutablity policy and the blob is leased.
If everything is correct but still not being deleted, you can debug your storage account by using audit logs and if you get warning that will explain why the policy is not working.
Reference:
azure - Lifecycle management rule not working for ADLS Gen2 Storage Account - Stack Overflow by Joel Cochran.

Need to deploy the Azure Policy for the Tags only for the VM

I am deploying the Azure policy for the Recommended Tags that need to be applied when anyone creates the new VM.
I found one in-built policy: Require a tag on resources
But when I deployed, it will be applied to all the resources and I need a policy for only VM resources.
Also how I can use more than one tag in a single policy?
In your policy rule, you must indicate that the policy is just for VMs
For example:
...
"policyRule": {
"if": {
"field": "type",
"in": [
"Microsoft.Compute/virtualMachines",
"Microsoft.ClassicCompute/virtualMachines"
]
},
"then": {
...
}
}
...
Hope this helps!

Azure Policy - Check Blob container access level

I am creating a policy to check if the Blob container access level is set to "Anonymous" in Azure Storage accounts.
This is the policy that i have created.
{
"properties": {
"displayName": "check if Blob container access level is set to Anonymous",
"description": "check the container access level",
"mode": "all",
"policyRule": {
"if": {
"allOf": [
{
"field": "type",
"equals": "Microsoft.Storage/storageAccounts/blobServices/containers"
},
{
"not": {
"field": "Microsoft.Storage/storageAccounts/containers/publicAccess",
"equals": "False"
}
}
]
},
"then": {
"effect": "Audit"
}
}
}
}
Its not able to detect the container access level.
As the issue said, Storage team is releasing public access setting on storage account towards Jun 30 2020. Customers can use it to control the public access on all containers in the storage account.
After it's released on storage, we will work with Azure Policy team to integrate the setting with Azure Policy so customers can us Azure Policy to audit and govern public access across storage accounts.
We work the best to ship above features and capabilities as early as possible.

Use Azure Container Instances via ARM to create an indeterminate number of containers

I am attempting to deploy an Azure Storage account along with an indeterminate number of tables via an ARM template.
Since MS are yet to provide a tables resource type for ARM, I'm instead using Azure Container Instances to spin up a container running azure-cli and then create the table that way.
As you can see in my example below, I'm using property iteration to create multiple containers - one for each table. This seemed to be working until the number of tables to create changed, and then I started getting errors.
The updates on container group 'your-aci-instance' are invalid. If you are going to update the os type, restart policy, network profile, CPU, memory or GPU resources for a container group, you must delete it first and then create a new one.
I understand what it's saying, but it does seem strange to me that you can create a container group yet not alter the group of containers within.
As ARM doesn't allow you do delete resources, I'd have to add a manual step to my deployment process to ensure that the ACI doesn't exist, which isn't really desirable.
Equally undesirable would be to use resource iteration to create multiple ACI's - there would be the possibility of many ACI's being strewn about the Resource Group that will never be used again.
Is there some ARM magic that I don't yet know about which can help me achieve the creation of tables that meets the following criteria?
Only a single ACI is created.
The number of tables to be created can change.
Notes
I have tried to use variable iteration to create a single 'command' array for a single container, but it seems that ACI considers all commands as a one liner, so this caused an error.
Further reading suggests that it is only possible to run one command on container startup.
How do I run multiple commands when deploying a container group?
Current attempt
Here is a snippet from my ARM template showing how I used property iteration to try and achieve my goal.
{
"condition": "[not(empty(variables('tables')))]",
"type": "Microsoft.ContainerInstance/containerGroups",
"name": "[parameters('containerInstanceName')]",
"apiVersion": "2018-10-01",
"location": "[resourceGroup().location]",
"properties": {
"copy": [
{
"name": "containers",
"count": "[max(length(variables('tables')), 1)]",
"input": {
"name": "[toLower(variables('tables')[copyIndex('containers')])]",
"properties": {
"image": "microsoft/azure-cli",
"command": [
"az",
"storage",
"table",
"create",
"--name",
"[variables('tables')[copyIndex('containers')]]"
],
"environmentVariables": [
{
"name": "AZURE_STORAGE_KEY",
"value": "[listkeys(parameters('storageAccount_Application_Name'), '2019-04-01').keys[0].value]"
},
{
"name": "AZURE_STORAGE_ACCOUNT",
"value": "[parameters('storageAccount_Application_Name')]"
},
{
"name": "DATETIME",
"value": "[parameters('dateTime')]"
}
],
"resources": {
"requests": {
"cpu": "1",
"memoryInGb": "1.5"
}
}
}
}
}
],
"restartPolicy": "OnFailure",
"osType": "Linux"
},
"dependsOn": [
"[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccount_Application_Name'))]"
],
"tags": {
"displayName": "Application Storage Account - Tables",
"Application": "[parameters('tagApplication')]",
"environment": "[parameters('tagEnvironment')]",
"version": "[parameters('tagVersion')]"
}
}
If it says the field is immutable - it is, there's nothing you can do about it really. You can always create a unique name for that container instance and use complete deployment mode and only deploy ACI to this particular resource group, that way it will always have only this ACI instance and others will get deleted and it will work around immutability.
you can call an azure function from inside the template (HTTP trigger) and pass in names of storage tables to create and it will do that, for example.
But either way its a hack.

Retention Policy for Azure Containers?

I'm looking to set up a policy for one of my containers so it deletes or only retains data for x days. So if x is 30, that container should only contain files that are less than 30 days old. If the files are sitting in the container for more than 30 days it should discard them. Is there any way I can configure that?
Currently such kind of thing is not supported by Azure Blob Storage. You would need to write something of your own that would run periodically to do this check and delete old blobs.
On a side note, this feature has been long pending (since 2011): https://feedback.azure.com/forums/217298-storage/suggestions/2474308-provide-time-to-live-feature-for-blobs.
UPDATE
If you need to do it yourself, there are two things to consider:
Code to fetch the list of blobs, find out the blobs that need to be deleted and then delete those blobs. To do this, you can use Azure Storage SDK. Azure Storage SDK is available for many programming languages like .Net, Java, Node, PHP etc. You just need to use the one that you're comfortable with.
Schedule this code to run once on a daily basis: To do this, you can use one of the many services available in Azure. You can use Azure WebJobs, Functions, Schedular, Azure Automation etc.
If you decide to use Azure Automation, there's a Runbook already available for you that you can use (no need to write your code). You can find more details about this here: https://gallery.technet.microsoft.com/scriptcenter/Remove-Storage-Blobs-that-aae4b761.
Azure Blob storage Lifecycle (Preview) is now available and using that we create Policies with different rules.
Here is the rule to delete the blobs which are older than 30 days
{
"version": "0.5",
"rules": [
{
"name": "expirationRule",
"type": "Lifecycle",
"definition": {
"filters": {
"blobTypes": [ "blockBlob" ]
},
"actions": {
"baseBlob": {
"delete": { "daysAfterModificationGreaterThan": 30}
}
}
}
}
]
}
For more details refer this Azure Blob storage Lifecycle

Resources