Retention Policy for Azure Containers? - azure

I'm looking to set up a policy for one of my containers so it deletes or only retains data for x days. So if x is 30, that container should only contain files that are less than 30 days old. If the files are sitting in the container for more than 30 days it should discard them. Is there any way I can configure that?

Currently such kind of thing is not supported by Azure Blob Storage. You would need to write something of your own that would run periodically to do this check and delete old blobs.
On a side note, this feature has been long pending (since 2011): https://feedback.azure.com/forums/217298-storage/suggestions/2474308-provide-time-to-live-feature-for-blobs.
UPDATE
If you need to do it yourself, there are two things to consider:
Code to fetch the list of blobs, find out the blobs that need to be deleted and then delete those blobs. To do this, you can use Azure Storage SDK. Azure Storage SDK is available for many programming languages like .Net, Java, Node, PHP etc. You just need to use the one that you're comfortable with.
Schedule this code to run once on a daily basis: To do this, you can use one of the many services available in Azure. You can use Azure WebJobs, Functions, Schedular, Azure Automation etc.
If you decide to use Azure Automation, there's a Runbook already available for you that you can use (no need to write your code). You can find more details about this here: https://gallery.technet.microsoft.com/scriptcenter/Remove-Storage-Blobs-that-aae4b761.

Azure Blob storage Lifecycle (Preview) is now available and using that we create Policies with different rules.
Here is the rule to delete the blobs which are older than 30 days
{
"version": "0.5",
"rules": [
{
"name": "expirationRule",
"type": "Lifecycle",
"definition": {
"filters": {
"blobTypes": [ "blockBlob" ]
},
"actions": {
"baseBlob": {
"delete": { "daysAfterModificationGreaterThan": 30}
}
}
}
}
]
}
For more details refer this Azure Blob storage Lifecycle

Related

Azure storage deletion policy does not work

I need to delete all blobs residing in a particular container in my storage account after 1 day since creation. To do that I have tried to set a policy but I cant make it work so far.
I have tried to create the policy both by using the azure portal and by using terraform. The policy code view shown in the portal is the following:
{
"rules": [
{
"enabled": true,
"name": "delete_old_records_csv_files",
"type": "Lifecycle",
"definition": {
"actions": {
"baseBlob": {
"delete": {
"daysAfterCreationGreaterThan": 1
}
}
},
"filters": {
"blobTypes": [
"blockBlob"
],
"prefixMatch": [
"queryresults"
]
}
}
}
]
}
I have waited 2 days for the files to get deleted but they did not. Is there something I am doing wrong?
Thanks!
I have waited 2 days for the files to get deleted but they did not. Is there something I am doing wrong?
If the blobs are created before the creation of the lifecycle management rule they will not be deleted.
To apply policy for existing blobs, you can move or copy them to a new container where the policy will be applied.
Make sure If you are updating or new policy it may take up to 48Hrs to complete.
Portal:
Ensure that the blobs you trying to delete match the prefix "queryresults" as same as in the specified policy.
Also check if any restriction is applied to a container or blob like Immutablity policy and the blob is leased.
If everything is correct but still not being deleted, you can debug your storage account by using audit logs and if you get warning that will explain why the policy is not working.
Reference:
azure - Lifecycle management rule not working for ADLS Gen2 Storage Account - Stack Overflow by Joel Cochran.

What exact blob containers do I need to purge during a WebJob SDK log cleanup process?

** Problem Background **
As we know, Azure WebJob SDK, has no way of defining a retention policy for logs. That means the execution or dashboard Blob storage can grow and impose problems including slowing down or crash the kudu Dashboard – which could compromise the stability of the other apps in the App Service plan.
The problem stated here:
https://github.com/Azure/azure-webjobs-sdk/issues/560
https://github.com/Azure/azure-webjobs-sdk/issues/1050
https://github.com/Azure/azure-webjobs-sdk/issues/107
My web job functions are extensively logging and they are running more than 100,000 times a day. That means I have a huge amount of log files piled up in my storage.
** The Workaround approach that I am planning: **
I am planning to add a time trigger Functions to my WebJob code that purges log entries older than 30 days.
We have the following blob containers created or used by the WebJobs SDK:
1.Storage Connection: AzureWebJobsDashboard
1.1. azure-webjobs-dashboard
1.2. azure-jobs-host-archive
1.3. Duplicates with AzureWebJobsStorage
1.3.1 azure-jobs-host-output
1.3.2 azure-webjobs-host
2.Storage AzureWebJobsStorage
2.1. azure-jobs-host-output
2.2. azure-webjobs-host
2.2.1 Heartbeats
2.2.2 Ids
2.2.3 Output-logs
I am thinking to create a process that deletes every file older than 30 days from above containers. But I am concern that some of the blobs might be required by the running WebJobs.
** Question **
Which of the above blob containers do I need to purge, to prevent blob file pile-up problem without interfering running WebJobs ?
As far as I know, AzureWebJobsDashboard connection string account is used to store logs from the WebJobs Dashboard. This connection string is optional.
It will generate two container 'azure-webjobs-dashboard'and 'azure-jobs-host-archive'.
Azure-webjobs-dashboard: WebJob dashboard to store host and execution endpoint (function) details
Azure-jobs-host-archive: This is used as an archive for execution logs.
So both of these containers could be deleted without interfering running WebJobs.
azure-jobs-host-output is the key for troubleshooting web jobs. This container hosts logs created by the WebJob runtime during initialization and termination of every execution. If you don't want this log , you could delete it.
Azure-webjobs-host container in-turn hosts three directories:
Heartbeats – Containing 0 byte blogs for every heartbeat check performed on the service. If you don't want it, you could delete the old file.
Ids – Containing the directory with a single blog holding a unique identifier for this service.I don't suggest you delete this container's file.
Output-logs – Hosts the output of the explicit logs for each run. Explicit logs being logs introduced by WebJob developers within the execution code. You could delete the old log.
We've just implemented Storeage Lifecycle Management and are testing this:
{
"version": "0.5",
"rules": [
{
"name": "DeleteOldLogs",
"type": "Lifecycle",
"definition": {
"actions": {
"baseBlob": {
"delete": {
"daysAfterModificationGreaterThan": 30
}
}
},
"filters": {
"blobTypes": [
"blockBlob"
],
"prefixMatch": [
"azure-webjobs-host/output-logs",
"azure-webjobs-dashboard/functions/recent",
"azure-webjobs-dashboard/functions/instances",
"azure-jobs-host-output",
"azure-jobs-host-archive"
]
}
}
}
]
}

Why can't configure Azure diagnostics to use Azure Table Storage via new Azure Portal?

I am developing a web api which will be hosted in Azure. I would like to use Azure diagnostics to log errors to Azure table storage.
In the Classic portal, I can configure the logs to go to Azure table storage.
Classic Portal Diagnostic Settings
However in the new Azure portal, the only storage option I have is to use Blob storage:
New Azure Portal Settings
It seems that if I was to make use of a web role, I could configure the data store for diagnostics but as I am developing a web api, I don't want to create a separate web role for every api just so that I can log to an azure table.
Is there a way to programmatically configure azure diagnostics to propagate log messages to a specific data store without using a web role? Is there any reason why the new Azure portal only has diagnostic settings for blob storage and not table storage?
I can currently work around the problem by using the classic portal but I am worried that table storage for diagnostics will eventually become deprecated since it hasn't been included in the diagnostic settings for the new portal.
(I'll do some necromancy on this question as this was the most relevant StackOverflow question I found while searching for a solution to this as it is no longer possible to do this through the classic portal)
Disclaimer: Microsoft has seemingly removed support for logging to Table in the Azure Portal, so I don't know if this is deprecated or will soon be deprecated, but I have a solution that will work now (31.03.2017):
There are specific settings determining logging, I first found out information on this from an issue in the Azure Powershell github: https://github.com/Azure/azure-powershell/issues/317
The specific settings we need are (from github):
AzureTableTraceEnabled = True, & AppSettings has:
DIAGNOSTICS_AZURETABLESASURL
Using the excellent resource explorer (https://resources.azure.com) under (GUI navigation):
/subscriptions/{subscriptionName}/resourceGroups/{resourceGroupName}/providers/Microsoft.Web/sites/{siteName}/config/logs
I was able to find the Setting AzureTableTraceEnabled in the Properties.
The property AzureTableTraceEnabled has Level and sasURL. In my experience updating these two values (Level="Verbose",sasUrl="someSASurl") will work, as updating the sasURL sets DIAGNOSTICS_AZURETABLESASURL in appsettings.
How do we change this? I did it in Powershell. I first tried the cmdlet Get-AzureRmWebApp, but could not find what i wanted - the old Get-AzureWebSite does display AzureTableTraceEnabled, but I could not get it to update (perhaps someone with more powershell\azure experience can come with input on how to use the ASM cmdlets to do this).
The solution that worked for me was setting the property through the Set-AzureRmResource command, with the following settings:
Set-AzureRmResource -PropertyObject $PropertiesObject -ResourceGroupName "$ResourceGroupName" -ResourceType Microsoft.Web/sites/config -ResourceName "$ResourceName/logs" -ApiVersion 2015-08-01 -Force
Where the $PropertiesObject looks like this:
$PropertiesObject = #{applicationLogs=#{azureTableStorage=#{level="$Level";sasUrl="$SASUrl"}}}
The Level corresponds to "Error", "Warning", "Information", "Verbose" and "Off".
It is also possible to do this in the ARM Template (important bits is in properties on the logs resource in the site):
{
"apiVersion": "2015-08-01",
"name": "[variables('webSiteName')]",
"type": "Microsoft.Web/sites",
"location": "[resourceGroup().location]",
"tags": {
"displayName": "WebApp"
},
"dependsOn": [
"[resourceId('Microsoft.Web/serverfarms/', variables('hostingPlanName'))]"
],
"properties": {
"name": "[variables('webSiteName')]",
"serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('hostingPlanName'))]"
},
"resources": [
{
"name": "logs",
"type": "config",
"apiVersion": "2015-08-01",
"dependsOn": [
"[resourceId('Microsoft.Web/sites/', variables('webSiteName'))]"
],
"tags": {
"displayName": "LogSettings"
},
"properties": {
"azureTableStorage": {
"level": "Verbose",
"sasUrl": "SASURL"
}
}
}
}
The issue with doing it in ARM is that I've yet to find a way to generate the correct SAS, it is possible to fetch out Azure Storage Account keys (from: ARM - How can I get the access key from a storage account to use in AppSettings later in the template?):
"properties": {
"type": "AzureStorage",
"typeProperties": {
"connectionString": "[concat('DefaultEndpointsProtocol=https;AccountName=', variables('storageAccountName'),';AccountKey=',listKeys(resourceId('Microsoft.Storage/storageAccounts', variables('storageAccountName')), providers('Microsoft.Storage', 'storageAccounts').apiVersions[0]).keys[0].value)]"
}
}
There are also some clever ways of generating them using linked templates (from: http://wp.sjkp.dk/service-bus-arm-templates/).
The current solution I went for (time constraints) was a custom Powershell script that looks something like this:
...
$SASUrl = New-AzureStorageTableSASToken -Name $LogTable -Permission $Permissions -Context $StorageContext -StartTime $StartTime -ExpiryTime $ExpiryTime -FullUri
$PropertiesObject = #{applicationLogs=#{azureTableStorage=#{level="$Level";sasUrl="$SASUrl"}}}
Set-AzureRmResource -PropertyObject $PropertiesObject -ResourceGroupName "$ResourceGroupName" -ResourceType Microsoft.Web/sites/config -ResourceName "$ResourceName/logs" -ApiVersion 2015-08-01 -Force
...
This is quite an ugly solution, as it is something extra you need to maintain in addition to the ARM template - but it is easy, fast and it works while we wait for updates to the ARM Templates (or for someone cleverer than I to come and enlighten us).
We don't typically recommend using Tables for log data - it can result in the append only pattern which at scale doesn't work effectively for Table Storage. See the log-data anti-pattern in this guide Table Design Guide. Often times we see that even though people think of log data as structured - they way they typically query it makes Blobs more efficient.
Excerpt from the Design Guide:
Log data anti-pattern
Typically, you should use the Blob service instead of the Table service to store log data.
Context and problem
A common use case for log data is to retrieve a selection of log entries for a specific date/time range: for example, you want to find all the error and critical messages that your application logged between 15:04 and 15:06 on a specific date. You do not want to use the date and time of the log message to determine the partition you save log entities to: that results in a hot partition because at any given time, all the log entities will share the same PartitionKey value (see the section Prepend/append anti-pattern).
...
Solution
The previous section highlighted the problem of trying to use the Table service to store log entries and suggested two, unsatisfactory, designs. One solution led to a hot partition with the risk of poor performance writing log messages; the other solution resulted in poor query performance because of the requirement to scan every partition in the table to retrieve log messages for a specific time span. Blob storage offers a better solution for this type of scenario and this is how Azure Storage Analytics stores the log data it collects.
This section outlines how Storage Analytics stores log data in blob storage as an illustration of this approach to storing data that you typically query by range.
Storage Analytics stores log messages in a delimited format in multiple blobs. The delimited format makes it easy for a client application to parse the data in the log message.
Storage Analytics uses a naming convention for blobs that enables you to locate the blob (or blobs) that contain the log messages for which you are searching. For example, a blob named "queue/2014/07/31/1800/000001.log" contains log messages that relate to the queue service for the hour starting at 18:00 on 31 July 2014. The "000001" indicates that this is the first log file for this period. Storage Analytics also records the timestamps of the first and last log messages stored in the file as part of the blob’s metadata. The API for blob storage enables you locate blobs in a container based on a name prefix: to locate all the blobs that contain queue log data for the hour starting at 18:00, you can use the prefix "queue/2014/07/31/1800."
Storage Analytics buffers log messages internally and then periodically updates the appropriate blob or creates a new one with the latest batch of log entries. This reduces the number of writes it must perform to the blob service.
If you are implementing a similar solution in your own application, you must consider how to manage the trade-off between reliability (writing every log entry to blob storage as it happens) and cost and scalability (buffering updates in your application and writing them to blob storage in batches).
Issues and considerations
Consider the following points when deciding how to store log data:
If you create a table design that avoids potential hot partitions, you may find that you cannot access your log data efficiently.
To process log data, a client often needs to load many records.
Although log data is often structured, blob storage may be a better solution.

Automatically Delete/Expire Azure Blobs after a time period

With Azure Blob storage is it possible to either have an individual blob or all blobs within a container delete themselves after a certain period of time similar to Amazon AWS S3's Object Expiration Feature? Or does Azure storage not provide such functionality?
It is possible with the Azure Blob storage lifecycle. Please take a look here
https://learn.microsoft.com/en-us/azure/storage/blobs/storage-lifecycle-management-concepts?tabs=azure-portal
and Configure a lifecycle management policy
Because I've missed the feature for years I wrote a small project with a nice 'Deploy to Azure button'. Not yet perfect but works https://github.com/nulllogicone/ExpireBlobFunction
And now I see that Microsoft has released this as a feature on March 27, 2019.
Excerpt from that article:
Azure Blob storage lifecycle management offers a rich, rule-based
policy for GPv2 and Blob storage accounts. Use the policy to
transition your data to the appropriate access tiers or expire at the
end of the data's lifecycle.
The lifecycle management policy lets you:
Transition blobs to a cooler storage tier (hot to cool, hot to archive, or cool to archive) to optimize for performance and cost
Delete blobs at the end of their lifecycles
Define rules to be run once per day at the storage account level Apply rules to containers or a subset of blobs (using prefixes as filters)
You can auto-delete in various ways. Since long time you can do it even with Logical App but, sometimes, it is not so clear.
Today you have it directly available in your storage account:
and there you have an easy and specific task generator (template) to delete old blobs.
Looks like the feature is planned now at least: https://feedback.azure.com/forums/217298-storage/suggestions/2474308-provide-time-to-live-feature-for-blobs
The Azure Storage Team recently posted (Oct 5, 2017) an update on expiring blobs. It seems that this is now possible using an Azure Logic App template and they will have a native blob storage solution later this year.
Link: Provide Time to live feature for Blobs
We are pleased to announce that we have made an Azure Logic Apps template available to expire old blobs. To set up this automated solution in your environment: Create a new Logic Apps instance, select the “Delete old Azure blobs” template, customize and run. We will release a blog post detailing instructions and providing more templates in the coming weeks.
Allowing users to define expiration policies on blobs natively from storage is still planned for the coming year. As soon as we have progress to share, we will do so. We will continue to provide updates at least once per quarter.
For any further questions, or to discuss your specific scenario, send us an email at azurestoragefeedback#microsoft.com.
Azure Storage does not have an expiration feature; you must delete blobs via your app. How you do this is up to you; you'll need to store your expiration date target somewhere (whether in a database or in blob properties).
You can effectively create TTL on blob access, via Shared Access Signatures (by setting an end-date on the SAS). This would let you have an effective way of removing access when it's time to remove access, and then have a follow-on process remove the now-expired blobs.
Yes, it's possible. Refer these two, it was hard finding out the sample code.
Rules reference: https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview?tabs=azure-portal
Python sample code reference: https://github.com/Azure-Samples/azure-samples-python-management/blob/master/samples/storage/manage_management_policy.py
Code snippet I used:
def add_expiry_rule(self):
token_credential = ClientSecretCredential(
tenant_id=tenant_id,
client_id=client_id,
client_secret=client_secret,
)
storage_client = StorageManagementClient(
credential=token_credential, subscription_id=subscription_id
)
rule = {
"id": "test",
"prefix": "test/",
"expiration": 91,
}
azure_rule = {
"enabled": True,
"name": rule.get("id"),
"type": "Lifecycle",
"definition": {
"filters": {"blob_types": ["blockBlob"], "prefix_match": [rule.get("prefix")]},
"actions": {
"base_blob": {
"delete": {
"days_after_modification_greater_than": str(rule.get("expiration"))
}
}
},
},
}
try:
management_policy = storage_client.management_policies.get(
group_name, storage_account, "default"
)
existing_rules = management_policy.policy.as_dict()
existing_rules.get("rules").append(azure_rule)
management_policy_rules = existing_rules
except Exception as e:
management_policy_rules = {"rules": [azure_rule]}
try:
management_policy = storage_client.management_policies.create_or_update(
group_name,
storage_account,
"default",
{"policy": management_policy_rules},
)
print("Azure: Added rule {} successfully".format(rule.get("id")))
except Exception as e:
if e.message.endswith("conflicting rule name."):
print("Azure: Rule ID: {} exists".format(rule.get("id")))
else:
raise Exception("Azure: Error adding rule. {}".format(e.message))

What is the best way to backup Azure Blob Storage contents

I know that the Azure Storage entities (blobs, tables and queues) have a built-in resiliency, meaning that they are replicated to 3 different servers in the same datacenter. On top of that they may also be replicated to a different datacenter altogether that is physically located in a different geographical region. The chance of losing your data in this case is close to zero for all practical purposes.
However, what happens if a sloppy developer (or the one under the influence of alcohol :)) accidentally deletes the storage account through the Azure Portal or the Azure Storage Explorer tool? Worst yet, what if a hacker gets hold of your account and clears the storage? Is there a way to retrieve the gigabytes of deleted blobs or is that it? Somehow I think there has to be an elegant solution that Azure infrastructure provides here but I cannot find any documentation.
The only solution I can think of is to write my own process (worker role) that periodically backs up my entire storage to a different subscription/account, thus essentially doubling the cost of storage and transactions.
Any thoughts?
Regards,
Archil
Depending on where you want to backup your data, there are two options available:
Backing up data locally - If you wish to backup your data locally in your infrastructure, you could:
a. Write your own application using either Storage Client Library or consuming REST API or
b. Use 3rd party tools like Cerebrata Azure Management Cmdlets (Disclosure: I work for Cerebrata).
Backing up data in the cloud - Recently, Windows Azure Storage team announced Asynchronous Copy Blob functionality which will essentially allow you to copy data from one storage account to another storage account without downloading the data locally. The catch here is that your target storage account should be created after 7th June 2012. You can read more about this functionality on Windows Azure Blog: http://blogs.msdn.com/b/windowsazurestorage/archive/2012/06/12/introducing-asynchronous-cross-account-copy-blob.aspx.
Hope this helps.
The accepted answer is fine, but it took me a few hours to decipher through everything.
I've put together solution which I use now in production. I expose method Backup() through Web Api which is then called by an Azure WebJob every day (at midnight).
Note that I've taken the original source code, and modified it:
it wasn't up to date so I changed few method names
added retry copy operation safeguard (fails after 4 tries for the same blob)
added a little bit of logging - you should swap it out with your own.
does the backup between two storage accounts (replicating containers & blobs)
added purging - it gets rid of old containers that are not needed (keeps 16 days worth of data). you can always disable this, as space is cheap.
the source can be found from: https://github.com/ChrisEelmaa/StackOverflow/blob/master/AzureStorageAccountBackup.cs
and this is how I use it in the controller (note your controller should be only callable by the azure webjob - you can check credentials in the headers):
[Route("backup")]
[HttpPost]
public async Task<IHttpActionResult> Backup()
{
try
{
await _blobService.Backup();
return Ok();
}
catch (Exception e)
{
_loggerService.Error("Failed to backup blobs " + e);
return InternalServerError(new Exception("Failed to back up blobs!"));
}
}
note: I wanted to add this code as part of the post, but wasted 6 minutes trying to get that code into this post, but failed. the formatting didn't work at all, and it broke completely.
I have used Azure Data Factory to backup Azure storage with great effect. It's really easy to use, cost effective and work very well.
Simply create a Data Factory (v2), set up data connections to your data sources (it currently supports Azure Tables, Azure Blobs and Azure Files) and then set up a data copy pipeline.
The pipelines can merge, overwrite, etc. and you can set up custom rules/wildcards.
Once you've set up the pipeline, you should then set up a schedule trigger. This will kick off the backup at an interval to suit your needs.
I've been using it for months and it's perfect. No code, no VMS, no custom PowerShell scripts or third party software. Pure Azure solution.
I have had exactly the same requirements: backing up blobs from Azure as we have millions of blobs of customers and you are right - a sloppy developer with full access can compromise the entire system.
Thus, I wrote an entire application "Blob To Local Backup", free and open source on github under the MIT license: https://github.com/smartinmedia/BlobToLocalBackup
It solves many of your issues, namely:
a) you can give only READ-access to this application, so that the application cannot destroy any data on Azure
b) backup to a server, where your sloppy developer or the hacker does not have the same access as to your Azure account.
c) The software provides versioning, so you can even protect yourself from e. g. ransom/encryption attacks.
d) I included a serialization method instead of a database, so you can even have millions of files on Azure and you are still able to keep the synch (we have 20 million files on Azure).
Here is how it works (for more detailed information, read the README on github):
You set up the appsettings.json file in the main folder. You can give the LoginCredentials here for the entire access or do it more granular on a storage account level:
{
"App": {
"ConsoleWidth": 150,
"ConsoleHeight": 42,
"LoginCredentials": {
"ClientId": "2ab11a63-2e93-2ea3-abba-aa33714a36aa",
"ClientSecret": "ABCe3dabb7247aDUALIPAa-anc.aacx.4",
"TenantId": "d666aacc-1234-1234-aaaa-1234abcdef38"
},
"DataBase": {
"PathToDatabases": "D:/temp/azurebackup"
},
"General": {
"PathToLogFiles": "D:/temp/azurebackup"
}
}
}
Set up a job as a JSON file like this (I have added numerous options):
{
"Job": {
"Name": "Job1",
"DestinationFolder": "D:/temp/azurebackup",
"ResumeOnRestartedJob": true,
"NumberOfRetries": 0,
"NumberCopyThreads": 1,
"KeepNumberVersions": 5,
"DaysToKeepVersion": 0,
"FilenameContains": "",
"FilenameWithout": "",
"ReplaceInvalidTargetFilenameChars": false,
"TotalDownloadSpeedMbPerSecond": 0.5,
"StorageAccounts": [
{
"Name": "abc",
"SasConnectionString": "BlobEndpoint=https://abc.blob.core.windows.net/;QueueEndpoint=https://abc.queue.core.windows.net/;FileEndpoint=https://abc.file.core.windows.net/;TableEndpoint=https://abc.table.core.windows.net/;SharedAccessSignature=sv=2019-12-12&ss=bfqt&srt=sco&sp=rl&se=2020-12-20T04:37:08Z&st=2020-12-19T20:37:08Z&spr=https&sig=abce3e399jdkjs30fjsdlkD",
"FilenameContains": "",
"FilenameWithout": "",
"Containers": [
{
"Name": "test",
"FilenameContains": "",
"FilenameWithout": "",
"Blobs": [
{
"Filename": "2007 EasyRadiology.pdf",
"TargetFilename": "projects/radiology/Brochure3.pdf"
}
]
},
{
"Name": "test2"
}
]
},
{
"Name": "martintest3",
"SasConnectionString": "",
"Containers": []
}
]
}
}
Run the application with your job with:
blobtolocal job1.json
Without referring to 3rd party solutions, you can achieve that using built in features in Azure now using the below steps might help secure your blob.
Soft delete for Azure Storage Blobs
The better step is first to enable soft delete which is now in GA:
https://azure.microsoft.com/en-us/blog/soft-delete-for-azure-storage-blobs-ga
Read-access geo-redundant storage
The second approach is enable geo-replication for RA-RGA, so if the first data center is down you can always read from a secondary replica in another region, you can find more information in here:
https://learn.microsoft.com/en-us/azure/storage/common/storage-redundancy-grs
You can make a snapshot of a blog container and then download the snapshot for a point in time backup.
https://learn.microsoft.com/en-us/azure/storage/storage-blob-snapshots
A snapshot is a read-only version of a blob that's taken at a point in
time. Snapshots are useful for backing up blobs. After you create a
snapshot, you can read, copy, or delete it, but you cannot modify it.+
A snapshot of a blob is identical to its base blob, except that the
blob URI has a DateTime value appended to the blob URI to indicate the
time at which the snapshot was taken. For example, if a page blob URI
is http://storagesample.core.blob.windows.net/mydrives/myvhd, the
snapshot URI is similar to
http://storagesample.core.blob.windows.net/mydrives/myvhd?snapshot=2011-03-09T01:42:34.9360000Z.

Resources