What is Azure Lease Blob?
Is it just a mechanism which provides an exclusive lock to the blob storage?
What is special about it? Where can I use it? I tried to read Azure documents, but it's not clear to me yet.
What is Azure Lease Blob? Is it just a mechanism which provides an
exclusive lock to the blob storage?
Your understanding is correct. When you lease a blob, you acquire an exclusive lock on that blob. As long as the blob is leased, no one other than lease holder can modify or delete the blob. You can either acquire a temporary lease, duration of which could be anywhere between 15 to 60 seconds or an infinite lease.
Where can I use it?
Most commonly this functionality is used by Azure Virtual Machines. As you know Azure VMs are backed by Azure Page Blobs. So when a VM is created, an infinite lease is acquired on the blob holding the VHD for that Azure VM so that no other process can modify or delete that blob.
Yet another place where you would want to use lease blob functionality is when you want to implement Leader Election pattern.
For example, consider a scenario where you would want to process a file stored in blob storage through a background Worker Role. Further assume that there are multiple instances of that Worker Role running where each instance wakes up after 30 seconds and checks for blobs in a container and if any blob is found that instance processes it.
Since Azure Worker Roles are stateless in nature, without locking that blob each instance will process that blob (not something you would want). To avoid this situation what you could do is have each instance try to acquire a lease on that blob. Only one instance will succeed (and hence elected as leader). Other instances will fail to acquire the lock on that blob. The leader instance will process the blob.
Another example would be where you want to distribute work in Competing Consumers where you would want to distribute work among them. In one of our products, we are using this pattern with Leader Election pattern. Through blob lease functionality, we find a leader (consumer which was able to acquire blob lease) and then this leader distributes work amongst other consumers.
In Microservice Architecture, we can also use it as a distributed lock. Imaging many instances/services try to do the same task on a resource/object/artifact, anyone can acquire the lock can do the job. Otherwise, it has to wait until lock gets released.
Related
I am reading the data from one blob storage lot of excel files (in For loop). out of 450 files 2 to 3 CSV files not creating in blob storage.
What is the reason for this? Same file it is processing next day. We are getting the below error.
BadRequest. The provided workflow action input is not valid.
It is showing the file in blob storage lease storage is Expired like below.
Thank you
Venu.
Can you provide the complete error message?
How are the Logic App , Workflows or API connections deployed? Do you see any error on the overview page of the Logic App?
Sometime NVA(Gateway Server) will be blocking the traffic to Azure services to save the workflows in Standard Logic App, (network may be blocked on the firewall to connect to Azure Connector endpoints.)please cross verify
Let me answer your expired part. Once a lease has expired, the lease ID is maintained by the Blob service until the blob is modified or leased again. A client may attempt to renew or release their lease using their expired lease ID and know that if the operation is successful, the blob has not been changed since the lease ID was last valid.
If the client attempts to renew or release a lease with their previous lease ID and the request fails, the client then knows that the blob was modified or leased again since their lease was last active. The client must then acquire a new lease on the blob.
If a lease expires rather than being explicitly released, a client may need to wait up to one minute before a new lease can be acquired for the blob. However, the client can renew the lease with their lease ID immediately if the blob has not been modified.
Note that a lease cannot be granted for a blob snapshot, since snapshots are read-only. Requesting a lease against a snapshot results in status code 400 (Bad Request).
The blob's Last-Modified-Time property is not updated by calls to Lease Blob.
The following tables show outcomes of actions on blobs with leases in various lease states. Letters (A), (B), and (C) represent lease IDs, and (X) represents a lease ID generated by the Blob service.
I am thinking of using Azure Blob Storage for document management system which I am developing. All Blobs ( images,videos, word/excel/pdf etc) will be stored in Azure Blob storage. As I understand, I need to create container and these files can be stored within the container.
I would like to know how to safeguard against accidental/malicious deletion of the container. If a container is deleted, all the files it contains will be lost. I am trying to figure out how to put backup and recovery mechanism in place for my storage account so that it is always guaranteed that if something happens to a container, I can recover files inside it.
Is there any way provided by Microsoft Azure for such backup and recovery or Do I need explicitly write a code in such a way that files are stored in two separate Blob storage account.
Anyone with access to your storage account's key (primary or secondary; there are two keys for a storage account) can manipulate the storage account in any way they see fit. The only way to ensure nothing happens? Don't give anyone access to the key(s). If you place the storage account within a resource group that only you have permissions on, you'll at least prevent others with access to the subscription from discovering the storage account and accessing it.
Within the subscription itself, you can place a lock on the actual resource (the storage account), so that nobody with access to the subscription accidentally deletes the entire storage account.
Note: with storage account keys, you do have the ability to regenerate the keys at any time. So if you ever suspected a key was compromised, you can perform a re-gen action.
Backups
There are several backup solutions offered for blob storage in case if containers get deleted.more product info can be found here:https://azure.microsoft.com/en-us/services/backup/
Redundancy
If you are concerned about availability, "The data in your Microsoft Azure storage account is always replicated to ensure durability and high availability. Replication copies your data, either within the same data center, or to a second data center, depending on which replication option you choose." , there are several replication options:
Locally redundant storage (LRS)
Zone-redundant storage (ZRS)
Geo-redundant storage (GRS)
Read-access geo-redundant storage (RA-GRS)
More details can be found here:
https://learn.microsoft.com/en-us/azure/storage/common/storage-redundancy
Managing Access
Finally, managing access to your storage account would be the best way to secure and ensure you'll avoid any loss on your data. You can provide read access only if you don't want anyone to delete files,folders etc.. through the use of SAS: Shared Access Signatures, allows you to create policies and provide access based on Read, Write, List, Delete, etc.. A quick GIF demo can be seen here: https://azure.microsoft.com/en-us/updates/manage-stored-access-policies-for-storage-accounts-from-within-the-azure-portal/
We are using blob to store documents and for documents management.
To prevent deletion of the blob, you can now enable soft deletion as described in here:
https://azure.microsoft.com/en-us/blog/soft-delete-for-azure-storage-blobs-ga/
You can also create your own automation around powershell,azcopy to do incremental and full backups.
The last element would be to use RA-GRS blobs where you can read from a secondary blob in read mode in another region in case the data center goes down.
Designing Highly Available Applications using RA-GRS
https://learn.microsoft.com/en-us/azure/storage/common/storage-designing-ha-apps-with-ragrs?toc=%2fazure%2fstorage%2fqueues%2ftoc.json
Use Microsoft's Azure Storage Explorer. It will allow you to download the full contents of blob containers including folders and subfolders with blobs. Conversely, you can upload to containers in the same way. Simple and free!
We have a very large blob container containing blob blocks which we Encrypted using our own encryption.
We now wish to move all these blobs to a new storage container which will use Azure's encryption at rest
The only way I can think of is to write a worker role that downloads to a stream decrypts it and uploads it. One at a time ...
That will probably take a while. Are they any other faster ways one can think of.Is there a way to parallelize this
As David Makogon mentioned that there are lots of ways to copy blobs.
Is there a way to parallelize this.
Based on my experience, we could use WebJob queue trigger to do that.
We could list the container blob names and write the blob url to storage queue or service bus queue.
In the webjob with your owner logic that download the blobs rid of your encryption and upload to another container.
Then we could scale out the App service plan or use multiple webjobs to do that.
Could you explain me how to delete BLOBs with blocked lease?
I have error message after try to delete:
There is currently a lease on the blob and no lease ID was specified in the request.
Property of Lease of that BLOBs objects is: Blocked
The article has a power shell script to unlock the lease of the blob files
https://msdn.microsoft.com/en-us/library/jj919145.aspx?f=255&MSPPError=-2147217396
Exract from the link
When backing up to or restoring from Windows Azure storage, SQL Server acquires an infinite lease in order to lock exclusive access to the blob. When the backup or restore process is successfully completed, the lease is released. If a backup or restore fails, the backup process attempts to clean up any invalid blob. However, if the backup fails due to prolonged or sustained network connectivity failure, the backup process may not be able gain access to the blob and the blob may remain orphaned. This means that the blob cannot be written to or deleted until the lease is released. This topic describes how to release the lease and deleting the blob
Azure Blob API support breaking the lease
https://msdn.microsoft.com/en-us/library/azure/ee691972.aspx
if you are using client library, look for those method.
or simply implement your own, below is blog post contains sample code that from Steve Marx
http://blog.smarx.com/posts/leasing-windows-azure-blobs-using-the-storage-client-library
I had this same problem and was able to delete the VHD blob through the old portal (manage.windowsazure.com).
Under the Storage blade, navigate to your storage account, click on containers, select the container, then select the VHD blob and click delete.
You will get the message stating that there is a lease on the blob, but you should also see an option to delete the disk and the blob. Select that, and it should delete the VHD blob (as well as the orphaned disk).
I just came to know that this can also be done from the Azure Storage Explorer tool(version: 0.8.13). Navigate to your .vhd file and right click where you will see 'Break Lease' option. This should do the trick.
I need to do an automatic periodic backup of an Azure blob storage to another Azure blob storage.
This is in order to guard against any kind of malfunction in the software.
Are there any services which do that? Azure doesn't seem to have this
As #Brent mentioned in the comments to Roberto's answer, the replicas are for HA; if you deleted a blob, that delete is replicated instantly.
For blobs, you can very easily create asynchronous copies to a separate blob (even in a separate storage account). You can also make snapshots which capture a blob at a current moment in time. At first, snapshots don't cost anything, but if you start modifying the blocks/pages referred to by the snapshot, then new blocks/pages are allocated. Over time, you'll want to start purging your snapshots. This is a great way to keep data "as-is" over time and revert back to a snapshot if there's a malfunction in your software.
With queues, the malfunction story isn't quite the same, as typically you'd only have a small number of queue items present (at least that's the hope; if you have thousands of queue messages, this is typically a sign that your software is falling behind). In any event: You could, when writing queue messages, write your queue messages to blob storage, for archive purposes, in case there's a malfunction. I wouldn't recommend using blob- based messaging for scaling/parallel processing, since they don't have the mechanisms in place that queues do, but you could use them manually in case of malfunction.
There's no copy function for tables. You'd need to write to two tables during your write.
Azure keeps 3 redundant copies of your data in different locations in the same data centre where your data is hosted (to guard against hardware failure).
This applies to blob, table and queue storage.
Additionally, You can enable geo-replication on all of your storage. Azure will automatically keep redundant copies of your data in separate data centres. This guards against anything happening to the data centre itself.
See Here