we are trying to achieve object replication in azure storage account. Currently we are able to achieve replication between source to destination, but wouldn't able to do destination to source. what we wanted to achieve is, each region will have its own specific storage account and ours is kind of blue/green deployment. so, we need two way replication. for e.g
our Env1 storage replicates to Env2 Storage account and then we bring Env3 storage which will start replicate from Env2 storage account, post that we will scrap Env1 storage account. I understand that this is currently not possible with Azure storage any alternate PaaS service which we can use?
I was thinking custom solution like, logic app/function app which might do the job. Is there any other way to achieve?
Two-way replication requires destination to be write-enabled. As per the docs, we have the following note:
“After you create the replication policy, write operations to the destination container aren't permitted. Any attempts to write to the destination container fail with error code 409 (Conflict). To write to a destination container for which a replication rule is configured, you must either delete the rule that is configured for that container, or remove the replication policy. Read and delete operations to the destination container are permitted when the replication policy is active.”
https://learn.microsoft.com/en-us/azure/storage/blobs/object-replication-overview#replication-rules
If it helps, you should be able to set up replication from B -> A on a different set of containers than the ones in the initial rule.
That said, if the container that is used to receive the replica needs to be write enabled in active-active fashion, can you please submit a detailed technical feedback on the feature and associated opportunity? We are also collecting use-cases and requests on active-active replication.
https://feedback.azure.com/d365community
Currently, two way replication is not possible. you have one storage account which acts as a driver and it will be replicated to other storage accounts where the replication enabled.
It is not possible to have replication happens from storage account 1 to 2 and vice versa, currently. it always happens from storage account 1 to 2.
Related
We have Terraform state file stored in the Azure Storage Account. In case storage account went down we will be screwed. What is the best way to store the file? where?
AFAIK, there are two methods to store a terraform state file i.e. Locally in your machine or in a Storage account in azure .
In case storage account went down we will be screwed. What is the best
way to store the file? where?
As confirmed , You are using Standard_LRS which is not preferred as per the Microsoft Document if you are looking for high availability.
Locally redundant storage (LRS) copies your data synchronously three
times within a single physical location in the primary region. LRS is
the least expensive replication option, but is not recommended for
applications requiring high availability or durability.
So, as a solution you can change the storage account type as per your requirement to Standard_GRS or Standard_ZRS so that your data is present in two locations i.e. replicated.
You can change it by going to your storage account>>Configuration>>replication as shown below:
If You want more details on Disaster recovery (if one location is down) or data protection from Accidental Deletes then please refer the below documents:
Disaster recovery and storage account failover - Azure Storage | Microsoft Docs
Soft delete for containers - Azure Storage | Microsoft Docs
Why do we need Azure backup for our VMs (disks) on azure, when azure storage account provides different replication options like LRS, ZRS, GRS, RA-GRS.
All the data is already replicate in different region (in case of GRS), what advantes I will get out of Azure Backup.
All the data is already replicate in different region (in case of
GRS), what advantes I will get out of Azure Backup.
Replication is not backup!
It is true that when you opt for GRS replication, 6 copies of your data is maintained (3 in primary and 3 in secondary) but when you delete the data from primary, data from secondary is automatically deleted.
UPDATE
You mean, if any data is deleted/corrupted due to some error/bug, can
be reproduced from backup and it is not possible in case of storage
replication.
You're absolutely correct!
But Microsoft sells "Azure backup and Site recovery" as a BCDR
strategy. In context of any disaster, why not just rely on Storage
replication. Any advantages of Azure backup/site recovery?
I have not used Azure backup so let me answer it from Storage Replication point of view. To put things simply, "In context of Azure, a disaster is not a disaster unless Microsoft thinks it is a disaster". Till the time that happens, you don't get access to secondary assuming you have opted for GRS replication (with RA-GRS, you obviously have an option to read the data from secondary at all times).
Furthermore if you choose LRS or Premium LRS replication and there's indeed a disaster in one data center, all of your data will be lost. With Azure Backup, you at least have a copy of your data lying somewhere safe and you could recreate your environment based on that backup.
I know this question is old but MS provide a solution for Disaster recovery by Storage account
We may have 2 solution for dealing with Disaster
https://learn.microsoft.com/en-us/azure/storage/common/storage-disaster-recovery-guidance?toc=/azure/storage/blobs/toc.json
it said :
If the primary endpoint becomes unavailable for any reason, the client is no longer able to write to the storage account. The following image shows the scenario where the primary has become unavailable, but no recovery has happened yet:
enter image description here
The customer initiates the account failover to the secondary endpoint. The failover process updates the DNS entry provided by Azure Storage so that the secondary endpoint becomes the new primary endpoint for your storage account, as shown in the following image:
enter image description here
I've searched the web and contacted technical support yet no one seems to be able to give me a straight answer on whether items in Azure Blob Storage are backed up or not.
What I mean is, do I need to create a twin storage account as a "backup" and program copies of all content from one storage to another, or are the contents of a client's Blob Storage automatically redundantly backed up by Microsoft?
I know with AWS, storage is redundantly backed up via onsite drives as well as across other nodes in the cluster.
do I need to create a twin storage account as a "backup" and program
copies of all content from one storage to another, or are the contents
of a client's Blob Storage automatically redundantly backed up by
Microsoft?
Yes, you will need to do backup manually. Azure Storage does not back up the contents of your storage account automatically.
Azure Storage does provide geo-redundant replication (provided you configure the redundancy level for your storage account as GRS or RA-GRS) but that is not back up. Once you delete content from your primary account (location, it will automatically be removed from secondary account (geo-redundant location).
Both AWS (EBS) and Azure(Blob Storage) options provides durability by replicating the data across different data centers. This is for the high availability and durability of the data to provide the guarantee by the cloud provider.
In order to ensure that your data is durable, Azure Storage has the
ability to keep (and manage) multiple copies of your data. This is
called replication, or sometimes redundancy. When you set up your
storage account, you select a replication type. In most cases, this
setting can be modified after the storage account is set up.
For more details refer the replication section in documentation.
If you need to capture changes to the storage and allow restore to previous versions (e.g In situations like data corruption or application feature requirements like restore points, backups), you need to take a SnapShot manually. This is common for both AWS and Azure.
For more details on creating a Snapshot of Blob in Azure refer the documentation.
For an HDInsight cluster there has to be at least one azure storage account which is its default storage account -- it is required so that it is treated as its fs (filesystem). This I get. But what about optional linked azure storage accounts? From ADF (Azure Data Factory) perspective at least, do we need to have a storage account added as linked storage account to an HDInsight cluster? Anyway the Azure storage account is accessible purely by providing just two pieces of information --- the account name and the key. Both these things are specified in Linked Servers in ADF. This guarantees the access of the storage account. What is the real benefit of having some account added as linked storage account, from ADF point of view or otherwise? Basically, what I am asking is -- is there anything that we can't do purely using account name and key without adding the account as linked storage for the given HDInsight cluster?
The main reason to have additional accounts is because they have limits. A storage account can have 500 TB of data in it and 20000 request per second. Depending on the size of your cluster and work load you might hit the request limit. If you are worried about those limits and you don't want to manage alot of storage accounts you should look into Azure Data Lake.
I think I sort of figured out the answer. With linked storage accounts the cluster, when used as a compute, can directly access BLOBS on those storage accounts without requiring us to separately specify the storage keys in queries. That's the use case for which linked storage is a must have.
I know Azure will geo-replication a copy of current storage account to another location,
my questions is: can I access another location in program, even just read only
I asked this, because this allow me to build another deploy in different geo-location for performance and disaster-proof like what Azure did. For current setup, if I use same source of storage in different geo-location, I have to pay extra bandwidth cost.
You can only access your storage account by its primary name. In the event of failover, that name will be mapped to the alternate datacenter. You cannot access the failover storage directly, nor can you choose when to trigger a failover. For a multi-site setup as you described, you'd need to duplicate your data (which would then add the cost of storage in datacenter #2). This does give you ultimate flexibility in your DR and performance planning, but at an added cost of storage and bandwidth (egress-only).
Last week the storage team announced read-only access to the failover storage: Windows Azure Storage Redundancy Options and Read Access Geo Redundant Storage.
This means you can now deploy your application in a different datacenter which can be used for "full" failover (meaning that the storage will also be available there). Even if it's only read-only, your application will still be online - but simply in "degraded" mode.
The steps on how you can implement this with traffic manager are described here: http://fabriccontroller.net/blog/posts/adding-failover-to-your-application-with-read-access-geo-redundant-storage-and-the-windows-azure-traffic-manager/