Backup ADLS gen2 - azure

Question

I have datalake & datawarehouse containing about 5-10 TBs of data in Azure ADLS gen2, CSV and Delta formats. ADLS's Performance/Tier=Standard/Hot, replication=GRS, type=StorageV2.
What is the best way to backup my ADLS gen2 data?
From data corruption perspective, I want to backup raw ingested data. This can be done incrementally, small amount of data, on regular basis.
From PROD availability perspective, I want to backup all 5-10 TBs rarely before complex PROD migrations. Yes, the data can be derived from raw data, but it may take up to few days or even a week (including reconsiliations, testing even more).
Considerations:
Azure Backup doesn't support ADLS
Copying data with help of Azure Storage Explorer is slow, because speed is unstable from 50 to 1000 Mbps. It may take days or week on my data volumes. Am I right Azure Storage Explorer speed doesn't depend on my local internet speed?
I haven't tried AzCopy, but expect it to have the same speed as Azure Storage Explorer
Mounting data_container to archive_container in DBFS, and trying to copy data with Databrick's dbutils.fs.cp works even slower then Azure Storage Explorer: 3GB/10 minutes on big 10 notes 30 DBUs cluster. Why?
ADF haven't tried, but I dislike the fact that Copy activity requires details on table/format level. I would like to backup the whole container, without implementing logic and depending on folders amount and naming.

Answer 1

For Raw data/folder backup I use Microsoft data movement service to copy blob directory from ADLS Gen2 into Storage Account.
For this create a daily time trigger function to do an incremental copy of the blob directory.
You can configure something like this.
Create a new folder with every Monday (date) full backup and keep incremental changes till Sunday. After a Month remove old backup folders.
here is my implementation.
public async Task<string> CopyBlobDirectoryAsync(BlobConfiguration sourceBlobConfiguration, BlobConfiguration destBlobConfiguration, string blobDirectoryName)
{
CloudBlobDirectory sourceBlobDir = await GetCloudBlobDirectoryAsync(sourceBlobConfiguration.ConnectionString, sourceBlobConfiguration.ContainerName, blobDirectoryName);
CloudBlobDirectory destBlobDir = await GetCloudBlobDirectoryAsync(destBlobConfiguration.ConnectionString, destBlobConfiguration.ContainerName, destBlobConfiguration.BlobDirectoryPath + "/" + blobDirectoryName);
// You can also replace the source directory with a CloudFileDirectory instance to copy data from Azure File Storage. If so:
// 1. If recursive is set to true, SearchPattern is not supported. Data movement library simply transfer all azure files
// under the source CloudFileDirectory and its sub-directories.
CopyDirectoryOptions options = new CopyDirectoryOptions()
{
Recursive = true
};
DirectoryTransferContext context = new DirectoryTransferContext();
context.FileTransferred += FileTransferredCallback;
context.FileFailed += FileFailedCallback;
context.FileSkipped += FileSkippedCallback;
// Create CancellationTokenSource used to cancel the transfer
CancellationTokenSource cancellationSource = new CancellationTokenSource();
TransferStatus trasferStatus = await TransferManager.CopyDirectoryAsync(sourceBlobDir, destBlobDir, CopyMethod.ServiceSideAsyncCopy, options, context, cancellationSource.Token);
return TransferStatusToString(blobDirectoryName, trasferStatus);
}