I am using java to write a Azure Function App which is eventgrid trigger and the trigger is blobcreated. So when ever blob is created it will be trigerred and the function is to copy a blob from one container to another. I am using startCopy function from com.microsoft.azure.storage.blob. It was working fine but sometimes, It uses to copy files of zero bytes which are actually containing some data in source location. So at destination sometimes it dumps zero bytes of files. I would like to have a little help on this so that I could understand how to possibly handle this situation
CloudBlockBlob cloudBlockBlob = container.getBlockBlobReference(blobFileName);
CloudStorageAccount storageAccountdest = CloudStorageAccount.parse("something");
CloudBlobClient blobClientdest = storageAccountdest.createCloudBlobClient();
CloudBlobContainer destcontainer = blobClientdest.getContainerReference("something");
CloudBlockBlob destcloudBlockBlob = destcontainer.getBlockBlobReference();
destcloudBlockBlob.startCopy(cloudBlockBlob);
Copying blobs across storage accounts is an async operation. When you call startCopy method, it just signals Azure Storage to copy a file. Actual file copy operation happens asynchronously and may take some time depending how how large file you're transferring.
I would suggest that you check the copy operation progress on the target blob to see how many bytes have been copied and if there's a failure in the copy operation. You can do so by fetching the properties of the target blob. A copy operation could potentially fail if the source blob is modified after the copy operation has started by Azure Storage.
had the same problem, and later figured out from the docs
Event Grid isn't a data pipeline, and doesn't deliver the actual
object that was updated
Event grid will tell you that something has changed and that the actual message has a size limit and as long as the data that you are copying is within that limit it will be successful if not it will be 0 bytes. I was able to copy upto 1mb and beyond that it resulted 0 bytes. You can try and see if azure has increased by size limit in the recent.
However if you want to copy the complete data then you need to use Event Hub or Service Bus. For mine, I went with service bus.
Related
I am building a service with an endpoint that images and other files will be uploaded to, and I need to stream the file directly to Blob Storage. This service will handle hundreds of images per second, so I cannot buffer the images into memory before sending it to Blob Storage.
I was following the article here and ran into this comment
Next, using the latest version (v12) of the Azure Blob Storage libraries and a Stream upload method. Notice that it’s not much better than IFormFile! Although BlobStorageClient is the latest way to interact with blob storage, when I look at the memory snapshots of this operation it has internal buffers (at least, at the time of this writing) that cause it to not perform too well when used in this way.
But, using almost identical code and the previous library version that uses CloudBlockBlob instead of BlobClient, we can see a much better memory performance. The same file uploads result in a small increase (due to resource consumption that eventually goes back down with garbage collection), but nothing near the ~600MB consumption like above
I tried this and found that yes, v11 has considerably less memory usage compared to v12! When I ran my tests with about a ~10MB file the memory, each new upload (after initial POST) jumped the memory usage 40MB, while v11 jumped only 20MB
I then tried a 100MB file. On v12 the memory seemed to use 100MB nearly instantly each request and slowly climbed after that, and was over 700MB after my second upload. Meanwhile v11 didn't really jump in memory, though it would still slowly climb in memory, and ended with around 430MB after the 2nd upload.
I tried experimenting with creating BlobUploadOptions properties InitialTransferSize, MaximumConcurrency, etc. but it only seemed to make it worse.
It seems unlikely that v12 would be straight up worse in performance than v11, so I am wondering what I could be missing or misunderstanding.
Thanks!
Sometimes this issue may occur due to Azure blob storage (v12) libraries.
Try to upload the large files in chunks [a technique called file chunking which breaks the large file into smaller chunks for each upload] instead of uploading whole file. Please refer this link
I tried producing the scenario in my lab
public void uploadfile()
{
string connectionString = "connection string";
string containerName = "fileuploaded";
string blobName = "test";
string filePath = "filepath";
BlobContainerClient container = new BlobContainerClient(connectionString, containerName);
container.CreateIfNotExists();
// Get a reference to a blob named "sample-file" in a container named "sample-container"
BlobClient blob = container.GetBlobClient(blobName);
// Upload local file
blob.Upload(filePath);
}
The output after uploading file.
I have datalake & datawarehouse containing about 5-10 TBs of data in Azure ADLS gen2, CSV and Delta formats. ADLS's Performance/Tier=Standard/Hot, replication=GRS, type=StorageV2.
What is the best way to backup my ADLS gen2 data?
From data corruption perspective, I want to backup raw ingested data. This can be done incrementally, small amount of data, on regular basis.
From PROD availability perspective, I want to backup all 5-10 TBs rarely before complex PROD migrations. Yes, the data can be derived from raw data, but it may take up to few days or even a week (including reconsiliations, testing even more).
Considerations:
Azure Backup doesn't support ADLS
Copying data with help of Azure Storage Explorer is slow, because speed is unstable from 50 to 1000 Mbps. It may take days or week on my data volumes. Am I right Azure Storage Explorer speed doesn't depend on my local internet speed?
I haven't tried AzCopy, but expect it to have the same speed as Azure Storage Explorer
Mounting data_container to archive_container in DBFS, and trying to copy data with Databrick's dbutils.fs.cp works even slower then Azure Storage Explorer: 3GB/10 minutes on big 10 notes 30 DBUs cluster. Why?
ADF haven't tried, but I dislike the fact that Copy activity requires details on table/format level. I would like to backup the whole container, without implementing logic and depending on folders amount and naming.
For Raw data/folder backup I use Microsoft data movement service to copy blob directory from ADLS Gen2 into Storage Account.
For this create a daily time trigger function to do an incremental copy of the blob directory.
You can configure something like this.
Create a new folder with every Monday (date) full backup and keep incremental changes till Sunday. After a Month remove old backup folders.
here is my implementation.
public async Task<string> CopyBlobDirectoryAsync(BlobConfiguration sourceBlobConfiguration, BlobConfiguration destBlobConfiguration, string blobDirectoryName)
{
CloudBlobDirectory sourceBlobDir = await GetCloudBlobDirectoryAsync(sourceBlobConfiguration.ConnectionString, sourceBlobConfiguration.ContainerName, blobDirectoryName);
CloudBlobDirectory destBlobDir = await GetCloudBlobDirectoryAsync(destBlobConfiguration.ConnectionString, destBlobConfiguration.ContainerName, destBlobConfiguration.BlobDirectoryPath + "/" + blobDirectoryName);
// You can also replace the source directory with a CloudFileDirectory instance to copy data from Azure File Storage. If so:
// 1. If recursive is set to true, SearchPattern is not supported. Data movement library simply transfer all azure files
// under the source CloudFileDirectory and its sub-directories.
CopyDirectoryOptions options = new CopyDirectoryOptions()
{
Recursive = true
};
DirectoryTransferContext context = new DirectoryTransferContext();
context.FileTransferred += FileTransferredCallback;
context.FileFailed += FileFailedCallback;
context.FileSkipped += FileSkippedCallback;
// Create CancellationTokenSource used to cancel the transfer
CancellationTokenSource cancellationSource = new CancellationTokenSource();
TransferStatus trasferStatus = await TransferManager.CopyDirectoryAsync(sourceBlobDir, destBlobDir, CopyMethod.ServiceSideAsyncCopy, options, context, cancellationSource.Token);
return TransferStatusToString(blobDirectoryName, trasferStatus);
}
I am trying to upload a contents to azure blob and the size is over 32MB. The c# code snippet below:
CloudBlockBlob blob = _blobContainer.GetBlockBlobReference(blobName);
blob.UploadFromByteArray(contents, 0, contents.Length, AccessCondition.GenerateIfNotExistsCondition(), options:writeOptions);
Everytime the blob is over 32MB, the above raises an exception:
Exception thrown: 'Microsoft.WindowsAzure.Storage.StorageException' in Microsoft.WindowsAzure.Storage.dll
Additional information: The remote server returned an error: (404) Not Found.
As per this
When a block blob upload is larger than the value in this property,
storage clients break the file into blocks.
Should there be a separate line of code to enable this.
Storage clients default to a 32 MB maximum single block upload. When a block blob upload is larger than the value in SingleBlobUploadThresholdInBytes property, storage clients break the file into blocks.
As Tamra said, the storage client handles the work of breaking the file into blocks. Here is my tests for you to have a better understanding of it.
Code Sample
CloudBlockBlob blob = container.GetBlockBlobReference(blobName);
var writeOptions = new BlobRequestOptions()
{
SingleBlobUploadThresholdInBytes = 50 * 1024 * 1024, //maximum for 64MB,32MB by default
};
blob.UploadFromByteArray(contents, 0, contents.Length, AccessCondition.GenerateIfNotExistsCondition(), options: writeOptions);
Scenario
If you are writing a block blob that is no more than the SingleBlobUploadThresholdInBytes property in size, you could upload it in its entirety with a single write operation.
You could understand it by capturing the Network Package via Fiddler when you invoke the UploadFromByteArray method.
When a block blob upload is larger than the value in SingleBlobUploadThresholdInBytes property in size, storage clients break the file into blocks automatically.
I upload a blob which size is nearly 90MB, then you could find the difference as follows:
Upon the snapshot, you could find that storage clients break the file into blocks with 4MB in size and upload the blocks simultaneously.
Every time the blob is over 32MB, the above raises an exception
You could try to set the SingleBlobUploadThresholdInBytes property or capture the Network Package when you invoke the UploadFromByteArray method to find the detailed error.
I created a WebJob project to backup images from Azure to Amazon that used the BlobTrigger attribute for the first parameter in the method
public static async Task CopyImage([BlobTrigger("images/{name}")] ICloudBlob image, string name, TextWriter log)
{
var imageStream = new MemoryStream();
image.DownloadToStream(imageStream);
await S3ImageBackupContext.UploadImageAsync(name, imageStream);
}
Then I read that the BlobTrigger is based on the best effort basis in the document How to use Azure blob storage and changed it to a QueueTrigger.
Both works perfectly fine :-) so it's not a problem but a question. Since I deployed the change the CPU and Memory usage of the WebJob looks like
Can somebody explain me the reason for the drop of memory and CPU usage? Also the data egress went down.
Very interesting.
I think you're the only one who can answer that question.
Do a remote profile for both blob and queue versions, see which method eats up that CPU time:
https://azure.microsoft.com/en-us/blog/remote-profiling-support-in-azure-app-service/
For memory consumption you probably need to grab a memory dump:
https://blogs.msdn.microsoft.com/waws/2015/07/01/create-a-memory-dump-for-your-slow-performing-web-app/
I upload gzipped files to an Azure Storage Container (input). I then have a WebJob that is supposed to pick up the Blobs, decompress them and drop them into another Container (output). Both containers use the same storage account.
My problem is that it doesn't process all Blobs. It always seems to miss 1. This morning I uploaded 11 blobs to the input Container and only 10 were processed and dumped into the output Container. If I upload 4 then 3 will be processed. The dashboard will show 10 invocations even though 11 blobs have been uploaded. It doesn't look like it gets triggered for the 11th blob. If I only upload 1 it seems to process it.
I am running the website in Standard Mode with Always On set to true.
I have tried:
Writing code like the Azure Samples (https://github.com/Azure/azure-webjobs-sdk-samples).
Writing code like the code in this article (http://azure.microsoft.com/en-us/documentation/articles/websites-dotnet-webjobs-sdk-get-started).
Using Streams for the input and output instead of CloudBlockBlobs.
Various combinations of closing the input, output and Gzip Streams.
Having the UnzipData code in the Unzip method.
This is my latest code. Am I doing something wrong?
public class Functions
{
public static void Unzip(
[BlobTrigger("input/{name}.gz")] CloudBlockBlob inputBlob,
[Blob("output/{name}")] CloudBlockBlob outputBlob)
{
using (Stream input = inputBlob.OpenRead())
{
using (Stream output = outputBlob.OpenWrite())
{
UnzipData(input, output);
}
}
}
public static void UnzipData(Stream input, Stream output)
{
GZipStream gzippedStream = null;
gzippedStream = new GZipStream(input, CompressionMode.Decompress);
gzippedStream.CopyTo(output);
}
}
As per Victor's comment above it looks like it is a bug on Microsoft's end.
Edit: I don't get the downvote. There is a problem and Microsoft are going to fix it. That is the answer to why some of my blobs are ignored...
"There is a known issue about some Storage log events being ignored. Those events are usually generated for large files. We have a fix for it but it is not public yet. Sorry for the inconvenience. – Victor Hurdugaci Jan 9 at 12:23"
Just as an workaround, what if you don't directly listen to the Blob instead bring a Queue in-between, when you write to the Input Blob Container also write a message about the new Blob in the Queue also, let the WebJob listen to this Queue, once a message arrived to the Queue , the WebJob Function read the File from the Input Blob Container and copied into the Output Blob Container.
Does this model work with you ?