I created a WebJob project to backup images from Azure to Amazon that used the BlobTrigger attribute for the first parameter in the method
public static async Task CopyImage([BlobTrigger("images/{name}")] ICloudBlob image, string name, TextWriter log)
{
var imageStream = new MemoryStream();
image.DownloadToStream(imageStream);
await S3ImageBackupContext.UploadImageAsync(name, imageStream);
}
Then I read that the BlobTrigger is based on the best effort basis in the document How to use Azure blob storage and changed it to a QueueTrigger.
Both works perfectly fine :-) so it's not a problem but a question. Since I deployed the change the CPU and Memory usage of the WebJob looks like
Can somebody explain me the reason for the drop of memory and CPU usage? Also the data egress went down.
Very interesting.
I think you're the only one who can answer that question.
Do a remote profile for both blob and queue versions, see which method eats up that CPU time:
https://azure.microsoft.com/en-us/blog/remote-profiling-support-in-azure-app-service/
For memory consumption you probably need to grab a memory dump:
https://blogs.msdn.microsoft.com/waws/2015/07/01/create-a-memory-dump-for-your-slow-performing-web-app/
Related
I am building a service with an endpoint that images and other files will be uploaded to, and I need to stream the file directly to Blob Storage. This service will handle hundreds of images per second, so I cannot buffer the images into memory before sending it to Blob Storage.
I was following the article here and ran into this comment
Next, using the latest version (v12) of the Azure Blob Storage libraries and a Stream upload method. Notice that it’s not much better than IFormFile! Although BlobStorageClient is the latest way to interact with blob storage, when I look at the memory snapshots of this operation it has internal buffers (at least, at the time of this writing) that cause it to not perform too well when used in this way.
But, using almost identical code and the previous library version that uses CloudBlockBlob instead of BlobClient, we can see a much better memory performance. The same file uploads result in a small increase (due to resource consumption that eventually goes back down with garbage collection), but nothing near the ~600MB consumption like above
I tried this and found that yes, v11 has considerably less memory usage compared to v12! When I ran my tests with about a ~10MB file the memory, each new upload (after initial POST) jumped the memory usage 40MB, while v11 jumped only 20MB
I then tried a 100MB file. On v12 the memory seemed to use 100MB nearly instantly each request and slowly climbed after that, and was over 700MB after my second upload. Meanwhile v11 didn't really jump in memory, though it would still slowly climb in memory, and ended with around 430MB after the 2nd upload.
I tried experimenting with creating BlobUploadOptions properties InitialTransferSize, MaximumConcurrency, etc. but it only seemed to make it worse.
It seems unlikely that v12 would be straight up worse in performance than v11, so I am wondering what I could be missing or misunderstanding.
Thanks!
Sometimes this issue may occur due to Azure blob storage (v12) libraries.
Try to upload the large files in chunks [a technique called file chunking which breaks the large file into smaller chunks for each upload] instead of uploading whole file. Please refer this link
I tried producing the scenario in my lab
public void uploadfile()
{
string connectionString = "connection string";
string containerName = "fileuploaded";
string blobName = "test";
string filePath = "filepath";
BlobContainerClient container = new BlobContainerClient(connectionString, containerName);
container.CreateIfNotExists();
// Get a reference to a blob named "sample-file" in a container named "sample-container"
BlobClient blob = container.GetBlobClient(blobName);
// Upload local file
blob.Upload(filePath);
}
The output after uploading file.
I am using java to write a Azure Function App which is eventgrid trigger and the trigger is blobcreated. So when ever blob is created it will be trigerred and the function is to copy a blob from one container to another. I am using startCopy function from com.microsoft.azure.storage.blob. It was working fine but sometimes, It uses to copy files of zero bytes which are actually containing some data in source location. So at destination sometimes it dumps zero bytes of files. I would like to have a little help on this so that I could understand how to possibly handle this situation
CloudBlockBlob cloudBlockBlob = container.getBlockBlobReference(blobFileName);
CloudStorageAccount storageAccountdest = CloudStorageAccount.parse("something");
CloudBlobClient blobClientdest = storageAccountdest.createCloudBlobClient();
CloudBlobContainer destcontainer = blobClientdest.getContainerReference("something");
CloudBlockBlob destcloudBlockBlob = destcontainer.getBlockBlobReference();
destcloudBlockBlob.startCopy(cloudBlockBlob);
Copying blobs across storage accounts is an async operation. When you call startCopy method, it just signals Azure Storage to copy a file. Actual file copy operation happens asynchronously and may take some time depending how how large file you're transferring.
I would suggest that you check the copy operation progress on the target blob to see how many bytes have been copied and if there's a failure in the copy operation. You can do so by fetching the properties of the target blob. A copy operation could potentially fail if the source blob is modified after the copy operation has started by Azure Storage.
had the same problem, and later figured out from the docs
Event Grid isn't a data pipeline, and doesn't deliver the actual
object that was updated
Event grid will tell you that something has changed and that the actual message has a size limit and as long as the data that you are copying is within that limit it will be successful if not it will be 0 bytes. I was able to copy upto 1mb and beyond that it resulted 0 bytes. You can try and see if azure has increased by size limit in the recent.
However if you want to copy the complete data then you need to use Event Hub or Service Bus. For mine, I went with service bus.
My azure function is calculating results of certain request jobs (cca. 5s-5min) where each job has unique jobId based on the hash of the request message. Execution leads to deterministic results. So it is functionally "pure function". Therefore we are caching results of already evaluated jobs in a blob storage based on the jobId. All great so far.
Now if a request for jobId comes three scenarios are possible.
Result is in the cache already => then it is served from the cache.
Result is not in the cache and no function is running the evaluation => new invocation
Result is not in the cache, but some function is already working on it => wait for result
We do some custom table storage based progress tracking magic to tell if function is working on given jobId or not yet.
It works somehow, up to the point of 5 x restart -> poison queue scenarios. There we are quite hopeless.
I feel like we are hacking around some of already reliably implemented feature of Azure Functions internals, because exactly the same info can be seen in the monitor page in azure portal or used to be visible in kudu webjobs monitor page.
How to reliably find out in c# if a given message (jobId) is currently being processed by some function and when it is not?
Azure Durable Functions provide a mechanism how to track progress of execution of smaller tasks.
https://learn.microsoft.com/en-us/azure/azure-functions/durable-functions-overview
Accroding to the "Pattern #3: Async HTTP APIs" the orchestrator can provide information about the function status in form like this:
{"runtimeStatus":"Running","lastUpdatedTime":"2017-03-16T21:20:47Z", ...}
This solves my problem about finding if given message is being processed.
How to reliably find out in c# if a given message (jobId) is currently being processed by some function and when it is not?
If you’d like to detect which message is being processed and get the message ID in queue triggered Azure function, you can try the following code:
#r "Microsoft.WindowsAzure.Storage"
using System;
using Microsoft.WindowsAzure.Storage.Queue;
public static void Run(CloudQueueMessage myQueueItem, TraceWriter log)
{
log.Info($"messageid: {myQueueItem.Id}, messagebody: {myQueueItem.AsString}");
}
I upload gzipped files to an Azure Storage Container (input). I then have a WebJob that is supposed to pick up the Blobs, decompress them and drop them into another Container (output). Both containers use the same storage account.
My problem is that it doesn't process all Blobs. It always seems to miss 1. This morning I uploaded 11 blobs to the input Container and only 10 were processed and dumped into the output Container. If I upload 4 then 3 will be processed. The dashboard will show 10 invocations even though 11 blobs have been uploaded. It doesn't look like it gets triggered for the 11th blob. If I only upload 1 it seems to process it.
I am running the website in Standard Mode with Always On set to true.
I have tried:
Writing code like the Azure Samples (https://github.com/Azure/azure-webjobs-sdk-samples).
Writing code like the code in this article (http://azure.microsoft.com/en-us/documentation/articles/websites-dotnet-webjobs-sdk-get-started).
Using Streams for the input and output instead of CloudBlockBlobs.
Various combinations of closing the input, output and Gzip Streams.
Having the UnzipData code in the Unzip method.
This is my latest code. Am I doing something wrong?
public class Functions
{
public static void Unzip(
[BlobTrigger("input/{name}.gz")] CloudBlockBlob inputBlob,
[Blob("output/{name}")] CloudBlockBlob outputBlob)
{
using (Stream input = inputBlob.OpenRead())
{
using (Stream output = outputBlob.OpenWrite())
{
UnzipData(input, output);
}
}
}
public static void UnzipData(Stream input, Stream output)
{
GZipStream gzippedStream = null;
gzippedStream = new GZipStream(input, CompressionMode.Decompress);
gzippedStream.CopyTo(output);
}
}
As per Victor's comment above it looks like it is a bug on Microsoft's end.
Edit: I don't get the downvote. There is a problem and Microsoft are going to fix it. That is the answer to why some of my blobs are ignored...
"There is a known issue about some Storage log events being ignored. Those events are usually generated for large files. We have a fix for it but it is not public yet. Sorry for the inconvenience. – Victor Hurdugaci Jan 9 at 12:23"
Just as an workaround, what if you don't directly listen to the Blob instead bring a Queue in-between, when you write to the Input Blob Container also write a message about the new Blob in the Queue also, let the WebJob listen to this Queue, once a message arrived to the Queue , the WebJob Function read the File from the Input Blob Container and copied into the Output Blob Container.
Does this model work with you ?
I've been running a cloud drive snapshot in dev for a while now with no probs. I'm now trying to get this working in Azure.
I can't for the life of me get it to work. This is my latest error:
Microsoft.WindowsAzure.Storage.CloudDriveException: Unknown Error HRESULT=D000000D --->
Microsoft.Window.CloudDrive.Interop.InteropCloudDriveException: Exception of type 'Microsoft.WindowsAzure.CloudDrive.Interop.InteropCloudDriveException' was thrown.
at ThrowIfFailed(UInt32 hr)
at Microsoft.WindowsAzure.CloudDrive.Interop.InteropCloudDrive.Mount(String url, SignatureCallBack sign, String mount, Int32 cacheSize, UInt32 flags)
at Microsoft.WindowsAzure.StorageClient.CloudDrive.Mount(Int32 cacheSize, DriveMountOptions options)
Any idea what is causing this? I'm running both the WorkerRole and Storage in Azure so it's nothing to do with the dev simulation environment disconnect.
This is my code to mount the snapshot:
CloudDrive.InitializeCache(localPath.TrimEnd('\\'), size);
var container = _blobStorage.GetContainerReference(containerName);
var blob = container.GetPageBlobReference(driveName);
CloudDrive cloudDrive = _cloudStorageAccount.CreateCloudDrive(blob.Uri.AbsoluteUri);
string snapshotUri;
try
{
snapshotUri = cloudDrive.Snapshot().AbsoluteUri;
Log.Info("CloudDrive Snapshot = '{0}'", snapshotUri);
}
catch (Exception ex)
{
throw new InvalidCloudDriveException(string.Format(
"An exception has been thrown trying to create the CloudDrive '{0}'. This may be because it doesn't exist.",
cloudDrive.Uri.AbsoluteUri), ex);
}
cloudDrive = _cloudStorageAccount.CreateCloudDrive(snapshotUri);
Log.Info("CloudDrive created: {0}", snapshotUri, cloudDrive);
string driveLetter = cloudDrive.Mount(size, DriveMountOptions.None);
The .Mount() method at the end is what's now failing.
Please help as this has me royally stumped!
Thanks in advance.
Dave
I finally got this to work last night. All I did was create a new container and upload my VHD to it so I'm not sure if there was something weird going on with the old container...? Can't think what. The old container must've been getting a bit long in the tooth..!?!
2 days of my life I'll never get back. Debugging live Azure issues is an excruciatingly tedious process.
It's a shame the Azure CloudDrive dev simulation doesn't more closely replicate the live environment.
One source of the D000000D InteropCloudDriveException is when the drive (or snapshot) being mounted is expandable rather than fixed size. Unfortunately, the MSDN documentation provides minimal information on restrictions, but this note is an excellent source of information:
http://convective.wordpress.com/2010/02/27/azure-drive/
I can confirm Dave's findings regarding the BLOB container (Love you Dave, I only spent one evening).
I also had problems debugging before changing BLOB container.
The error message I had was "there was an error attaching the debugger to the IIS worker process for url ...".
Hope this helps some poor Azure dev, having a challenging time with the debugger.