I am getting System.OutOfMemoryException exception for Blob Trigger Azure Function.
When I am executing locally it is working fine.
Blog Trigger Azure Function:
public static class ProcessEvent
{
[FunctionName(nameof(ProcessEvent))]
public static async Task Run([BlobTrigger(BlobStorageContainer.Name + "/{name}",
Connection = "AzureWebJobsStorage")]
Stream eventBlob, string name,
[Inject] ILoggingService loggingService,
[Inject] IEventProcessorService eventProcessor,
[Inject] IBlobClient blobClient)
{
var logger = new Logger(loggingService);
try
{
logger.Info($"Starting blob job tracker for file name {name}",
nameof(ProcessEvent));
var eventContent = eventBlob.ReadAsString();
var result = await eventProcessor.HandleProcessor(eventContent, logger);
if (result)
{
await blobClient.DeleteBlobAsync(BlobStorageContainer.Name, name);
logger.Info($"Blob deleted successfully file name: {name}");
}
else
{
logger.Warning($"Unable to process blob job for file with name: {name}");
}
}
catch (Exception ex)
{
logger.Error($"Unable to process blob job for file with name: {name}", ex,
nameof(ProcessEvent));
}
}
}
My App Service Plan:
You can diagnose the memory usage in portal->your function app->Diagnose and solve problem->Memory Analysis->View Full Report.
It shows the overall percent physical memory usage per instance over the last 24 hours.
https://learn.microsoft.com/en-us/azure/app-service/overview-diagnostics#health-checkup-graphs
Related
I am using a file upload component that posts the file to an API controller and this works ok but i need to get the progres of the upload.
[HttpPost("upload/single")]
public async Task SingleAsync(IFormFile file)
{
try
{
// Azure connection string and container name passed as an argument to get the Blob reference of the container.
var container = new BlobContainerClient(azureConnectionString, "upload-container");
// Method to create our container if it doesn’t exist.
var createResponse = await container.CreateIfNotExistsAsync();
// If container successfully created, then set public access type to Blob.
if (createResponse != null && createResponse.GetRawResponse().Status == 201)
await container.SetAccessPolicyAsync(Azure.Storage.Blobs.Models.PublicAccessType.Blob);
// Method to create a new Blob client.
var blob = container.GetBlobClient(file.FileName);
// If a blob with the same name exists, then we delete the Blob and its snapshots.
await blob.DeleteIfExistsAsync(Azure.Storage.Blobs.Models.DeleteSnapshotsOption.IncludeSnapshots);
// Create a file stream and use the UploadSync method to upload the Blob.
uploadFileSize = file.Length;
var progressHandler = new Progress<long>();
progressHandler.ProgressChanged += UploadProgressChanged;
using (var fileStream = file.OpenReadStream())
{
await blob.UploadAsync(fileStream, new BlobHttpHeaders { ContentType = file.ContentType },progressHandler:progressHandler);
}
Response.StatusCode = 400;
}
catch (Exception e)
{
Response.Clear();
Response.StatusCode = 204;
Response.HttpContext.Features.Get<IHttpResponseFeature>().ReasonPhrase = "File failed to upload";
Response.HttpContext.Features.Get<IHttpResponseFeature>().ReasonPhrase = e.Message;
}
}
private double GetProgressPercentage(double totalSize, double currentSize)
{
return (currentSize / totalSize) * 100;
}
private void UploadProgressChanged(object sender, long bytesUploaded)
{
uploadPercentage = GetProgressPercentage(uploadFileSize, bytesUploaded);
}
I am posting this file and it does upload but the file upload progress event is inaccurate it says the file upload is complete after a few seconds when in reality the file takes ~90 secs on my connection to appear in the Azure Blob Storage container.
So in the code above i have the progress handler which works (I can put a break point on it and see it increasing) but how do I return this value to the UI?
I found one solution that used Microsoft.AspNetCore.SignalR but I can't manage to integrate this into my own code and I'm not even sure if I'm on the right track.
using BlazorReportProgress.Server.Hubs;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.SignalR;
using System.Threading;
namespace BlazorReportProgress.Server.Controllers
{
[ApiController]
[Route("api/[controller]")]
public class SlowProcessController : ControllerBase
{
private readonly ILogger<SlowProcessController> _logger;
private readonly IHubContext<ProgressHub> _hubController;
public SlowProcessController(
ILogger<SlowProcessController> logger,
IHubContext<ProgressHub> hubContext)
{
_logger = logger;
_hubController = hubContext;
}
[HttpGet("{ClientID}")]
public IEnumerable<int> Get(string ClientID)
{
List<int> retVal = new();
_logger.LogInformation("Incoming call from ClientID : {ClientID}", ClientID);
_hubController.Clients.Client(ClientID).SendAsync("ProgressReport", "Starting...");
Thread.Sleep(1000);
for (int loop = 0; loop < 10; loop++)
{
_hubController.Clients.Client(ClientID).SendAsync("ProgressReport", loop.ToString());
retVal.Add(loop);
Thread.Sleep(500);
}
_hubController.Clients.Client(ClientID).SendAsync("ProgressReport", "Done!");
return retVal;
}
}
}
I read the Steve Sandersen blog but this says not to use the code as its been superceded by inbuilt blazor functionality.
My application is only for a few users and so i'm not too worried about backend APIs etc, If the upload component used a service not a controller I could more easily get the progress, but the compoents all seem to post to controllers.
Can anyone please enlighten me as to the best way to solve this?
I have an Azure durable function triggered by a message that then uses the client to start an Orchestration trigger which starts several activity functions. I have set breakpoints in Orchestration client , trigger and each activity function.
But, it only hits the breakpoints in Orchestration client function and the others are getting ignored. But underneath it seems to execute the activity functions although the breakpoints are not hit.
Investigative information
Programming language used = C#
Visual Studio Enterprise 2019 version = 16.8.3
Azure Functions Core Tools
Core Tools Version: 3.0.3216
Function Runtime Version: 3.0.15193.0
Below here, I have included a code snippet. (have not added every activity function)
[FunctionName(nameof(InitiateExport))]
public static async Task InitiateExport(
[ServiceBusTrigger("%ExportQueueName%", Connection = "AzureSBConnection")]Message message,
[DurableClient(TaskHub = "%FunctionHubName%")] IDurableOrchestrationClient orchestrationClient,
[Inject]IServiceProvider rootServiceProvider, ILogger log)
{
var DataQueuedDetails = JsonConvert.DeserializeObject<DataQueuedDetails>(Encoding.UTF8.GetString(message.Body));
using (var scope = rootServiceProvider.CreateScope())
{
log.LogInformation($"{nameof(ExportData)} function execution started at: {DateTime.Now}");
var services = scope.ServiceProvider;
services.ResolveRequestContext(message);
var requestContext = services.GetRequiredService<RequestContext>();
await orchestrationClient.StartNewAsync(nameof(TriggerDataExport), null, (DataQueuedDetails, requestContext));
log.LogInformation($"{nameof(ExportData)} timer triggered function execution finished at: {DateTime.Now}");
}
}
[FunctionName(nameof(TriggerDataExport))]
public static async Task TriggerDataExport(
[OrchestrationTrigger] IDurableOrchestrationContext orchestrationContext,
[Inject] IServiceProvider rootServiceProvider, ILogger log)
{
using (var scope = rootServiceProvider.CreateScope())
{
var services = scope.ServiceProvider;
var (DataOperationInfo, requestContext) = orchestrationContext.GetInput<(DataQueuedDetails, RequestContext)>();
if (!orchestrationContext.IsReplaying)
log.LogInformation($"Starting Export data Id {DataOperationInfo.Id}");
var blobServiceFactory = services.GetRequiredService<IBlobServiceFactory>();
requestContext.CustomerId = DataOperationInfo.RelatedCustomerId;
try
{
await orchestrationContext.CallActivityAsync(
nameof(UpdateJobStatus),
(DataOperationInfo.Id, DataOperationStatus.Running, string.Empty, string.Empty, requestContext));
// some other activity functions
---
---
} catch (Exception e)
{
await orchestrationContext.CallActivityAsync(
nameof(UpdateJobStatus),
(DataOperationInfo.Id, DataOperationStatus.Failed, string.Empty, string.Empty, requestContext));
}
}
}
[FunctionName(nameof(UpdateJobStatus))]
public static async Task RunAsync(
[ActivityTrigger] IDurableActivityContext activityContext,
[Inject]IServiceProvider rootServiceProvider)
{
using (var scope = rootServiceProvider.CreateScope())
{
try
{
var (DataOperationId, status, blobReference, logFileBlobId, requestContext) = activityContext.GetInput<(string, string, string, string, RequestContext)>();
var services = scope.ServiceProvider;
services.ResolveRequestContext(requestContext.CustomerId, requestContext.UserId, requestContext.UserDisplayName, requestContext.Culture);
var dataService = services.GetRequiredService<IDataService>();
var DataOperationDto = new DataOperationDto
{
Id = DataOperationId,
OperationStatusCode = status,
BlobReference = blobReference,
LogBlobReference = logFileBlobId
};
await dataService.UpdateAsync(DataOperationDto);
}
catch (Exception e)
{
throw e;
}
}
}
While you are debugging your Function, you should make sure that you are either using the local storage emulator, or an Azure Storage Account that is different from the one that is also being used by the Function already deployed in Azure.
If you are using the same storage account that another running Function is using, then it could be that parts of the execution is actually happening in Azure instead of your dev machine.
I have a asp.net core app that uploads file to azure blob storage. The upload completes successfully but when CloudBlobContainer calls CreateIfNotExistAsync the web page crashes with a "The container already exists error".
var container = BlobClient.GetContainerReference(containerName.ToString().ToLower());
await container.CreateIfNotExistsAsync();
return container;
I have tried using surronding CreateIfNotExistsAsync() with the following if
if (! await container.ExistsAsync())
but that still errors.
The container name and the AccountName= in the connection string is lowercase.
I am using the latest stable Microsoft.Azure.Storage.Blob NuGet package 10.0.2
I have tried to catch the StroageExeception but the exception is not called
try
{
await container.CreateIfNotExistsAsync();
}
catch (StorageException ex)
{
Logger.LogError(ex.ToString());
}
I have been through all the points in this previous question but none of them apply/work in my scenario Azure Blob 400 Bad request on Creation of container
public class CloudBlobStorageProvider : ICloudBlobStorageProvider
{
private CloudBlobClient BlobClient { get; }
private ILogger Logger { get; }
public CloudBlobStorageProvider(IConfiguration configuration, ILogger<CloudBlobStorageProvider> logger)
{
var connectionString = configuration.GetConnectionString("AzureStorageAccount");
if (!CloudStorageAccount.TryParse(connectionString, out CloudStorageAccount storageAccount))
{
logger.LogError($"The supplied connection string for Azure blob storage could not be parsed: {connectionString}");
}
BlobClient = storageAccount.CreateCloudBlobClient();
}
public async Task<CloudBlobContainer> GetContainerAsync(CloudBlobContainerName containerName)
{
var container = BlobClient.GetContainerReference(containerName.ToString().ToLower());
await container.CreateIfNotExistsAsync(BlobContainerPublicAccessType.Off, null, null);
return container;
}
}
public interface ICloudBlobStorageProvider
{
Task<CloudBlobContainer> GetContainerAsync(CloudBlobContainerName containerName);
}
Which is called by
public async Task<CloudBlockBlob> UploadServiceUserAttachmentAsync(IFormFile formFile)
{
var fileExtenion = RegularExpressionHelpers.GetFileExtension(formFile.FileName);
string attachmentFileName = (string.IsNullOrEmpty(fileExtenion)) ? $"{Guid.NewGuid().ToString()}" : $"{Guid.NewGuid().ToString()}{fileExtenion}";
var userAttachmentContainer = await CloudBlobStorageProvider.GetContainerAsync(CloudBlobContainerName.userattachments);
var blobBlockReference = userAttachmentContainer.GetBlockBlobReference(attachmentFileName);
try
{
using (var stream = formFile.OpenReadStream())
{
await blobBlockReference.UploadFromStreamAsync(stream);
await blobBlockReference.FetchAttributesAsync();
var blobProperties = blobBlockReference.Properties;
blobProperties.ContentType = formFile.ContentType;
await blobBlockReference.SetPropertiesAsync();
}
}
catch (Exception e)
{
Logger.LogWarning(e, $"Exception encountered while attempting to write profile photo to blob storage");
}
return blobBlockReference;
}
I've tried to come up with something from the example in the WebJobsSDK gitHub
var eventHubConfig = new EventHubConfiguration();
string eventHubName = "MyHubName";
eventHubConfig.AddSender(eventHubName,"Endpoint=sb://test.servicebus.windows.net/;SharedAccessKeyName=SendRule;SharedAccessKey=xxxxxxxx");
eventHubConfig.AddReceiver(eventHubName, "Endpoint=sb://test.servicebus.windows.net/;SharedAccessKeyName=ReceiveRule;SharedAccessKey=yyyyyyy");
config.UseEventHub(eventHubConfig);
JobHost host = new JobHost(config);
But I'm afraid that's not far enough for someone of my limited "skillset"!
I can find no instance of JobHostConfiguration that has a UseEventHub property (using the v1.2.0-alpha-10291 version of the Microsoft.AzureWebJobs package), so I can't pass the EventHubConfiguration to the JobHost.
I've used EventHub before, not within the WebJob context. I don't see if the EventHostProcessor is still required if using the WebJob triggering...or does the WebJob trigger essentially act as the EventHostProcessor?
Anyway, if anyone has a more complete example for a simpleton like me that would be really sweet! Thanks
From the documentation here, you should have all the information you need.
What you are missing is a reference of the Microsoft.Azure.WebJobs.ServiceBus.1.2.0-alpha-10291 nuget package.
The UseEventHub is an extension method that is declared in this package.
Otherwise your configuration seems ok.
Here is an example on how to receive or send messages from/to an EventHub:
public class BasicTest
{
public class Payload
{
public int Counter { get; set; }
}
public static void SendEvents([EventHub("MyHubName")] out Payload x)
{
x = new Payload { Counter = 100 };
}
public static void Trigger(
[EventHubTrigger("MyHubName")] Payload x,
[EventHub("MyHubName")] out Payload y)
{
x.Counter++;
y = x;
}
}
EventProcessorHost is still required, as the WebJob just provides the hosting environment for running it. As far as I know, EventProcessorHost is not integrated so deeply into WebJob, so its triggering mechanism cannot be used for processing EventHub messages. I use WebJob for running EventProcessorHost continuously:
public static void Main()
{
RunAsync().Wait();
}
private static async Task RunAsync()
{
try
{
using (var shutdownWatcher = new WebJobsShutdownWatcher())
{
await Console.Out.WriteLineAsync("Initializing...");
var eventProcessorHostName = "eventProcessorHostName";
var eventHubName = ConfigurationManager.AppSettings["eventHubName"];
var consumerGroupName = ConfigurationManager.AppSettings["eventHubConsumerGroupName"];
var eventHubConnectionString = ConfigurationManager.ConnectionStrings["EventHub"].ConnectionString;
var storageConnectionString = ConfigurationManager.ConnectionStrings["EventHubStorage"].ConnectionString;
var eventProcessorHost = new EventProcessorHost(eventProcessorHostName, eventHubName, consumerGroupName, eventHubConnectionString, storageConnectionString);
await Console.Out.WriteLineAsync("Registering event processors...");
var processorOptions = new EventProcessorOptions();
processorOptions.ExceptionReceived += ProcessorOptions_ExceptionReceived;
await eventProcessorHost.RegisterEventProcessorAsync<CustomEventProcessor>(processorOptions);
await Console.Out.WriteLineAsync("Processing...");
await Task.Delay(Timeout.Infinite, shutdownWatcher.Token);
await Console.Out.WriteLineAsync("Unregistering event processors...");
await eventProcessorHost.UnregisterEventProcessorAsync();
await Console.Out.WriteLineAsync("Finished.");
}
catch (Exception ex)
{
await HandleErrorAsync(ex);
}
}
}
private static async void ProcessorOptions_ExceptionReceived(object sender, ExceptionReceivedEventArgs e)
{
await HandleErrorAsync(e.Exception);
}
private static async Task HandleErrorAsync(Exception ex)
{
await Console.Error.WriteLineAsync($"Critical error occured: {ex.Message}{ex.StackTrace}");
}
I have an Azure Web Job built using the Azure SDK whose only job is to call a web service (Web API) and then log a response based on the return value (a class). The problem is that as soon as it calls the HttpClient PostAsJsonAsync method to call the service, it exits out of the web job without executing any of the response handling. My code is:
public class Result
{
// Properties ---------------------------------------------------------
public bool Success { get; set; }
public string Error { get; set; }
}
public class Functions
{
// This function will be triggered based on the schedule you have set for this WebJob
// This function will enqueue a message on an Azure Queue called queue
[NoAutomaticTrigger]
public async static void ManualTrigger(TextWriter log, int value)
{
using (var client = new HttpClient())
{
var rootUrl = ConfigurationManager.AppSettings.Get("WebJobTargetUrl");
client.BaseAddress = new System.Uri(rootUrl);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
Console.WriteLine("Call service");
var response = await client.PostAsJsonAsync("api/Reminder/ProcessDueReminders", new { ItemID = 1 });
Console.WriteLine("After service");
var result = await response.Content.ReadAsAsync<Result>();
Console.WriteLine("After result");
if (result.Success)
Console.WriteLine("Reminders Processed");
else
Console.WriteLine("Reminder process error: " + result.Error);
}
}
}
and the execution logs from the portal are:
I believe it has something to do with the asynchronous operation but I can't figure out a pattern that will work. Any help would be appreciated.
You must define the return value of your own async method as Task instead of void.
On a related note, you should suffix the name of your method with Async. That's not going to solve the problem, but it indicates that you're using the async/await pattern.
There is probably an exception in your PostAsJsonAsync call. Try to put a try catch around it to and log the error:
try {
var response = await client.PostAsJsonAsync("api/Reminder/ProcessDueReminders", new { ItemID = 1 });
} catch (Exception ex){
Console.WriteLine("Exception: "+ ex);
}
Console.WriteLine("After service");