some of my continuous running web job function(random) show message of Timeout value of 00:30:00 exceeded by function '<myfunction>' (Id: '<id>'). Initiating cancellation.
after this message this function will not execute itself until and unless manually stop and start the azure web job.
Thanks in advance.
some of my continuous running web job function(random) show message of Timeout value of 00:30:00 exceeded by function '<myfunction>' (Id: '<id>'). Initiating cancellation.
Based on your error, I found the related code from Microsoft.Azure.WebJobs.Host under FunctionExecutor.cs as follows:
internal static void OnFunctionTimeout(System.Timers.Timer timer, FunctionDescriptor method, Guid instanceId, TimeSpan timeout, bool timeoutWhileDebugging,
TraceWriter trace, ILogger logger, CancellationTokenSource cancellationTokenSource, Func<bool> isDebuggerAttached)
{
timer.Stop();
bool shouldTimeout = timeoutWhileDebugging || !isDebuggerAttached();
string message = string.Format(CultureInfo.InvariantCulture,
"Timeout value of {0} exceeded by function '{1}' (Id: '{2}'). {3}",
timeout.ToString(), method.ShortName, instanceId,
shouldTimeout ? "Initiating cancellation." : "Function will not be cancelled while debugging.");
trace.Error(message, null, TraceSource.Execution);
logger?.LogError(message);
trace.Flush();
// Only cancel the token if not debugging
if (shouldTimeout)
{
// only cancel the token AFTER we've logged our error, since
// the Dashboard function output is also tied to this cancellation
// token and we don't want to dispose the logger prematurely.
cancellationTokenSource.Cancel();
}
}
I assumed that you specified the TimeoutAttribute for your function as follows:
I would recommend you could use a CancellationToken parameter in your function and it would be canceled whenever a timeout occurs or host shutdown, and you could exit your function gracefully as follows:
Related
I have a CRM system, when a contact is added, I want to add them to an accounting system.
I have setup a webhook in the CRM system that passes the contact to an Azure Function. The Azure function connects to the accounting system API and creates them there.
There is a little other processing I need to do before the user can be added to the accounting system.
I need about a 5 minute delay after receiving the webhook before I can add the user to the accounting system.
I would rather not add a pause or delay statement in the Azure Function as there is a timeout limit, and also It's a consumption plan so I want each function to action quickly.
I am using Powershell core.
Is a Service Bus Queue the best way to do this?
You could use a Timer in a Durable Function for this. Then you won't need an extra component like a queue. A Durable Function is all you need. For example (warning: not compiled this):
Note: Durable Functions do support powershell but I don't ;-) So the code below is to understand the concept.
[FunctionName("Orchestration_HttpStart")]
public static async Task<HttpResponseMessage> HttpStart(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequestMessage req,
[DurableClient] IDurableOrchestrationClient starter,
ILogger log)
{
// Function input comes from the request content.
string content = await req.Content.ReadAsStringAsync();
string instanceId = await starter.StartNewAsync("Orchestration", content);
log.LogInformation($"Started orchestration with ID = '{instanceId}'.");
return starter.CreateCheckStatusResponse(req, instanceId);
}
[FunctionName("Orchestration")]
public static async Task Run(
[OrchestrationTrigger] IDurableOrchestrationContext context)
{
var requestContent = context.GetInput<string>();
DateTime waitAWhile = context.CurrentUtcDateTime.Add(TimeSpan.FromMinutes(5));
await context.CreateTimer(waitAWhile, CancellationToken.None);
await context.CallActivityAsync("ProcessEvent", requestContent);
}
[FunctionName("ProcessEvent")]
public static string ProcessEvent([ActivityTrigger] string requestContent, ILogger log)
{
// Do something here with requestContent
return "Done!";
}
I would rather not add a pause or delay statement in the Azure Function as there is a timeout limit, and also It's a consumption plan so I want each function to action quickly.
The 5 minutes delay introduced by the timer won't count as active time so you won't run out of time on the consumption plan for those minutes.
Is a Service Bus Queue the best way to do this?
You can use it, but Azure Storage Queue is cheaper for your scenario.
What you can do is create a time triggered functions (* */5 * * * *) and will check for a message in a queue. If the time between the execution and the time the message was created is greater than minutes, then you process and complete the message, otherwise, don't complete the message and it will return to the queue for the next execution.
I have a scenario in which I am calling RegisterMessageHandler of SubscriptionClient class of Azure Service Bus library.
Basically I am using trigger based approach while receiving the messages from Service Bus in one of my services in Service Fabric Environment as a stateless service.
So I am not closing the subscriptionClient object immediately, rather I am keeping it open for the lifetime of the Service so that it keeps on receiving the message from azure service bus topics.
And when the service needs to shut down(due to some reasons), I want to handle the cancellation token being passed into the service of Service Fabric.
My question is how can I handle the cancellation token in the RegisterMessageHandler method which gets called whenever a new message is received?
Also I want to handle the closing of the Subscription client "Gracefully", i.e I want that if a message is already being processed, then I want that message to get processed completely and then I want to close the connection.
Below is the code I am using.
Currently We are following the below approach:
1. Locking the process of the message using semaphore lock and releasing the lock in finally block.
2. Calling the cancellationToken.Register method to handle cancellation token whenever cancellation is done. Releasing the lock in the Register Method.
public class AzureServiceBusReceiver
{
private SubscriptionClient subscriptionClient;
private static Semaphore semaphoreLock;
public AzureServiceBusReceiver(ServiceBusReceiverSettings settings)
{
semaphoreLock = new Semaphore(1, 1);
subscriptionClient = new SubscriptionClient(
settings.ConnectionString, settings.TopicName, settings.SubscriptionName, ReceiveMode.PeekLock);
}
public void Receive(
CancellationToken cancellationToken)
{
var options = new MessageHandlerOptions(e =>
{
return Task.CompletedTask;
})
{
AutoComplete = false,
};
subscriptionClient.RegisterMessageHandler(
async (message, token) =>
{
semaphoreLock.WaitOne();
if (subscriptionClient.IsClosedOrClosing)
return;
CancellationToken combinedToken = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, token).Token;
try
{
// message processing logic
}
catch (Exception ex)
{
await subscriptionClient.DeadLetterAsync(message.SystemProperties.LockToken);
}
finally
{
semaphoreLock.Release();
}
}, options);
cancellationToken.Register(() =>
{
semaphoreLock.WaitOne();
if (!subscriptionClient.IsClosedOrClosing)
subscriptionClient.CloseAsync().GetAwaiter().GetResult();
semaphoreLock.Release();
return;
});
}
}
Implement the message client as ICommunicationListener, so when the service is closed, you can block the call until message processing is complete.
Don't use a static Semaphore, so you can safely reuse the code within your projects.
Here is an example of how you can do this.
And here's the Nuget package created by that code.
And feel free to contribute!
I created a very simple Azure functionApp using visual Studio Code which does nothing fancy but logs the received message(Function App listen to Eventhub message in Azure). This works and get hosted well when i debug the function app locally. However, when I publish this into minikube as container, it throws below error:
The listener for function 'EventHubCSharpFunc' was unable to start.
Microsoft.Azure.WebJobs.Host.Listeners.FunctionListenerException: The listener for function 'EventHubCSharpFunc' was unable to start. ---> Microsoft.Azure.EventHubs.Processor.EventProcessorConfigurationException: Encountered error while fetching the list of EventHub PartitionIds ---> System.Net.Sockets.SocketException: Resource temporarily unavailable
at Microsoft.Azure.EventHubs.Amqp.AmqpEventHubClient.CreateConnectionAsync(TimeSpan timeout)
at Microsoft.Azure.Amqp.FaultTolerantAmqpObject`1.OnCreateAsync(TimeSpan timeout)
at Microsoft.Azure.Amqp.Singleton`1.CreateValue(TaskCompletionSource`1 tcs, TimeSpan timeout)
at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout)
at Microsoft.Azure.EventHubs.Amqp.Management.AmqpServiceClient.OpenRequestResponseLinkAsync(String type, String address, Nullable`1 entityType, String[] requiredClaims, TimeSpan timeout)
at Microsoft.Azure.EventHubs.Amqp.Management.AmqpServiceClient.OpenLinkAsync(TimeSpan timeout)
at Microsoft.Azure.Amqp.FaultTolerantAmqpObject`1.OnCreateAsync(TimeSpan timeout)
at Microsoft.Azure.Amqp.Singleton`1.CreateValue(TaskCompletionSource`1 tcs, TimeSpan timeout)
at Microsoft.Azure.Amqp.Singleton`1.GetOrCreateAsync(TimeSpan timeout)
at Microsoft.Azure.EventHubs.Amqp.Management.AmqpServiceClient.GetRuntimeInformationAsync()
at Microsoft.Azure.EventHubs.Amqp.AmqpEventHubClient.OnGetRuntimeInformationAsync()
at Microsoft.Azure.EventHubs.EventHubClient.GetRuntimeInformationAsync()
at Microsoft.Azure.EventHubs.Processor.PartitionManager.GetPartitionIdsAsync()
--- End of inner exception stack trace ---
at Microsoft.Azure.EventHubs.Processor.PartitionManager.GetPartitionIdsAsync()
at Microsoft.Azure.EventHubs.Processor.PartitionManager.GetPartitionIdsAsync()
at Microsoft.Azure.EventHubs.Processor.PartitionManager.InitializeStoresAsync()
at Microsoft.Azure.EventHubs.Processor.PartitionManager.StartAsync()
at Microsoft.Azure.EventHubs.Processor.EventProcessorHost.RegisterEventProcessorFactoryAsync(IEventProcessorFactory factory, EventProcessorOptions processorOptions)
at Microsoft.Azure.WebJobs.EventHubs.EventHubListener.StartAsync(CancellationToken cancellationToken) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Extensions.EventHubs\Listeners\EventHubListener.cs:line 46
at Microsoft.Azure.WebJobs.Host.Listeners.FunctionListener.StartAsync(CancellationToken cancellationToken, Boolean allowRetry) in C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Listeners\FunctionListener.cs:line 68
I tried to check if my cluster is having internet access, but I am successfully able to ping internet IP from other containers in minikube.
public static class EventHubCSharpFunc
{
[FunctionName("EventHubCSharpFunc")]
public static async Task Run([EventHubTrigger("containertestevthub", Connection = "EventGridPerfEventHun_RootManageSharedAccessKey_EVENTHUB")] EventData[] events, ILogger log)
{
var exceptions = new List<Exception>();
foreach (EventData eventData in events)
{
try
{
string messageBody = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);
// Replace these two lines with your processing logic.
log.LogInformation($"C# Event Hub trigger function processed a message: {messageBody}");
await Task.Yield();
}
catch (Exception e)
{
// We need to keep processing the rest of the batch - capture this exception and continue.
// Also, consider capturing details of the message that failed processing so it can be processed again later.
exceptions.Add(e);
}
}
// Once processing of the batch is complete, if any messages in the batch failed processing throw an exception so that there is a record of the failure.
if (exceptions.Count > 1)
throw new AggregateException(exceptions);
if (exceptions.Count == 1)
throw exceptions.Single();
}
}
looking into "EventHubCSharpFunc was unable to start" and "Resource temporarily unavailable", please try to connect from Your APP POD to Eventhub.
Please verify if your connection and credentials are working inside Your APP POD.
You can find more about how to pass credentials as Secrets into your POD or Create a Secret based on existing Docker credentials here and here.
For troubleshooting purposes please use those below commands to get more information from your cluster:
kubectl log <your_pod_with_application>
kubectl describe <your_pod_with_application>
kubectl get events
kubectl get all --all-namespaces
When an exception is thrown from webjob, it exits without logging to the application insights. Observed that flushing the logs to application insights takes few minutes, so we are missing the exceptions here. How to handle this?
Also, is there a way to move the message which hit the exception to poison queue automatically without manually inserting that message to poison queue?
I am using latest stable 3.x versions for the 2 NuGet packages:
Microsoft.Azure.WebJobs and Microsoft.Azure.WebJobs.Extensions
Created a host that implemented IHost as below:
var builder = new HostBuilder()
.UseEnvironment("Development")
.ConfigureWebJobs(b =>
{
...
})
.ConfigureLogging((context, b) =>
{
string appInsightsKey = context.Configuration["APPINSIGHTS_INSTRUMENTATIONKEY"];
if (!string.IsNullOrEmpty(appInsightsKey))
{
b.AddApplicationInsights(o => o.InstrumentationKey = appInsightsKey);
appInsights.TrackEvent("Application Insights is starting!!");
}
})
.ConfigureServices(services =>
{
….
})
.UseConsoleLifetime();
var host = builder.Build();
using (host)
{
host.RunAsync().Wait();
}
and Function.cs
public static async void ProcessQueueMessageAsync([QueueTrigger("queue")] Message message, int dequeueCount, IBinder binder, ILogger logger)
{
switch (message.Name)
{
case blah:
...break;
default:
logger.LogError("Invalid Message object in the queue.", message);
logger.LogWarning("Current dequeue count: " + dequeueCount);
throw new InvalidOperationException("Simulated Failure");
}
}
My questions here are:
1) When the default case is hit, webjob is terminating immediately and the loggers are not getting flushed into app insights even after waiting and starting the web job again. As it takes few minutes to reflect in app insights, and webjob stops, I am losing the error logs. How to handle this?
2) From the sample webjobs here, https://github.com/Azure/azure-webjobs-sdk-samples/blob/master/BasicSamples/QueueOperations/Functions.cs they are using JobHost host = new JobHost(); and if the 'FailAlways' function fails, it automatically retries for 5 times and pushed the message into poison queue. But this is not happening in my code. Is it because of different Hosts? or do I have to add any more configurations?
Try changing your function to return Task instead of void:
public static async Task ProcessQueueMessageAsync([QueueTrigger("queue")] Message message, int dequeueCount, IBinder binder, ILogger logger)
This worked for me where even though I was logging the error and throwing the exception, Application Insights would either show a successful invocation or no invocation occurring.
After inspecting the source code of the Application Insights SDK it became apparent that to get an Exception in Application Insights you must pass an exception object into the LogError call.
log.Error(ex, "my error message") - will result in Application Insight Exception
log.Error("my error message") - will result in Application Insight Trace.
is there a way to move the message which hit the exception to poison queue automatically without manually inserting that message to poison queue?
You could set config.Queues.MaxDequeueCount = 1; in webjob. The number of times to try processing a message before moving it to the poison queue.
And where is the MaxDequeueCount configuration should be added in the code?
You could set the property in JobHostConfiguration in program.cs
I have a Service Bus Trigger Azure function, which is triggered every time a topic receives a message.
Messages arrive at regular intervals, for example every 30 minutes. Between lots, no activity.
The function does nothing special, it does an asynchronous posting of the message via HttpClient. The function is regularly stopped with a TaskCanceledException.
The HttpClient is static
public static class SampleEventTrigger
{
private static DefaultHttpWebHook webHook = new DefaultHttpWebHook(new Uri("https://nonexistent.invalid/sampleWebHook"), "/event/sampleEvent");
[FunctionName("SampleEventTrigger")]
public static async Task Run(
[ServiceBusTrigger("sampleevent", "SampleEvent.Subs", AccessRights.Manage, Connection = GlobalConfiguration.ServiceBusConnection)]BrokeredMessage message,
TraceWriter log)
{
log.Info("launch sample event subscription");
try
{
var resp = await webHook.Post(message, log);
log.Info($"{resp.StatusCode}, {resp.ReasonPhrase}");
}
catch (Exception ex)
{
log.Error($"exception in webhook: {ex.Message}", ex);
throw;
}
}
}
If I raise it again just after, this time it passes.
Where does this exception come from? How do we avoid that?
Is it related to a timeout, or to launching the function that would be too slow?
My function is in Consumption mode.
Chances are that your Http call is timing out. Awaited Http calls that time out throw TaskCanceledException . I'm not sure what your DefaultHttpWebHook class does under the covers, but it should be using PostAsync in the Post method (which itself should have the Async suffix).
To verify you could catch TaskCanceledException and examine the inner exception. If you are still struggling, convert your code to non-async during local development to get a better handle on what's happening - it'll give you back a true exception rather than bubbling it up as a TCE.