I have a Service Bus Trigger Azure function, which is triggered every time a topic receives a message.
Messages arrive at regular intervals, for example every 30 minutes. Between lots, no activity.
The function does nothing special, it does an asynchronous posting of the message via HttpClient. The function is regularly stopped with a TaskCanceledException.
The HttpClient is static
public static class SampleEventTrigger
{
private static DefaultHttpWebHook webHook = new DefaultHttpWebHook(new Uri("https://nonexistent.invalid/sampleWebHook"), "/event/sampleEvent");
[FunctionName("SampleEventTrigger")]
public static async Task Run(
[ServiceBusTrigger("sampleevent", "SampleEvent.Subs", AccessRights.Manage, Connection = GlobalConfiguration.ServiceBusConnection)]BrokeredMessage message,
TraceWriter log)
{
log.Info("launch sample event subscription");
try
{
var resp = await webHook.Post(message, log);
log.Info($"{resp.StatusCode}, {resp.ReasonPhrase}");
}
catch (Exception ex)
{
log.Error($"exception in webhook: {ex.Message}", ex);
throw;
}
}
}
If I raise it again just after, this time it passes.
Where does this exception come from? How do we avoid that?
Is it related to a timeout, or to launching the function that would be too slow?
My function is in Consumption mode.
Chances are that your Http call is timing out. Awaited Http calls that time out throw TaskCanceledException . I'm not sure what your DefaultHttpWebHook class does under the covers, but it should be using PostAsync in the Post method (which itself should have the Async suffix).
To verify you could catch TaskCanceledException and examine the inner exception. If you are still struggling, convert your code to non-async during local development to get a better handle on what's happening - it'll give you back a true exception rather than bubbling it up as a TCE.
Related
I have a CRM system, when a contact is added, I want to add them to an accounting system.
I have setup a webhook in the CRM system that passes the contact to an Azure Function. The Azure function connects to the accounting system API and creates them there.
There is a little other processing I need to do before the user can be added to the accounting system.
I need about a 5 minute delay after receiving the webhook before I can add the user to the accounting system.
I would rather not add a pause or delay statement in the Azure Function as there is a timeout limit, and also It's a consumption plan so I want each function to action quickly.
I am using Powershell core.
Is a Service Bus Queue the best way to do this?
You could use a Timer in a Durable Function for this. Then you won't need an extra component like a queue. A Durable Function is all you need. For example (warning: not compiled this):
Note: Durable Functions do support powershell but I don't ;-) So the code below is to understand the concept.
[FunctionName("Orchestration_HttpStart")]
public static async Task<HttpResponseMessage> HttpStart(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequestMessage req,
[DurableClient] IDurableOrchestrationClient starter,
ILogger log)
{
// Function input comes from the request content.
string content = await req.Content.ReadAsStringAsync();
string instanceId = await starter.StartNewAsync("Orchestration", content);
log.LogInformation($"Started orchestration with ID = '{instanceId}'.");
return starter.CreateCheckStatusResponse(req, instanceId);
}
[FunctionName("Orchestration")]
public static async Task Run(
[OrchestrationTrigger] IDurableOrchestrationContext context)
{
var requestContent = context.GetInput<string>();
DateTime waitAWhile = context.CurrentUtcDateTime.Add(TimeSpan.FromMinutes(5));
await context.CreateTimer(waitAWhile, CancellationToken.None);
await context.CallActivityAsync("ProcessEvent", requestContent);
}
[FunctionName("ProcessEvent")]
public static string ProcessEvent([ActivityTrigger] string requestContent, ILogger log)
{
// Do something here with requestContent
return "Done!";
}
I would rather not add a pause or delay statement in the Azure Function as there is a timeout limit, and also It's a consumption plan so I want each function to action quickly.
The 5 minutes delay introduced by the timer won't count as active time so you won't run out of time on the consumption plan for those minutes.
Is a Service Bus Queue the best way to do this?
You can use it, but Azure Storage Queue is cheaper for your scenario.
What you can do is create a time triggered functions (* */5 * * * *) and will check for a message in a queue. If the time between the execution and the time the message was created is greater than minutes, then you process and complete the message, otherwise, don't complete the message and it will return to the queue for the next execution.
I have a scenario in which I am calling RegisterMessageHandler of SubscriptionClient class of Azure Service Bus library.
Basically I am using trigger based approach while receiving the messages from Service Bus in one of my services in Service Fabric Environment as a stateless service.
So I am not closing the subscriptionClient object immediately, rather I am keeping it open for the lifetime of the Service so that it keeps on receiving the message from azure service bus topics.
And when the service needs to shut down(due to some reasons), I want to handle the cancellation token being passed into the service of Service Fabric.
My question is how can I handle the cancellation token in the RegisterMessageHandler method which gets called whenever a new message is received?
Also I want to handle the closing of the Subscription client "Gracefully", i.e I want that if a message is already being processed, then I want that message to get processed completely and then I want to close the connection.
Below is the code I am using.
Currently We are following the below approach:
1. Locking the process of the message using semaphore lock and releasing the lock in finally block.
2. Calling the cancellationToken.Register method to handle cancellation token whenever cancellation is done. Releasing the lock in the Register Method.
public class AzureServiceBusReceiver
{
private SubscriptionClient subscriptionClient;
private static Semaphore semaphoreLock;
public AzureServiceBusReceiver(ServiceBusReceiverSettings settings)
{
semaphoreLock = new Semaphore(1, 1);
subscriptionClient = new SubscriptionClient(
settings.ConnectionString, settings.TopicName, settings.SubscriptionName, ReceiveMode.PeekLock);
}
public void Receive(
CancellationToken cancellationToken)
{
var options = new MessageHandlerOptions(e =>
{
return Task.CompletedTask;
})
{
AutoComplete = false,
};
subscriptionClient.RegisterMessageHandler(
async (message, token) =>
{
semaphoreLock.WaitOne();
if (subscriptionClient.IsClosedOrClosing)
return;
CancellationToken combinedToken = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, token).Token;
try
{
// message processing logic
}
catch (Exception ex)
{
await subscriptionClient.DeadLetterAsync(message.SystemProperties.LockToken);
}
finally
{
semaphoreLock.Release();
}
}, options);
cancellationToken.Register(() =>
{
semaphoreLock.WaitOne();
if (!subscriptionClient.IsClosedOrClosing)
subscriptionClient.CloseAsync().GetAwaiter().GetResult();
semaphoreLock.Release();
return;
});
}
}
Implement the message client as ICommunicationListener, so when the service is closed, you can block the call until message processing is complete.
Don't use a static Semaphore, so you can safely reuse the code within your projects.
Here is an example of how you can do this.
And here's the Nuget package created by that code.
And feel free to contribute!
I have an azure function which returns a service bus message. However, I want to conditionally return a service bus message, instead of being forced to return the message every time.
here is an example
[FunctionName("ServiceBusOutput")]
[return: ServiceBus("myqueue", Connection = "ServiceBusConnection")]
public static string ServiceBusOutput([HttpTrigger] dynamic input, ILogger log)
{
log.LogInformation($"C# function processed: {input.Text}");
// check condition here, abort return completely
// Otherwise, return
return input.Text;
}
Said another way, I want to return a message on a service bus when certain conditions apply within the function code block. Is this possible?
One idea that does not work is to throw an exception. However, this just results in the message being placed into the DL queue. I want to completely abort the operation of returning the message on the service bus, and avoid DL.
Another idea that does not work is to simply execute
return;
But this results in compile-time error, which is sort of expected
"An object of a type convertible to 'MyReturnType1' is required"
I can think of a hack which I dont like, which is to return null, and handle the null later in the chain. But this is sort of dirty to me.
You could just bind ServiceBus as MessageSender type, then use the SendAsync() method to send the message.
The below is my test code, if the request name equals "george", it will send the name to the message queue.
public static class Function1
{
[FunctionName("Function1")]
public static async System.Threading.Tasks.Task RunAsync(
[HttpTrigger(AuthorizationLevel.Function, "get", "post", Route = null)] HttpRequest req,
[ServiceBus("myqueue", Connection = "ServiceBusConnection")] MessageSender messagesQueue,
ILogger log)
{
log.LogInformation("C# HTTP trigger function processed a request.");
string name = req.Query["name"];
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
dynamic data = JsonConvert.DeserializeObject(requestBody);
name = name ?? data?.name;
if (name.Equals("george")) {
byte[] bytes = Encoding.ASCII.GetBytes(name);
Message m1 = new Message();
m1.Body = bytes;
await messagesQueue.SendAsync(m1);
}
}
}
Suppose this is what you want, hope this could help you, if you still have other problem please feel free to let me know.
Instead of using the output binding available in Azure Function, you can send a message to the queue from a custom queue client created inside the function.
Posting the message based on a condition is not possible with the bindings.
When an exception is thrown from webjob, it exits without logging to the application insights. Observed that flushing the logs to application insights takes few minutes, so we are missing the exceptions here. How to handle this?
Also, is there a way to move the message which hit the exception to poison queue automatically without manually inserting that message to poison queue?
I am using latest stable 3.x versions for the 2 NuGet packages:
Microsoft.Azure.WebJobs and Microsoft.Azure.WebJobs.Extensions
Created a host that implemented IHost as below:
var builder = new HostBuilder()
.UseEnvironment("Development")
.ConfigureWebJobs(b =>
{
...
})
.ConfigureLogging((context, b) =>
{
string appInsightsKey = context.Configuration["APPINSIGHTS_INSTRUMENTATIONKEY"];
if (!string.IsNullOrEmpty(appInsightsKey))
{
b.AddApplicationInsights(o => o.InstrumentationKey = appInsightsKey);
appInsights.TrackEvent("Application Insights is starting!!");
}
})
.ConfigureServices(services =>
{
….
})
.UseConsoleLifetime();
var host = builder.Build();
using (host)
{
host.RunAsync().Wait();
}
and Function.cs
public static async void ProcessQueueMessageAsync([QueueTrigger("queue")] Message message, int dequeueCount, IBinder binder, ILogger logger)
{
switch (message.Name)
{
case blah:
...break;
default:
logger.LogError("Invalid Message object in the queue.", message);
logger.LogWarning("Current dequeue count: " + dequeueCount);
throw new InvalidOperationException("Simulated Failure");
}
}
My questions here are:
1) When the default case is hit, webjob is terminating immediately and the loggers are not getting flushed into app insights even after waiting and starting the web job again. As it takes few minutes to reflect in app insights, and webjob stops, I am losing the error logs. How to handle this?
2) From the sample webjobs here, https://github.com/Azure/azure-webjobs-sdk-samples/blob/master/BasicSamples/QueueOperations/Functions.cs they are using JobHost host = new JobHost(); and if the 'FailAlways' function fails, it automatically retries for 5 times and pushed the message into poison queue. But this is not happening in my code. Is it because of different Hosts? or do I have to add any more configurations?
Try changing your function to return Task instead of void:
public static async Task ProcessQueueMessageAsync([QueueTrigger("queue")] Message message, int dequeueCount, IBinder binder, ILogger logger)
This worked for me where even though I was logging the error and throwing the exception, Application Insights would either show a successful invocation or no invocation occurring.
After inspecting the source code of the Application Insights SDK it became apparent that to get an Exception in Application Insights you must pass an exception object into the LogError call.
log.Error(ex, "my error message") - will result in Application Insight Exception
log.Error("my error message") - will result in Application Insight Trace.
is there a way to move the message which hit the exception to poison queue automatically without manually inserting that message to poison queue?
You could set config.Queues.MaxDequeueCount = 1; in webjob. The number of times to try processing a message before moving it to the poison queue.
And where is the MaxDequeueCount configuration should be added in the code?
You could set the property in JobHostConfiguration in program.cs
I am testing the Poison message handling of the Webjob that I am building.
Everything seems to be working as expected except, one strange thing:
When a message is moved to the “-poison” queue, its ghost seems to remain hidden (invisible) in the main job queue. That means if I have 6 poison messages moved to the “-poison” queue, storage explorer shows “Showing 0 of 6 messages in queue”. I can not see the 6 hidden messages in the Storage Explorer.
I tried to delete the job queue and recreating it, but the strange issue still happening after I run my tests. Storage explorer shows “Showing 0 of 6 messages in queue”.
What is happening behind the scene?
Update 1
I did some investigation and I think WebJob SDK does not delete the poison message.
I went through WebJob SDK source code and I think this line of code is not being executed for some reason:
https://github.com/Azure/azure-webjobs-sdk/blob/dev/src/Microsoft.Azure.WebJobs.Host/Queues/QueueProcessor.cs#L119
Here is my Function that can help reproducing the issue:
public class Functions
{
public static void ProcessQueueMessage([QueueTrigger("%QueueName%")] string message, TextWriter log)
{
if (message.Contains("Break"))
{
throw new Exception($"Error while processing message {message}");
}
log.WriteLine($"Processed message {message}");
}
}
Update 2
Here is the WebJob SDK I am using:
As far as I know, the azure storage SDK 8.+ is not work well with the Azure webjobs SDK2.0 (related issue).
If you use storage SDK 8.+ the poison messages stay undeleted-but-invisible.
Workaround method is using the low azure storage SDK 7.2.1.
It will work well.
And this issue will be solved in the future SDK version.
I have the same problem.
The problem is when then Message copy in poison queue pass by ref without visibility time https://github.com/Azure/azure-webjobs-sdk/blob/dev/src/Microsoft.Azure.WebJobs.Host/Queues/QueueProcessor.cs#L145 and when try to delete the message from original queue the service returns 404 not found. Is a problem in azure-webjobs-sdk and the solution is to make this change
await AddMessageAndCreateIfNotExistsAsync(poisonQueue, new CloudQueueMessage(message.AsString), cancellationToken);
in https://github.com/Azure/azure-webjobs-sdk/blob/dev/src/Microsoft.Azure.WebJobs.Host/Queues/QueueProcessor.cs#L145
we wait new version with this fix
Custom solution
To solve this create your own CustomProcessor and in CopyMessageToPoisonQueueAsync function create new CloudMessage from original to pass in poison queue, see example below.
var config = new JobHostConfiguration
config.Queues.QueueProcessorFactory = new CustomQueueProcessorFactory();
public QueueProcessor Create(QueueProcessorFactoryContext context)
{
// demonstrates how the Queue.ServiceClient options can be configured
context.Queue.ServiceClient.DefaultRequestOptions.ServerTimeout = TimeSpan.FromSeconds(30);
// demonstrates how queue options can be customized
context.Queue.EncodeMessage = true;
// return the custom queue processor
return new CustomQueueProcessor(context);
}
/// <summary>
/// Custom QueueProcessor demonstrating some of the virtuals that can be overridden
/// to customize queue processing.
/// </summary>
private class CustomQueueProcessor : QueueProcessor
{
private QueueProcessorFactoryContext _context;
public CustomQueueProcessor(QueueProcessorFactoryContext context)
: base(context)
{
_context = context;
}
public override async Task CompleteProcessingMessageAsync(CloudQueueMessage message, FunctionResult result, CancellationToken cancellationToken)
{
await base.CompleteProcessingMessageAsync(message, result, cancellationToken);
}
protected override async Task CopyMessageToPoisonQueueAsync(CloudQueueMessage message, CloudQueue poisonQueue, CancellationToken cancellationToken)
{
var msg = new CloudQueueMessage(message.AsString);
await base.CopyMessageToPoisonQueueAsync(msg, poisonQueue, cancellationToken);
}
protected override void OnMessageAddedToPoisonQueue(PoisonMessageEventArgs e)
{
base.OnMessageAddedToPoisonQueue(e);
}
}
For anyone out there still having this issue. This should be fixed since 2.1.0-beta1-10851. The downside is that there is currently no stable released version of 2.1.0 yet.