.NET Core 2.2, WebJobs SDK 3.0
I have a webjob that takes the messages from a queue. A standard QueueTrigger like this
public void ProcessQueueMessage(
[QueueTrigger("%WebJobs:WorkerQueueName%")] CloudQueueMessage queueMessage,
ILogger log)
At the end of the process I write the message to another queue (archive).
The function finishes successfully but the message is kept in the source queue
In Storage Explorer I see this (in this example I had 3 messages pending)
and the message is dequeued once again after 10 minutes.
How can I make it so the message is dequeued when my function is successful?
Btw my queue config is
BatchSize 1
MaxDequeueCount 2
MaxPollingInterval 00:00:04
VisibilityTimeout 02:00:00
The SDK should already handle that.
The message will be leased (or become invisible) for 30 seconds by default. If the job takes longer than that, then the lease will be renewed. The message will not become available to another instance of the function unless the host crashes or the function throws an exception. When the function completes successfully, the message is deleted by the SDK.
When a message is retrieved from the queue, the response includes the message and a pop receipt value, which is required to delete the message. The message is not automatically deleted from the queue, but after it has been retrieved, it is not visible to other clients for the time interval specified by the visibilitytimeout parameter. The client that retrieves the message is expected to delete the message after it has been processed, and before the time specified by the TimeNextVisible element of the response, which is calculated based on the value of the visibilitytimeout parameter. The value of visibilitytimeout is added to the time at which the message is retrieved to determine the value of TimeNextVisible.
So you shouldn't need to write any special code for deleting message from queue.
For more details you could refer to this article and this one.
It turns out that I was using the queueMessage object I got as a parameter to directly put it in another queue which probably confused the SDK.
public void ProcessQueueMessage(
[QueueTrigger("%WebJobs:WorkerQueueName%")] CloudQueueMessage queueMessage,
ILogger log)
So I changed that and I create a new CloudQueueMessage object before I put it in another queue.
var newMessage = new CloudQueueMessage(queueMessage.AsString);
Now the message is properly deleted from the queue when my function returns.
Related
I've set up a function to hit an API endpoint for a newly created entity that isn't immediately available. If the endpoint returns a status of "pending", the function throws an error. If the endpoint returns a status of "active", the function then deletes the SQS message and triggers several other microservices to do their things using SNS. The SQS queue that triggers the function has a visibility timeout of 2 minutes, and the function itself has a 1 minute timeout.
What I'm expecting to happen is that if the endpoint returns a "pending" status, and the function throws an error, then after the 2 minute visibility timeout, the message would trigger the function again. This should happen every 2 minutes until the api call returns an "active" status and the message is deleted, or until the message retention period is surpassed (currently 1 hour). This seemed like a nice serverless way to poll my newly created entity to check if it was ready for other post-processing.
What's actually happening after adding a message to the SQS queue is that the CloudWatch logs are showing that the function is throwing an error like I'd expect, but the function is only being triggered one time. I can't tell if the message is just not not visible for some reason, or if it somehow was deleted. I don't know. I'm am new to using SQS for a Lambda trigger, am I thinking about this wrong?
A few possible causes here:
your Lambda function handler did not actually throw an exception to the Lambda runtime environment, so Lambda thought the function had successfully processed the message and the Lambda service then deleted the message from the queue (so that it would not get processed again)
your SQS queue has a configured DLQ with maximum receives set to 1, so the message is delivered once, the Lambda function fails, and the message is subsequently moved to the DLQ
the SQS message was re-delivered to the Lambda function and was logged but the logs were made to an earlier log stream (because this invocation was warm) and so it wasn't obvious that the Lambda function had actually been invoked multiple times with the same failed message
To verify this all works normally, I set up a simple test with both FIFO and non-FIFO queues and configured the queues to trigger a Lambda function that simply logged the SQS message and then threw an exception. As expected, I saw the same SQS message delivered to the Lambda function every 2 minutes (which is the queue's message visibility timeout). That continued until it hit the max receive count on the SQS redrive policy (defaults to 10 attempts) at which point the failed message was correctly moved to the associated DLQ.
In Azure WebJobs, in the OnMessageOptions class, I'm calling the QueueClient.Complete(Guid) method by setting the AutoComplete flag to true and messages seem to dequeue just fine when running the ProcessQueue function. Active messages count goes down by 1 after successful processing of each message. However, when I want to requeue a message (because it cannot be processed currently) back to the queue that triggers the service bus function, as a new brokered message after a minute, using BrokeredMessage.ScheduledEnqueueTimeUtc, it seems like it isn't working. Scheduled messages count seems to go up initially. I go back to the queue after a few hours and see active messages in the thousands. The copies are of the same message. What is happening? I'd expect the message to be taken off the queue because of QueueClient.Complete(Guid) and the new scheduled message to be its replacement.
Some detail:
To send the message I do the following:
var queueclient = QueueClient.CreateFromConnectionString(connectionString, queueName);
queueclient.Send(message);
queueclient.close();
Inside the WebJob I created a ServiceBusConfiguration object which requires a onMessageOptions object where I set the AutoComplete=true. I pass the ServiceBusConfiguration object to the JobHostConfiguration.UserServiceBus
method.
Inside the WebJob service bus queue triggered function I again do the following to requeue, by first creating a new instance of the brokered message again.
//if not available yet for processing please requeue...
var queueclient = QueueClient.CreateFromConnectionString(connectionString, queueName);
queueclient.Send(message);
queueclient.close();
I don't do the following/use callbacks which is may be why it isn't working?
var options = new OnMessageOptions();
options.AutoComplete = false; // to call complete ourselves
Callback to handle received messages
client.OnMessage(m =>
{
var clone = m.Clone();
clone.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddSeconds(60);
client.Send(clone);
m.Complete();
}, options);
when I want to requeue a message (because it cannot be processed currently) back to the queue that triggers the service bus function, as a new brokered message after a minute, using BrokeredMessage.ScheduledEnqueueTimeUtc, it seems like it isn't working
If you fail to process your message, do not re-queue it. Instead, abandon (with a reason) and it will be picked up again.
BrokeredMessage.ScheduledEnqueueTimeUtc is intended to be used for messages added to the queue. When you receive a message, you can complete, dead-letter, defer, or abandon. If you abandon a message, it will be retried, but you can't control when that will happen. If you have no other messages in the queue, it will be retried almost immediately.
Note: when you see a behaviour that you suspect is not right, having a simple repro to share would be very helpful.
Our code is just a copy paste from some online tutorial in getting the messages from an azure storage queue.
public int? GetQueueMessageCount(CloudQueue queue, TextWriter textWriter)
{
int? messageCount;
try
{
queue.FetchAttributes();
// Retrieve the cached approximate message count.
messageCount = queue.ApproximateMessageCount;
}
catch (Exception exception)
{
LogHelper.LogInfo(logger, textWriter, $"GetQueueMessageCount failed for {queue.Name}." + exception);
throw;
}
return messageCount;
}
However, we found that randomly some messages may get stuck in the queue and our queue trigger never got fired.
public static void ProcessUnitsForCacheItem(
[QueueTrigger(QueueClient.RefreshUnitsQueue)] string projectUnitsMessage, TextWriter textWriter)
When I open my queue with storage explorer, I can see the explorer will not show any message, instead just display a status text "displaying 0 of 199 messages". So storage explorer must somehow know that these messages are not right (expired or something).
Is there some status I can retrieve to see the status of the message or anyone know how storage explorer decide to show a message or not?
Storage explorer shows info exactly what it retrieves from Storage account/emulator.
displaying 0 of 199 messages means the messages are invisible for now because they have been dequeued and being processed, it's a feature of queue message and handled by Storage service automatically once your queue trigger gets messages from a queue. See Storage queue doc.
Typically, when a consumer retrieves a message via Get Messages, that message is usually reserved for deletion until the visibilitytimeout interval expires, but this behavior is not guaranteed. After the visibilitytimeout interval expires, the message again becomes visible to other consumers.
As for the problem
get stuck in the queue and our queue trigger never got fired
If I understand correctly, your code from some tutorial is a custom queuetrigger, which may have no guarantee on the behavior. Have a look at Azure Function Queuetrigger example.
I am encountering one major road block issue when trying to use ServiceBusTrigger in azureFunction. I am trying to abandon, or deadletter, a service bus message in V2 ServiceBusTrigger, How can I do so? I've tried the following solution, but I didn't get anywhere.
Here is the codeSample I used:
public async static Task Run(Message myQueueItem, TraceWriter log, ExecutionContext context)
{
log.Info($"C# ServiceBus queue trigger function processed message delivery count: {myQueueItem.SystemProperties.DeliveryCount}");
QueueClient queueClient = new QueueClient("[connectionstring]","[queueName]");
////await queueClient.DeadLetterAsync(myQueueItem.SystemProperties.LockToken);
await queueClient.AbandonAsync(myQueueItem.SystemProperties.LockToken);
}
Solution 1: I tried to substitute Message myQueueItem for BrokeredMessage like in V1, I then can call myQueueItem.Abandon, or deadletter, on the message lever. However It came back with exception:
Microsoft.Azure.WebJobs.Host: Exception binding parameter 'myQueueItem'. System.Private.DataContractSerialization: There was an error deserializing the object of type Microsoft.ServiceBus.Messaging.BrokeredMessage. The input source is not correctly formatted. System.Private.DataContractSerialization: The input source is not correctly formatted."
At least I can go one step further. to
solution 2. Solution 2: is to use:
QueueClient queueClient = new QueueClient("[connectionstring]","[queueName]");
////await queueClient.DeadLetterAsync(myQueueItem.SystemProperties.LockToken);
await queueClient.AbandonAsync(myQueueItem.SystemProperties.LockToken);
I can use the lock provided in the Message Object, however, when I try to send it with queueClient, It said the message gone from the queue. or no longer available.
Can anybody let me know if i am on the right track? If I am not, please kindly guide me in the right track.
Service Bus messages are automatically completed or abandoned by Azure Functions runtime based on the success/failure of the function call, docs:
The Functions runtime receives a message in PeekLock mode. It calls Complete on the message if the function finishes successfully, or calls Abandon if the function fails.
So, the suggested way to Abandon your message is to throw an exception from function invocation.
I'm using Azure WebJobs to poll a queue and then process the message.
Part of the message processing includes a hit to 3rd party HTTP endpoint. (e.g. a Weather api or some Stock market api).
Now, if the hit to the api fails (network error, 500 error, whatever) I try/catch this in my code, log whatever and then ... what??
If I continue .. then I assume the message will be deleted by the WebJobs SDK.
How can I:
1) Say to the SDK - please don't delete this message (so it will be retried automatically at the next queue poll and when the message is visible again).
2) Set the invisibility time value, when the SDK pops a message off the queue for processing.
Thanks!
Now, if the hit to the api fails (network error, 500 error, whatever) I try/catch this in my code, log whatever and then ... what??
The Webjobs SDK behaves like this: If your method throws an uncaught exception, the message is returned to the Queue with its dequeueCount property +1. Else, if all is well, the message is considered successfully processed and is deleted from the Queue - i.e. queue.DeleteMessage(retrievedMessage);
So don't gracefully catch the HTTP 500, throw an exception so the SDK gets the hint.
If I continue .. then I assume the message will be deleted by the WebJobs SDK.
From https://github.com/Azure/azure-content/blob/master/articles/app-service-web/websites-dotnet-webjobs-sdk-get-started.md#contosoadswebjob---functionscs---generatethumbnail-method:
If the method fails before completing, the queue message is not deleted; after a 10-minute lease expires, the message is released to be picked up again and processed. This sequence won't be repeated indefinitely if a message always causes an exception. After 5 unsuccessful attempts to process a message, the message is moved to a queue named {queuename}-poison. The maximum number of attempts is configurable.
If you really dislike the hardcoded 10-minute visibility timeout (the time the message stays hidden from consumers), you can change it. See this answer by #mathewc:
From https://stackoverflow.com/a/34093943/4148708:
In the latest v1.1.0 release, you can now control the visibility timeout by registering your own custom QueueProcessor instances via JobHostConfiguration.Queues.QueueProcessorFactory. This allows you to control advanced message processing behavior globally or per queue/function.
https://github.com/Azure/azure-webjobs-sdk-samples/blob/master/BasicSamples/MiscOperations/CustomQueueProcessorFactory.cs#L63
protected override async Task ReleaseMessageAsync(CloudQueueMessage message, FunctionResult result, TimeSpan visibilityTimeout, CancellationToken cancellationToken)
{
// demonstrates how visibility timeout for failed messages can be customized
// the logic here could implement exponential backoff, etc.
visibilityTimeout = TimeSpan.FromSeconds(message.DequeueCount);
await base.ReleaseMessageAsync(message, result, visibilityTimeout, cancellationToken);
}