Our code is just a copy paste from some online tutorial in getting the messages from an azure storage queue.
public int? GetQueueMessageCount(CloudQueue queue, TextWriter textWriter)
{
int? messageCount;
try
{
queue.FetchAttributes();
// Retrieve the cached approximate message count.
messageCount = queue.ApproximateMessageCount;
}
catch (Exception exception)
{
LogHelper.LogInfo(logger, textWriter, $"GetQueueMessageCount failed for {queue.Name}." + exception);
throw;
}
return messageCount;
}
However, we found that randomly some messages may get stuck in the queue and our queue trigger never got fired.
public static void ProcessUnitsForCacheItem(
[QueueTrigger(QueueClient.RefreshUnitsQueue)] string projectUnitsMessage, TextWriter textWriter)
When I open my queue with storage explorer, I can see the explorer will not show any message, instead just display a status text "displaying 0 of 199 messages". So storage explorer must somehow know that these messages are not right (expired or something).
Is there some status I can retrieve to see the status of the message or anyone know how storage explorer decide to show a message or not?
Storage explorer shows info exactly what it retrieves from Storage account/emulator.
displaying 0 of 199 messages means the messages are invisible for now because they have been dequeued and being processed, it's a feature of queue message and handled by Storage service automatically once your queue trigger gets messages from a queue. See Storage queue doc.
Typically, when a consumer retrieves a message via Get Messages, that message is usually reserved for deletion until the visibilitytimeout interval expires, but this behavior is not guaranteed. After the visibilitytimeout interval expires, the message again becomes visible to other consumers.
As for the problem
get stuck in the queue and our queue trigger never got fired
If I understand correctly, your code from some tutorial is a custom queuetrigger, which may have no guarantee on the behavior. Have a look at Azure Function Queuetrigger example.
Related
.NET Core 2.2, WebJobs SDK 3.0
I have a webjob that takes the messages from a queue. A standard QueueTrigger like this
public void ProcessQueueMessage(
[QueueTrigger("%WebJobs:WorkerQueueName%")] CloudQueueMessage queueMessage,
ILogger log)
At the end of the process I write the message to another queue (archive).
The function finishes successfully but the message is kept in the source queue
In Storage Explorer I see this (in this example I had 3 messages pending)
and the message is dequeued once again after 10 minutes.
How can I make it so the message is dequeued when my function is successful?
Btw my queue config is
BatchSize 1
MaxDequeueCount 2
MaxPollingInterval 00:00:04
VisibilityTimeout 02:00:00
The SDK should already handle that.
The message will be leased (or become invisible) for 30 seconds by default. If the job takes longer than that, then the lease will be renewed. The message will not become available to another instance of the function unless the host crashes or the function throws an exception. When the function completes successfully, the message is deleted by the SDK.
When a message is retrieved from the queue, the response includes the message and a pop receipt value, which is required to delete the message. The message is not automatically deleted from the queue, but after it has been retrieved, it is not visible to other clients for the time interval specified by the visibilitytimeout parameter. The client that retrieves the message is expected to delete the message after it has been processed, and before the time specified by the TimeNextVisible element of the response, which is calculated based on the value of the visibilitytimeout parameter. The value of visibilitytimeout is added to the time at which the message is retrieved to determine the value of TimeNextVisible.
So you shouldn't need to write any special code for deleting message from queue.
For more details you could refer to this article and this one.
It turns out that I was using the queueMessage object I got as a parameter to directly put it in another queue which probably confused the SDK.
public void ProcessQueueMessage(
[QueueTrigger("%WebJobs:WorkerQueueName%")] CloudQueueMessage queueMessage,
ILogger log)
So I changed that and I create a new CloudQueueMessage object before I put it in another queue.
var newMessage = new CloudQueueMessage(queueMessage.AsString);
Now the message is properly deleted from the queue when my function returns.
C# .NetCore 2.2 -
Azure Service Bus 3.4.0
I have 3 queues in Azure Service Bus with same properties. While sending messages to these queues, the messages in one of the queues always get delivered to Dead letter queues, while other 2 queues receive active messages.
I have tried playing with the properties - increase TTL, maximum delivery count etc. The properties of all 3 queues are same, the only difference is the name of the queues.
I have used this tutorial - https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-get-started-with-queues
queue properties image
static async Task SendMessagesAsync(int numberOfMessagesToSend)
{
try
{
for (var i = 0; i < numberOfMessagesToSend; i++)
{
// Create a new message to send to the queue.
string messageBody = $"Message {i}";
var message = new Message(Encoding.UTF8.GetBytes(messageBody));
Console.WriteLine($"Sending message: {messageBody}");
// Send the message to the queue.
await queueClient.SendAsync(message);
}
}
catch (Exception exception)
{
Console.WriteLine($"{DateTime.Now} :: Exception: {exception.Message}");
}
}
How do I prevent messages from going to Dead Letter Queue? Why does it happen with only 1 queue, not the other 2?
When messages are dead-lettered, there is a reason user property gets added. Check that property to see the reason and troubleshoot accordingly. Specifically, check for DeadLetterReason and DeadLetterErrorDescription custom properties.
The common reasons for a message to be dead-lettered are
Maximum transfer hop count exceeded
Session Id Is Null
TTLExpiredException
HeaderSizeExceeded
The messages might also have got dead-lettered due to some errors while receiving the message from the Queue. As Sean Feldman mentioned, looking into the DeadLetterReason and DeadLetterDescription property will help you diagnose the error reason clearly.
Also try to increase or set the time to live of the message sent if the DeadLetterReason is TTLExpiredException. Because if you have set the time to live of the message to a lower value then it will override the time to live property of the Queue.
Check whether the Queue whether the queue where the messages are getting dead-lettered is a Session enabled queue and the message sent has the Session Id value set.
Without seeing your app / messages it's hard to help. But probably there's an error with the application that is trying to consume the message. As it could not be completed, the message goes to the dead letter queue.
Log the messages from this particular queue and see if there's any missing required properties. Sometimes you're trying to deserialize to an uncompatible type.
The purpose of the dead-letter queue is to hold messages that cannot
be delivered to any receiver, or messages that could not be processed.
I have a queue receiver, which reads messages from the queue and process the message (do some processing and inserts some data to the azure table or retrieves the data).
What I observed was that any exception that my processing method (SendResponseAsync()) throws results in retry i.e. redelivery of the message to the default 10 times.
Can this behavior be customized i.e. I only retry for certain exception and ignore for other. Like if there is some network issue, then it makes sense to retry but if it is BadArgumentException(poisson message), then I may not want to retry.
Since retry is taken care by ServiceBus client library, can we customize this behavior ?
This is the code at the receiver end
public MessagingServer(QueueConfiguration config)
{
this.requestQueueClient = QueueClient.CreateFromConnectionString(config.ConnectionString, config.QueueName);
this.requestQueueClient.OnMessageAsync(this.DispatchReplyAsync);
}
private async Task DispatchReplyAsync(BrokeredMessage message)
{
await this.SendResponseAsync(message);
}
I'm using Azure WebJobs to poll a queue and then process the message.
Part of the message processing includes a hit to 3rd party HTTP endpoint. (e.g. a Weather api or some Stock market api).
Now, if the hit to the api fails (network error, 500 error, whatever) I try/catch this in my code, log whatever and then ... what??
If I continue .. then I assume the message will be deleted by the WebJobs SDK.
How can I:
1) Say to the SDK - please don't delete this message (so it will be retried automatically at the next queue poll and when the message is visible again).
2) Set the invisibility time value, when the SDK pops a message off the queue for processing.
Thanks!
Now, if the hit to the api fails (network error, 500 error, whatever) I try/catch this in my code, log whatever and then ... what??
The Webjobs SDK behaves like this: If your method throws an uncaught exception, the message is returned to the Queue with its dequeueCount property +1. Else, if all is well, the message is considered successfully processed and is deleted from the Queue - i.e. queue.DeleteMessage(retrievedMessage);
So don't gracefully catch the HTTP 500, throw an exception so the SDK gets the hint.
If I continue .. then I assume the message will be deleted by the WebJobs SDK.
From https://github.com/Azure/azure-content/blob/master/articles/app-service-web/websites-dotnet-webjobs-sdk-get-started.md#contosoadswebjob---functionscs---generatethumbnail-method:
If the method fails before completing, the queue message is not deleted; after a 10-minute lease expires, the message is released to be picked up again and processed. This sequence won't be repeated indefinitely if a message always causes an exception. After 5 unsuccessful attempts to process a message, the message is moved to a queue named {queuename}-poison. The maximum number of attempts is configurable.
If you really dislike the hardcoded 10-minute visibility timeout (the time the message stays hidden from consumers), you can change it. See this answer by #mathewc:
From https://stackoverflow.com/a/34093943/4148708:
In the latest v1.1.0 release, you can now control the visibility timeout by registering your own custom QueueProcessor instances via JobHostConfiguration.Queues.QueueProcessorFactory. This allows you to control advanced message processing behavior globally or per queue/function.
https://github.com/Azure/azure-webjobs-sdk-samples/blob/master/BasicSamples/MiscOperations/CustomQueueProcessorFactory.cs#L63
protected override async Task ReleaseMessageAsync(CloudQueueMessage message, FunctionResult result, TimeSpan visibilityTimeout, CancellationToken cancellationToken)
{
// demonstrates how visibility timeout for failed messages can be customized
// the logic here could implement exponential backoff, etc.
visibilityTimeout = TimeSpan.FromSeconds(message.DequeueCount);
await base.ReleaseMessageAsync(message, result, visibilityTimeout, cancellationToken);
}
I need some help solving a problem involving the Task Parallel Library with Azure instances. Below is code for my Worker Role.
Whenever I upload multiple files, a request is inserted into the queue and the worker process continously process queries Queues and gets the message. Once a message is retrieved, I do some long runnning process. I used task schedulder so that mutliple request are served by multiple task instance on multiple instances.
Now the uestion is if one instance take a message from a queue and assigns the message to a task and it process, now i see another instance also retrieves the same message from Queue and process it. Because of that my tasks are executed multiple times.
Please help me on this problem. My requirement is only one Azure instance of one Ccre handles one task operation not by mutliple by task.
public override void Run()
{
//Step1 : Get the message from Queue
//Step 2:
Task<string>.Factory.StartNew(() =>
{
//Message delete from Queue
PopulateBlobtoTable(uri, localStoragePath);
}
catch (Exception ex)
{
Trace.WriteLine(ex.Message);
throw;
}
finally
{
}
}
return "Finished!";
})
catch (AggregateException ae)
{
foreach (var exception in ae.InnerExceptions)
{
Trace.WriteLine(exception.Message);
}
}
I'm assuming you are using Windows Azure Storage queues, which have a default invisibility timeout of 90 seconds, when using the storage client APIs. If your message is not completely processed and explicitly deleted within that time period, it will reappear on the queue.
While you can increase this invisibility timeout to up to seven days when you add the message to the queue, you should be using operations that are idempotent, meaning it doesn't matter if the message is processed multiple times. It's your job to ensure idempotence, perhaps by recording a unique id (in table storage, SQL database, etc.) associated with each message and ignoring the message if you see it a second time and you find it's already been marked complete.
You might also look at Windows Azure Queues and Windows Azure Service Bus Queues - Compared and Constrasted. You'll note Service Bus queues have some additional constructs you can use to guarantee at-most-once (and at-least-once) delivery.
Now the uestion is if one instance take a message from a queue and assigns the message to a task and it process, now i see another instance also retrieves the same message from Queue and process it. Because of that my tasks are executed multiple times.
Are you getting the messages via "GET" semantics? If that's the case, then what's the visibility timeout you have set for your messages. When you "GET" a message, it should become invisible to other callers (read "instances" in your case) for a particular period of time which you can specify using visibility timeout period. Check out the documentation here for this: http://msdn.microsoft.com/en-us/library/windowsazure/ee758454.aspx