According to this WebJobs documentation page, for POCO output queue messages, "A queue message is always created, even if the object is null."
In my scenario, I only want to conditionally output queue messages from my WebJob. Currently I am getting a ton of null messages to my downstream WebJob using the "out" queue:
[Queue("myoutqueue")] out myPOCO outputQueueMessage
Is the only way to do this to not use the WebJobs Queue attribute and to queue the message myself using the client library?
You can conditionally enqueue an output message by using the ICollector<T> binding. For example:
[Queue("myoutqueue")] ICollector<MyPoco> outMessages
Then, only messages added to the collector via outMessages.Add(message) will be sent (one or more). More on ICollector<T> and other Queue bindings can be found here.
Related
I've been recently having problems with my Service Bus queue. Random messages (one can pass and the other not) are placed on the deadletter queue with the error message saying:
"DeadLetterReason": "Moved because of Unable to get Message content There was an error deserializing the object of type System.String. The input source is not correctly formatted."
"DeadLetterErrorDescription": "Des"
This happens even before my consumer has the chance to receive the message from the queue.
The weird part is that when I requeue the message through Service Bus Explorer it passes and is successfully received and handled by my consumer.
I am using the same version of Service Bus either for sending and receiving the messages:
Azure.Messaging.ServiceBus, version: 7.2.1
My message is being sent like this:
await using var client = new ServiceBusClient(connString);
var sender = client.CreateSender(endpointName);
var message = new ServiceBusMessage(serializedMessage);
await sender.SendMessageAsync(message).ConfigureAwait(true);
So the solution I have for now for the described issue is that I implemented a retry policy for the messages that land on the dead-letter queue. The message is cloned from the DLQ and added again to the ServiceBus queue and for the second time there is no problems and the message completes successfully. I suppose that this happens because of some weird performance issues I might have in the Azure infrastructure. But this approach bought me some time to investigate further.
I have dot net code which read loan-numbers from Azure service queue and calls my API for each loan-number.
This is my code which calls the api
private async Task ProcessMessagesAsync(Message message, CancellationToken token)
{
try
{
string loanNumber = Encoding.UTF8.GetString(message.Body);
_logger.LogInformation($"Received message: SequenceNumber:{message.SystemProperties.SequenceNumber} LoanNumber:{loanNumber}");
//API CALL HERE
await _apiClient.getResult(loanNumber);
await _queueClient.CompleteAsync(message.SystemProperties.LockToken);
}
catch (Exception ex)
{
//sending failed messages to Dead Letter queue
await _queueClient.AbandonAsync(message.SystemProperties.LockToken);
}
}
Failed loan-numbers are successfully sent to Dead Letter queue. When the server is down or bad request from API response.
I want to call the api after certain duration on the loan-numbers which are in Dead Letter Queue.
Is there any way to convert the messages in Dead Letter queue to active messages after some interval??
I am new to azure. Please help me to resolve the issue.
Thanks in advance.
Is there any way to convert the messages in Dead Letter queue to
active messages after some interval??
Automatically, no. However what you could do is read the messages from dead letter queue and then post them as new message in your main queue.
As far as automating the whole process, one possible solution would be to run a timer triggered Azure Function that reads messages from a dead letter queue periodically and post them in your main queue.
#akhil, it's worth noting that these messages will automatically be re-queued on the main queue until their DeliveryCount exceeds the MaxDeliveryCount of your main queue. The default value for MaxDeliveryCount is 10 so any failed requests to the API be retried ten times by this handler before being moved to the DLQ.
If you wanted to be a bit 'smarter' about this you could delay the retries using the ScheduledEnqueueTimeUtc property on the message in your catch block.
As #Gaurav Mantri has said, the framework offers nothing to process dead-letter messages 'for free'; you'll have to code a handler yourself as you would for a normal queue.
No, there is no way to convert the DLQ message to active message.
you need to process the DLQ and fix the reason as to why it ended in DLQ and then re transmit it back to your queue.
A sample from Github can be useful here.
https://github.com/Azure/azure-service-bus/tree/master/samples/DotNet/Microsoft.ServiceBus.Messaging/DeadletterQueue
(This sample shows how to move messages to the Dead-letter queue, how to retrieve messages from it, and resubmit corrected message back into the main queue.)
This could be an old answer but we can use ServiceBusExplorer from Microsoft. It can pull the messages from dead letter queue and then those can be requeued using
Let me know if you have any further questions.
C# .NetCore 2.2 -
Azure Service Bus 3.4.0
I have 3 queues in Azure Service Bus with same properties. While sending messages to these queues, the messages in one of the queues always get delivered to Dead letter queues, while other 2 queues receive active messages.
I have tried playing with the properties - increase TTL, maximum delivery count etc. The properties of all 3 queues are same, the only difference is the name of the queues.
I have used this tutorial - https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-get-started-with-queues
queue properties image
static async Task SendMessagesAsync(int numberOfMessagesToSend)
{
try
{
for (var i = 0; i < numberOfMessagesToSend; i++)
{
// Create a new message to send to the queue.
string messageBody = $"Message {i}";
var message = new Message(Encoding.UTF8.GetBytes(messageBody));
Console.WriteLine($"Sending message: {messageBody}");
// Send the message to the queue.
await queueClient.SendAsync(message);
}
}
catch (Exception exception)
{
Console.WriteLine($"{DateTime.Now} :: Exception: {exception.Message}");
}
}
How do I prevent messages from going to Dead Letter Queue? Why does it happen with only 1 queue, not the other 2?
When messages are dead-lettered, there is a reason user property gets added. Check that property to see the reason and troubleshoot accordingly. Specifically, check for DeadLetterReason and DeadLetterErrorDescription custom properties.
The common reasons for a message to be dead-lettered are
Maximum transfer hop count exceeded
Session Id Is Null
TTLExpiredException
HeaderSizeExceeded
The messages might also have got dead-lettered due to some errors while receiving the message from the Queue. As Sean Feldman mentioned, looking into the DeadLetterReason and DeadLetterDescription property will help you diagnose the error reason clearly.
Also try to increase or set the time to live of the message sent if the DeadLetterReason is TTLExpiredException. Because if you have set the time to live of the message to a lower value then it will override the time to live property of the Queue.
Check whether the Queue whether the queue where the messages are getting dead-lettered is a Session enabled queue and the message sent has the Session Id value set.
Without seeing your app / messages it's hard to help. But probably there's an error with the application that is trying to consume the message. As it could not be completed, the message goes to the dead letter queue.
Log the messages from this particular queue and see if there's any missing required properties. Sometimes you're trying to deserialize to an uncompatible type.
The purpose of the dead-letter queue is to hold messages that cannot
be delivered to any receiver, or messages that could not be processed.
I have two applications App1 Console app and App2 Azure func App
App1 is for enroll new customer
App2 is for activate account for new customer
We need to ensure that each message that App1 sends is stored in a queue for 10 minutes before App2 uses the message
From App1 I'm taking inputs like customer-id , customer-name and send this message to queue.
using simple code -
// Create a new message to send to the queue
string messageBody = $"Message {i}";
var message = new Message(Encoding.UTF8.GetBytes(messageBody));
// Write the body of the message to the console
Console.WriteLine($"Sending message: {messageBody}");
// Send the message to the queue
await queueClient.SendAsync(message);
do i need to add any setting for 10 minutes scenario?
From App2 I need to read this message but need to ensure for 10 minutes before App2 uses the message.
How to achieve this to ensure message there for 10 minutes in queue.
When i search on stackoverflow i found this
If a message is added with a 5 minute timeout it will only be processed after that.
it means while sending message i need to set something ?
It looks like you're looking for Scheduled messages.
You can submit messages to a queue or topic for delayed processing; for example, to schedule a job to become available for processing by a system at a certain time. This capability realizes a reliable distributed time-based scheduler.
Scheduled messages do not materialize in the queue until the defined enqueue time. Before that time, scheduled messages can be canceled. Cancellation deletes the message.
You can schedule messages either by setting the ScheduledEnqueueTimeUtc property when sending a message through the regular send path, or explicitly with the ScheduleMessageAsync API. The latter immediately returns the scheduled message's SequenceNumber, which you can later use to cancel the scheduled message if needed. Scheduled messages and their sequence numbers can also be discovered using message browsing.
I understand that MS Azure Queue service document http://msdn.microsoft.com/en-us/library/windowsazure/dd179363.aspx says first out (FIFO) behavior is not guaranteed.
However, our application is such that ALL the messages have to be read and processed in FIFO order. Could anyone please suggest how to achieve a guaranteed FIFO using Azure Queue Service?
Thank you.
The docs say for Azure Storage queues that:
Messages in Storage queues are typically first-in-first-out, but sometimes they can be out of order; for example, when a message's
visibility timeout duration expires (for example, as a result of a
client application crashing during processing). When the visibility
timeout expires, the message becomes visible again on the queue for
another worker to dequeue it. At that point, the newly visible message
might be placed in the queue (to be dequeued again) after a message
that was originally enqueued after it.
Maybe that is good enough for you? Else use Service bus.
The latest Service Bus release offers reliable messaging queuing: Queues, topics and subscriptions
Adding to #RichBower answer... check out this... Azure Storage Queues vs. Azure Service Bus Queues
MSDN (link retired)
http://msdn.microsoft.com/en-us/library/windowsazure/hh767287.aspx
learn.microsoft.com
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted
Unfortunately, many answers misleads to Service Bus Queues but I assume the question is about Storage Queues from the tags mentioned. In Azure Storage Queues, FIFO is not guranteed, whereas in Service Bus, FIFO message ordering is guaranteed and that too, only with the use of a concept called Sessions.
A simple scenario could be, if any consumer receives a message from the queue, it is not visible to you when you are the second receiver. So you assume the second message you received is actually the first message (Where FIFO failed :P)
Consider using Service Bus if this is not your requirement.
I don't know how fast do you want to process the messages, but if you need to have a real FIFO, don't allow Azure's queue to get more than one message at a time.
Use this at your "program.cs" at the top of the function.
static void Main()
{
var config = new JobHostConfiguration();
if (config.IsDevelopment)
{
config.UseDevelopmentSettings();
}
config.Queues.BatchSize = 1; //Number of messages to dequeue at the same time.
config.Queues.MaxPollingInterval = TimeSpan.FromMilliseconds(100); //Pooling request to the queue.
JobHost host = new JobHost(config);
....your initial information...
// The following code ensures that the WebJob will be running continuously
host.RunAndBlock();
This will get one message at a time with a wait period of 100 miliseconds.
This is working perfectly with a logger webjob to write to files the traze information.
As mentioned here https://www.jayway.com/2013/12/20/message-ordering-on-windows-azure-service-bus-queues/ ordering is not guaranteed also in service bus, except of using recieve and delete mode which is risky
You just need to follow below steps to ensure Message ordering.:
1) Create a Queue with session enabled=false.
2) While saving message in the queue, provide the session id like below:-
var message = new BrokeredMessage(item);
message.SessionId = "LB";
Console.WriteLine("Response from Central Scoring System : " + item);
client.Send(message);
3) While creating receiver for reviving message:-
queueClient.OnMessage(s =>
{
var body = s.GetBody<string>();
var messageId = s.MessageId;
Console.WriteLine("Message Body:" + body);
Console.WriteLine("Message Id:" + messageId);
});
4) While having the same session id, it would automatically ensure order and give the ordered message.
Thanks!!