When publishing messages to a service bus topic, if I loop over 3 messages:
{ A, B, C }
And await the SendAsync() each time, I'd expect them to be published to the topic in the order:
{ A, B, C }
public async Task PublishMessage(string topic, string json, string sessionId)
{
var topicClient = new TopicClient(_connectionString, topic);
var busMessage = new Message(Encoding.UTF8.GetBytes(json));
busMessage.SessionId = sessionId;
await topicClient.SendAsync(busMessage);
}
A number of employees have suggested this isn't guaranteed to be the case, and that in certain scenarios (i.e large messages), this publishing order isn't guaranteed. I've never encountered a scenario of this myself, does ASB not guarantee publish ordering even when the sending of messages is awaited like the above?
This article https://devblogs.microsoft.com/premier-developer/ordering-messages-in-azure-service-bus/ uses this quote:
"While Azure Service Bus allows for a FIFO approach (First-In-First-Out), we cannot guarantee that messages are entered in the order we want them to be processed"
This all seems quite baffling to me, as I'd have assumed SendAsync() would only return a successful result once the message has been added into the topic. Do we really need to write layers of complexity around this to manage it?
Please note this only relates to the publishing of messages, we use SessionIds to handle consumption.
Even if you wait for one message to be sent and then only send the next one, the FIFO is not guaranteed. This is due to too many probable causes. In order to ensure you get guaranteed ordering, you need to use session enabled queues or subscriptions.
Related
I try to get 32 messages per request from Azure Queue.
queue.ApproximateMessageCount;
This code gives me the result of 1509. Telling me the connection is OK and it has records. Also I check in queue it really has 1509 records.
But when I try to retrieve records I don't get any record.
I do the following:
var messages = await queue.GetMessagesAsync(configuration.MessageBatchSize);
if (!messages.Any()) {
return;
}
It always goes in the if and returns.
What is going on here and what am I missing?
Do do that, receiving messages in batch mode, i use this kind of code :
var messages = await queueClient?.ReceiveBatchAsync(Max_Messages);
foreach (var message in messages)
{
await dispatcher.Dispatch(message); // do something with each message
}
But, for receiving messages with ReceiveBatchAsync, the queue have to be configured with the EnableBatchedOperations flag to true.
ApproximateMessageCount property represents the total number of messages available in queue at that particular moment. It does not represent that all messages (max #32 messages in a pull) are ready to be dequeued. You can use this property to infer that how many messages are in queue.
queue.ApproximateMessageCount;
If you could not retrieve the message by, GetMessagesAsync(numberOfMessages), then it says that all messages are not available or invisible for current QueueClient.
var cloudQueueMessages = await cloudQueue.GetMessagesAsync(numberOfMessages);
You could try polling the queue after sometime to see if messages came back to surface.
Note that, be advised of setting adequate visibility timeout for any message being dequeued to avoid indefinite starvation :)
I am continuously receiving messages in peek mode and abandoning them if processing fails (Not the delivery). However, the message immediately becomes available again and is received for processing again. It fails quickly again and after max deliveries it is dead-lettered.
Is there a way to configure the topic/subscription to wait before releasing the message after it is abandoned? Preferably in exponential manner.
Of course I am open for suggestions via the code as well.
There is not a way to set an exponential back-off in the Service Bus configuration. I've experienced the same issue, and have done the following:
Dequeue the message, and marked the message as received.
Perform processing within a try/catch block. If there is an exception, enqueue a new message with a scheduled delivery at a certain point in the future.
We've wrapped our Service Bus Message Queue payload in a class that specifies the number of delivery attempts. We multiply the number of delivery attempts times a constant, then add that number to the current dateTime for scheduled delivery in the future. After the number of delivery attempts that we want to try are exceeded, we explicitly dead letter the message.
Edit 7-17-2020
Consider using Azure Durable Task Framework which has a customizable retry policy built right in.
Another option is using MassTransit which has support for Azure Service Bus.
Take a look at its extensive retry configuration.
Note that MassTransit is effectively doing the retries in memory after the message has been received so you'll need to adjust your topic subscription's MaxDeliveryCount and MaxAutoRenewDuration settings appropriately.
Your configuration might look something like:
var busControl = Bus.Factory.CreateUsingAzureServiceBus(cfg =>
{
var host = cfg.Host(serviceBusConnectionString, hst => { });
cfg.UseMessageRetry(retryConfigurator =>
RetryConfigurationExtensions.Exponential(retryConfigurator, ...);
cfg.SubscriptionEndpoint(
"subscriptionName",
"topicPath",
e =>
{
e.Consumer<SomeConsumer>();
// Let MassTransit do the retries
e.MaxDeliveryCount = 1;
e.MaxAutoRenewDuration = TimeSpan.FromMinutes(10);
});
});
You can use Scheduled Messages for this.
Essentially, when you need to retry a message you just schedule it in the future and Service Bus will add it to the queue again once enough time has passed:
ServiceBusSender queue = ...
int secs = // calculate the delay
var msg = new ServiceBusMessage("My scheduled message body")
{
ApplicationProperties =
{
["RetryCount"] = retryCount,
},
};
logger.LogInformation("Scheduling for " + secs + " secs");
await queue.ScheduleMessageAsync(msg, DateTimeOffset.Now.AddSeconds(secs));
You must add information about retry count in a header or the body. Otherwise you won't know how many times you have tried it and cannot really calculate the future date.
I am looking into this topic as well and I came across the class RetryExponential class from Microsoft.
RetryExponential Class
Namespace: Microsoft.ServiceBus
Assembly: Microsoft.ServiceBus.dll
Represents an implementation of a retry policy. For each time the messaging operation must be retried, the delay between retries grows in a staggered, exponential manner.
public sealed class RetryExponential : Microsoft.ServiceBus.RetryPolicy
Why would someone want to do that? I have to unit-test exception handling mechanism in our application.
I presumed that dead letter queue is literally azure service bus queue, where I could publish messages using QueueClient
string dlQ = #"sb://**.servicebus.windows.net/**/Subscriptions/DefaultSubscription/$DeadLetterQueue";
string connectionString = CloudConfigurationManager.GetSetting("Microsoft.ServiceBus.ConnectionString");
NamespaceManager _namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
QueueDescription qd = _namespaceManager.GetQueue(dataPromotionDLQ);
var queueClient = QueueClient.CreateFromConnectionString(connectionString, "DefaultSubscription/$DeadLetterQueue");
BrokeredMessage brokeredMessage = new BrokeredMessage("Message to PublishToDLQ");
try
{
queueClient.Send(brokeredMessage);
}
catch (Exception)
{
}
But I get MessagingEntityNotFoundException. What could be wrong?
You would never want to publish directly to a dead letter queue. It's where poisoned messages that can't be processed are placed.
There are two ways of placing messages onto the dead letter queue. The service bus itself dead-letters messages that have exceeded the maximum number of delivery attempts. You can also explicitly dead-letter a message that you have received using the DeadLetter() method.
Create your messages with a very short TTL via the BrokeredMessage.TimeToLive property.
The Subscription must have EnableDeadLetteringOnMessageExpiration set to true.
Though late here, adding to the answers of #Mikee and #Ben Morris may help someone. You can make use of #Mike's suggestion of making use of message.DeadLetter() or message.DeadLetterAsync() to dead-letter a message. Another suggestion can be to set very less or 0 second TimeToLive to move the messages to Dead letter.
After you perform any of these and try to view the messages in the Active end queue, you may still find that message is available sometimes (Which you are currently facing). The reason is that the messages that are dead-lettered due to TTLExpiredException, HeaderSizeExceeded or any system defined Errors, or manually Dead-Lettered messages like DeadLetter() methods are cleaned up by an asynchronous "garbage collection" program periodically. This doesn't occur immediately which we expect it to.
When you perform Peek operation, you can still see that the message is in the Active queue. You have to wait for the garbage collector to run or you can perform a Receive operation which forces the garbage collector to run first, thereby moving the messages to dead-letter before retrieval is done.
I'm beginning a project which will involve Azure Queue (not Service Bus).
I'm trying to figure out what is the best practice to find out whether there are messages waiting in the Queue.
AFAIK, there are two methods for that:
Using the ApproximateMessageCount property of the Queue object
Calling GetMessage, and if the returned value is null - there are no messages.
Which one is better performance-wise? Is there any difference?
From a billing POV, I understand there is a transaction cost for both of them, is that correct?
Thanks!
GetMessage is both faster and cheaper. GetMessage is also more correct from a logic perspective since the message count will return both messages that have already been retrieved by another reader as well as messages that have expired without being deleted.
I have also used this code in the past:
var cnnString = "the connection string";
var queueName = "the queue name";
var nsManager = NamespaceManager.CreateFromConnectionString(cnnString);
return nsManager.GetQueue(queueName).MessageCount;
That said - this was from about 4 months ago.
Any reason you need to do this (i.e. are you not just consuming messages off the queue?)
I've got about 50K messages I wish to add to an azure queue.
I'm not sure if the code I have is safe. It feels/smells bad.
Basically, give a collection of POCO's, serialize the POCO to some json, then add that json text to the queue.
public void AddMessage(T content)
{
content.ShouldNotBe(null);
var json = JsonConvert.SerializeObject(content);
var message = new CloudQueueMessage(json);
Queue.AddMessage(message);
}
public void AddMessages(ICollection<T> contents)
{
contents.ShouldNotBe(null);
Parallel.ForEach(contents, AddMessage);
}
Can someone tell me what I should be doing to fix this up -- and most importantly, why?
I feel that the Queue might not be thread safe, in this scenario.
A few things I have observed regarding Parallel.ForEach and dealing with Azure Storage (my experience has been with uploading blobs/blocks in parallel):
Azure storage operations are Network (IO) based operations and not processor intensive operations. If I am not mistaken, Parallel.ForEach is more suitable for processor intensive applications.
Another thing we noticed with uploading a large number of blobs (or blocks) using Parallel.ForEach is that we started to get a lot of Timeout exceptions and actually slowed down the entire operation. I believe the reason for this is when you iterate over a collection with large number of items using this approach, you're essentially handling the control to underlying framework which decides how to deal with that collection. In this case, a lot of Context Switching will take place which slows down the operation. Not sure how this would work in your scenario considering the payload is smaller.
My recommendation would be have the application control the number of parallel threads it can spawn. A good criteria would be the number of logical processor. Another good criteria would be the number of ports IE can open. So you would spawn that many number of parallel threads. Then you could either wait for all threads to finish to spawn next set of parallel threads or start a new thread as soon as one task finishes.
Pseudo Code:
ICollection<string> messageContents;
private void AddMessages()
{
int maxParallelThreads = Math.Min(Environment.ProcessorCount, messageContents.Count);
if (maxParallelThreads > 0)
{
var itemsToAdd = messageContents.Take(maxParallelThreads);
List<Task> tasks = new List<Task>();
for (var i = 0; i < maxParallelThreads; i++)
{
tasks.Add(Task.Factory.StartNew(() =>
{
AddMessage(itemsToAdd[i]);
RemoveItemFromCollection();
}));
}
Task.WaitAll(tasks.ToArray());
AddMessages();
}
}
Your code looks fine to me at a high level. Gaurav's additions make sense, so you have more controls over the parallel processing of your requests. Make sure you add some form of retry logic, and perhaps setting the DefaultConnectionLimit to something greater than its default value (which is 2). You may also consider adding multiple Azure Queues across multiple storage accounts if you hit a form of throttling, depending on the type of errors you are getting.
For anyone looking to add a large number of non-POCO/string messages in bulk/batch to a queue, an alternate/better solution will be to add the list of messages as a single message or blob, and then in a queue/blob trigger traverse & add each message to a [seperate] queue.
var maxDegreeOfParallelism = Math.Min(Environment.ProcessorCount,cloudQueueMessageCollection.Count());
var parallelOptions=new ParallelOptions { MaxDegreeOfParallelism = maxDegreeOfParallelism };
Parallel.ForEach(cloudQueueMessageCollection, parallelOptions,
async (m) => await AddMessageAsync(queue, connectionStringOrKey, m));