One can receive messages in azure service bus using either of the the two methods..
queueClient.BeginReceiveBatch OR messageReceiver.ReceiveBatchAsync
Is there any difference between these two methods speedwise or in any other way.
Thanks
If you don't need to the batch receive functionalilty, I prefer the method of wiring up a callback on the OnMessage event of the queue client. We have some fairly high throughput services relying on this pattern of message processing without any issues (1M+ messages/day)
I like that you end up with less, and simpler code, and can easily control the options of how many messages to process in parallel, which receive mode (peek and lock, vs receive and delete), etc
There's a sample of it in this documentation:
string connectionString =
CloudConfigurationManager.GetSetting("Microsoft.ServiceBus.ConnectionString");
QueueClient Client =
QueueClient.CreateFromConnectionString(connectionString, "TestQueue");
// Configure the callback options
OnMessageOptions options = new OnMessageOptions();
options.AutoComplete = false;
options.AutoRenewTimeout = TimeSpan.FromMinutes(1);
// Callback to handle received messages
Client.OnMessage((message) =>
{
try
{
// Process message from queue
Console.WriteLine("Body: " + message.GetBody<string>());
Console.WriteLine("MessageID: " + message.MessageId);
Console.WriteLine("Test Property: " +
message.Properties["TestProperty"]);
// Remove message from queue
message.Complete();
}
catch (Exception)
{
// Indicates a problem, unlock message in queue
message.Abandon();
}
};
Related
I am working with an azure service bus queue configured to be FIFO (First input first output). I work on an order application with the following states "Pending", "Received" and "Sent". therefore I have grouped the messages by the "SessionId" service bus option, setting the orderId as sessionId so that it processes the messages in order in case of horizontal scaling.
So far it works perfectly, the problem I have found is when a message in "pending" or "Received" status fails due to a timeout and goes to the dead letter queue. The message in "sent" status is processed correctly and then when the support team re-sends the "Pending" or "Received" status message to the queue it is processed correctly marking the order in a previous status instead of "sent" ".
I can think of several ways to control this, for example that the support team looks at the status of the order before reprocessing the message from the dead letter queue :) but I would like to know if service bus offers the possibility that if there is a message in the dead letter queu all the messages in the session queue that have the same sessionId go to the dead letter queu. Finallly, my question is:
Is there a way to configure azure service bus so that if there are any messages in the dead letter queue it sends all messages with the same sessionId to the dead letter queue?
Thank you very much!!!
I would like to know if service bus offers the possibility that if there is a message in the dead letter queue all the messages in the session queue that have the same sessionId go to the dead letter queue.
No, there is no such offering by Service Bus by default.
Is there a way to configure azure service bus so that if there are any messages in the dead letter queue it sends all messages with the same sessionId to the dead letter queue?
Yes, you can do that. You can first peek the messages in your dead-letter queue to fetch all the session ids. Then you can receive the messages in your main queue whose session id is in the DLQ, and then move those messages to DLQ. Here's one such logic I've implemented in dot net using the latest version of Service Bus SDK.
var queueName = "<queue>";
var connectionString = "<connection-string>";
var client = new ServiceBusClient(connectionString);
var sessionIdInDLQList = new List<string>();
var receiver = client.CreateReceiver(queueName, new ServiceBusReceiverOptions { SubQueue = SubQueue.DeadLetter });
var message = await receiver.PeekMessageAsync();
while (message != null)
{
if (!sessionIdInDLQList.Contains(message.SessionId))
sessionIdInDLQList.Add(message.SessionId);
message = await receiver.PeekMessageAsync();
}
foreach (var sessionId in sessionIdInDLQList)
{
var session = await client.AcceptSessionAsync(queueName, sessionId);
message = await session.ReceiveMessageAsync(TimeSpan.FromSeconds(20));
while (message != null)
{
await session.DeadLetterMessageAsync(message, "Message with this session is to be dead-lettered!");
message = await session.ReceiveMessageAsync(TimeSpan.FromSeconds(20));
}
}
In your case, you need to do this before your consumers start reading the messages, probably you can write this in your consumer application or any trigger application like Azure Function or worker role. That’s upto your method of handling.
You can try this code to read Dead Letter from Queue.
public static async Task GetMessage()
{
string topic = "myqueue1";
string connectionString = "Endpoint = sb://xxx.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=xxx";
var servicebusclient = new ServiceBusClient(connectionString);
var reciveroptions = new ServiceBusReceiverOptions { SubQueue = SubQueue.DeadLetter };
var reciver = servicebusclient.CreateReceiver(topic, reciveroptions);
// 10 number of message read from Queue
await receiver.PeekMessageAsync(10);
}
after receiving message from Dead Letter you can send to queue.
As per Microsoft official documents
There's no automatic cleanup of the DLQ. Messages remain in the DLQ
until you explicitly retrieve them from the DLQ and call Complete() on
the dead-letter message.
These following document help you.
Thanks Casually Coding for posting post on Read Message from the Dead Letter Queue
Microsoft Documents Using Dead-Letter Queues to Handle Message Transfer Failures , Receive Message from Dead letter queue
I am trying a sample code of Azure Event Hub Producer and trying to send some message to Azure Event Hub.
The eventhub and its policy is correctly configured for sending and listening messages. I am using Dotnet core 3.1 console application. However, the code doesn't move beyond CreateBatchAsync() call. I tried debugging and the breakpoint doesn't go to next line. Tried Try-catch-finally and still no progress. Please guide what I am doing wrong here. The Event hub on Azure is shows some number of successful incoming requests.
class Program
{
private const string connectionString = "<event_hub_connection_string>";
private const string eventHubName = "<event_hub_name>";
static async Task Main()
{
// Create a producer client that you can use to send events to an event hub
await using (var producerClient = new EventHubProducerClient(connectionString, eventHubName))
{
// Create a batch of events
using EventDataBatch eventBatch = await producerClient.CreateBatchAsync();
// Add events to the batch. An event is a represented by a collection of bytes and metadata.
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("First event")));
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Second event")));
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Third event")));
// Use the producer client to send the batch of events to the event hub
await producerClient.SendAsync(eventBatch);
Console.WriteLine("A batch of 3 events has been published.");
}
}
}
The call to CreateBatchAsync would be the first need to create a connection to Event Hubs. This indicates that you're likely experiencing a connectivity or authorization issue.
In the default configuration you're using, the default network timeout is 60 seconds and up to 3 retries are possible, with some back-off between them.
Because of this, a failure to connect or authorize may take up to roughly 5 minutes before it manifests. That said, the majority of connection errors are not eligible for retries, so the failure would normally surface after roughly 1 minute.
To aid in your debugging, I'd suggest tweaking the default retry policy to speed things up and surface an exception more quickly so that you have the information needed to troubleshoot and make adjustments. The options to do so are discussed in this sample and would look something like:
var connectionString = "<< CONNECTION STRING FOR THE EVENT HUBS NAMESPACE >>";
var eventHubName = "<< NAME OF THE EVENT HUB >>";
var options = new EventHubProducerClientOptions
{
RetryOptions = new EventHubsRetryOptions
{
// Allow the network operation only 15 seconds to complete.
TryTimeout = TimeSpan.FromSeconds(15),
// Turn off retries
MaximumRetries = 0,
Mode = EventHubsRetryMode.Fixed,
Delay = TimeSpan.FromMilliseconds(10),
MaximumDelay = TimeSpan.FromSeconds(1)
}
};
await using var producer = new EventHubProducerClient(
connectionString,
eventHubName,
options);
I created a simple Azure Service bus (Queue) and a client that is sending message to service bus. Using below code to send message:
using Microsoft.Azure.ServiceBus;
using Microsoft.Extensions.Configuration;
public async Task SendMessageAsync<T>(T message, string queueName)
{
try
{
var queueClient = new QueueClient(_config.GetConnectionString("AzureServiceBus"), queueName);
string messageBody = JsonSerializer.Serialize(message);
var byteMessage = new Message(Encoding.UTF8.GetBytes(messageBody));
queueClient.SendAsync(byteMessage);
Console.WriteLine((message as Employee).FirstName);
}
catch (Exception ex)
{
var c = ex;
}
}
Sending message using:
using SenderApp;
Console.WriteLine("Hello, World!");
QueueService service = new QueueService();
for (int i = 0; i < 100; i++)
{
Employee e = new Employee();
e.FirstName = "1 " + i.ToString();
e.LastName = "2 " + i.ToString();
service.SendMessageAsync<Employee>(e, "employeequeue");
}
When I try to see active messages, There is nothing in the queue:
However I do see some traffic. But the number of message I sent (over 100) is not equal to number of incoming request show (62) at the bottom of the image. I am not sure what is happening to my messages? This defeats the purpose of the queue.
Please guide me why I am not seeing any messages. What is the best way to handle this ?
I am using following nuget packages:
<PackageReference Include="Microsoft.Azure.ServiceBus" Version="5.2.0" />
<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="6.0.0" />
<PackageReference Include="Microsoft.Extensions.Hosting" Version="6.0.1" />
A message sent to an Azure Service Bus queue will be delivered to the queue unless operation is failing. In that case, an exception will be thrown. Check the following:
Exception handling doesn't swollow exceptions
Await asynchronous send operations to ensure messages are dispatched
Namespace/queue used for sending is what you use to receive
There are no competing consumers, actively receiving messages from the queue.
Validate TCP ports needed for AMQP are not blocked. If those ports are blocked, you could configure your client to use WebSockets.
So I still dont know what caused this issue. But I realized Microsoft.Azure.ServiceBus package was deprecated and later I started using Azure.Messaging.ServiceBus package to connect to service bus and things started to work.
I used following code to send message to queue:
string connectionString = "Endpoint=sb://test.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=f3f+qMYTyVwE18YNl5J6ygJFi30v6J/Smph5HZvyQyE=";
string queueName = "employeequeue";
// since ServiceBusClient implements IAsyncDisposable we create it with "await using"
await using var client = new ServiceBusClient(connectionString);
// create the sender
ServiceBusSender sender = client.CreateSender(queueName);
// create a message that we can send. UTF-8 encoding is used when providing a string.
ServiceBusMessage message = new ServiceBusMessage("Hello world! " + id);
// send the message
await sender.SendMessageAsync(message);
return "Sent";
Used following code to receive message:
string queueName = "employeequeue";
// since ServiceBusClient implements IAsyncDisposable we create it with "await using"
await using var client = new ServiceBusClient(connectionString);
// create a receiver that we can use to receive and settle the message
ServiceBusReceiver receiver = client.CreateReceiver(queueName);
// the received message is a different type as it contains some service set properties
ServiceBusReceivedMessage receivedMessage = await receiver.ReceiveMessageAsync();
string body = receivedMessage.Body.ToString();
// complete the message, thereby deleting it from the service
await receiver.CompleteMessageAsync(receivedMessage);
More info is available # https://github.com/Azure/azure-sdk-for-net/blob/Azure.Messaging.ServiceBus_7.7.0/sdk/servicebus/Azure.Messaging.ServiceBus/README.md
I have created a service bus queue in Azure and it works well. And if the message is not getting delivered within default try (10 times), it is correctly moving the message to the dead letter queue.
Now, I would like to resubmit this message from the dead letter queue back to the queue where it originated and see if it works again. I have tried the same using service bus explorer. But it gets moved to the dead letter queue immediately.
Is it possible to do the same, and if so how?
You'd need to send a new message with the same payload. ASB by design doesn't support message resubmission.
We had a batch of around 60k messages, which need to be reprocessed from the dead letter queue. Peeking and send the messages back via Service Bus Explorer took around 6 minutes per 1k messages from my machine. I solved the issue by setting a forward rule for DLQ messages to another queue and from there auto forward it to the original queue. This solution took around 30 seconds for all 60k messages.
Try to remove dead letter reason
resubmittableMessage.Properties.Remove("DeadLetterReason");
resubmittableMessage.Properties.Remove("DeadLetterErrorDescription");
full code
using Microsoft.ServiceBus.Messaging;
using System.Transactions;
namespace ResubmitDeadQueue
{
class Program
{
static void Main(string[] args)
{
var connectionString = "";
var queueName = "";
var queue = QueueClient.CreateFromConnectionString(connectionString, QueueClient.FormatDeadLetterPath(queueName), ReceiveMode.PeekLock);
BrokeredMessage originalMessage
;
var client = QueueClient.CreateFromConnectionString(connectionString, queueName);
do
{
originalMessage = queue.Receive();
if (originalMessage != null)
{
using (var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
{
// Create new message
var resubmittableMessage = originalMessage.Clone();
// Remove dead letter reason and description
resubmittableMessage.Properties.Remove("DeadLetterReason");
resubmittableMessage.Properties.Remove("DeadLetterErrorDescription");
// Resend cloned DLQ message and complete original DLQ message
client.Send(resubmittableMessage);
originalMessage.Complete();
// Complete transaction
scope.Complete();
}
}
} while (originalMessage != null);
}
}
}
Thanks to some other responses here!
We regularly need to resubmit messages. The answer from #Baglay-Vyacheslav helped a lot. I've pasted some updated C# code that works with the latest Azure.Messaging.ServiceBus Nuget Package.
Makes it much quicker/easier to process DLQ on both queues/topics/subscribers.
using Azure.Messaging.ServiceBus;
using System.Collections.Generic;
using System.Threading.Tasks;
using NLog;
namespace ServiceBus.Tools
{
class TransferDeadLetterMessages
{
// https://github.com/Azure/azure-sdk-for-net/blob/Azure.Messaging.ServiceBus_7.2.1/sdk/servicebus/Azure.Messaging.ServiceBus/README.md
private static Logger logger = LogManager.GetCurrentClassLogger();
private static ServiceBusClient client;
private static ServiceBusSender sender;
public static async Task ProcessTopicAsync(string connectionString, string topicName, string subscriberName, int fetchCount = 10)
{
try
{
client = new ServiceBusClient(connectionString);
sender = client.CreateSender(topicName);
ServiceBusReceiver dlqReceiver = client.CreateReceiver(topicName, subscriberName, new ServiceBusReceiverOptions
{
SubQueue = SubQueue.DeadLetter,
ReceiveMode = ServiceBusReceiveMode.PeekLock
});
await ProcessDeadLetterMessagesAsync($"topic: {topicName} -> subscriber: {subscriberName}", fetchCount, sender, dlqReceiver);
}
catch (Azure.Messaging.ServiceBus.ServiceBusException ex)
{
if (ex.Reason == Azure.Messaging.ServiceBus.ServiceBusFailureReason.MessagingEntityNotFound)
{
logger.Error(ex, $"Topic:Subscriber '{topicName}:{subscriberName}' not found. Check that the name provided is correct.");
}
else
{
throw;
}
}
finally
{
await sender.CloseAsync();
await client.DisposeAsync();
}
}
public static async Task ProcessQueueAsync(string connectionString, string queueName, int fetchCount = 10)
{
try
{
client = new ServiceBusClient(connectionString);
sender = client.CreateSender(queueName);
ServiceBusReceiver dlqReceiver = client.CreateReceiver(queueName, new ServiceBusReceiverOptions
{
SubQueue = SubQueue.DeadLetter,
ReceiveMode = ServiceBusReceiveMode.PeekLock
});
await ProcessDeadLetterMessagesAsync($"queue: {queueName}", fetchCount, sender, dlqReceiver);
}
catch (Azure.Messaging.ServiceBus.ServiceBusException ex)
{
if (ex.Reason == Azure.Messaging.ServiceBus.ServiceBusFailureReason.MessagingEntityNotFound)
{
logger.Error(ex, $"Queue '{queueName}' not found. Check that the name provided is correct.");
}
else
{
throw;
}
}
finally
{
await sender.CloseAsync();
await client.DisposeAsync();
}
}
private static async Task ProcessDeadLetterMessagesAsync(string source, int fetchCount, ServiceBusSender sender, ServiceBusReceiver dlqReceiver)
{
var wait = new System.TimeSpan(0, 0, 10);
logger.Info($"fetching messages ({wait.TotalSeconds} seconds retrieval timeout)");
logger.Info(source);
IReadOnlyList<ServiceBusReceivedMessage> dlqMessages = await dlqReceiver.ReceiveMessagesAsync(fetchCount, wait);
logger.Info($"dl-count: {dlqMessages.Count}");
int i = 1;
foreach (var dlqMessage in dlqMessages)
{
logger.Info($"start processing message {i}");
logger.Info($"dl-message-dead-letter-message-id: {dlqMessage.MessageId}");
logger.Info($"dl-message-dead-letter-reason: {dlqMessage.DeadLetterReason}");
logger.Info($"dl-message-dead-letter-error-description: {dlqMessage.DeadLetterErrorDescription}");
ServiceBusMessage resubmittableMessage = new ServiceBusMessage(dlqMessage);
await sender.SendMessageAsync(resubmittableMessage);
await dlqReceiver.CompleteMessageAsync(dlqMessage);
logger.Info($"finished processing message {i}");
logger.Info("--------------------------------------------------------------------------------------");
i++;
}
await dlqReceiver.CloseAsync();
logger.Info($"finished");
}
}
}
It may be "duplicate message detection" as Peter Berggreen indicated or more likely if you are directly moving the BrokeredMessage from the dead letter queue to the live queue then the DeliveryCount would still be at maximum and it would return to the dead letter queue.
Pull the BrokeredMessage off the dead letter queue, get the content using GetBody(), create in new BrokeredMessage with that data and send it to the queue. You can do this in a safe manor, by using peek to get the message content off the dead letter queue and then send the new message to the live queue before removing the message from the dead letter queue. That way you won't lose any crucial data if for some reason it fails to write to the live queue.
With a new BrokeredMessage you should not have an issue with "duplicate message detection" and the DeliveryCount will be reset to zero.
The Service Bus Explorer tool always creates a clone of the original message when you repair and resubmit a message from the deadletter queue. It could not be any different as by default Service Bus messaging does not provide any message repair and resubmit mechanism. I suggest you to investigate why your message gets ends up in the deadletter queue as well as its clone when you resubmit it. Hope this helps!
It sounds like it could be related to ASB's "duplicate message detection" functionality.
When you resubmit a message in ServiceBus Explorer it will clone the message and thereby the new message will have the same Id as the original message in the deadletter queue.
If you have enabled "Requires Duplicate Detection" on the queue/topic and you try to resubmit the message within the "Duplicate Detection History Time Window", then the message will immediately be moved to the deadletter queue again.
If you want to use Service Bus Explorer to resubmit deadletter messages, then I think that you will have to disable "Requires Duplicate Detection" on the queue/topic.
I'm using a Service Bus queue with Sessions enabled and I'm sending 5 messages with the same SessionId. My receiving code uses AcceptMessageSessionAsync to get a session lock so that it will receive all the messages for that session. It then uses session.ReceiveBatchAsync to try and get all the messages for the session. However, it only seems to get the first message, then when another attempt is made, it gets all the others. You should be able to see that there is a gap of almost a minute between the two batches even though all these messages were sent at once:
Session started:AE8DC914-8693-4110-8BAE-244E42A302D5
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_1_08:03:03.36523
Session started:AE8DC914-8693-4110-8BAE-244E42A302D5
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_2_08:03:04.22964
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_3_08:03:04.29515
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_4_08:03:04.33959
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_5_08:03:04.39587
My code to process these is a function in a WebJob:
[NoAutomaticTrigger]
public static async Task MessageHandlingLoop(TextWriter log, CancellationToken cancellationToken)
{
var connectionString = ConfigurationManager.ConnectionStrings["ServiceBusListen"].ConnectionString;
var client = QueueClient.CreateFromConnectionString(connectionString, "myqueue");
while (!cancellationToken.IsCancellationRequested)
{
MessageSession session = null;
try
{
session = await client.AcceptMessageSessionAsync(TimeSpan.FromMinutes(1));
log.WriteLine("Session started:" + session.SessionId);
foreach (var msg in await session.ReceiveBatchAsync(100, TimeSpan.FromSeconds(5)))
{
log.WriteLine("Message received:" + msg.MessageId);
msg.Complete();
}
}
catch (TimeoutException)
{
log.WriteLine("Timeout occurred");
await Task.Delay(5000, cancellationToken);
}
catch (Exception ex)
{
log.WriteLine("Error:" + ex);
}
}
}
This is called from my WebJob Main using:
JobHost host = new JobHost();
host.Start();
var task = host.CallAsync(typeof(Functions).GetMethod("MessageHandlingLoop"));
task.Wait();
host.Stop();
Why don't I get all my messages in the first call of ReceiveBatchAsync?
This was answered in the MSDN forum by Hillary Caituiro Monge: https://social.msdn.microsoft.com/Forums/azure/en-US/9a84f319-7bc6-4ff8-b142-4fc1d5f1e2fa/service-bus-session-receivebatchasync-only-receiving-1-message?forum=servbus
Service Bus does not guarantee you will receive the message count you
specify in receive batch even if your queue has them or more. Having
say that, you can change your code to try to get the 100 messages in
the first call, buy remember that your application should not assume
that as a guaranteed behavior.
Below this line of code varclient =
QueueClient.CreateFromConnectionString(connectionString, "myqueue");
add client.PrefetchCount = 100;
The reason that you are getting only 1 message at all times in the
first call is due to that when you accept a session it may be also
getting 1 prefetched message with it. Then when you do receive batch,
the SB client will give you that 1 message.
Unfortunately I found that setting the PrefetchCount didn't have an affect, but the reason given for only receiving one message seemed likely so I accepted it as the answer.