How to handle cancellation token in azure service bus topic receiver? - azure

I have a scenario in which I am calling RegisterMessageHandler of SubscriptionClient class of Azure Service Bus library.
Basically I am using trigger based approach while receiving the messages from Service Bus in one of my services in Service Fabric Environment as a stateless service.
So I am not closing the subscriptionClient object immediately, rather I am keeping it open for the lifetime of the Service so that it keeps on receiving the message from azure service bus topics.
And when the service needs to shut down(due to some reasons), I want to handle the cancellation token being passed into the service of Service Fabric.
My question is how can I handle the cancellation token in the RegisterMessageHandler method which gets called whenever a new message is received?
Also I want to handle the closing of the Subscription client "Gracefully", i.e I want that if a message is already being processed, then I want that message to get processed completely and then I want to close the connection.
Below is the code I am using.
Currently We are following the below approach:
1. Locking the process of the message using semaphore lock and releasing the lock in finally block.
2. Calling the cancellationToken.Register method to handle cancellation token whenever cancellation is done. Releasing the lock in the Register Method.
public class AzureServiceBusReceiver
{
private SubscriptionClient subscriptionClient;
private static Semaphore semaphoreLock;
public AzureServiceBusReceiver(ServiceBusReceiverSettings settings)
{
semaphoreLock = new Semaphore(1, 1);
subscriptionClient = new SubscriptionClient(
settings.ConnectionString, settings.TopicName, settings.SubscriptionName, ReceiveMode.PeekLock);
}
public void Receive(
CancellationToken cancellationToken)
{
var options = new MessageHandlerOptions(e =>
{
return Task.CompletedTask;
})
{
AutoComplete = false,
};
subscriptionClient.RegisterMessageHandler(
async (message, token) =>
{
semaphoreLock.WaitOne();
if (subscriptionClient.IsClosedOrClosing)
return;
CancellationToken combinedToken = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, token).Token;
try
{
// message processing logic
}
catch (Exception ex)
{
await subscriptionClient.DeadLetterAsync(message.SystemProperties.LockToken);
}
finally
{
semaphoreLock.Release();
}
}, options);
cancellationToken.Register(() =>
{
semaphoreLock.WaitOne();
if (!subscriptionClient.IsClosedOrClosing)
subscriptionClient.CloseAsync().GetAwaiter().GetResult();
semaphoreLock.Release();
return;
});
}
}

Implement the message client as ICommunicationListener, so when the service is closed, you can block the call until message processing is complete.
Don't use a static Semaphore, so you can safely reuse the code within your projects.
Here is an example of how you can do this.
And here's the Nuget package created by that code.
And feel free to contribute!

Related

Azure Service Bus - Messages body not getting printed in console. Code not getting in the for loop and taking a lot of time

with servicebus_client:
receiver = servicebus_client.get_subscription_receiver(topic_name=TOPIC_NAME, subscription_name=SUBSCRIPTION_NAME, max_wait_time=5)
with receiver:
for msg in receiver:
print("Received: " + str(msg))
receiver.complete_message(msg)
This code is taken from the Azure service bus code snippet in Microsoft portal. The code works fine till the it sends the messages, but when it reaches this part of the code, it is not getting in the for loop. The print statement is not getting execute. But I can see the Incoming and outcoming messages on the particular metrics page of that topic. Can I get some help here as I am new to Azure Service Bus?
Code for receive message from topic. its working as expected. you can try again with all library reference and follow all steps.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Azure.Messaging.ServiceBus;
namespace SubscriptionReceiver
{
internal class Program
{
// connection string to your Service Bus namespace
static string connectionString = "Endpoint=sb://xxx/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=xxx";
// name of the Service Bus topic
static string topicName = "mytopic";
// name of the subscription to the topic
static string subscriptionName = "S1";
// the client that owns the connection and can be used to create senders and receivers
static ServiceBusClient client;
// the processor that reads and processes messages from the subscription
static ServiceBusProcessor processor;
static async Task Main()
{
// The Service Bus client types are safe to cache and use as a singleton for the lifetime
// of the application, which is best practice when messages are being published or read
// regularly.
//
// Create the clients that we'll use for sending and processing messages.
client = new ServiceBusClient(connectionString);
// create a processor that we can use to process the messages
processor = client.CreateProcessor(topicName, subscriptionName, new ServiceBusProcessorOptions());
try
{
// add handler to process messages
processor.ProcessMessageAsync += MessageHandler;
// add handler to process any errors
processor.ProcessErrorAsync += ErrorHandler;
// start processing
await processor.StartProcessingAsync();
Console.WriteLine("Wait for a minute and then press any key to end the processing");
Console.ReadKey();
// stop processing
Console.WriteLine("\nStopping the receiver...");
await processor.StopProcessingAsync();
Console.WriteLine("Stopped receiving messages");
}
finally
{
// Calling DisposeAsync on client types is required to ensure that network
// resources and other unmanaged objects are properly cleaned up.
await processor.DisposeAsync();
await client.DisposeAsync();
}
}
// handle received messages
static async Task MessageHandler(ProcessMessageEventArgs args)
{
string body = args.Message.Body.ToString();
Console.WriteLine($"Received: {body} from subscription: {subscriptionName}");
// complete the message. messages is deleted from the subscription.
await args.CompleteMessageAsync(args.Message);
}
// handle any errors when receiving messages
static Task ErrorHandler(ProcessErrorEventArgs args)
{
Console.WriteLine(args.Exception.ToString());
return Task.CompletedTask;
}
}
}
Output :- Receive Message from Topic Subscription

How to add retry options (retry policy) to publisher/subscriber client which uses Azure Service Bus Java SDK?

I'm testing retry options for Azure Service Bus publisher/subscriber client because after a sudden connection failure the client will not retry to send or receive messages.
Following is the code for publisher client sendMessage() method and I have set maximum delivery count to 1000 for the subscription. Still the client uses default retryPolicy values and I cannot see it retries as I have given in amqpRetryOptions.
static void sendMessage() {
// create Retry Options for the Service Bus client
AmqpRetryOptions amqpRetryOptions = new AmqpRetryOptions();
amqpRetryOptions.setDelay(Duration.ofSeconds(1));
amqpRetryOptions.setMaxRetries(120);
amqpRetryOptions.setMaxDelay(Duration.ofMinutes(5));
amqpRetryOptions.setMode(AmqpRetryMode.EXPONENTIAL);
amqpRetryOptions.setTryTimeout(Duration.ofSeconds(5));
// create a Service Bus Sender client for the queue
ServiceBusSenderClient senderClient = new ServiceBusClientBuilder()
.connectionString(connectionString)
.retryOptions(amqpRetryOptions)
.sender()
.topicName(topicName)
.buildClient();
// send one message to the topic
senderClient.sendMessage(new ServiceBusMessage("Hello, World! "));
System.out.println("Sent a single message to the topic");
}
Is my approach wrong?
If so, what is the standard way?
If not how can approach retry mechanism?
If not how to
I was able to get retrying mechanism work using ServiceBusSenderAsyncClient. Also, I could catch exceptions to check whether the cause is transient or not.
static void sendMessage() {
// create Retry Options for the Service Bus client
AmqpRetryOptions amqpRetryOptions = new AmqpRetryOptions();
amqpRetryOptions.setDelay(Duration.ofSeconds(1));
amqpRetryOptions.setMaxRetries(5);
amqpRetryOptions.setMaxDelay(Duration.ofSeconds(15));
amqpRetryOptions.setMode(AmqpRetryMode.EXPONENTIAL);
amqpRetryOptions.setTryTimeout(Duration.ofSeconds(5));
// instantiate a client that will be used to call the service
ServiceBusSenderAsyncClient serviceBusSenderAsyncClient = new ServiceBusClientBuilder()
.connectionString(connectionString)
.retryOptions(amqpRetryOptions)
.sender()
.topicName(topicName)
.buildAsyncClient();
// create a message
ServiceBusMessage serviceBusMessage = new ServiceBusMessage("Hello, World!\n")
// send the message to the topic
serviceBusSenderAsyncClient.sendMessage(serviceBusMessage).subscribe(
unused -> System.out.println("Message sent successfully"),
error -> {
ServiceBusException serviceBusException = (ServiceBusException) error;
System.out.println(serviceBusException.isTransient());
},
() -> {
System.out.println("Message sent successfully");
}
);
}

Masstransit not creating Error queue for Azure Function event subscriber

We followed this example (http://masstransit-project.com/MassTransit/usage/azure-functions.html) to try to set up Azure Functions as Azure Service Bus event (topic) subscribers using MassTransit (for .Net CORE 2.1, Azure Functions 2.0).
When using Azure Webjobs this is as simple as using RabbitMQ, configure the publisher, let the subscriber configure and set up its queue, and have Masstransit automatically create one topic per event, redirect to queue and to "queue_error" after all retries have failed. You do not have to setup anything manually.
But with Azure Functions we seem to manually (through Service Bus Explorer or ARM templates) have to add the subscribers to the topic (which is created by the publisher on the first event it publishes) and the queues as well (though these don't even seem to be necessary, the events are handled directly by the consuming Azure Function topic subscribers.).
Maybe we are doing something wrong, I cannot see from the docs that MT will not, as it normally does, set up the subscriber andd creating queues when using Azure Functions. But it works, except for when the consumer throws an exception and after all setup retries have been executed. We simply do not get the event in the deadletter queue and the normally MT-generated error queue does not even get generated.
So how do we get MT to create the error queues, and MOVE the failed events there?
Our code:
[FunctionName("OrderShippedConsumer")]
public static Task OrderShippedConsumer(
[ServiceBusTrigger("xyz.events.order/iordershipped", "ordershippedconsumer-queue", Connection = "AzureServiceBus")] Message message,
IBinder binder,
ILogger logger,
CancellationToken cancellationToken,
ExecutionContext context)
{
var config = CreateConfig(context);
var handler = Bus.Factory.CreateBrokeredMessageReceiver(binder, cfg =>
{
var serviceBusEndpoint = Parse.ConnectionString(config["AzureServiceBus"])["Endpoint"];
cfg.CancellationToken = cancellationToken;
cfg.SetLog(logger);
cfg.InputAddress = new Uri($"{serviceBusEndpoint}{QueueName}");
cfg.UseRetry(x => x.Intervals(TimeSpan.FromSeconds(5)));
cfg.Consumer(() => new OrderShippedConsumer(cfg.Log, config));
});
return handler.Handle(message);
}
And the Consumer code:
public OrderShippedConsumer(ILog log, IConfigurationRoot config)
{
this.config = config;
this.log = log;
}
public async Task Consume(ConsumeContext<IOrderShipped> context)
{
// Handle the event
}
}

Resubmitting a message from dead letter queue - Azure Service Bus

I have created a service bus queue in Azure and it works well. And if the message is not getting delivered within default try (10 times), it is correctly moving the message to the dead letter queue.
Now, I would like to resubmit this message from the dead letter queue back to the queue where it originated and see if it works again. I have tried the same using service bus explorer. But it gets moved to the dead letter queue immediately.
Is it possible to do the same, and if so how?
You'd need to send a new message with the same payload. ASB by design doesn't support message resubmission.
We had a batch of around 60k messages, which need to be reprocessed from the dead letter queue. Peeking and send the messages back via Service Bus Explorer took around 6 minutes per 1k messages from my machine. I solved the issue by setting a forward rule for DLQ messages to another queue and from there auto forward it to the original queue. This solution took around 30 seconds for all 60k messages.
Try to remove dead letter reason
resubmittableMessage.Properties.Remove("DeadLetterReason");
resubmittableMessage.Properties.Remove("DeadLetterErrorDescription");
full code
using Microsoft.ServiceBus.Messaging;
using System.Transactions;
namespace ResubmitDeadQueue
{
class Program
{
static void Main(string[] args)
{
var connectionString = "";
var queueName = "";
var queue = QueueClient.CreateFromConnectionString(connectionString, QueueClient.FormatDeadLetterPath(queueName), ReceiveMode.PeekLock);
BrokeredMessage originalMessage
;
var client = QueueClient.CreateFromConnectionString(connectionString, queueName);
do
{
originalMessage = queue.Receive();
if (originalMessage != null)
{
using (var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
{
// Create new message
var resubmittableMessage = originalMessage.Clone();
// Remove dead letter reason and description
resubmittableMessage.Properties.Remove("DeadLetterReason");
resubmittableMessage.Properties.Remove("DeadLetterErrorDescription");
// Resend cloned DLQ message and complete original DLQ message
client.Send(resubmittableMessage);
originalMessage.Complete();
// Complete transaction
scope.Complete();
}
}
} while (originalMessage != null);
}
}
}
Thanks to some other responses here!
We regularly need to resubmit messages. The answer from #Baglay-Vyacheslav helped a lot. I've pasted some updated C# code that works with the latest Azure.Messaging.ServiceBus Nuget Package.
Makes it much quicker/easier to process DLQ on both queues/topics/subscribers.
using Azure.Messaging.ServiceBus;
using System.Collections.Generic;
using System.Threading.Tasks;
using NLog;
namespace ServiceBus.Tools
{
class TransferDeadLetterMessages
{
// https://github.com/Azure/azure-sdk-for-net/blob/Azure.Messaging.ServiceBus_7.2.1/sdk/servicebus/Azure.Messaging.ServiceBus/README.md
private static Logger logger = LogManager.GetCurrentClassLogger();
private static ServiceBusClient client;
private static ServiceBusSender sender;
public static async Task ProcessTopicAsync(string connectionString, string topicName, string subscriberName, int fetchCount = 10)
{
try
{
client = new ServiceBusClient(connectionString);
sender = client.CreateSender(topicName);
ServiceBusReceiver dlqReceiver = client.CreateReceiver(topicName, subscriberName, new ServiceBusReceiverOptions
{
SubQueue = SubQueue.DeadLetter,
ReceiveMode = ServiceBusReceiveMode.PeekLock
});
await ProcessDeadLetterMessagesAsync($"topic: {topicName} -> subscriber: {subscriberName}", fetchCount, sender, dlqReceiver);
}
catch (Azure.Messaging.ServiceBus.ServiceBusException ex)
{
if (ex.Reason == Azure.Messaging.ServiceBus.ServiceBusFailureReason.MessagingEntityNotFound)
{
logger.Error(ex, $"Topic:Subscriber '{topicName}:{subscriberName}' not found. Check that the name provided is correct.");
}
else
{
throw;
}
}
finally
{
await sender.CloseAsync();
await client.DisposeAsync();
}
}
public static async Task ProcessQueueAsync(string connectionString, string queueName, int fetchCount = 10)
{
try
{
client = new ServiceBusClient(connectionString);
sender = client.CreateSender(queueName);
ServiceBusReceiver dlqReceiver = client.CreateReceiver(queueName, new ServiceBusReceiverOptions
{
SubQueue = SubQueue.DeadLetter,
ReceiveMode = ServiceBusReceiveMode.PeekLock
});
await ProcessDeadLetterMessagesAsync($"queue: {queueName}", fetchCount, sender, dlqReceiver);
}
catch (Azure.Messaging.ServiceBus.ServiceBusException ex)
{
if (ex.Reason == Azure.Messaging.ServiceBus.ServiceBusFailureReason.MessagingEntityNotFound)
{
logger.Error(ex, $"Queue '{queueName}' not found. Check that the name provided is correct.");
}
else
{
throw;
}
}
finally
{
await sender.CloseAsync();
await client.DisposeAsync();
}
}
private static async Task ProcessDeadLetterMessagesAsync(string source, int fetchCount, ServiceBusSender sender, ServiceBusReceiver dlqReceiver)
{
var wait = new System.TimeSpan(0, 0, 10);
logger.Info($"fetching messages ({wait.TotalSeconds} seconds retrieval timeout)");
logger.Info(source);
IReadOnlyList<ServiceBusReceivedMessage> dlqMessages = await dlqReceiver.ReceiveMessagesAsync(fetchCount, wait);
logger.Info($"dl-count: {dlqMessages.Count}");
int i = 1;
foreach (var dlqMessage in dlqMessages)
{
logger.Info($"start processing message {i}");
logger.Info($"dl-message-dead-letter-message-id: {dlqMessage.MessageId}");
logger.Info($"dl-message-dead-letter-reason: {dlqMessage.DeadLetterReason}");
logger.Info($"dl-message-dead-letter-error-description: {dlqMessage.DeadLetterErrorDescription}");
ServiceBusMessage resubmittableMessage = new ServiceBusMessage(dlqMessage);
await sender.SendMessageAsync(resubmittableMessage);
await dlqReceiver.CompleteMessageAsync(dlqMessage);
logger.Info($"finished processing message {i}");
logger.Info("--------------------------------------------------------------------------------------");
i++;
}
await dlqReceiver.CloseAsync();
logger.Info($"finished");
}
}
}
It may be "duplicate message detection" as Peter Berggreen indicated or more likely if you are directly moving the BrokeredMessage from the dead letter queue to the live queue then the DeliveryCount would still be at maximum and it would return to the dead letter queue.
Pull the BrokeredMessage off the dead letter queue, get the content using GetBody(), create in new BrokeredMessage with that data and send it to the queue. You can do this in a safe manor, by using peek to get the message content off the dead letter queue and then send the new message to the live queue before removing the message from the dead letter queue. That way you won't lose any crucial data if for some reason it fails to write to the live queue.
With a new BrokeredMessage you should not have an issue with "duplicate message detection" and the DeliveryCount will be reset to zero.
The Service Bus Explorer tool always creates a clone of the original message when you repair and resubmit a message from the deadletter queue. It could not be any different as by default Service Bus messaging does not provide any message repair and resubmit mechanism. I suggest you to investigate why your message gets ends up in the deadletter queue as well as its clone when you resubmit it. Hope this helps!
It sounds like it could be related to ASB's "duplicate message detection" functionality.
When you resubmit a message in ServiceBus Explorer it will clone the message and thereby the new message will have the same Id as the original message in the deadletter queue.
If you have enabled "Requires Duplicate Detection" on the queue/topic and you try to resubmit the message within the "Duplicate Detection History Time Window", then the message will immediately be moved to the deadletter queue again.
If you want to use Service Bus Explorer to resubmit deadletter messages, then I think that you will have to disable "Requires Duplicate Detection" on the queue/topic.

Service Bus Session ReceiveBatchAsync only receiving 1 message

I'm using a Service Bus queue with Sessions enabled and I'm sending 5 messages with the same SessionId. My receiving code uses AcceptMessageSessionAsync to get a session lock so that it will receive all the messages for that session. It then uses session.ReceiveBatchAsync to try and get all the messages for the session. However, it only seems to get the first message, then when another attempt is made, it gets all the others. You should be able to see that there is a gap of almost a minute between the two batches even though all these messages were sent at once:
Session started:AE8DC914-8693-4110-8BAE-244E42A302D5
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_1_08:03:03.36523
Session started:AE8DC914-8693-4110-8BAE-244E42A302D5
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_2_08:03:04.22964
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_3_08:03:04.29515
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_4_08:03:04.33959
Message received:AE8DC914-8693-4110-8BAE-244E42A302D5_5_08:03:04.39587
My code to process these is a function in a WebJob:
[NoAutomaticTrigger]
public static async Task MessageHandlingLoop(TextWriter log, CancellationToken cancellationToken)
{
var connectionString = ConfigurationManager.ConnectionStrings["ServiceBusListen"].ConnectionString;
var client = QueueClient.CreateFromConnectionString(connectionString, "myqueue");
while (!cancellationToken.IsCancellationRequested)
{
MessageSession session = null;
try
{
session = await client.AcceptMessageSessionAsync(TimeSpan.FromMinutes(1));
log.WriteLine("Session started:" + session.SessionId);
foreach (var msg in await session.ReceiveBatchAsync(100, TimeSpan.FromSeconds(5)))
{
log.WriteLine("Message received:" + msg.MessageId);
msg.Complete();
}
}
catch (TimeoutException)
{
log.WriteLine("Timeout occurred");
await Task.Delay(5000, cancellationToken);
}
catch (Exception ex)
{
log.WriteLine("Error:" + ex);
}
}
}
This is called from my WebJob Main using:
JobHost host = new JobHost();
host.Start();
var task = host.CallAsync(typeof(Functions).GetMethod("MessageHandlingLoop"));
task.Wait();
host.Stop();
Why don't I get all my messages in the first call of ReceiveBatchAsync?
This was answered in the MSDN forum by Hillary Caituiro Monge: https://social.msdn.microsoft.com/Forums/azure/en-US/9a84f319-7bc6-4ff8-b142-4fc1d5f1e2fa/service-bus-session-receivebatchasync-only-receiving-1-message?forum=servbus
Service Bus does not guarantee you will receive the message count you
specify in receive batch even if your queue has them or more. Having
say that, you can change your code to try to get the 100 messages in
the first call, buy remember that your application should not assume
that as a guaranteed behavior.
Below this line of code varclient =
QueueClient.CreateFromConnectionString(connectionString, "myqueue");
add client.PrefetchCount = 100;
The reason that you are getting only 1 message at all times in the
first call is due to that when you accept a session it may be also
getting 1 prefetched message with it. Then when you do receive batch,
the SB client will give you that 1 message.
Unfortunately I found that setting the PrefetchCount didn't have an affect, but the reason given for only receiving one message seemed likely so I accepted it as the answer.

Resources