Azure Service Bus - Messages body not getting printed in console. Code not getting in the for loop and taking a lot of time - azure

with servicebus_client:
receiver = servicebus_client.get_subscription_receiver(topic_name=TOPIC_NAME, subscription_name=SUBSCRIPTION_NAME, max_wait_time=5)
with receiver:
for msg in receiver:
print("Received: " + str(msg))
receiver.complete_message(msg)
This code is taken from the Azure service bus code snippet in Microsoft portal. The code works fine till the it sends the messages, but when it reaches this part of the code, it is not getting in the for loop. The print statement is not getting execute. But I can see the Incoming and outcoming messages on the particular metrics page of that topic. Can I get some help here as I am new to Azure Service Bus?

Code for receive message from topic. its working as expected. you can try again with all library reference and follow all steps.
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Azure.Messaging.ServiceBus;
namespace SubscriptionReceiver
{
internal class Program
{
// connection string to your Service Bus namespace
static string connectionString = "Endpoint=sb://xxx/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=xxx";
// name of the Service Bus topic
static string topicName = "mytopic";
// name of the subscription to the topic
static string subscriptionName = "S1";
// the client that owns the connection and can be used to create senders and receivers
static ServiceBusClient client;
// the processor that reads and processes messages from the subscription
static ServiceBusProcessor processor;
static async Task Main()
{
// The Service Bus client types are safe to cache and use as a singleton for the lifetime
// of the application, which is best practice when messages are being published or read
// regularly.
//
// Create the clients that we'll use for sending and processing messages.
client = new ServiceBusClient(connectionString);
// create a processor that we can use to process the messages
processor = client.CreateProcessor(topicName, subscriptionName, new ServiceBusProcessorOptions());
try
{
// add handler to process messages
processor.ProcessMessageAsync += MessageHandler;
// add handler to process any errors
processor.ProcessErrorAsync += ErrorHandler;
// start processing
await processor.StartProcessingAsync();
Console.WriteLine("Wait for a minute and then press any key to end the processing");
Console.ReadKey();
// stop processing
Console.WriteLine("\nStopping the receiver...");
await processor.StopProcessingAsync();
Console.WriteLine("Stopped receiving messages");
}
finally
{
// Calling DisposeAsync on client types is required to ensure that network
// resources and other unmanaged objects are properly cleaned up.
await processor.DisposeAsync();
await client.DisposeAsync();
}
}
// handle received messages
static async Task MessageHandler(ProcessMessageEventArgs args)
{
string body = args.Message.Body.ToString();
Console.WriteLine($"Received: {body} from subscription: {subscriptionName}");
// complete the message. messages is deleted from the subscription.
await args.CompleteMessageAsync(args.Message);
}
// handle any errors when receiving messages
static Task ErrorHandler(ProcessErrorEventArgs args)
{
Console.WriteLine(args.Exception.ToString());
return Task.CompletedTask;
}
}
}
Output :- Receive Message from Topic Subscription

Related

change default protocol of Azure EventHub from AMQP to HTTPS

in my case AMOP protocol is blocked by firewall. I just can work with Azure eventhub with https protocol.
I just find:
.NET EventHubConnectionOptions.TransportType property with EventHubsTransportType.AmqpTcp or EventHubsTransportType.AmqpWebSockets
I do not know, how to change the protocol in receiver of my application.
this is my sample code:
static async Task Main()
{
// Read from the default consumer group: $Default
string consumerGroup = EventHubConsumerClient.DefaultConsumerGroupName;
// Create a blob container client that the event processor will use
storageClient = new BlobContainerClient(blobStorageConnectionString, blobContainerName);
// Create an event processor client to process events in the event hub
processor = new EventProcessorClient(storageClient, consumerGroup, ehubNamespaceConnectionString, eventHubName);
// Register handlers for processing events and handling errors
processor.ProcessEventAsync += ProcessEventHandler;
processor.ProcessErrorAsync += ProcessErrorHandler;
// Start the processing
await processor.StartProcessingAsync();
// Wait for 30 seconds for the events to be processed
await Task.Delay(TimeSpan.FromSeconds(30));
// Stop the processing
await processor.StopProcessingAsync();
}
static async Task ProcessEventHandler(ProcessEventArgs eventArgs)
{
// Write the body of the event to the console window
Console.WriteLine("\tReceived event: {0}", Encoding.UTF8.GetString(eventArgs.Data.Body.ToArray()));
// Update checkpoint in the blob storage so that the app receives only new events the next time it's run
await eventArgs.UpdateCheckpointAsync(eventArgs.CancellationToken);
}
It's a bit convoluted process.
You would want to use EventProcessorClient(BlobContainerClient, String, String, EventProcessorClientOptions) constructor and then specify ConnectionOptions in your client options.
There's a property called TransportType in ConnectionOptions which is of type EventHubsTransportType.
You can specify TransportType as EventHubsTransportType.AmqpWebSockets and then you should be able to use AMQP over WebSockets.

How to add retry options (retry policy) to publisher/subscriber client which uses Azure Service Bus Java SDK?

I'm testing retry options for Azure Service Bus publisher/subscriber client because after a sudden connection failure the client will not retry to send or receive messages.
Following is the code for publisher client sendMessage() method and I have set maximum delivery count to 1000 for the subscription. Still the client uses default retryPolicy values and I cannot see it retries as I have given in amqpRetryOptions.
static void sendMessage() {
// create Retry Options for the Service Bus client
AmqpRetryOptions amqpRetryOptions = new AmqpRetryOptions();
amqpRetryOptions.setDelay(Duration.ofSeconds(1));
amqpRetryOptions.setMaxRetries(120);
amqpRetryOptions.setMaxDelay(Duration.ofMinutes(5));
amqpRetryOptions.setMode(AmqpRetryMode.EXPONENTIAL);
amqpRetryOptions.setTryTimeout(Duration.ofSeconds(5));
// create a Service Bus Sender client for the queue
ServiceBusSenderClient senderClient = new ServiceBusClientBuilder()
.connectionString(connectionString)
.retryOptions(amqpRetryOptions)
.sender()
.topicName(topicName)
.buildClient();
// send one message to the topic
senderClient.sendMessage(new ServiceBusMessage("Hello, World! "));
System.out.println("Sent a single message to the topic");
}
Is my approach wrong?
If so, what is the standard way?
If not how can approach retry mechanism?
If not how to
I was able to get retrying mechanism work using ServiceBusSenderAsyncClient. Also, I could catch exceptions to check whether the cause is transient or not.
static void sendMessage() {
// create Retry Options for the Service Bus client
AmqpRetryOptions amqpRetryOptions = new AmqpRetryOptions();
amqpRetryOptions.setDelay(Duration.ofSeconds(1));
amqpRetryOptions.setMaxRetries(5);
amqpRetryOptions.setMaxDelay(Duration.ofSeconds(15));
amqpRetryOptions.setMode(AmqpRetryMode.EXPONENTIAL);
amqpRetryOptions.setTryTimeout(Duration.ofSeconds(5));
// instantiate a client that will be used to call the service
ServiceBusSenderAsyncClient serviceBusSenderAsyncClient = new ServiceBusClientBuilder()
.connectionString(connectionString)
.retryOptions(amqpRetryOptions)
.sender()
.topicName(topicName)
.buildAsyncClient();
// create a message
ServiceBusMessage serviceBusMessage = new ServiceBusMessage("Hello, World!\n")
// send the message to the topic
serviceBusSenderAsyncClient.sendMessage(serviceBusMessage).subscribe(
unused -> System.out.println("Message sent successfully"),
error -> {
ServiceBusException serviceBusException = (ServiceBusException) error;
System.out.println(serviceBusException.isTransient());
},
() -> {
System.out.println("Message sent successfully");
}
);
}

Posting events to Azure Event Hubs using Policy defined on Event Hub Namespace

I have created an Event Hub Namespace and 2 event hubs. I defined a Shared Access Policy (SAP) on the Event Hub Namespace. However, when I use the connection string defined on the namespace, I am able to send events to only one of the hubs even though I create the client using the correct event hub name
function void SendEvent(connectionString, eventHubName){
await using(var producerClient = new EventHubProducerClient(connectionString, eventHubName)) {
// Create a batch of events
using EventDataBatch eventBatch = await producerClient.CreateBatchAsync();
var payload = GetEventModel(entity, entityName);
// Add events to the batch. An event is a represented by a collection of bytes and metadata.
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes(payload.ToString())));
// Use the producer client to send the batch of events to the event hub
await producerClient.SendAsync(eventBatch);
System.Diagnostics.Debug.WriteLine($"Event for {entity} sent to Hub {eventHubName}");
}
}
The above code is called for sending events to Hub1 and Hub2. When I use the connection string from the SAP defined on the Namespace, I can only send events to Hub1 or Hub2 whichever happens to be called first. I am specifying the eventHubName as Hub1 or Hub2 as appropriate.
I call the function SendEvent in my calling code.
The only way I can send to both hubs is to define SAP on each hub and use that connection string when creating the EventHubProducer
Am I missing something or is this by design?
I did a quick test at my side, and it can work well at my side.
Please try the code below, and let me know if it does not meet your need:
class Program
{
//the namespace level sas
private const string connectionString = "Endpoint=sb://yyeventhubns.servicebus.windows.net/;SharedAccessKeyName=mysas;SharedAccessKey=xxxx";
//I try to send data to the following 2 eventhub instances.
private const string hub1 = "yyeventhub1";
private const string hub2 = "yyeventhub2";
static async Task Main()
{
SendEvent(connectionString, hub1);
SendEvent(connectionString, hub2);
Console.WriteLine("**completed**");
Console.ReadLine();
}
private static async void SendEvent(string connectionString, string eventHubName)
{
// Create a producer client that you can use to send events to an event hub
await using (var producerClient = new EventHubProducerClient(connectionString, eventHubName))
{
// Create a batch of events
using EventDataBatch eventBatch = await producerClient.CreateBatchAsync();
// Add events to the batch. An event is a represented by a collection of bytes and metadata.
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("First event: "+eventHubName)));
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Second event: "+eventHubName)));
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Third event: "+eventHubName)));
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Fourth event: " + eventHubName)));
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Fifth event: " + eventHubName)));
// Use the producer client to send the batch of events to the event hub
await producerClient.SendAsync(eventBatch);
Console.WriteLine("A batch of 3 events has been published to: "+ eventHubName);
}
}
}
After running the code, I can see the data are sent to both of the 2 eventhub instances. Here is the screenshot:

How to handle cancellation token in azure service bus topic receiver?

I have a scenario in which I am calling RegisterMessageHandler of SubscriptionClient class of Azure Service Bus library.
Basically I am using trigger based approach while receiving the messages from Service Bus in one of my services in Service Fabric Environment as a stateless service.
So I am not closing the subscriptionClient object immediately, rather I am keeping it open for the lifetime of the Service so that it keeps on receiving the message from azure service bus topics.
And when the service needs to shut down(due to some reasons), I want to handle the cancellation token being passed into the service of Service Fabric.
My question is how can I handle the cancellation token in the RegisterMessageHandler method which gets called whenever a new message is received?
Also I want to handle the closing of the Subscription client "Gracefully", i.e I want that if a message is already being processed, then I want that message to get processed completely and then I want to close the connection.
Below is the code I am using.
Currently We are following the below approach:
1. Locking the process of the message using semaphore lock and releasing the lock in finally block.
2. Calling the cancellationToken.Register method to handle cancellation token whenever cancellation is done. Releasing the lock in the Register Method.
public class AzureServiceBusReceiver
{
private SubscriptionClient subscriptionClient;
private static Semaphore semaphoreLock;
public AzureServiceBusReceiver(ServiceBusReceiverSettings settings)
{
semaphoreLock = new Semaphore(1, 1);
subscriptionClient = new SubscriptionClient(
settings.ConnectionString, settings.TopicName, settings.SubscriptionName, ReceiveMode.PeekLock);
}
public void Receive(
CancellationToken cancellationToken)
{
var options = new MessageHandlerOptions(e =>
{
return Task.CompletedTask;
})
{
AutoComplete = false,
};
subscriptionClient.RegisterMessageHandler(
async (message, token) =>
{
semaphoreLock.WaitOne();
if (subscriptionClient.IsClosedOrClosing)
return;
CancellationToken combinedToken = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, token).Token;
try
{
// message processing logic
}
catch (Exception ex)
{
await subscriptionClient.DeadLetterAsync(message.SystemProperties.LockToken);
}
finally
{
semaphoreLock.Release();
}
}, options);
cancellationToken.Register(() =>
{
semaphoreLock.WaitOne();
if (!subscriptionClient.IsClosedOrClosing)
subscriptionClient.CloseAsync().GetAwaiter().GetResult();
semaphoreLock.Release();
return;
});
}
}
Implement the message client as ICommunicationListener, so when the service is closed, you can block the call until message processing is complete.
Don't use a static Semaphore, so you can safely reuse the code within your projects.
Here is an example of how you can do this.
And here's the Nuget package created by that code.
And feel free to contribute!

Resubmitting a message from dead letter queue - Azure Service Bus

I have created a service bus queue in Azure and it works well. And if the message is not getting delivered within default try (10 times), it is correctly moving the message to the dead letter queue.
Now, I would like to resubmit this message from the dead letter queue back to the queue where it originated and see if it works again. I have tried the same using service bus explorer. But it gets moved to the dead letter queue immediately.
Is it possible to do the same, and if so how?
You'd need to send a new message with the same payload. ASB by design doesn't support message resubmission.
We had a batch of around 60k messages, which need to be reprocessed from the dead letter queue. Peeking and send the messages back via Service Bus Explorer took around 6 minutes per 1k messages from my machine. I solved the issue by setting a forward rule for DLQ messages to another queue and from there auto forward it to the original queue. This solution took around 30 seconds for all 60k messages.
Try to remove dead letter reason
resubmittableMessage.Properties.Remove("DeadLetterReason");
resubmittableMessage.Properties.Remove("DeadLetterErrorDescription");
full code
using Microsoft.ServiceBus.Messaging;
using System.Transactions;
namespace ResubmitDeadQueue
{
class Program
{
static void Main(string[] args)
{
var connectionString = "";
var queueName = "";
var queue = QueueClient.CreateFromConnectionString(connectionString, QueueClient.FormatDeadLetterPath(queueName), ReceiveMode.PeekLock);
BrokeredMessage originalMessage
;
var client = QueueClient.CreateFromConnectionString(connectionString, queueName);
do
{
originalMessage = queue.Receive();
if (originalMessage != null)
{
using (var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
{
// Create new message
var resubmittableMessage = originalMessage.Clone();
// Remove dead letter reason and description
resubmittableMessage.Properties.Remove("DeadLetterReason");
resubmittableMessage.Properties.Remove("DeadLetterErrorDescription");
// Resend cloned DLQ message and complete original DLQ message
client.Send(resubmittableMessage);
originalMessage.Complete();
// Complete transaction
scope.Complete();
}
}
} while (originalMessage != null);
}
}
}
Thanks to some other responses here!
We regularly need to resubmit messages. The answer from #Baglay-Vyacheslav helped a lot. I've pasted some updated C# code that works with the latest Azure.Messaging.ServiceBus Nuget Package.
Makes it much quicker/easier to process DLQ on both queues/topics/subscribers.
using Azure.Messaging.ServiceBus;
using System.Collections.Generic;
using System.Threading.Tasks;
using NLog;
namespace ServiceBus.Tools
{
class TransferDeadLetterMessages
{
// https://github.com/Azure/azure-sdk-for-net/blob/Azure.Messaging.ServiceBus_7.2.1/sdk/servicebus/Azure.Messaging.ServiceBus/README.md
private static Logger logger = LogManager.GetCurrentClassLogger();
private static ServiceBusClient client;
private static ServiceBusSender sender;
public static async Task ProcessTopicAsync(string connectionString, string topicName, string subscriberName, int fetchCount = 10)
{
try
{
client = new ServiceBusClient(connectionString);
sender = client.CreateSender(topicName);
ServiceBusReceiver dlqReceiver = client.CreateReceiver(topicName, subscriberName, new ServiceBusReceiverOptions
{
SubQueue = SubQueue.DeadLetter,
ReceiveMode = ServiceBusReceiveMode.PeekLock
});
await ProcessDeadLetterMessagesAsync($"topic: {topicName} -> subscriber: {subscriberName}", fetchCount, sender, dlqReceiver);
}
catch (Azure.Messaging.ServiceBus.ServiceBusException ex)
{
if (ex.Reason == Azure.Messaging.ServiceBus.ServiceBusFailureReason.MessagingEntityNotFound)
{
logger.Error(ex, $"Topic:Subscriber '{topicName}:{subscriberName}' not found. Check that the name provided is correct.");
}
else
{
throw;
}
}
finally
{
await sender.CloseAsync();
await client.DisposeAsync();
}
}
public static async Task ProcessQueueAsync(string connectionString, string queueName, int fetchCount = 10)
{
try
{
client = new ServiceBusClient(connectionString);
sender = client.CreateSender(queueName);
ServiceBusReceiver dlqReceiver = client.CreateReceiver(queueName, new ServiceBusReceiverOptions
{
SubQueue = SubQueue.DeadLetter,
ReceiveMode = ServiceBusReceiveMode.PeekLock
});
await ProcessDeadLetterMessagesAsync($"queue: {queueName}", fetchCount, sender, dlqReceiver);
}
catch (Azure.Messaging.ServiceBus.ServiceBusException ex)
{
if (ex.Reason == Azure.Messaging.ServiceBus.ServiceBusFailureReason.MessagingEntityNotFound)
{
logger.Error(ex, $"Queue '{queueName}' not found. Check that the name provided is correct.");
}
else
{
throw;
}
}
finally
{
await sender.CloseAsync();
await client.DisposeAsync();
}
}
private static async Task ProcessDeadLetterMessagesAsync(string source, int fetchCount, ServiceBusSender sender, ServiceBusReceiver dlqReceiver)
{
var wait = new System.TimeSpan(0, 0, 10);
logger.Info($"fetching messages ({wait.TotalSeconds} seconds retrieval timeout)");
logger.Info(source);
IReadOnlyList<ServiceBusReceivedMessage> dlqMessages = await dlqReceiver.ReceiveMessagesAsync(fetchCount, wait);
logger.Info($"dl-count: {dlqMessages.Count}");
int i = 1;
foreach (var dlqMessage in dlqMessages)
{
logger.Info($"start processing message {i}");
logger.Info($"dl-message-dead-letter-message-id: {dlqMessage.MessageId}");
logger.Info($"dl-message-dead-letter-reason: {dlqMessage.DeadLetterReason}");
logger.Info($"dl-message-dead-letter-error-description: {dlqMessage.DeadLetterErrorDescription}");
ServiceBusMessage resubmittableMessage = new ServiceBusMessage(dlqMessage);
await sender.SendMessageAsync(resubmittableMessage);
await dlqReceiver.CompleteMessageAsync(dlqMessage);
logger.Info($"finished processing message {i}");
logger.Info("--------------------------------------------------------------------------------------");
i++;
}
await dlqReceiver.CloseAsync();
logger.Info($"finished");
}
}
}
It may be "duplicate message detection" as Peter Berggreen indicated or more likely if you are directly moving the BrokeredMessage from the dead letter queue to the live queue then the DeliveryCount would still be at maximum and it would return to the dead letter queue.
Pull the BrokeredMessage off the dead letter queue, get the content using GetBody(), create in new BrokeredMessage with that data and send it to the queue. You can do this in a safe manor, by using peek to get the message content off the dead letter queue and then send the new message to the live queue before removing the message from the dead letter queue. That way you won't lose any crucial data if for some reason it fails to write to the live queue.
With a new BrokeredMessage you should not have an issue with "duplicate message detection" and the DeliveryCount will be reset to zero.
The Service Bus Explorer tool always creates a clone of the original message when you repair and resubmit a message from the deadletter queue. It could not be any different as by default Service Bus messaging does not provide any message repair and resubmit mechanism. I suggest you to investigate why your message gets ends up in the deadletter queue as well as its clone when you resubmit it. Hope this helps!
It sounds like it could be related to ASB's "duplicate message detection" functionality.
When you resubmit a message in ServiceBus Explorer it will clone the message and thereby the new message will have the same Id as the original message in the deadletter queue.
If you have enabled "Requires Duplicate Detection" on the queue/topic and you try to resubmit the message within the "Duplicate Detection History Time Window", then the message will immediately be moved to the deadletter queue again.
If you want to use Service Bus Explorer to resubmit deadletter messages, then I think that you will have to disable "Requires Duplicate Detection" on the queue/topic.

Resources