I'm using the code to implement Azure service bus topics that can be found here:
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-how-to-use-topics-subscriptions
I've tried to run two instances of the subscriber program, which holds these methods:
private static void RegisterOnMessageHandlerAndReceiveMessages()
{
var messageHandlerOptions = new MessageHandlerOptions(ExceptionReceivedHandler)
{
MaxConcurrentCalls = 1,
AutoComplete = false
};
_subscriptionClient.RegisterMessageHandler(ProcessMessagesAsync, messageHandlerOptions);
}
private static async Task ProcessMessagesAsync(Message message, CancellationToken token)
{
Console.WriteLine($"Received #{message.SystemProperties.SequenceNumber} message: {Encoding.UTF8.GetString(message.Body)}");
await _subscriptionClient.CompleteAsync(message.SystemProperties.LockToken);
}
However, this doesn't allow both subscribers to recieve the message, it is recieved one message each split across both subscribers.
How does I ensure both subscribers get all the messages coming from the bus?
Azure Service Bus is a broker with Competing Consumer pattern when it comes to retrieving messages. This is expected behaviour. What you have here is a scaled-out processing endpoint, where both are trying to process messages from the same subscription. The messages are distributed between these two competing consumers. If you need to distribute the same message to more than a single subscriber, you should have different subscription entities created and listened to.
It's worth stating - as it's not obvious from any of the documentation - that each topic should be setup with a subscription for each subscriber.
I.e. if I have 3 subscribers:
/myservicebus/mytopic/mysubscription1
/myservicebus/mytopic/mysubscription2
/myservicebus/mytopic/mysubscription3
I was expecting - probably same as OP - that topics and subscribers were automatically configured as one:many.
Essentially, if multiple subscribers are configured to the same subscription, then this becomes a 'first sub wins' scenario - the same as if it were a queue.
I also ran into this.
I resolved it by making an extra micro service just for forwarding the messages. The forwarder service itself must subscribe to the messages that each of your subscribers must receive. Each Subscriber registers itself (with a unique id) with the event forwarder service (by sending a message to it on startup). The subscribers must then use the message name + the unique id to subscribe to the messages. The message forwarder service will receive the messages and forward them to all the registered subscribers by changing the subject/label of the message (the unique id is added to the subject/label).
This workaround also has it's limitations, there can only be one instance of the message forwarder service or otherwise you have the same problem again. And the message forwarder service must also be running before the other subscribers otherwise it doesn't receive the registrations.
Related
I'm new to Azure Service Bus and MassTransit. Now I'm experimenting with messages and I want to send a Message that goes into the Dead Letter Queue of the Azure Service Bus. What should such a message look like? Will it only appear in the Dead Letter Queue if there is no consumer? And how should I consume a Dead Letter Message by creating a dead letter consumer?
Thanks in advance :)
Consumer:
public Task Consume(ConsumeContext<IEventCreated> context)
{
try
{
var serializedMessage = JsonSerializer.Serialize(context.Message, new JsonSerializerOptions());
logger.LogInformation("Event consumed: {serializedMessage}", serializedMessage);
return Task.CompletedTask;
}
catch (Exception ex)
{
logger.LogError(ex, "Error while processing message {messageId}. Retry attempt {retryAttempt}", context.MessageId, context.GetRetryAttempt());
throw;
}
}
My consumer definition:
public class EventCreatedConsumerDefinition : ConsumerDefinition<EventCreatedConsumer>
{
protected override void ConfigureConsumer(IReceiveEndpointConfigurator endpointConfigurator,IConsumerConfigurator<EventCreatedConsumer> consumerConfigurator)
{
endpointConfigurator.UseMessageRetry(r => r.Interval(5, 500));
endpointConfigurator.ConfigureDeadLetterQueueDeadLetterTransport();
endpointConfigurator.ConfigureDeadLetterQueueErrorTransport();
}
}
When configured to use the dead-letter queue (DLQ), MassTransit will move faulted or skipped messages to the DLQ instead of creating separate _error or _skipped queues.
If a message consumer throws an exception, and all retry/redelivery limits are reached, the message is dead-lettered using the Azure SDK along with additional message headers detailing the reason/exceptions that occurred.
As for how to deal with messages in the DLQ, you can use Azure Service Bus Explorer (in the Azure Portal) or the old client Windows application to examine/move the messages, or consider other options like a SaaS service that can monitor/move those messages for you.
Message Aid is an early stage tool that does this, author is the co-founder of MassTransit.
I have the following scenario: I have 2 microservices subscribes in an Azure Service Bus, one microservice to send push notifications and another to send emails, these microservice will never work together, so the message will execute only by one microservice.
I was searching and looks like a good idea to send the message to a Topic and have a filter to choose what microservice should execute the operation.
So I have two questions:
Should I use Topics and filters? Not sure if's the better practices.
If yes, exist a way to send the message to the correct subscription? For example, I publish a message to subscription X instead of publishing for all subscriptions.
Thanks
Should I use Topics and filters? Not sure if's the better practices.
Yes. This is a classic use case to topics and subscriptions.
If yes, exist a way to send the message to the correct subscription? For example, I publish a message to subscription X instead of publishing for all subscriptions.
The way publishing works is that you publish it to the topic, the topic takes the messages and validates against the filters on all the subscriptions. Subscriptions with the satisfying filter criteria receive the message. Subscriptions that do not have filter criteria satisfied are skipped. There are three types of filters, Boolean, SQL and Correlation. Boolean filters are less relevant to you. Correlation or SQL filters will do the job. You can find more info on filters in my post.
What I would suggest here is to "stamp" each message with the method of notification. Since you're going to use a single notification method per method, the Correlation filter would be the simplest and most efficient. Notification method value could be assigned to the Label system property of the outgoing messages and a correlation filter on both subscriptions (one for email service and one for push notification services). For example (pseudo code):
var message = new Message();
// custom logic to determine what notification method to use
if (notificationMethod == NotificationMethod.Email)
{
message.Label = "email";
}
else
{
message.Label = "push";
}
// publish the message to the topic
And the two filters setup would be:
// email notification service subscription
var emailFilter = new CorrelationFilter();
filter.Label = "email";
// push notifications service subscription
var pushFilter = new new CorrelationFilter();
filter.Label = "push";
Creating subscription with a given filter (email for example):
var management = new ManagementClient(...);
await management.CreateSubscriptionAsync(new SubscriptionDescription("topic", "subscription"));
var client = new SubscriptionClient("topic", "subscription");
await client.AddRuleAsync(new RuleDescription(
{
Filter = emailFilter,
Name = "email filter"
}));
I have a server which will need to send messages to several clients to let the clients know that something needs to be done.
I am trying to achieve this by using Azure Event Hub.
I use the following code to send the message:
await eventHubClient.SendAsync(
new EventData(Encoding.UTF8.GetBytes(String.Format("Message {0}, {1}", i, sMessage))),
"1")
.ConfigureAwait(continueOnCapturedContext: false);
await eventHubClient.CloseAsync();
I use two WPF application as listeners which will create the listener at startup and will save the EventProcessorHost in a private variable.
When I send a message it's random which of the listeners will process the message.
Is is possible to send messages to multiple recipients with Azure Event Hub?
You need to put each listener to a separate Consumer Group.
Listeners of the same consumer group are "Competing Consumers", i.e. the first one who takes a lock on an event hub partition wins.
I have 2 question for which I failed to find any solution.
Is there any built in way in azure service bus queue, where I can receive messages only from one session (for that sessionId) in my session handler.
Also is there any way I can receive messages only from the deadlettered queue for that session alone.
I can do this programmatically in my code, but I don't want to write if else logic in my code.
Is there any built in way in azure service bus queue, where I can receive messages only from one session (for that sessionId) in my session handler.
As Sean Feldman mentioned QueueClient.AcceptMessageSessionAsync(sessionID) allows to achieve that for a normal queue.
If want to use QueueClient.AcceptMessageSessionAsync(sessionID) to receive queue message, we need to create the with RequiresSession true,
var description = new QueueDescription(QueueName)
{
RequiresSession = true
};
namespaceManager.CreateQueue(description);
Also is there any way I can receive messages only from the deadlettered queue for that session alone.
Base on my test we can't get the dead-letter queue message from the deadletter queue with following code
var dlqName= QueueClient.FormatDeadLetterPath(queueName);
var dlqClient = QueueClient.CreateFromConnectionString(connectionstring, dfQueue);
var gid = Guid.NewGuid().ToString();
var messageSession = client.AcceptMessageSession(gid);
var receiveMessage = messageSession.Receive();
Cannot create a MessageSession for a sub-queue. TrackingId:06a39820-7bf6-412d-ab31-80ef5c174a12_G20_B31, SystemTracker:tomsbservice:Queue: queuename |$DeadLetterQueue
My workaround is that we could get the dead-letter queue message and submit to queue or another normal queue.
I don't think you can do that for Service Bus queues. You can archieve something similar with Service Bus topic and filtered subscription.
So, you send your messages to a topic, and then you'd have to create a subscription per session ID similar to this:
var filter = new SqlFilter(#"sys.SessionId = \"SESSIONID\"");
namespaceManager.CreateSubscription("YOUR_TOPIC", "SESSIONID_SUBSCRIPTION", filter);
and then just receive messages from this subscription:
var client = new SubscriptionClient("...", "YOUR_TOPIC", "SESSIONID_SUBSCRIPTION");
var message = client.Receive();
Message Sessions feature is specifically designed for that.
QueueClient.AcceptMessageSessionAsync(sessionID) allows to achieve that for a normal queue.
As for the DLQ - not sure if that works.
In Azure Service Bus, you can send a brokered message using QueueClient and MessageFactory. I would like to know why would you want to use one over the other.
Azure Service Bus provides different way to send/receive messages.
You can use the QueueClient to send and receive message to/from a queue.
You can use the TopicClient to send message to a topic
And you can use the SubscriptionClient to receive message from a subscription.
Using MessageSender and MessageReceiver, you create sender and receiver that are entity type invariant:
var factory = MessagingFactory.CreateFromConnectionString("MyConnectionString");
A MessageSender can send messages to both topic or queue:
var sender = factory.CreateMessageSender("Queue ou topic path");
A MessageReceiver ca receive messages from both queue and subscription:
var receiver = factory.CreateMessageReceiver("Queue ou subscription path");
Theses abstractions can give you more flexibility if you need to switch from a queue to a topic or vice versa because you just need to change the path of the service bus entity (This could be in your configuration file) so no code change needed. Using QueueClient, TopicClient, SubscriptionClient, you'll have to change your code if you want to move from a queue to a topic.
So my advice is to always use a MessageReceiver/MessageSender when you have to send/receive message from/to a an Azure ServiceBus queue topic/subscription.
NOTE: This does not apply for Eventhub which has a different implementation.