I am working on Azure ServiceBus Topic-Subscriptions. I created one topic and two subscriptions in Service Bus Namespace. In my current application, I am able to send messages to topic after that I receive messages by using subscribers.
But whenever one subscriber will read messages from topic then the messages are gone into Dead-Letter-Queue. That’s why second subscriber will be unable to read messages from topic.
I wrote this line in my code after process message from Topic
await subscriptionClient.CompleteAsync(message.SystemProperties.LockToken);
I followed this documentation for implementing above scenario.
So, can anyone suggest me how to read messages from topic using multiple subscribers at a time.
whenever one subscriber will read messages from topic then the messages are gone into Dead-Letter-Queue. That’s why second subscriber will be unable to read messages from topic.
Either there's a processing logic error and message is retried until it gets moved to the dead-letter queue or you've got MaxDeliveryCount set to one processing attempt. Either way, check DeadLetterReason and DeadLetterErrorDescription headers/properties for the reason message got moved into dead-letter queue.
can anyone suggest me how to read messages from topic using multiple subscribers at a time.
Azure Service Bus supports competing consumer pattern, you don't have to do anything special.
Related
We're noticing that exactly 50 percent of the messages produced to my Pulsar topic are reaching my app. Everything was working fine yesterday where our Pulsar consumer app was getting 100% of messages that were produced to the topic. We haven't made any setting changes in our app. What is happening with the missing messages? Where are they going?
Pulsar isn't losing your messages.
It looks like you're using a shared subscription and more than one consumer connected. That other consumer is receiving your other messages since the topic will dispatch them in a round-robin when using a shared subscription. This behavior can occur by design if your consumers are auto-scaling on a shared subscription.
If you check topic stats ($ pulsar-admin topics stats options, documented here), in the response, in "subscriptions", look for your subscription by its name. In that object, you can see the "type", which will be marked as "shared," and you will see a list of "consumers". I'd expect that you have more than one consumer in that list.
I might be confused how EventHubs supposed to be used or need guidance on how to reliably process events posted into Eventhub. I export Azure ActivityLog to Eventhub and currently just using console application to read those messages. What I don't understand is what I'm supposed to do with events which I already read and processed. Say I want to write content of all messages into Storage account AppendLog. For this I need to delete messages which I already processed (like it would be done if it will be message queue), how do I do that with eventhub?
You cannot delete them. From the docs:
Event Hubs retains data for a configured retention time that applies across all partitions in the event hub. Events expire on a time basis; you cannot explicitly delete them.
Back to your question:
Say I want to write content of all messages into Storage account AppendLog. For this I need to delete messages which I already processed
I am not sure why you need this though. You can keep a pointer to the last read message so you are able to process only new messages. Why should you need to delete the older ones? You can read about offsets and ceckpointing here.
What technique are you using for reading the messages?
If you need a pattern of popping messages, you need the Queue or Topic from the Azure Service Bus.
When you ack that message, it is popped from the queue.
My service consumes messages from an Azure Service Bus subscription. A dependency of my service was down for a while, which caused a lot of messages to end up in the deadletter queue (DLQ). Now that the service is back up, I want to reprocess all messages from the DLQ. How can I move/resubmit all messages from the DLQ back in to the main queue.
Restrictions:
It's thousands of messages, so manually handling them isn't feasible.
The topic has about ten subscriptions. I don't want to resubmit the messages to the topic, because then all subscriptions would receive the messages, leading to double-processing.
I don't want to run the service against the DLQ directly, because some messages are broken and cause permanent errors, i.e. they would end up in the DLQ again, which would lead to an infinite loop. Moreover, the broken messages are put back at the front of the queue, effectively starving healthy messages that come after the broken ones.
I realize this is a while after the original post but if anyone else stumbles on this problem, there is a fairly handy solution baked into the Service Bus Explorer (which I have found to be incredibly handy with ASB development).
After connecting to your Service Bus and finding the needed namespace, find the desired topic and subscription with the deadletters in it. From there Right Click and Receive Deadletter Queue Messages and hit OK.
From there, highlight which you would like to send back to the main queue and hit Resubmit Selected Messages in Batch Mode.
Thomas, you probably already found your answer since this is quite awhile ago. think of DLQ (or any existing queue that you have) as just another collection variable like in a PC app, but residing on the cloud. just like a PC-app or in-memory collection variable from your tool-kit, you have many ways of utilising it. off course there are limitations and differences between these 2 types of collection variables, but that's how you design your solution as though the DLQ is just another collection variable by knowing those limitations and differences.
For some queuing implementations, one of the solutions would be to have another instance of the same app pointing to the DLQ, but with a fairly long visibility timeout (e.g. 6 or 12 or even 24 hours depending on your SLA), since you don't want to repeat them too often. However, this is not applicable to Azure service bus, as it limits the visibility timeout to at most 5 minutes.
if the DLQ contains broken un-recoverable jobs, you should fix the app to delete them based on the error messages when the unknown exception occurred. once the fix is deployed, such broken un-recoverable jobs would have been removed by your app and never get sent to the DLQ in the first place. and those already in the DLQ will be removed by the fixed app.
The only option to replay DLQ messages is to receive them from DLQ, create new message with same content and send it again to the topic. They will end up at the end of subscription queue.
You can't send messages directly to the subscription. There is a trick to add a metadata property to the message, and then adjust all except one subscription to filter out such messages. It's up to you to decide if it's going to help in your scenario.
As for tooling, we always did that with custom code, because we always needed some extra work to be done, like logging each replayed message for further analysis.
The quick answer is that you cannot directly move messages back to the main queue of a subscription. This is by design with how Microsoft implemented their topics and subscriptions.
Option #1
There is the option to use Azure Service Bus topic filters https://learn.microsoft.com/en-us/azure/service-bus-messaging/topic-filters and define/tag your messages in a manner that would only allow them to be received on the targeted subscription.
Option #2
The other option would be to change your current implementation. You would set up "delivery queues" (regular service bus queues) and configure each corresponding subscription to auto forward its messages to these delivery queues. Your message processing logic would then listen on these "delivery queues" vs the subscription. Any failures would then result in DLQ messages on these associated "delivery queues" which could then be handled outside of the topic/subscriptions.
IN Azure Service bus topic how the messages are moved from dead letter queue to topic?
whether it will automatically moved to topic or we need to configure the properties of topic in portal or whether there is any other way to do it?(i prefer not to use any code here i wish to do only configuration changes)
We had a batch of around 60k messages, which need to be reprocessed from the dead letter queue. Peeking and send the messages back via Service Bus Explorer took around 6 minutes per 1k messages from my machine. I solved the issue by setting a forward rule for DLQ messages to another queue and from there auto forward it to the original queue. This solution took around 30 seconds for all 60k messages. This works well for queues and topics.
I'm just getting started with Windows Azure Service Bus (Topics & Queues) and I'm trying to implement a Competing-Consumers messaging pattern.
Essentially, I want to have a set of message Producers and a set of message Consumers. Once a message is produced, I want the first available Consumer to process the message. No other Consumers should get the message.
Is there a way to do this in Azure?
Simple. Just make two (or more) receivers that concurrently receive from a single queue and you're done. Any retrieved message goes to exactly one of those receivers since the cursor over the mesasage log is advanced as a message is taken. Competing consumers are an inherent capability of a networked queue so there's really nothing special needed.
If you need the opposite - each message goes to each consumer - you make a subscrioption per consumer which gives you an isolated cusor over the message log that can move independent of other receivers. For kicks, you can obviously also have competing consumers on a subscription.
Clemens
Topics are a feature of brokered messaging, but are a one-to-many "publish/subscribe" pattern. Queues are one-to-one message communication. So yes, it sounds like you should simply use queues. Also see http://msdn.microsoft.com/en-us/library/hh689723(VS.103).aspx.
You probably don't want Topics then, but rather Brokered Messaging.
You can emulate Topic-like functionality in Brokered Messaging by using the message's Label and/or Content Type properties along with the PeekLock receive mode.