I'm just getting started with Windows Azure Service Bus (Topics & Queues) and I'm trying to implement a Competing-Consumers messaging pattern.
Essentially, I want to have a set of message Producers and a set of message Consumers. Once a message is produced, I want the first available Consumer to process the message. No other Consumers should get the message.
Is there a way to do this in Azure?
Simple. Just make two (or more) receivers that concurrently receive from a single queue and you're done. Any retrieved message goes to exactly one of those receivers since the cursor over the mesasage log is advanced as a message is taken. Competing consumers are an inherent capability of a networked queue so there's really nothing special needed.
If you need the opposite - each message goes to each consumer - you make a subscrioption per consumer which gives you an isolated cusor over the message log that can move independent of other receivers. For kicks, you can obviously also have competing consumers on a subscription.
Clemens
Topics are a feature of brokered messaging, but are a one-to-many "publish/subscribe" pattern. Queues are one-to-one message communication. So yes, it sounds like you should simply use queues. Also see http://msdn.microsoft.com/en-us/library/hh689723(VS.103).aspx.
You probably don't want Topics then, but rather Brokered Messaging.
You can emulate Topic-like functionality in Brokered Messaging by using the message's Label and/or Content Type properties along with the PeekLock receive mode.
Related
Ok so i'm relatively new to the servicebus. Working on a project where we use Azure servicebus for queueing messages. Our architecture roughly looks like the following:
So the idea is that in our SourceSystem all kinds of stuff happens, which leads to messages being put on the servicebustopics. Now our responsibility is syncing these events to the external client so they are aware of what we are doing.
Now the issue is that currently we dont use servicebus sessions so message order isnt guaranteed. Also consider the following scenario:
OrderCreated
OrderUpdate 1
OrderUpdate 2
OrderClosed
What happens now is if the externalclients API is down for say OrderUpdate 1 and OrderUpdate 2, we could potentially send the messages in order: OrderCreated, OrderClosed, OrderUpdate 1, OrderUpdate 2.
Currently we just retry a message a few times and then it moves into the deadletter queue for manual reprocessing.
What steps should we take to better guarantee message order? I feel like in the scope of an order, message order needs to be guaranteed.
Should we force the sourcesystem to put all messages for a order in a servicebus session? But how can we handle this with multiple topics? And what do we do if message 1 from a session ends up in the deadletter?
There are a lot of considerations here, should we use a single topic so its easier to manage the sessions? But this opens up other problems with different message structures being in a single topic?
Id love to hear your opinions on this
Have a look at Durable Functions in Azure. You can use the 'Async Http API' or one of the other patterns to achieve the orchestration you need to do.
NServicebus' Sagas might also be a good option, here is an article that does a very good comparison between NServicebus and Durable Functions.
If the external client has to receive all those events and order matters, sending those messages to multiple topics where a topic is per message type will make your mission extremely hard to accomplish. For ordered messaging first you need to use a single entity (queue or topic) with Sessions enabled. That way you can guarantee ordered message processing. In case you have multiple external clients, you'd need to have a session-enabled entity (topic) per external client.
Another option is to implement a pattern known as Process Manager. The process manager would be responsible to make the decisions about the incoming messages and conclude when the work for a given order is completed or not.
There are also libraries (MassTransit, NServiceBus, etc) that can help you. NServiceBus implements Process Manager via a feature called Saga (tutorial) and MassTransit has it as well (documentation).
According to this doc service bus supports two modes Receive-and-Delete and Peek-Lock.
If using Peek-Lock Mode if the consumer crashes/hangs/do a very long GC right after processing the message, but before the messageId is "Completed" and visibility time expires there's a chance that same message is delivered twice.
Then how does Microsoft says that Service Bus supports at most once delivery mode. Is it because of the Receive-and-Delete mode which sends messages only once.But then again, if something happens while consumers are processing the message then that valuable info is lost.
If yes then what is the best way to ensure exact once delivery using Azure Services Bus as Queue and Azure Functions as Consumers.
P.S. The one approach I can think of is storing MessageID's in blob but since in my case number of MessageID's could be very large storing and loading all of them is not right approach.
Azure Functions will always consume Service Bus messages in Peek-Lock mode. Exactly Once delivery is basically not possible in general case: there's always a chance that consuming application will crash at wrong time just before completing the message, and then the message will be re-delivered.
You should strive to implement Effectively Once processing. This is usually achieved with idempotent message processor.
Storing MessageID's (consumer-side de-duplication) is one option. You could have a policy to clean up old Message IDs to keep the size of such storage manageable. To make this 100% reliable you would have to store Message ID in the same transaction as other modifications done by processor.
Other options really depend on your processing scenario. Find a way to make it idempotent - so that processing the same message multiple times is functionally same as processing it just once.
My service consumes messages from an Azure Service Bus subscription. A dependency of my service was down for a while, which caused a lot of messages to end up in the deadletter queue (DLQ). Now that the service is back up, I want to reprocess all messages from the DLQ. How can I move/resubmit all messages from the DLQ back in to the main queue.
Restrictions:
It's thousands of messages, so manually handling them isn't feasible.
The topic has about ten subscriptions. I don't want to resubmit the messages to the topic, because then all subscriptions would receive the messages, leading to double-processing.
I don't want to run the service against the DLQ directly, because some messages are broken and cause permanent errors, i.e. they would end up in the DLQ again, which would lead to an infinite loop. Moreover, the broken messages are put back at the front of the queue, effectively starving healthy messages that come after the broken ones.
I realize this is a while after the original post but if anyone else stumbles on this problem, there is a fairly handy solution baked into the Service Bus Explorer (which I have found to be incredibly handy with ASB development).
After connecting to your Service Bus and finding the needed namespace, find the desired topic and subscription with the deadletters in it. From there Right Click and Receive Deadletter Queue Messages and hit OK.
From there, highlight which you would like to send back to the main queue and hit Resubmit Selected Messages in Batch Mode.
Thomas, you probably already found your answer since this is quite awhile ago. think of DLQ (or any existing queue that you have) as just another collection variable like in a PC app, but residing on the cloud. just like a PC-app or in-memory collection variable from your tool-kit, you have many ways of utilising it. off course there are limitations and differences between these 2 types of collection variables, but that's how you design your solution as though the DLQ is just another collection variable by knowing those limitations and differences.
For some queuing implementations, one of the solutions would be to have another instance of the same app pointing to the DLQ, but with a fairly long visibility timeout (e.g. 6 or 12 or even 24 hours depending on your SLA), since you don't want to repeat them too often. However, this is not applicable to Azure service bus, as it limits the visibility timeout to at most 5 minutes.
if the DLQ contains broken un-recoverable jobs, you should fix the app to delete them based on the error messages when the unknown exception occurred. once the fix is deployed, such broken un-recoverable jobs would have been removed by your app and never get sent to the DLQ in the first place. and those already in the DLQ will be removed by the fixed app.
The only option to replay DLQ messages is to receive them from DLQ, create new message with same content and send it again to the topic. They will end up at the end of subscription queue.
You can't send messages directly to the subscription. There is a trick to add a metadata property to the message, and then adjust all except one subscription to filter out such messages. It's up to you to decide if it's going to help in your scenario.
As for tooling, we always did that with custom code, because we always needed some extra work to be done, like logging each replayed message for further analysis.
The quick answer is that you cannot directly move messages back to the main queue of a subscription. This is by design with how Microsoft implemented their topics and subscriptions.
Option #1
There is the option to use Azure Service Bus topic filters https://learn.microsoft.com/en-us/azure/service-bus-messaging/topic-filters and define/tag your messages in a manner that would only allow them to be received on the targeted subscription.
Option #2
The other option would be to change your current implementation. You would set up "delivery queues" (regular service bus queues) and configure each corresponding subscription to auto forward its messages to these delivery queues. Your message processing logic would then listen on these "delivery queues" vs the subscription. Any failures would then result in DLQ messages on these associated "delivery queues" which could then be handled outside of the topic/subscriptions.
I was using AzureQueue to communicate between roles. My messages like "GoToMaintenanceMode", "StopSendingEmails", "DoNotAcceptRequests" etc. But I realized that, it won't work for my scenario when I have multiple instances due to queue message will shows up only 1 instance at a time.
So my question is beside the options below is there an elegant way to handle this issue something like Role.AllInstances.Run() etc.?
the method I'm using it right now:
instance peeks the message, adds it's own instance id to the message and puts it back to the queue, and does not peek the message if it contains it's own instance id.
P.S. I do not want to implement TCP listener, asking for native solution if there is one.
You could use Windows Azure Service Bus Topics/Subscriptions instead of queues. They support multicasting (i.e. multiple receivers).
A short how-to guide can be found here
http://www.windowsazure.com/en-us/develop/net/how-to-guides/service-bus-topics/
Basically, your queue would become a Topic all your instances would become Subscriber to the Topic.
As you do not want TCP listener option or Service bus option. How about extending your same idea with multiple queues. Instance1 will read from Queue1 and Instance2 will read from Queue2 and so on. The only thing you need to handle is the number of queues and simultaneous adding of queue messages to all the queues.
If there are no longer any publishers or subscribers reading nor writing to a Queue, Topic, or Subscription, because of crashes or other abnormal terminations (instance restart, etc.), is that Queue/Topic/Subscription effectively orphaned?
I tested this by creating a few Queues, and then terminating the applications. Those Queues were still on the Service Bus a long time later. It seems that they will just stay there forever. That would be wonderful if we WANTED that behavior, but in this case, we do not.
How can we detect and delete these Queues, Topics, and Subscriptions? They will count towards Azure limits, etc, and we cannot have these orphaned processes every time an instance is restarted/patched/crashes.
If it helps make the question clearer, this is a unique situation in which the Queues/Topics/Subscriptions have special names, or special Filters, and a very limited set of publishers (1) and subscribers (1) for a limited time. This is not a case where we want survivability. These are instance-specific response channels. Whether we use Queues or Subscriptions is immaterial. If the instance is gone, so is the need for that Queue (or Subscription).
This is part of a solution where each web role has a dedicated response channel that it monitors. At any time, this web role may have dozens of requests pending via other messaging channels (Queues/Topics), and it is waiting for the answers on multiple threads. We need the response to come back to the thread that placed the message, so that the web role can respond to the caller. It is no good in this situation to simply have a Subscription based on the machine, because it will be receiving messages for other threads. We need each publishing thread to establish a dedicated response channel, so that the only thing on that channel is the response for that thread.
Even if we use Subscriptions (with some kind of instance-related filter) to do a long-polling receive operation on the Subscription, if the web role instance dies, that Subscription will be orphaned, correct?
This question can be boiled down like so:
If there are no more publishers or subscribers to a Queue/Topic/Subscription, then that service is effectively orphaned. How can those orphans be detected and cleaned up?
In this scenario you are looking for the Queue/Subscriptions to be "dynamic" in nature. They would be created and removed based on use as opposed to the current explicit provisioning model for these entities. Service Bus provides you with the APIs to perform create/delete operations so you can plug these on role OnStart/OnStop events appropriately. If those operations fail for some reason then the orphaned entities will exist. Again you can run clean up operation on them based on some unique identifier for the name of the entities. An example of this can be seen here: http://windowsazurecat.com/2011/08/how-to-simplify-scale-inter-role-communication-using-windows-azure-service-bus/
In the near future we will add more metadata and query capabilities to Queues/Topics/Subscriptions so you can see when they were last accessed and make cleanup decisions.
Service Bus Queues are built using the “brokered messaging” infrastructure designed to integrate applications or application components that may span multiple communication protocols, data contracts, trust domains, and/or network environments. The allows for a mechanism to communicate reliably with durable messaging.
If a client (publisher) sends a message to a service bus queue and then crashes the message will be stored on the Queue until as consumer reads the message off the queue. Also if your consumer dies and restarts it will just poll the queue and pick up any work that is waiting for it (You can scale out and have multiple consumers reading from queue to increase throughput), Service Bus Queues allow you to decouple your applications via durable cloud gateway analogous to MSMQ on-premises (or other queuing technology).
What I'm really trying to say is that you won't get an orphaned queue, you might get poisoned messages that you will need to handled, this blog post gives some very detailed information re: Service Bus Queues and their Capacity and Quotas which might give you a better understanding http://msdn.microsoft.com/en-us/library/windowsazure/hh767287.aspx
Re: Queue Management, you can do this via Visual Studio (1.7 SDK & Tools) or there is an excellent tool called Service Bus Explorer that will make your life easier for queue managagment: http://code.msdn.microsoft.com/windowsazure/Service-Bus-Explorer-f2abca5a
*Note the default maximum number of queues is 10,000 (per service namespace, this can be increased via a support call)
As Abhishek Lai mentioned there is no orphan detecting capability supported.
Orphan detection can be implement externally in multiple ways.
For example, whenever you send/receive a message, update a timestamp in an SQL database to indicate that the queue/tropic/subscription is still active. This timestamp can then be used to determine orphans.
If your process will crash which is very much possible there will be issue with the message delivery within the queue however queue will still be available to process your request. Handling Application Crashes and Unreadable Messages with Windows Azure Service Bus queues are described here:
The Service Bus provides functionality to help you gracefully recover from errors in your application or difficulties processing a message. If a receiver application is unable to process the message for some reason, then it can call the Abandon method on the received message (instead of the Complete method). This will cause the Service Bus to unlock the message within the queue and make it available to be received again, either by the same consuming application or by another consuming application.
In the event that the application crashes after processing the message but before the Complete request is issued, then the message will be redelivered to the application when it restarts. This is often called At Least Once Processing, that is, each message will be processed at least once but in certain situations the same message may be redelivered. If the scenario cannot tolerate duplicate processing, then application developers should add additional logic to their application to handle duplicate message delivery. This is often achieved using the MessageId property of the message, which will remain constant across delivery attempts.
If there are no longer any processes reading nor writing to a queue, because of crashes or other abnormal terminations (instance restart, etc.), is that queue effectively orphaned?
No the queue is in place to allow communication to occur via Brokered Messages, if all your apps die for some reason then the queue still exists and will be there when they become alive again, it's the communication channel for loosely decoupled applications. Regards Billing 'Messages are charged based on the number of messages sent to, or delivered by, the Service Bus during the billing month' you won't be charged if a queue exists but nobody is using it.
I tested this by creating a few queues, and then terminating the
applications. Those queues were still on the machine a long time
later.
The whole point of the queue is to guarantee message delivery of loosely decoupled applications. Think of the queue as an entity or application in its own right with high availability (SLA) as its hosted in Azure, your producer/consumers can die/restart and the queue will be active in Azure. *Note I got a bit confused with your wording re: "still on the machine a long time later", the queue doesn't actually live on your machine, it sits up in Azure in a designated service bus namespace. You can view and managed the queues via the tools I pointed out in the previous answer.
How can we detect and delete these queues, as they will count towards
Azure limits, etc.
As stated above the default maximum number of queues is 10,000 (per service namespace, this can be increased via a support call), queue management can be done via the tools stated in the other answer. You should only be looking to delete queue's when you no longer have producer/consumers looking to write to them (i.e. never again). You can of course create and delete queues in your producer/consumer applications via the namespaceManager.QueueExists, more information here How to Use Service Bus Queues
If it helps make the question clearer, this is a unique situation in which the queues have special names, and a very limited set of publishers (1) and subscribers (1) for a limited time.
It sounds like you need to use Topics & Subscriptions How to Use Service Bus Topics/Subscriptions, this link also has a section on 'How to Delete Topics and Subscriptions' If you have a very limited lifetime then you could handle topic creation/deletion in your app's otherwise you could have have a separate Queue/Topic/Subscription setup/deletion script to handle this logic...