I want to create a Service Bus Topic with a couple of subscriptions using filters for different message types. However I need to guarantee that all messages sent to the Topic will be received and successfully processed by at least one subscription, even if all of the subscribing processes go offline.
Is there a better way than auto-forwarding to queues for each filter, and a way to capture messages ignored by all filtering subscribers without capturing all messages?
Edit: my motivation is to provide a queue-like mechanism with prioritisation without creating a queue for each message type/priority level, or at least manage the complexity of multiple queues on the listening side. A queue generally guarantees a consumer. Rather than have the publisher have to push to different queues I would like to use a topic and use filters to manage priority.
Based on my current knowledge of the SB I suspect that I just need to make sure the subscriptions are in place for a topic including an inverse catch-all filter subscription before exposing the topic for use. I don't know whether subscriptions are completely reliable.
However I need to guarantee that all messages sent to the Topic will be received and successfully processed by at least one subscription, even if all of the subscribing processes go offline.
There's a problem in that statement. Topics and subscriptions are there to implement pub/sub and decouple publishers from subscribers. The broker itself does not guarantee there will be subscribers.
While topics support EnableFilteringMessagesBeforePublishing (TopicDescription.EnableFilteringMessagesBeforePublishing) it is not recommended for production use.
Update
Based on the updated question, the general answer remains the same. Topics/subscriptions are for pub/sub and decoupling. If you want to ensure that no message is lost once subscriber is coming online, you will need to ensure that subscription is created first.
I don't know whether subscriptions are completely reliable.
Yes, subscriptions are reliable. Behind the scenes subscription is a queue.
In case you want to route your messages to different processors based on message type, publishing that message to a topic and having forwarding subscriptions is a good approach. You do need to be mindful of the quotas (how many subscriptions per topic you can create), but those are fairly high. And if you get to that point, it's possible to reduce number of subscriptions when a given processor handles multiple message types by having more complex SQL filtering rules.
Related
I have an online service hosted on Azure, that asynchronously sends data to on-premise clients.
Each client is identified by an unique code.
Actually there is a single topic, with a subscription for each client which has a filter on the unique code, that is sent as a parameter in the message. No message will ever be broadcasted to all the clients.
I feel that using topic this way is wrong.
The alternative that comes to my mind is to use a dedicated queue for each client, that is created on first contact
Could this be a better approach?
Thanks
In my opinion using Topics and Subscriptions is the right way to go. Here's the reason why:
Currently the routing logic (which message needs to go to which subscription) is handled by Azure Service Bus based on the rules you have configured. If you go with queues, the routing logic will need to come to your hosted service. You'll need to ensure that the queue exists before sending each message. I think it will increase the complexity at your service level somehow.
Furthermore, topics and subscriptions would enable you to do build an audit trail kind of functionality (not sure if you're looking for this kind of functionality). You can create a separate subscription that has a rule to deliver all messages (True SQL Rule) to that subscription along with client specific subscription.
Creating a separate Queue for each client is not advisable. This is the problem solved by Topics.
If you have separate Queue for each client, then you need to send messages to multiple Queues from Server. This will become tedious when the number of clients increases.
Having a single Topic and multiple Subscriptions is easy to manage as the message will be sent only to a single Topic from Server.
I have a single client application that will publish the message to a single location/target, and I need that message to then end up in 3 separate other queues as well ( and subsequently processed)
Basically here is the use case:
A website collects customers information in a lead form. That lead information is pushed to an restful web api. The restfull web api then publishes a message to a single location and then returns a success result to client. In the background, the message ends up on 4 queues, and ultimately sent to 4 different other web services (some external, some internal)
The system needs to be performant with respect to how quickly all 4 of the queues are processed from the 4 queues. But the volume of leads is not necessarily extremely high. (perhaps a few hundred leads per day)
Here is an image of what I am thinking
You could use a Topic in combination with the autoforwarding feature.
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-auto-forwarding
A single publish to a topic could then be setup to auto-forward to 4 separate queues.
Instead of uses Queues, you should use Topics:
a queue is often used for point-to-point communication, topics are useful in publish/subscribe scenarios.
Topics and subscriptions provide a "one-to-many" form of communication, using a publish/subscribe pattern.
If you really need to use a Queue there is no other way than to send copies of the message to the different queues.
One best solution to solve your business scenario, is using a Service Bus Topic with four Topic Subscriptions. You can send the message to the topic. You can create filters (or) Topic subscription rules to filter the messages received by the Service Bus Topic.
You can then set the auto-forward property of each topic subscription to the desired Service Bus Queue.
I'm looking into using Azure Service Bus for publishing/delivering messages between different services in our application, and are trying to find a decent design for the topic and subscriptions within a Service Bus namespace.
The intent of the system is for service-a to publish a message with type service-a.test-event to a bus, and have any service listening on that event type get the message delivered. It will be a fairly low volume
I'm a bit torn on which of the following designs to use:
The Service Bus namespace has one topic events where all messages from all services are delivered. Any service subscribing to events from any other service create a subscription in this topic using filters to get the message types they want - one subscription per message type (eg. service-b-service-a-test-event).
The Service Bus namespace has one topic per publisher (eg. events-service-a). Any subscribing service to events from this service create a subscription in the topic using filters to get the message types they want - one subscription per message type (eg. service-b-test-event).
Service Bus seems to have a limit of 2000 subscriptions per topic, which as far as I can tell will be sufficient for us. If I had suspicions otherwise, option #2 would probably be the best choice (as I can have 10.000 topics per namespace). None of the other limitations of Service Bus, as far as I can tell, really impacts which of these options I should go for.
One additional requirement is that I want to have a service subscribing to any event from any service for recording reasons. If I went for option #1, that'd be very simple to implement. For option #2 however, that service needs to somehow make sure it has a subscription within any event topic in the namespace - and reconfigure itself once new topics are added and old topics are removed. That's outside the scope of this question, but a requirement for the design none the less.
When deciding about a topology, take into consideration things that matter to you the most. One of the principles of pub/sub is to decouple between publishers and subscribers. With topology #2 (topic per publisher) this principle is violates since every subscriber would have to know the name of the publisher. And if publisher of an event would change, all your subscribers would have to get that knowledge in order to re-subscribe. Topology #1 solves that problem by abstracting publishers away by using a single topic.
Also, the 2nd requirement you had (to audit all of the events) would be much easier to implement with this topology as you only have to maintain a single wiretap subscription on that topic.
You're correct about limitation of 2000 subscriptions per topic. Saying that, 2000 subscriptions is quite a lot. Once your system get to that number of subscribers, you'd probably want to review your architecture/topology anyway.
These two topologies are sound extremely similar to the NServiceBus on Azure Service Bus as a transport topologies. You could have a look to see some of the other ideas you could use for your topology, including benefits of one over another.
Disclaimer: I am working on NServiceBus Azure Service Bus transport. You don't have to use NServiceBus, though a lot of questions like the one you're posting here become a non-issue when you do.
Does the Azure Service Bus and its on-premise version, Service Bus for Windows Server, replicate a message for every subscriber?
For example, let's say that there is a single topic with five subscribers, then is that message stored in the service bus' database five times - once for each subscriber - or is that message only stored once with business logic to determine which subscribers have read the message?
It would be nice if there is an official site and/or documentation to provide as a reference.
The behavior the Azure Service Bus seems to be that it is keeping a copy per subscriber. I tested this by creating a topic with two subscriptions. I sent in a single message and I see that the size of the Topic in Bytes is 464 (using topic.SizeInBytes). When I receive one message of a subscription the size the drops in half to 232. I tested it with three subscriptions and same behavior occurred: 696 bytes.
Even if they aren't keeping a copy of the message per subscription they are counting the size of the message times the number of subscriptions against the maximum size of the topic, which may be what you were trying to determine.
I agree it would be nice if they documented the behavior, especially for Service Bus for Windows Server since that could affect planning for the amount of storage you need to set aside. As for the Azure Service Bus side, I'm not sure the implementation behind the scenes matters as much as knowing how it factors towards the max size of the topic.
A subscription to a topic resembles a virtual queue that receives
copies of the messages that were sent to the topic. You can optionally
register filter rules for a topic on a per-subscription basis, which
allows you to filter/restrict which messages to a topic are received
by which topic subscriptions.
I think it copies messages. If it does not copy, it should check always, did all subscribers get the messages ? Additionally, if there is filter, it should check just these subscribers to delete message. I think, copying and applying simple consume implemation cost is less than without copying cost.
Article
I have potentially tens or even hudnreds of thousands of clients who need to communicate with a central server.
Communication is in the form of:
receive command from central servers (process it on the client)
respond with a status to central servers
I would like to avoid having the client machines talk to any intermediate web/API servers and instead, I want them to go directly to ASB
No client can see each other's messages. Whatsoever. I understand I can use SAS tokens to provide temporary privilges to clients and renew them on a scheduled basis and that's great and works within my architecture. However, I'm not sure if I can utilize the same ASB topic and have each client have their own topic inside?
Is ASB even the right technology for this? Can I somehow maintain only two queues/service-bus subscriptions for this (request/reply) or must I create an individual queue for each indivdiual client?
TIA
It’s difficult to tell without knowing more about the nature of the messages you are sending – e.g. how many are being sent. However, with this many clients you may be coming up against the quotas which are shown here:
https://msdn.microsoft.com/en-us/library/azure/ee732538.aspx
The salient limitations are:
100 concurrent connections per entity (i.e. topic, queue or subscription)
2,000 subscriptions per topic
10,000 queues or topics per service bus namespace
100,000 correlation filters per topic
It’s worth taking a look at the Azure scalability scenarios described in the second half of this document:
https://msdn.microsoft.com/en-us/library/azure/hh528527.aspx
It may be possible to get the broadcast side of things going by getting clients to connect with correlation filters though I have not tried using them on this scale.
If you want to have lots of senders going to a single queue then you should consider using the Service Bus REST API for message sending.
Otherwise, I'm afraid you may want to consider a proxy...