Does the Azure Service Bus and its on-premise version, Service Bus for Windows Server, replicate a message for every subscriber?
For example, let's say that there is a single topic with five subscribers, then is that message stored in the service bus' database five times - once for each subscriber - or is that message only stored once with business logic to determine which subscribers have read the message?
It would be nice if there is an official site and/or documentation to provide as a reference.
The behavior the Azure Service Bus seems to be that it is keeping a copy per subscriber. I tested this by creating a topic with two subscriptions. I sent in a single message and I see that the size of the Topic in Bytes is 464 (using topic.SizeInBytes). When I receive one message of a subscription the size the drops in half to 232. I tested it with three subscriptions and same behavior occurred: 696 bytes.
Even if they aren't keeping a copy of the message per subscription they are counting the size of the message times the number of subscriptions against the maximum size of the topic, which may be what you were trying to determine.
I agree it would be nice if they documented the behavior, especially for Service Bus for Windows Server since that could affect planning for the amount of storage you need to set aside. As for the Azure Service Bus side, I'm not sure the implementation behind the scenes matters as much as knowing how it factors towards the max size of the topic.
A subscription to a topic resembles a virtual queue that receives
copies of the messages that were sent to the topic. You can optionally
register filter rules for a topic on a per-subscription basis, which
allows you to filter/restrict which messages to a topic are received
by which topic subscriptions.
I think it copies messages. If it does not copy, it should check always, did all subscribers get the messages ? Additionally, if there is filter, it should check just these subscribers to delete message. I think, copying and applying simple consume implemation cost is less than without copying cost.
Article
Related
I'm trying to find a solution for receiving large messages on Azure Service Bus. The essential pattern I was thinking is to publish a large messages in parts -- along with a correlation id, a page, and an "of".
So if I have a four-part message, they would all have the same correlation id, each would have an "of" of 4, and the page would be 0 - 3. The set would be published as a batch.
The listener could listen for only messages with a page of 0, and then pull the remaining messages according to the transaction id.
Publishing these messages is easy enough. ServiceBusMessage has a CorrelationId field, and a dictionary field called ApplicationProperties that I can add my custom "page" and "of" fields to. I can assemble them into a ServiceBusMessageBatch before publishing.
What I'm not sure about is how to receive the messages. I'm using Function Apps, so it's easy to setup a listener.
[FunctionName("GeneralLogger")]
public static void Run([ServiceBusTrigger("queueName", Connection = "AzureWebJobsServiceBus")] string myQueueItem, ApplicationProperties ap, ILogger log)
{ /// process message }
But I don't see how to filter here. Also, I can pull messages by adding a handler to the message processor, described here: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-dotnet-get-started-with-queues But likewise I don't see how to filter.
The only Azure Service Bus filtering I see how to do is between a topic and subscription. There is a lot of capability there, but nothing dynamically I can set during runtime.
I feel like I'm either trying to miss-use something or re-inventing the wheel. Is anyone else doing something like this with Azure Service Bus?
I'm trying to find a solution for receiving large messages on Azure Service Bus.
A solution is already there. It's Azure Service Bus premium tier. Capable of sending messages up to 100MB in size. It comes with a price. Assuming you're looking to spit up the file either because the premium is much to pay for or because messages could be larger than 100MB, the claim-check pattern is the way to go. There's just one issue when the claim-check pattern is used over the premium tier - you cannot have a deterministic clean-up when a message is an event, and there are multiple receivers. You'd need to come up with some policy to clean up those blobs, given that those are large blobs and will quickly add to the storage consumption over time, depending on the number of messages flowing through the system. With the premium tier, the problem of clean-up doesn't exist. Nor do you have to provide a storage account. Therefore, if your large messages will not exceed 100MB, it could be a more suitable solution for your production environment.
It isn't possible to apply filters on a queue; they only operate on topics/subscriptions.
Generally, the Claim Check pattern is recommended when you're looking to send a payload too large for a single message. In a nutshell, you would write your payload to some form of durable storage and then your Service Bus message would provide the location for consumers.
An example implementation using the Azure.Messaging.ServiceBus package can be found in this sample.
I have a single client application that will publish the message to a single location/target, and I need that message to then end up in 3 separate other queues as well ( and subsequently processed)
Basically here is the use case:
A website collects customers information in a lead form. That lead information is pushed to an restful web api. The restfull web api then publishes a message to a single location and then returns a success result to client. In the background, the message ends up on 4 queues, and ultimately sent to 4 different other web services (some external, some internal)
The system needs to be performant with respect to how quickly all 4 of the queues are processed from the 4 queues. But the volume of leads is not necessarily extremely high. (perhaps a few hundred leads per day)
Here is an image of what I am thinking
You could use a Topic in combination with the autoforwarding feature.
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-auto-forwarding
A single publish to a topic could then be setup to auto-forward to 4 separate queues.
Instead of uses Queues, you should use Topics:
a queue is often used for point-to-point communication, topics are useful in publish/subscribe scenarios.
Topics and subscriptions provide a "one-to-many" form of communication, using a publish/subscribe pattern.
If you really need to use a Queue there is no other way than to send copies of the message to the different queues.
One best solution to solve your business scenario, is using a Service Bus Topic with four Topic Subscriptions. You can send the message to the topic. You can create filters (or) Topic subscription rules to filter the messages received by the Service Bus Topic.
You can then set the auto-forward property of each topic subscription to the desired Service Bus Queue.
I want to create a Service Bus Topic with a couple of subscriptions using filters for different message types. However I need to guarantee that all messages sent to the Topic will be received and successfully processed by at least one subscription, even if all of the subscribing processes go offline.
Is there a better way than auto-forwarding to queues for each filter, and a way to capture messages ignored by all filtering subscribers without capturing all messages?
Edit: my motivation is to provide a queue-like mechanism with prioritisation without creating a queue for each message type/priority level, or at least manage the complexity of multiple queues on the listening side. A queue generally guarantees a consumer. Rather than have the publisher have to push to different queues I would like to use a topic and use filters to manage priority.
Based on my current knowledge of the SB I suspect that I just need to make sure the subscriptions are in place for a topic including an inverse catch-all filter subscription before exposing the topic for use. I don't know whether subscriptions are completely reliable.
However I need to guarantee that all messages sent to the Topic will be received and successfully processed by at least one subscription, even if all of the subscribing processes go offline.
There's a problem in that statement. Topics and subscriptions are there to implement pub/sub and decouple publishers from subscribers. The broker itself does not guarantee there will be subscribers.
While topics support EnableFilteringMessagesBeforePublishing (TopicDescription.EnableFilteringMessagesBeforePublishing) it is not recommended for production use.
Update
Based on the updated question, the general answer remains the same. Topics/subscriptions are for pub/sub and decoupling. If you want to ensure that no message is lost once subscriber is coming online, you will need to ensure that subscription is created first.
I don't know whether subscriptions are completely reliable.
Yes, subscriptions are reliable. Behind the scenes subscription is a queue.
In case you want to route your messages to different processors based on message type, publishing that message to a topic and having forwarding subscriptions is a good approach. You do need to be mindful of the quotas (how many subscriptions per topic you can create), but those are fairly high. And if you get to that point, it's possible to reduce number of subscriptions when a given processor handles multiple message types by having more complex SQL filtering rules.
My scenario: I have an Azure Storage Queue where messages can come in at any time. If I have 10 items in that queue, it's imperative that they be processed in order. I'm using c# and the windows azure storage SDK.
If the first item fails after, say, 2 seconds it remains invisible on the queue for another 28 seconds (30 second invisibility by default).
Now, my worker will just continue to check a queue for messages and process them as and when. If a queue message fails, it remains invisible and so the next queue item will be processed before the first message is retried.
This seems like really basic functionality for anyone needing a queue where the items are processed in order.
No, I can't set the timeout to a smaller amount because tasks can take varying lengths of time.
George, if you are looking for a messaging queue solution that processes items in order, you should consider using Azure Service Bus Queues:
As a solution architect/developer, you should consider using Service Bus queues when:
Your solution must be able to receive messages without having to poll the queue. With Service Bus, this can be achieved through the use of the long-polling receive operation using the TCP-based protocols that Service Bus supports.
Your solution requires the queue to provide a guaranteed first-in-first-out (FIFO) ordered delivery.
You want a symmetric experience in Azure and on Windows Server (private cloud).
For more information, see Service Bus for Windows Server.
Your solution must be able to support automatic duplicate detection.
There is a good article comparing both Storage Queues and Service Bus: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted , you may find the latter better suitable for your case.
I was under the impression that this was not available with storage queue but after investigating I can't find proof of this.
MSDN articles say At-Least-Once but the most information I can find is that the first consumer gets the message and sets the message to invisible.
Then when it becomes visible again it could be picked up again.
However I could set invisible to a large TimeSpan and I could check Dequeue count to limit it to At-Most-Once delivery.
This is using the assumption that competing consumers can't grab the same message at the same time which I can't verify.
If your question is whether Storage Queues offer at most once delivery, the answer is no. If you need at most once, use Service Bus queues. See the Foundational Capabilities section here: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted