Let's say I've got an azure service bus in a microservice scenario.
One microservice pushes master data changes to the other services with a subscription.
Now let's say a new service is introduced and subscribes to the master data service. How can I make sure that the new service receives all neccessary data?
Do I have to resend all master data on the master data service or does the azure service bus (or alternatives) provide some features for that?
As far as I know there is no way to achieve what you want within the capabilities of Azure Service Bus. Also, I don't think this what Service Bus is there for.
Of course there is a configurable "time to live" value for messages within queues and topics, which could probably be set to some really high value, but this would still not make your master data be infinitely available for future services. And - but this is just my opinion and I'm far from being an expert - I wouldn't want to load up my service bus with potentially thousands or even millions of messages (depending on what you're doing) without them being processed quickly.
For your specific concern I'd rather implement something like a "master data import service" without any service bus integration. Details of this, however, depend on your environment and specific requirements.
Couple of points:
1) This is not possible with Azure Service bus. Even If you set TTL at Topic level, the messages will only be delivered to available subscriptions at that point of time. you cant read messages directly from Topic.
2) you can consider Eventhub option where you can create new consumer group with offset from when you want to start reading messages but Eventhub has maximum retention period as 7 days. If you need message retention beyond 7 days, enabling Event Hubs Capture on your event hub pulls the data from your event hub to the Storage account. But in this case you would require additional logic to read from this storage account to replay the messages.
Related
Currently I am using Azure Service Bus as a means to communicate and keep data consistent among the different services in my platform. However, let's say that one of my services (subscribers) goes down for an extended period of time and is unable to receive any events. Suddenly this service is in an inconsistent state.
Does Azure Service Bus have any type of "event sourcing" solution in place in order to replay my events? I understand that Azure Event Hubs has this feature where I can store events in an append only fashion to azure blob storage. However, the only thing I am finding for Azure Service Bus is the dead letter queue and my understanding that this is only used when no subscribers are capable of processing an event.
Is this something that I will have to build myself?
All events stored in a subscription will be delivered once the consumer is up and running unless the subscription has DefaultMessageTimeToLive (TTL) set to purge messages.
Our application consist of 7 microservices that have some intercommunication. Currently we're using simple storage queues that a microservice publish events to (the number of events is relative low). Then we have a azurefunction for each queue that might call another microservices. This is working fine for us right now the services uses about 20 queues with a corresponding function.
Now we need to handle an blobstorage event, and I did some googling and a started to get really confused. Suddenly there was a lot of questions:
Should we switch to Azure Event Grid
It handles blobstorage without any limitations (functions blobstorage trigger has some)
It allows for multiple subscribers (storage queues does not)
It has a lot of fuz - maybe this is the new recommended way
I like the idea of one central thing, but it reminds me a bit about biztalk...
Should I switch to Azure Service Bus
It has a nice tool (ServiceBusExplorer) for monitoring the queues and listners, and I could to a repost of any failed events
It visulizes my azure functions subscribers nicely
Should I continue with only storage queues
A bit difficult to monitor, but it works nice
I'll be really thankful for any advice or insights to this question.
Thanks
EventGrid is great when you have notifications floating to multiple subscribers. Is that the case for you?
An example would be deferring messages. With queues you can defer a message, not with EventGrid. Whenever to choose Storage Queues or Service Bus depends on the specific requirement that you have. Do you need de-duplication? Or ordered delivery? If you do, Service Bus is the way. Otherwise Storage Queues is enough.
First of All, I would like to recommend these two articles, it will clarify most of your doubts about these services:
Choose between Azure services that deliver messages
Storage queues and Service Bus queues compared
Regarding Event Grid, it acts like a bridge between the publisher and the subscriber, where publisher will send messages and forget whether it has been processed or not, and the Event Grid will handle the retry if the receiver\subscriber does not acknowledge that it was processed successfully.
As you mentioned, storage queues has limitations, as such blob triggered functions, and maybe Service Bus, but it will depend on your design requirements. I would like to point out some things you might consider before moving to Event Grid.
Storage queues & Service Bus does not care about your message schema, in Event Grid you have to create a custom event based on their schema to wrap your event, so the publisher and subscriber has to understand Event Grid for that, not that is a big deal, but now you have both sides coupled to Event Grid.
If you want to send the event straight to your micro-service, you have to implement the subscription validation in your service, otherwise the service won't be able to receive the events
Event Grid only retry the delivery of your messages for 24 hours, if your service is down or not process the message correctly for longer than 24h, it will make the event dead. Currently, there is no way to query dead messages. Storage Queues and Service Bus are configurable how long you keep the message and it can be kept for many days.
Your service web-hook must acknowledge the receipt(http 200 or 202) of an event within 60 seconds, otherwise it will consider failed. If your operation is longer that that, you should send it to a queue and handle the locking from your service.
Probably there are more limitations, but these are the ones I remember right now that might change anytime soon, I think Event Grid is a great technology still on early days, and there is much to improve, I would recommencement only as a hub for Azure management events, I don't think it is ready for use as an application integrator.
Regarding your comment for queue manager, for Service Bus your have the Service Bus Explorer, and for Azure Storage you have the Azure Storage Explorer, where you can check the messages in the queue, is not the same as Service Bus, but helps.
It very much depends on how are you consuming the queue messages, you can take a look at this comparison: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted
If you don't need ordering and if you don't have a strong limit on message volume, size or TTL, you can stick to storage queues.
I'm currently building a hybrid-cloud solution that needs to write messages to a queue for later processing. It is absolutely imperative that the queue is highly available (99.999+% uptime).
My options are to read/write messages to a local ZeroMQ high availability pair, or an Azure Service Bus. I would prefer to go the Azure Service Bus route, but can't find any documentation regarding high availability configuration for Azure Service Bus.
Has anyone had success setting up Azure Service Bus for high availability? I understand that the SLA for a single instance of any Azure service cannot be changed. I'm thinking more along the lines of the failover capabilities of Azure Web Apps.
The main thing you can do for consuming a service at a higher than SLA value is to ensure you are handling retry logic. The key here will be the temporal nature of any outage, and tuning a retry backoff to handle edge cases. Some use linear or exponential backoffs to wait even longer for the service to come back up.
Also, you can have more than one service bus in a different region for georedundancy, and either load balancing messages across the two or use one as a hot backup. This can get you around any regional outages and keep your service up when one data center is not meeting its local SLA.
You can find the for SLA for Azure Service Bus here: legal/sla/service-bus/v1_0/
For Service Bus Relays, we guarantee that at least 99.9% of the time,
properly configured applications will be able to establish a
connection to a deployed Relay. For Service Bus Queues and Topics, we
guarantee that at least 99.9% of the time, properly configured
applications will be able to send or receive messages or perform other
operations on a deployed Queue or Topic. For Service Bus Basic and
Standard Notification Hub tiers, we guarantee that at least 99.9% of
the time, properly configured applications will be able to send
notifications or perform registration management operations with
respect to a Notification Hub. For Event Hubs Basic and Standard
tiers, we guarantee that at least 99.9% of the time, properly
configured applications will be able to send or receive messages or
perform other operations on the Event Hub.
We've had Service Bus Relay up and running for 5+ years and have had one outage. It was an outage at the specific data center the relay was provisioned in and touched many services. After that we implemented redundancy by implementing a secondary Service Bus Relay namespace in a different data center location. The reconfigured code was set to check the connectivity on every connection and switch the primary and secondary connections. We treated them as equals so once we "failed over" that namespace would become primary.
Service Bus now supports Geo-disaster recovery and Geo-replication at the namespace level.
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-geo-dr
I'm looking to use Azure Service Bus with topics but need to handle the scenario where a subscriber might not be listening for a message it's interested in (e.g. server being rebooted etc.). This is the typical durable subscriber pattern as described here http://www.eaipatterns.com/DurableSubscription.html.
What I can't work out is how to apply this with Azure Service Bus and I can't seem to find any examples or discussion of this in the documentation. Is this something that Azure service bus provides or should I start looking at alternatives to Azure Service Bus?
This is built straight into Service Bus. As long as a subscription is created it is durable. You create a topic and then create one or more subscriptions. One or more consumers then listen to a subscription when they are active. If they go inactive, such as the server being rebooted, then the subscription stores the messages until a consumer comes back up and asks for one.
Service Bus would only be nondurable if you were creating and destroying subscriptions on the fly as each consumer becomes active or becomes inactive. If there are no subscriptions then messages sent to a topic are lost. Once you create a subscription, any messages sent to the topic (if they pass any filters applied) will be available on the subscription regardless if there are any active consumers using that subscription. Subscriptions exist until you remove them or, if you have the idle removal feature turned on, they surpass the idle deletion time.
You can verify this with a simple console application, or using LinqPad to set up code that does the following:
Create a topic.
Create a subscription on that topic (no filters)
Send a few messages to the topic.
In a different script or console app, create a MessageReceiver for that subscription and pull down the messages.
The messages within a subscription are durable for the life of that subscription, until they are processed (completed, etc.), they are forwarded somewhere else or they expire.
I am not sure where you looked for documentation, following are good to read:
1) http://azure.microsoft.com/en-us/documentation/articles/service-bus-dotnet-how-to-use-topics-subscriptions/
2) http://code.msdn.microsoft.com/windowsazure/Simple-Publish-Subscribe-d406eb03
I have a pusher which brokers message x into an Azure Service bus. This is a customer request for some order.
A consumer comes along and wants to know "what is the state of message x"? I'd like to know as much as I can about message x, but at a minimum I need to answer "yes/no" is it still in the queue? Is there a recommended pattern for this within Azure Service Bus?
In View content of an Azure Service Bus queue Clemens Vasters explains that there is no browse feature available in Service Bus Queues/Topics at the moment. The only way to know the state is by storing it externally (take Table Storage for example).