I am using azure service bus topic and I have enable session for it's subscription.
With in the my logic app i am inserting data using sql transaction which are coming from topic
I am using topic Subscription(peek-lock) and
in subscriber level concurrency is set to default as follows
According to my Understanding, my logic app(subscriber) should read ALL the messages and have to processed in FIFO
my logic app is like
which means it should insert data to the table in ordered manner
however when i checked in the trigger log it shows correct order but in database level you can see the order is not happening
Message ordering is a delicate business. You can have either message ordering or concurrent processing, but not both. The moment you make message ordering a must, you lose the ability to have concurrent processing. This is correct for both, Azure Service Bus Sessions and Logic Apps concurrency control. You could process multiple sessions, but each session would be still restricted to a single processor. Here's a post about it.
Related
I am working on a service fabric application and want to publish few events from this application and subscribe or process those publish events in another application.
I have tried EventGrid concept and observed that there is a delay while publishing and processing the events. So, now I am looking for other alternatives like EventHub or Queues, etc..
If anyone had already used EventGrid, EventHud or Queues, etc.. , Please do suggest which one will give more performance when we deal with more events.
Design Approach
We have migrated the tables from SQL service to Service Fabric. There is a view in SQL Service, and we are planning to implement that as a service in service fabric.
The implementation logic follows below.
Table 1 implemented service and we publish an event for each CRUD operation to EventGrid/ EventHud.
Table 2 implemented service and we publish an event for each CRUD operation to EventGrid/ EventHud.
We have created a view service where it listens to the events when any event sent to EventGrid/ EventHud, it will perform required calculations and store in the ViewService( It is a background job)
We are looking for a messaging service which gives more performance.
Have you seen this comparison and this one?
Anyway, can you clarify your requirements in terms of throughput and performance? It depends on a lot of factors including, but not limited to, the message size and the amount of messages.
Having used both Event Grid and Event Hub I'd say Event Hub works very well for many messages per second, say data streams from iot devices, but the performance of the downstream processing can be a bottleneck. You have to process them very fast in order to receive new events. Then there are partitions and consumer groups that can be of help to balance the load and have different processors for the same data but with different view of the data stream. (A fast processor for live displaying of sensor data and a slower one for storing the data for later analysis)
If you're talking about a few events generated by an application that triggers other apps to start doing some work based on those events Event Grid is a good fit. I haven't experienced much delay in receiving those events.
But bottom line, I think all services (Event Grid, Event Hub, Service Bus etc) support different use cases and that should be your first decision point.
Can you describe your publisher, subscriber, etc. and show your metrics of the Azure Event Grid usage?
You can use the portal screen snippets on the topic (publisher) and subscription (subscriber).
The following screen snippets are from my tester when manually have been fired few events.
Publisher side:
Subscriber side:
Metrics on the portal:
As you can see, the delivery destination processing time is ~1ms. The latency time on the publisher side (custom topic) is between 2-4ms.
Note, that the AEG is a PUSH->PUSH-ACK or PUSH->PULL-ACK eventing loosely decupled Pub/Sub model instead of the Event Hub model which is based on the PUSH->PULL mechanism, in other words, the Event Hub needs to host a listener and receiver for pulling an event from the partition.
I have an online service hosted on Azure, that asynchronously sends data to on-premise clients.
Each client is identified by an unique code.
Actually there is a single topic, with a subscription for each client which has a filter on the unique code, that is sent as a parameter in the message. No message will ever be broadcasted to all the clients.
I feel that using topic this way is wrong.
The alternative that comes to my mind is to use a dedicated queue for each client, that is created on first contact
Could this be a better approach?
Thanks
In my opinion using Topics and Subscriptions is the right way to go. Here's the reason why:
Currently the routing logic (which message needs to go to which subscription) is handled by Azure Service Bus based on the rules you have configured. If you go with queues, the routing logic will need to come to your hosted service. You'll need to ensure that the queue exists before sending each message. I think it will increase the complexity at your service level somehow.
Furthermore, topics and subscriptions would enable you to do build an audit trail kind of functionality (not sure if you're looking for this kind of functionality). You can create a separate subscription that has a rule to deliver all messages (True SQL Rule) to that subscription along with client specific subscription.
Creating a separate Queue for each client is not advisable. This is the problem solved by Topics.
If you have separate Queue for each client, then you need to send messages to multiple Queues from Server. This will become tedious when the number of clients increases.
Having a single Topic and multiple Subscriptions is easy to manage as the message will be sent only to a single Topic from Server.
Our application consist of 7 microservices that have some intercommunication. Currently we're using simple storage queues that a microservice publish events to (the number of events is relative low). Then we have a azurefunction for each queue that might call another microservices. This is working fine for us right now the services uses about 20 queues with a corresponding function.
Now we need to handle an blobstorage event, and I did some googling and a started to get really confused. Suddenly there was a lot of questions:
Should we switch to Azure Event Grid
It handles blobstorage without any limitations (functions blobstorage trigger has some)
It allows for multiple subscribers (storage queues does not)
It has a lot of fuz - maybe this is the new recommended way
I like the idea of one central thing, but it reminds me a bit about biztalk...
Should I switch to Azure Service Bus
It has a nice tool (ServiceBusExplorer) for monitoring the queues and listners, and I could to a repost of any failed events
It visulizes my azure functions subscribers nicely
Should I continue with only storage queues
A bit difficult to monitor, but it works nice
I'll be really thankful for any advice or insights to this question.
Thanks
EventGrid is great when you have notifications floating to multiple subscribers. Is that the case for you?
An example would be deferring messages. With queues you can defer a message, not with EventGrid. Whenever to choose Storage Queues or Service Bus depends on the specific requirement that you have. Do you need de-duplication? Or ordered delivery? If you do, Service Bus is the way. Otherwise Storage Queues is enough.
First of All, I would like to recommend these two articles, it will clarify most of your doubts about these services:
Choose between Azure services that deliver messages
Storage queues and Service Bus queues compared
Regarding Event Grid, it acts like a bridge between the publisher and the subscriber, where publisher will send messages and forget whether it has been processed or not, and the Event Grid will handle the retry if the receiver\subscriber does not acknowledge that it was processed successfully.
As you mentioned, storage queues has limitations, as such blob triggered functions, and maybe Service Bus, but it will depend on your design requirements. I would like to point out some things you might consider before moving to Event Grid.
Storage queues & Service Bus does not care about your message schema, in Event Grid you have to create a custom event based on their schema to wrap your event, so the publisher and subscriber has to understand Event Grid for that, not that is a big deal, but now you have both sides coupled to Event Grid.
If you want to send the event straight to your micro-service, you have to implement the subscription validation in your service, otherwise the service won't be able to receive the events
Event Grid only retry the delivery of your messages for 24 hours, if your service is down or not process the message correctly for longer than 24h, it will make the event dead. Currently, there is no way to query dead messages. Storage Queues and Service Bus are configurable how long you keep the message and it can be kept for many days.
Your service web-hook must acknowledge the receipt(http 200 or 202) of an event within 60 seconds, otherwise it will consider failed. If your operation is longer that that, you should send it to a queue and handle the locking from your service.
Probably there are more limitations, but these are the ones I remember right now that might change anytime soon, I think Event Grid is a great technology still on early days, and there is much to improve, I would recommencement only as a hub for Azure management events, I don't think it is ready for use as an application integrator.
Regarding your comment for queue manager, for Service Bus your have the Service Bus Explorer, and for Azure Storage you have the Azure Storage Explorer, where you can check the messages in the queue, is not the same as Service Bus, but helps.
It very much depends on how are you consuming the queue messages, you can take a look at this comparison: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted
If you don't need ordering and if you don't have a strong limit on message volume, size or TTL, you can stick to storage queues.
I have a web api application which performs different type of actions an a Domain entity, called Application. Actions like "Copy", "Update", "Recycle", "Restore" etc.
This actions needs to be executed, per Application, in First In First Out order, not randomly or simultaneous. However, it can process simultaneously two Actions as long as they are for two separate Applications.
Is some kind of a queue, but not a big queue for all the requests, but a queue of actions for each Application in database.
Knowing this, i think that azure service bus queue is a good solution for this scenario.
However, the solution i can think of right now is to programmatically create a queue for each Application i have in database, and start listening to that queue.
Is possible to get messages from the queue based on a filter? (using FIFO principle) So i have to subscribe only to one queue? (instead of having a queue for each Application - which is very hard to maintain)
What you want is Azure Service Bus Topics/Subscriptions.
Subscriptions allow you to filter messages that are published to a topic using a SqlFilter on the message headers.
The article linked above should provide enough examples to meet your needs.
I think u can solve this by using Sessions.
I just came across this very clear article: https://dev.to/azure/ordered-queue-processing-in-azure-functions-4h6c which explains in to detail how Azure Service Bus Queue sessions work.
In short: by defining a SessionId on the messages you can force the ordering of the processing within a session, the downside is that there will be no parallelization for all messages in a session between multiple consumers of the queue.
I have been struggling with this concept for a while. I am attempting to come up with a loosely coupled Azure component design that is completely scalable using Queues and worker roles, which dequeue and process the items. I can scale the worker roles at will, and publishing to the queue is never an issue. So far so good, but, it seems that the only real world model this could work in is fire and forget. It would work fantastic for logging and other one way operations, but let's say I want to up load a file using queues/worker roles, save it to blob, then get a response back once it is complete. Or should this type of model not be used for online apps? What is the best way to send a notification back once an operation is completed? Do I create a response Q, then (somehow) retrieve the associated response? Any help is greatly appreciated!!!!!
I usually do a polling model.
Client (usually a browser) sends a request to do some work.
Front-end (web role) enqueues the work and replies with an ID.
Back-end (worker role) processes the queue and stores the result in a blob or table entity named .
Client polls ("Is done yet?") at some interval.
Front-end checks to see if the blob or table entity is there and replies accordingly.
See http://blog.smarx.com/posts/web-page-image-capture-in-windows-azure for one example of this pattern.
you could also look into the servicebus appfabric instead of using queues. with the servicebus you can send messages, use queues etc all from the servicebus appfabric. you could go to publish and subscribe instead of polling then!