Can I trigger multiple azure function on a single queue trigger.
Use case is that i am storing auth token in a queue, and multiple functions take up the token to call the different endpoints respectively.
Or will the first one grab the message (token) and remove it from the queue.
In this scenario, I would use ServiceBusTrigger function.
You can create a topic in a Service Bus, then for this one topic you can have multiple subscribers.
So, even if 1 subscriber has finished going trough the messages, you can still have a subscriber which has not even started once and don't worry about this subscriber missing any message.
EDIT
Useful links
ServiceBusTrigger Example
Topics and Subscribers
#dhruv Yes you can create multiple functions with the same queue, all works in parallel as likewise event grid subscribers, which queue you're using, is it storage queue or service bus etc?
You could use a durable function for this.
Related
We’re trying to create a functionality where we can create individual user queues so that the downstream apps can process each user messages at a different pace in-order.
And when any of those queues have messages, we’d like to trigger the same Azure function for all the queues.
Not supported by ServiceBusTrigger on Azure function, since it requires a function per queue to be configured at deployment time.
All suggestions welcome!
PS: We’re okay to deploy new pieces of wiring when new queues are created.
Using Azure Service Bus - Topics, I want to implement a solution wherein messages are sent/notified to end consumers once the producer sends the message to Topic (like Queues).
I understand that Topics work as Pub/Sub model wherein subscribers need to read messages from subscriptions. But I'm looking for a workaround that works some what similar to Queue (where it triggers a web job / service when any message is received).
I have few thoughts like
1. Using Auto-Forwarding in subscriptions to forward messages to Queues but again I think if this kills the purpose of Topics
2. Schedule a job to process these requests but again I think if I'm delaying the process
First, I want to know if Service Bus Topic is right option to go with? Next, If possible to implement a workaround what is the best/better way?
PS: I have to send messages which has information - I guess I can't use Relays
Just to be clear, Queues and Topics in Service Bus are different. As you noted, Topics are useful in publish/subscribe scenarios.
Since you are looking for something that gets triggered, Azure functions might be what you need.
Azure Functions supports trigger and output bindings for Service Bus
queues and topics
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus
I think that #William is right, you can use/attach other process to the subscription to make what you trying to do.
he mentioned Azure Functions which is a good tool and I want to suggest Azure Logic Apps as well in case you want to take some decisions based in the message that you received.
With Azure Logic Apps you can create a logic Workflow and integrate many services using connectors provided by this tool.
You will find more in:
https://learn.microsoft.com/en-us/azure/connectors/connectors-create-api-servicebus
And for answer your question
First, I want to know if Service Bus Topic is right option to go with?
The quick answer is yes, using messaging patterns is the best way to create reliable solutions.
In your case you want as well notify another system after receiving a message.
The only thing that you need to be aware is, whenever you did not receive the notification what you'll do? you need to think about this scenario.
From discussion above.
Azure functions with Queues/Topics
Regardless of queues or topics. you can trigger azure function with both. This function will process the message . Now you can create two methods in same function SendEmail(), sendPhoneNotifcation() and parrellize the tasks using C# task parallel library. So same function will do both tasks in parallel.
Every time you get a message , your function is triggered. Process the message and notify user. The benefit is this function will scale automatically if you have large number of message coming through queue.
In Azure, we have two separate messaging technologies and it's not very well documented when to use what? While EventGrid is really cool, I did not come across when to use EventGrid(scenarios) vs the Storage/ServiceBus queue? Can someone help?
E.g. if I have the following scenario :
A status of a flag changes and based on that, I want to trigger an algorithm that would do recalculations, few inserts/updates etc. in the database.
For implementing this - I can either use EventGrid or Storage Queue. How do we figure what to use in such scenario? I was looking for some kind of guidance.
Basically, Azure Event Grid handles events and Azure ServiceBus handles messages.A message is raw data produced by a service to be consumed or stored. Events are also messages (lightweigth), but they don’t generally convey a publisher intent, other than to inform.
1) If the purpose is to just to store the information ServiceBus can be used.
2) If the information received is used to trigger another service Azure Event Grid can be used.
Find more info here
https://learn.microsoft.com/en-us/azure/event-grid/compare-messaging-services
https://azure.microsoft.com/en-us/blog/events-data-points-and-messages-choosing-the-right-azure-messaging-service-for-your-data/
Events are like notifications from a service to inform the world that something happened in the domain of the publisher (similar to an email notification). There is no expectations from the publisher to have any actions taken. A message is a command you send to a specific receiver with the expectation of the message to be processed (like an asynchronous post request).
Events will work in pub/sub pattern and multiple subscribers could be configured to the events. The service that needs to react to an event will get notified by the event grid when an event occurs (http call from event grid to the receiver). The event will remain in the event grid until deletion (cleanup) and there is no garantie of keeping the original order (no FIFO).
In the other hand, messages will be added to a queue and will be deleted once the “message processor” is done with it. The messages in the queue will keep the original order (FIFO). The message processor has to pull messages from the queue.
In your scenario, you could use a combination of both. Service A sends an event “StatusChanged”, then you can configure a subscription to that event and send a message to a queue, then have your logic to process that message. This will end up with a fully async communication pattern. This is ideal to support scenarios where you processor is down or too busy. The incoming messages will simply get accumulated in the queue and eventually being processed once the service is back up and running. And without affecting the original service that sent the “StatusChanged” event..
Is there a way to make an Azure function triggerable by multiple Service Bus event queues? For example, if there is a function which logic is valid for multiple cases(event start, event end- each inserted into a different Service Bus queue) and I want to reuse it for these events can I subscribe to both of them in the Service Bus from the same function?
I was looking for an answer to this question, but so far everywhere I checked it seems to be impossible.
Azure Functions can be triggered by a single source queue or subscription.
If you'd like to consolidate multiple sources to serve as a trigger for a single function, you could forward messages to a single entity (let's assume a queue) and configure Function to be triggered by messages in that queue. Azure Service Bus support Auto-Forwarding natively.
Note that there cannot be more than 3 hops and you cannot necessarily know what the source was if message was forwarded from a queue. For subscriptions, there's a possible workaround to stamp messages.
If your goal is to simply reuse code, what about refactoring that Function to create a class which is then used in multiple functions.
If your goal is implementing events aggregation, you could probably create an Azure Durable Function Workflow that would do a fan-in on multiple Events.
Excerpt from https://github.com/Azure/azure-functions-durable-extension/issues/166:
Processing Azure blobs in hourly batches.
New blob notifications are sent to a trigger function using Event Grid trigger.
The event grid trigger uses the singleton pattern to create a single orchestration instance of a well-known name and raises an event to the instance containing the blob payload(s).
To protect against race conditions in instance creation, the event grid trigger is configured as a singleton using SingletonAttribute.
Blob payloads are aggregated into a List and sent to another function for processing - in this case, aggregated into a single per-batch output blob.
A Durable Timer is used to determine the one-hour time boundaries.
You might want to consider switching the pattern around by using only one queue but multiple Topics/Subscriptions for the clients.
Then the Function in question can be triggered by the Start-End Topic.
Some other Function can be triggered by the Working Topic, etc.
I have a web api application which performs different type of actions an a Domain entity, called Application. Actions like "Copy", "Update", "Recycle", "Restore" etc.
This actions needs to be executed, per Application, in First In First Out order, not randomly or simultaneous. However, it can process simultaneously two Actions as long as they are for two separate Applications.
Is some kind of a queue, but not a big queue for all the requests, but a queue of actions for each Application in database.
Knowing this, i think that azure service bus queue is a good solution for this scenario.
However, the solution i can think of right now is to programmatically create a queue for each Application i have in database, and start listening to that queue.
Is possible to get messages from the queue based on a filter? (using FIFO principle) So i have to subscribe only to one queue? (instead of having a queue for each Application - which is very hard to maintain)
What you want is Azure Service Bus Topics/Subscriptions.
Subscriptions allow you to filter messages that are published to a topic using a SqlFilter on the message headers.
The article linked above should provide enough examples to meet your needs.
I think u can solve this by using Sessions.
I just came across this very clear article: https://dev.to/azure/ordered-queue-processing-in-azure-functions-4h6c which explains in to detail how Azure Service Bus Queue sessions work.
In short: by defining a SessionId on the messages you can force the ordering of the processing within a session, the downside is that there will be no parallelization for all messages in a session between multiple consumers of the queue.