Is there a way to make an Azure function triggerable by multiple Service Bus event queues? For example, if there is a function which logic is valid for multiple cases(event start, event end- each inserted into a different Service Bus queue) and I want to reuse it for these events can I subscribe to both of them in the Service Bus from the same function?
I was looking for an answer to this question, but so far everywhere I checked it seems to be impossible.
Azure Functions can be triggered by a single source queue or subscription.
If you'd like to consolidate multiple sources to serve as a trigger for a single function, you could forward messages to a single entity (let's assume a queue) and configure Function to be triggered by messages in that queue. Azure Service Bus support Auto-Forwarding natively.
Note that there cannot be more than 3 hops and you cannot necessarily know what the source was if message was forwarded from a queue. For subscriptions, there's a possible workaround to stamp messages.
If your goal is to simply reuse code, what about refactoring that Function to create a class which is then used in multiple functions.
If your goal is implementing events aggregation, you could probably create an Azure Durable Function Workflow that would do a fan-in on multiple Events.
Excerpt from https://github.com/Azure/azure-functions-durable-extension/issues/166:
Processing Azure blobs in hourly batches.
New blob notifications are sent to a trigger function using Event Grid trigger.
The event grid trigger uses the singleton pattern to create a single orchestration instance of a well-known name and raises an event to the instance containing the blob payload(s).
To protect against race conditions in instance creation, the event grid trigger is configured as a singleton using SingletonAttribute.
Blob payloads are aggregated into a List and sent to another function for processing - in this case, aggregated into a single per-batch output blob.
A Durable Timer is used to determine the one-hour time boundaries.
You might want to consider switching the pattern around by using only one queue but multiple Topics/Subscriptions for the clients.
Then the Function in question can be triggered by the Start-End Topic.
Some other Function can be triggered by the Working Topic, etc.
Related
We’re trying to create a functionality where we can create individual user queues so that the downstream apps can process each user messages at a different pace in-order.
And when any of those queues have messages, we’d like to trigger the same Azure function for all the queues.
Not supported by ServiceBusTrigger on Azure function, since it requires a function per queue to be configured at deployment time.
All suggestions welcome!
PS: We’re okay to deploy new pieces of wiring when new queues are created.
I am designing Http trigger function. This function will be used to authenticate and to call an external API. Then get the response and pass it on to the caller(~in this case web application).
This function can be called thousand time/min(~during peak load).
I am going through event grids and having hard time deciding why to use it.
Microsoft says to safeguard your events so that it's not lost due to any reason I should use it along with my http trigger azure function.
I can very well design a queue trigger function which will process these requests in queue.
I am referring to this article of MS which says to use Event grid to gain more control on your serverless function for example:
MS Event Grid
I am going through event grids and having hard time deciding why to
use it.
I can very well design a queue trigger function which will process
these requests in queue.
I think this needs to be based on your needs. The event grid is discrete based on event triggers. And event grid has higher scalability.
If you use a queue trigger, it is not triggered based on an event, is it? All in all, it is not necessary to use event grid. Please refer to your use case for specific use.
If you want a comparison study of the messaging services you may refer to : https://tsuyoshiushio.medium.com/azure-messaging-service-in-a-picture-f8113cec54cd
https://hackernoon.com/azure-functions-choosing-between-queues-and-event-hubs-dac4157eee1c
For event grid, Azure Event Grid is an eventing service for the cloud. Azure Functions is one of the supported event handlers.
Azure Event Grid allows you to easily build applications with event-based architectures. First, select the Azure resource you would like to subscribe to, and then give the event handler or WebHook endpoint to send the event to. Event Grid has built-in support for events coming from Azure services, like storage blobs and resource groups. Event Grid also has support for your own events, using custom topics. reference
Can I trigger multiple azure function on a single queue trigger.
Use case is that i am storing auth token in a queue, and multiple functions take up the token to call the different endpoints respectively.
Or will the first one grab the message (token) and remove it from the queue.
In this scenario, I would use ServiceBusTrigger function.
You can create a topic in a Service Bus, then for this one topic you can have multiple subscribers.
So, even if 1 subscriber has finished going trough the messages, you can still have a subscriber which has not even started once and don't worry about this subscriber missing any message.
EDIT
Useful links
ServiceBusTrigger Example
Topics and Subscribers
#dhruv Yes you can create multiple functions with the same queue, all works in parallel as likewise event grid subscribers, which queue you're using, is it storage queue or service bus etc?
You could use a durable function for this.
I have one requirement to do file merging based on event driven architecture. I am having two blob containers and i need to merge files as soon as they are available in their respective containers. Correlation will happen based on file name.
That means suppose i have two containers, container A and container B. When file comes to container A then it should wait for the file to come in container B and then event should trigger which will get subscribed by ADF or logic app for further processing. Please suggest some way to achieve this.
Event Grid Microsoft.Storage.BlobCreated event will be raised per container and will not wait for another container to raise an event.
One option I could think of is to handle your events using Durable Function where you could use Correlation value as Durable Function instance ID to dentify an existing function or start a new one. If a function instance with a given ID is already running, you'd be able to either perform the merge or raise a new custom event and handle it separately.
another option is creating a simple Distributed Event Aggregator, like is shown in the following screen snippet:
The concept is based on the Lease Blob where is stored the State of EventAggregator with all received event messages. The HttpTrigger function has a responsibility for handling and managing received event messages, create and update the State and handling a retry delivery for reliable updating a State. In this case, the dead-lettering is used as a Watchdog Timer.
Creating or Updating a Lease Blob will generate an event for subscriber and its logic can see the State of EventAggregator with an array of received event messages.
In the case when we are not using an eventing for Lease Blob (separate storage account), the finally event message with an EventAggregator State can be sent to the Storage queue by HttpTrigger function - EventAggregator.
I can get the messages from all the partitions of event hub in azure function but I want to get messages from a particular event hub partition in azure function. Is there a way to do that ? And one other thing I want to do is to increase (scale out) the number of azure functions to process messages if there are large number of backlogs messages to process. How can I do that ? Is there any formulae to solve my second problem ?
In the Azure Functions Consumption plan, scale out is handled automatically for you. If we see that your function is not keeping up with the event stream, we'll add new instances. Those instances will cooperate to process the event stream in parallel.
For reading of the event stream, we rely on the Event Hubs EventProcessorHost as described in their documentation here. This host manages coordination of partition leases with other instances when the Function App starts - this isn't something you can (or should want to) control.