I am using Consumption Plan Function App.
I have IoT Devices which are communicating with the IoT Hub. The IoT Hub triggers an Azure Function from my Function App.
The image below was obtained from the Azure Function App setting, and it shows the IoT Hub-triggered function has an execution count of over 250.
Does this mean that there are 250 instances of Azure Function App? Normal?
If was to introduce batch processing for IoT Hub messages, what would be classified as batch of messages? Do they need to have a timestamp within a certain limit?
Your Consumption Plan Function App won't scale over 200 instances (100 for Linux), that's a hard limit. You can use the Function's metrics to check the amount of instances running. Select a metric and split it by 'instance'
Edit with info from comments:
It's possible that you're handling all those requests from 1 instance. The Function will scale out automatically. How that works is described in the docs; however, no exact logic for scaling is described. it's not a hard link to the number of messages available on the Event Hub.
Related
Suppose I have an Azure Function App in a Consumption Plan listening on an Azure Event Hub. According to this documentation, the number of function app instances can scale down to 0 if there are no incoming events. In my understanding, every function app instance also instantiates an Event Processor Host which is listening on partitions of the event hub. But if the number of instances is 0, who is then listening on the event hub to determine if messages must be processed? Does an Event Processor Host exist when the number of instances of a function is 0? Here is the (incomplete) picture I have so far about Azure Functions + Azure Event Hubs which I'd like to complete:
In said documentation I also read about the scale controller which apparently adds new function host instances when needed. Does this mean, the scale controller must also listen on the event hub? Where is the scale controller located, is it part of the function app resource I can create in Azure or is the scale controller hosted in an isolated part of Azure independent of the existence of my function app?
Edit: Adopted image based on answers.
This is what the managed Function runtime on Azure is doing for you. It listens to the Event Hub (or Service Bus, or other supported trigger sources) for you - and you do not have to pay for that compute. Once it detects new messages, it will spin up one or more instances of your Function - at which point only the billing starts.
Source
If you are running Functions somewhere else, e.g. in a k8s cluster, you have to host the scale controller yourself and keep it running all the time.
I am using the Azure function to process message from IOT hub and output to Blob storage.
enter image description here
But the function missed IOT messages when I send in high frequency.
For example, I send 30 messages from 20:40:16 to 20:40:23 but only 3 are processed and stored in to the Blob storage and I have no ideal where the rest 27 went.
enter image description here
I am using the function consumption plan and Azure declares it will auto scaling depends on the load.
But from the above activity log, it only one thread is running and not even queue the input and cause some message lost.
So, what I should do to catch all messages from IOT hub?
Found the solution myself.
The trigger need to change from Azure Event Hubs to Event Grid Trigger as the images show below.
Azure Event Hubs
Azure Grid Trigger
Azure Functions on a consumption plan can handle this load, but you might want to make a separate Consumer Group in your IoT Hub that the Function can use. In the Azure Portal, go to Built-in endpoints and add a new Consumer Group.
You then have to specify in your Function which Consumer Group to use
[FunctionName("Function1")]
public static async Task Run([IoTHubTrigger("messages/events",ConsumerGroup = "functions", Connection = "EventHubConnectionAppSetting")]EventData message,
I tested this with a consumption plan Function listening to IoT Hub default endpoint and writing to blob storage with an 8 second delay to make it more like your function. I'm seeing no loss in messages, whether I send 30 or 100 messages. Make sure that no other applications are using your new Consumer Group!
I have an Azure Event Hub over which I would like to send various types of messages. Each message should be handled by a separate Azure Function, based on their message type. What is the best way to accomplish this?
Actually, I could create some JSON container with a type and payload property and let one parent Azure Function dispatch all the messages payloads - based on their type - to other functions, but that feels a bit hacky.
This question basically asks the same - however it is answered how it can be done using the IoT Hub and message routing. In the Event Hub configuration I cannot find any setting to configure message routing though.
Or should I switch to an Azure Message Queue to get this functionality?
I would use Azure Streaming Analytics to route it to the different Azure Functions. ASAs allow you to specify Event Hubs as a source and several sinks (one of which can be multiple Azure Functions). You can read more about setting up Azure Streaming Analytics services through the Azure Portal here. You'll need to set up the Event Hub as your source (docs). You'll also need to set up your sink (docs). You write some MS SQL-like code to route the messages to the various sinks. However, ASAs are costly relative to other services since you're paying for a fixed amount of compute.
I put some pseudo code below. You'll have to swap it out based on how you configure you're ASA using the information from the attached MS Documentation.
SELECT
*
INTO
[YourOutputAlias]
FROM
[YourInputAlias]
HAVING
[CONDITION]
SELECT
*
INTO
[YourAlternateOutputAlias]
FROM
[YourInputAlias]
HAVING
[CONDITION]
Based on your additional info about the business requirements and assuming that the event size < 64KB (1MB in preview), the following screen snippet shows an example of your solution:
The concept of the above solution is based on the pushing a batch of the events to the Event Domain Endpoint of the AEG. The EventHub Trigger function has a responsibility for mapping each event message type in the batch to the domain topic before its publishing to the AEG.
Note, that using the Azure IoT Hub for ingestion of the events, the AEG can be directly integrated to the IoT Hub and each event message can be distributed in the loosely decoupled Pub/Sub manner. Besides that, for this business requirements can be used the B1 scale tier for IoT Hub ($10/month) comparing to the Basic Event Hubs ($11.16).
The IoT Hub has built-in a message routing mechanism (with some limitations), but a recently new feature of the IoT/AEG integration such as publishing a device telemetry message is giving a good support in the serverless architecture.
I ended up using Azure Durable Functions using the Fan Out/Fan In pattern.
In this approach, all events are handled by a single Orchestrator Function which in fact is a Durable Azure Function (F1). This deserializes incoming JSON to the correct DTO. Based on the content of the DTO, a corresponding activity function (F2) is invoked which processes it.
I have an Azure function that is triggered every 1 second.
Every time it is triggered, it reads a batch of messages from a Service Bus queue and processes them.
It runs on App service plan which scales on the active messages in the queue.
However as the service plan scales out I do not see any increase in the throughput of the function.
Does the Time triggered Azure function scale as the increase in the instances of the App service plan?
No, Timer triggers are singletons. That means that at any given moment only one instance will be firing Function calls on timer.
Obviously, different Functions have independent invocations.
To scale processing of Service Bus messages you should use Service Bus Trigger directly, which can manage scaling for you.
I'm playing around with creating a Pub/Sub system using Azure Service Bus Topics and an Azure Function app where the functions are decorated with ServiceBusTrigger. I'm using the standard tier for Service Bus and Consumption Plan for the Function app.
The concept is working fine and will serve my purposes, however, I'm having difficulty understanding how this will affect my Azure costs.
Running just one function with that attribute seems to generate a couple request per second to the Service Bus. In the end, I expect to have quite a few Topics with Subscriptions and an Azure Function for each Subscription.
If I understand correctly, each of the requests that the ServiceBusTrigger generates counts against the number of Service Bus operations you get. With a few functions, I'll go over 13M ops pretty easily.
Some of these subscribers aren't so time sensitive that they'd need to check the Subscription several times per second. Is there a way to change the frequency that ServiceBusTrigger checks the Subscription to reduce costs? Am I approaching this problem from entirely the wrong angle?