Available methods to monitor Azure Event hub parititons queue size - azure

Kafka provides capability to monitor current offset and latest offset. Similarly does azure eventhub expose any api to continously monitor partition's current offset and latest available offset?

Extending above answer, you can see offset by 2 ways.
Print offset in log file where you are listening EventHub
e.g using Azure function
public static async Task Run([EventHubTrigger("EventHubname", ConsumerGroup = "ConsumerGroupname", Connection = "EventHubConnection")]EventData eventMessage,
[Inject]IService service, [Inject]ILog log)
{
log.Info($"PartitionKey {eventMessage.PartitionKey}, Offset {eventMessage.Offset} and SequenceNumber {eventMessage.SequenceNumber}");
}
I am listening Eventhub by Azure Functions, You can see below location where Azure function maintain offset by partition.
Option 3 (Latest)
Offset is not the correct way to measure the depth of Eventhub, Specially when you want to check how many messages need to process.
Now We are using Eventhub message SequenceNumber instead Offset. We have created TimerTrigger Azure Function. On every 5 minutes, we are getting LastEnqueuedSequenceNumber from Eventhub and SequenceNumber for each partition from blob storage (checkpoint location), then we storing difference in ApplicationInsight customMetrics.
Then ApplicationInsights help us to PIN Eventhub depth information in Azure dashboard and set up an alert.
Timer Trigger Code
I hope this will help!

Looking at Features and terminology in Azure Event Hubs - Event consumers - Stream offsets:
An offset is the position of an event within a partition. You can think of an offset as a client-side cursor. The offset is a byte numbering of the event. This offset enables an event consumer (reader) to specify a point in the event stream from which they want to begin reading events. You can specify the offset as a timestamp or as an offset value. Consumers are responsible for storing their own offset values outside of the Event Hubs service. Within a partition, each event includes an offset.
And also under Common consumer tasks - Read events:
As events are sent to the client, each event data instance contains important metadata such as the offset and sequence number that are used to facilitate checkpointing on the event sequence.
There do not seem to be any methods you can use to monitor the offset since you need to do this yourself.

Related

Slow down EventHubTrigger in Azure Function

I have a simple Azure function which:
as input uses EventHubTrigger
internally write some events to the Azure Storage
During some part of the day the average batch size is around 250+, which is great because I write less block to Azure Storage, but for most of the time the batch size is less then 10.
Is there anyway to force EventHubTrigger to wait until there are more than 50/100/200 messages to process, so I can reduce append blocks in Azure Storage.
Update--
The host.json file contains settings that control Event Hub trigger behavior. See the host.json settings section for details regarding available settings.
You can specify a timestamp such that you receive events enqueued only after the given timestamp.
initialOffsetOptions/enqueuedTimeUtc: Specifies the enqueued time of the event in the stream from which to start processing. When initialOffsetOptions/type is configured as fromEnqueuedTime, this setting is mandatory. Supports time in any format supported by DateTime.Parse(), such as 2020-10-26T20:31Z. For clarity, you should also specify a timezone. When timezone isn't specified, Functions assumes the local timezone of the machine running the function app, which is UTC when running on Azure. For more information, see the EventProcessorOptions documentation.
---
If I understand your ask correctly, you want to hold all the received events until a certain threshold and then process them at once in a singe Azure function run.
To receive events in a batch, make string or EventData an array. In the binding configuration properties that you set in the function.json file and the EventHubTrigger attribute. Set function.json property "cardinality" to "many" in order to enable batching. If omitted or set to one, a single message is passed to the function. In C#, this property is automatically assigned whenever the trigger has an array for the type.
Note: When receiving in a batch you cannot bind to method parameters with event metadata and must receive these from each EventData object
Further, I could not find an explicit way to do this task. You would have to modify your function, like using a timer trigger or use the blob queue storage API to read all messages in a buffer, and then write to a blob via blob binding.

Azure Functions Eventhub Input Trigger scaling

We have an EventHub with a retention of 1 day, containing millions of messages. To consume this, we have an Azure Function which reads from this event hub via the Event Hub Binding. The function basically reads the raw bytes, deserializes it into a json, does some transformations, and outputs it to another event hub.
It takes an EventData[] as input, to allow us to receive a batch of EventData at once. We have configured it to receive 1024 messages per batch.
When we start the function, and it needs to reprocess the last 24 hours, it's only using 1 node of the available 5 we have in the app service plan, as can be seen in the metrics:
According to the docs, scaling should behave like this:
When your function is first enabled, there is only one instance of the function. Let's call this function instance Function_0. Function_0 has a single EventProcessorHost instance that has a lease on all ten partitions. This instance is reading events from partitions 0-9. From this point forward, one of the following happens:
New function instances are not needed: Function_0 is able to process all 1000 events before the Functions scaling logic kicks in. In this case, all 1000 messages are processed by Function_0.
An additional function instance is added: The Functions scaling logic determines that Function_0 has more messages than it can process. In this case, a new function app instance (Function_1) is created, along with a new EventProcessorHost instance. Event Hubs detects that a new host instance is trying read messages. Event Hubs load balances the partitions across the its host instances. For example, partitions 0-4 may be assigned to Function_0 and partitions 5-9 to Function_1.
N more function instances are added: The Functions scaling logic determines that both Function_0 and Function_1 have more messages than they can process. New function app instances Function_2...Functions_N are created, where N is greater than the number of event hub partitions. In our example, Event Hubs again load balances the partitions, in this case across the instances Function_0...Functions_9.
I believe we're hitting option #1, even though we have 24 hours of data on the event hub with only 1 node processing the data. At this rate it takes many hours to process, while 4 nodes are idle.
How does Azure Function know when to scale in this scenario, and can we influence this behavior?

When Batch Sending Events in Azure Event Hubs how can you calculate the total message size?

I have this problem where I am reading messages from SQL server (data could vary some fields are varchar(max)) and batch sending them by serializing them to JSON Azure Event Hubs Batch sending mechanism has a limit of 250Kb, is there a easy to check the size of the list of eventdata that gets trasmitted before the submission? Most of the times the data is tiny so I would want to batch up several messages. How can I know what size will the payload is going to be. I really don't want to count characters and calculate the size and thinking there is a better way you guys can help with?
If you are on .NET SDK WindowsAzure.ServiceBus, you can use Serialized​Size​In​Bytes property (see docs). Then sum them up for the batch.
Unfortunately, I don't see it in the new Azure Event Hubs Client for .NET.
You can use TryAdd function available in EventHub SDK.
Event Hub Data batch - TryAdd (EventData data)

Azure Functions Event Hub trigger bindings

Just have a couple of questions regarding the usage of Azure Functions with an EventHub in an IoT scenario.
EventHub has partitions. Typically messages from a specific device go to the same partition. How are the instances of an Azure Function distributed across EventHub partitions? Is it based on the performance? In case one instance of an Azure Function manages to process events from all partitions then it is enough otherwise one might end up with one instance of an Azure Function per EventHub partition?
What about the read-offset? Does this binding somehow records where it stopped reading the event stream? I thought the functions are meant to be stateless and here we have some state.
Thanks
Each instance of an Event Hub-Triggered Function is backed by only 1 EventProcessorHost(EPH) instance. Event Hub ensures that only 1 EPH can get a lease on a given partition.
Answer to Question 1:
Let's elaborate on this with a contrived example. Suppose we begin with the following setup and assumptions for an EventHub:
10 partitions.
1000 events distributed evenly across all partitions => 100 messages in each partition.
When your Function is first enabled, there is only 1 instance of the Function. Let's call this Function instance Function_0. Function_0 will have 1 EPH that manages to get a lease on all 10 partitions. Let this EPH be called EPH_0, and it will start reading events from partitions 0-9. From this point forward, one of the following will happen:
Only 1 Function instance is needed - Function_0 is able to process all 1000 before the Azure Functions' scaling logic kicks in.
Hence, all 1000 messages are processed by Function_0.
Add 1 more Function instance - Azure Functions' scaling logic determines that Function_0 seems sluggish, so a new instance
Function_1 is created, resulting in EPH_1. Event Hub detects that a new EPH instance is trying read messages. Event Hub will start load
balancing the partitions across the EPH instances, e.g., partitions
0-4 are assigned to EPH_0 and partitions 5-9 are assigned to EPH_1.
If all Function execution succeed without errors, both EPH_0 and
EPH_1 checkpoints successfully and all 1000 messages are processed. When check-pointing succeeds, all 1000 messages should never be retrieved again.
Add N more function instances - Azure Functions' scaling logic determines that both Function_0 and Function_1 are still sluggish and
will repeat workflow 2 again for Function_2...N, where N>9. Event Hub will load balance the partitions across Function_0...9 instances.
Unique to Azure Functions' current scaling logic is the fact that N is >(number of partitions). This is done to ensure
that there are always instances of EPH readily available to quickly
get a lock on the partition(s). As a customer, you are only charged for the resources used when your Function instance executes, but you are not charged for this over-provisioning.
Answer to Question 2:
EPH uses a check-pointing mechanism to mark the last known successfully read message. An EventHub-Triggered Function can be setup to process 1 message or a batch of messages at a time. The option you choose needs to consider the following:
1. Speed of message processing - Processing messages in batches instead of a single message at a time is one of the factors that will speed up the ability of your Azure Function workflow to keep up with the incoming messages in your Event Hub.
2. Tolerance for duplicates - If check-pointing fails due to errors in your Function code/(Updated Aug 24th, 2017) timeout/partition least lost, then the next EPH that gets a lease on that partition will start retrieving messages from the last known checkpoint. Event Hub guarantees at-least-once delivery but not at-most-once delivery. Azure Functions will not attempt to change that behavior. If not having duplicate messages is a priority, then you will need to mitigate it in your workflow. As such, when check-pointing fails, there are more duplicate messages to manage if your Function is processing messages at batch level.
Function Apps are based on WebJobs SDK, which use EventHostProcessor to consume events from Event Hubs. So you can lookup information about EventHostProcessor and it will be applicable to your Function App.
Particularly, you can find the implementation of IEventProcessor
here.
To your questions:
Not sure what you mean by "one instance". One listener will be created per partition, but they can be both hosted inside a single App Plan Instance if the load is low. On the high level, you should not care much: in Consumption Plan you pay per execution time, no matter how many servers/processes/threads are running. Of course, you should care whether the auto-scaling works good enough for high load, but that needs to be tested anyway.
Functions are stateless in a sense that you can't save anything in-memory between two function executions. You are totally fine to save state in external storage. Function App will use PartitionContext.CheckpointAsync() for checkpointing of the current offset. Azure Storage is used internally; again you can read more about how it works in Event Hubs and EventHostProcessor docs, e.g. here.

Azure Functions - Service Bus Scaling

I have an azure function listening to a service bus queue trigger using the dynamic consumption plan. Based on this documentation of the host.json config...
https://github.com/Azure/azure-webjobs-sdk-script/wiki/host.json
... you can set the following values
"serviceBus": {
// The maximum number of concurrent calls to the callback the message
// pump should initiate. The default is 16.
"maxConcurrentCalls": 16,
// The default PrefetchCount that will be used by the underlying MessageReceiver.
"prefetchCount": 100
},
Is there any documentation on setting the above for use with functions - particularly using a consumption plan.
The service bus performance best practices documentation suggests:
https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-performance-improvements
When using the default lock expiration of 60 seconds, a good value for >SubscriptionClient.PrefetchCount is 20 times the maximum processing rates of >all receivers of the factory. For example, a factory creates 3 receivers, and >each receiver can process up to 10 messages per second. The prefetch count >should not exceed 20*3*10 = 600. By default, QueueClient.PrefetchCount is set >to 0, which means that no additional messages are fetched from the service.
Can somebody please shed some light on how these are/should be used within functions?
Thanks!
Looking at the ASB code for Azure WebJobs (base for Functions) it looks like there's a single receiver that is created. Hence the settings that you see that take into consideration a single receiver created.
ASB performance documentation is describing a scenario where you create your own message pumps and control number of factories and receivers.

Resources