In one of the recent project, I need to add messages(>200kb) to Azure Event Hub through an endpoint exposed by Azure API Management Service. Then, the Stream Analytics job reads this message from Event Hub and writes it to the respective tables in SQL Server.
I was using "log-to-eventhub" policy to log the messages to event hub. But it has a size limitation associated with it, which is 200kb.
What would be the best approach to overcome this size limitation or should I consider a different way to log the payload to Event Hub?
Any help is much appreciated.
Here is a limit about this described in official docs.
The maximum supported message size that can be sent to an event hub
from this API Management policy is 200 kilobytes (KB). If a message
that is sent to an event hub is larger than 200 KB, it will be
automatically truncated, and the truncated message will be transferred
to event hubs.
You could consider using Azure Event Hubs output binding for Azure Functions.
About How Function consume Event Hubs events, you could try using multiple parallel Functions instances under consumption plan.
Related
We designed an eventhub trigger which reads the messages from event hub and inserts the messages into cosmos. During this process if any unhandled exceptions/throttling for cosmos we are moving these messages to blob.
Do we have any way we can move the messages back to event hub from blob through azure portal?. This helps azure admin to move the messages to eventhub in the production
No I don't think there is a way to move the message back in the portal UI.
I would use a combination of approaches here. First I would use autoscaling of Cosmos DB to lower the risk of you being throttled and to keep you from overprovisioning and thus overspending on Cosmos DB. Secondly I would implement a retry logic with exponential back-off in your trigger to further minimize the risk of throttling being a problem.
If you still get failed events, you might not have to push them to separate storage after all. All events remain by default in Event Hubs for seven days. You can just reread the entire thing if you want.
If that is not a good approach, I would push the failed messages to a queue (Storage queue or Service Bus Queue) and have an Azure Function on a timer trigger to process the queue and send the messages back to the Event Hub again. Then it would be fully automatic and admin does not have to do anything.
Do we have any way we can move the messages back to event hub from blob through azure portal?
One of the workarounds is to use logic apps where you can create a workflow from Azure blob storage to event hub. Here is a sample flow of my logic app.
I am using the Azure function to process message from IOT hub and output to Blob storage.
enter image description here
But the function missed IOT messages when I send in high frequency.
For example, I send 30 messages from 20:40:16 to 20:40:23 but only 3 are processed and stored in to the Blob storage and I have no ideal where the rest 27 went.
enter image description here
I am using the function consumption plan and Azure declares it will auto scaling depends on the load.
But from the above activity log, it only one thread is running and not even queue the input and cause some message lost.
So, what I should do to catch all messages from IOT hub?
Found the solution myself.
The trigger need to change from Azure Event Hubs to Event Grid Trigger as the images show below.
Azure Event Hubs
Azure Grid Trigger
Azure Functions on a consumption plan can handle this load, but you might want to make a separate Consumer Group in your IoT Hub that the Function can use. In the Azure Portal, go to Built-in endpoints and add a new Consumer Group.
You then have to specify in your Function which Consumer Group to use
[FunctionName("Function1")]
public static async Task Run([IoTHubTrigger("messages/events",ConsumerGroup = "functions", Connection = "EventHubConnectionAppSetting")]EventData message,
I tested this with a consumption plan Function listening to IoT Hub default endpoint and writing to blob storage with an 8 second delay to make it more like your function. I'm seeing no loss in messages, whether I send 30 or 100 messages. Make sure that no other applications are using your new Consumer Group!
I have an Azure Event Hub over which I would like to send various types of messages. Each message should be handled by a separate Azure Function, based on their message type. What is the best way to accomplish this?
Actually, I could create some JSON container with a type and payload property and let one parent Azure Function dispatch all the messages payloads - based on their type - to other functions, but that feels a bit hacky.
This question basically asks the same - however it is answered how it can be done using the IoT Hub and message routing. In the Event Hub configuration I cannot find any setting to configure message routing though.
Or should I switch to an Azure Message Queue to get this functionality?
I would use Azure Streaming Analytics to route it to the different Azure Functions. ASAs allow you to specify Event Hubs as a source and several sinks (one of which can be multiple Azure Functions). You can read more about setting up Azure Streaming Analytics services through the Azure Portal here. You'll need to set up the Event Hub as your source (docs). You'll also need to set up your sink (docs). You write some MS SQL-like code to route the messages to the various sinks. However, ASAs are costly relative to other services since you're paying for a fixed amount of compute.
I put some pseudo code below. You'll have to swap it out based on how you configure you're ASA using the information from the attached MS Documentation.
SELECT
*
INTO
[YourOutputAlias]
FROM
[YourInputAlias]
HAVING
[CONDITION]
SELECT
*
INTO
[YourAlternateOutputAlias]
FROM
[YourInputAlias]
HAVING
[CONDITION]
Based on your additional info about the business requirements and assuming that the event size < 64KB (1MB in preview), the following screen snippet shows an example of your solution:
The concept of the above solution is based on the pushing a batch of the events to the Event Domain Endpoint of the AEG. The EventHub Trigger function has a responsibility for mapping each event message type in the batch to the domain topic before its publishing to the AEG.
Note, that using the Azure IoT Hub for ingestion of the events, the AEG can be directly integrated to the IoT Hub and each event message can be distributed in the loosely decoupled Pub/Sub manner. Besides that, for this business requirements can be used the B1 scale tier for IoT Hub ($10/month) comparing to the Basic Event Hubs ($11.16).
The IoT Hub has built-in a message routing mechanism (with some limitations), but a recently new feature of the IoT/AEG integration such as publishing a device telemetry message is giving a good support in the serverless architecture.
I ended up using Azure Durable Functions using the Fan Out/Fan In pattern.
In this approach, all events are handled by a single Orchestrator Function which in fact is a Durable Azure Function (F1). This deserializes incoming JSON to the correct DTO. Based on the content of the DTO, a corresponding activity function (F2) is invoked which processes it.
in our usecase, we receive messages on Azure IoT Hub, and would like to route the data to different Event Hubs or Service Bus topics.
IoT Hub routes and endpoints are no option, because the data is binary data (protobuf), and there are only 10 different endpoints possible (we need more).
Our requierements are:
Splitting the message
Transform the data (maybe json)
Routing to different endpoints based on the payload (different parts of message could be routed to different endpoints)
(optional) enrich the data with additional payload
I see different options:
Azure Stream Analytics
Azure Functions
Spark or Flink
Do it yourself (write an Application and run it in Service Fabric or Kubernets)
Which techology would you recommend?
Regards,
Markus
There is also another option for your scenario such as using an Azure Event Grid. In this case, the telemetry data from the Azure IoT Hub are pushed to the Event Grid via its custom topic endpoint. Note, that there is a limit for the event message such as 64KB, see more details here.
The Event Grid allows to subscribe unlimited number of the Event Hubs, more details about the Event Grid are here and here.
Based on the above, the following screen snippet shows your another option for routing a small telemetry data to more than 10 Event Hubs, basically to any kind of subscriber.
I would like to get some recommendation, for designing a routing of IoT messages in Azure.
I have following scenario:
Senors sending messages to Azure IoT Hub in Google Protobuf format. Depending of the type of a message, I want to route the message to different applications inside a service fabric.
My current approach is to use a service fabric application to receive all messages from the IoT hub, parse the protobuf message, send the message depending on their type (attribute inside the protobuf) to an type-specific Azure event hub. Now the applications fetches the messages from their "own" event hub and process the messages.
I'm not sure if this is the best approach. I don't like the fact, to have one event hub for each type of message. Service Bus Topics are probably not an option, because I have a lot of messages (~30k per second).
Do I realy need a event hub, to decoupling this process, or does it make sense, to send the messages from the "routing application" direct to the different "type applications"?
What do you think?
Regards,
Markus
If you really need high performance you should take a look at IoT Hub and Event Hubs. Azure Event Hubs is a highly scalable data streaming platform and event ingestion service capable of receiving and processing millions of events per second. Event Hubs can process and store events, data, or telemetry produced by distributed software and devices. Data sent to an event hub can be transformed and stored using any real-time analytics provider or batching/storage adapters.
In other hand if you need only 30k messages per second you can go with Premium Messaging.
Comparison of Azure IoT Hub and Azure Event Hubs
Premium Messaging: How fast is it?
What is Event Hubs?