Pipe Events from Azure Event Hub to Azure Service Bus - azure

Question

I am listening to Event hub for various events.
Each event is high value and cannot be missed.
Events are partitioned based on device id.
Events from one device id are sparse and not very frequent (couple of events per few dasy) . It only occurs in response to a user action which is infrequent.
The number of devices are huge , so I will have a lot of events for a variety of device Ids.
For each event , I need to make 3-4 API calls to Systems which are not super reliable. And since some of these are cross Geo Calls it might take some time.
I am planning to take the events from Event hub and put them into Service Bus. My reasons are as follows.
Event hub can be scaled to only 32 partitions and if one event takes time , the entire partition gets blocked.
Service bus on the other hand is more horizontally scalable. If the throughput drops I can just add more subscribers to the Service Bus.
I have been looking for patterns like this but I have not seen patterns where we get data from a log based messaging system and push them to a queue based one.
Are there better approach to handle such scenarios ?

Answer 1

I think you can use Event hub trigger and service bus output binding to achieve what you want.
For example, I want to monitor Event hub 'test' and I am using C# library:
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.Azure.EventHubs;
using Microsoft.Azure.WebJobs;
using Microsoft.Extensions.Logging;
namespace FunctionApp68
{
public static class Function1
{
[FunctionName("Function1")]
[return: ServiceBus("test1", Connection = "ServiceBusConnection")]
public static string Run([EventHubTrigger("test", Connection = "str")] EventData[] events, ILogger log)
{
var exceptions = new List<Exception>();
string messageBodyt = "";
foreach (EventData eventData in events)
{
try
{
string messageBody = Encoding.UTF8.GetString(eventData.Body.Array, eventData.Body.Offset, eventData.Body.Count);
messageBodyt = messageBodyt + messageBody;
// Replace these two lines with your processing logic.
log.LogInformation($"C# Event Hub trigger function processed a message: {messageBody}");
//await Task.Yield();
}
catch (Exception e)
{
// We need to keep processing the rest of the batch - capture this exception and continue.
// Also, consider capturing details of the message that failed processing so it can be processed again later.
exceptions.Add(e);
}
}
// Once processing of the batch is complete, if any messages in the batch failed processing throw an exception so that there is a record of the failure.
if (exceptions.Count > 1)
throw new AggregateException(exceptions);
if (exceptions.Count == 1)
throw exceptions.Single();
return messageBodyt;
}
}
}
The above code will collect from event hub 'test' and save to service bus queue 'test1'.
Have a look of these doc:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-event-hubs-trigger?tabs=csharp
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus-output?tabs=csharp#example

Answer 2

What you need is actually a private queue per device-Id. As soon as event comes to event hub, Pull events from it and put that into device-Id's private queue, and then process it serially.
How to build queue per device-Id:
Simple way to build queue is to use SQL database(mostly it works if request per second are not very very high, for sql-db 100 req/second are normal.)
another horizontally scalable way is to use azure append blobs(if your event processors are stateless).
You can also use advanced methodology like using Azure Service Fabric Reliable Queue.