Azure Event Hub - Can't understand Java flow - azure

from Microsoft EventHub Java SDK examples (https://learn.microsoft.com/en-us/azure/event-hubs/event-hubs-java-get-started-send), these are the steps that needs to be taken to be able to consume messages from an Even-Hub via the java SDK :
1.Create a storage account
2.Create a new class called EventProcessorSample. Replace the placeholders with the values used when you created the event hub and storage account:
3.
String consumerGroupName = "$Default";
String namespaceName = "----NamespaceName----";
String eventHubName = "----EventHubName----";
String sasKeyName = "----SharedAccessSignatureKeyName----";
String sasKey = "----SharedAccessSignatureKey----";
String storageConnectionString = "----AzureStorageConnectionString----";
String storageContainerName = "----StorageContainerName----";
String hostNamePrefix = "----HostNamePrefix----";
ConnectionStringBuilder eventHubConnectionString = new ConnectionStringBuilder()
.setNamespaceName(namespaceName)
.setEventHubName(eventHubName)
.setSasKeyName(sasKeyName)
.setSasKey(sasKey);
There are several things i don't understand about this flow -
A. Why is a storage account required? Why does it needs to be created only when creating a consumer and not when creating the event hub itself?
B. What is 'hostNamePrefix' and why is it required?
C. More of a generalaztion of A, but i am failing to understand why is this flow so complicated and needs so much configuration. Event Hub is the default and only way of exporting metrics/monitoring data from Azure which is a pretty straightforward flow - Azure -> Event Hub -> Java Application. Am i missing a simpler way or a simpler client option?

All your questions are around consuming events from Event hub.
Why is a storage account required?
Read the event only once: Whenever your application will read event from event hub, you need to store the offset(identifier for the amount of event already read) value somewhere. The storing of this information is known as 'CheckPointing' and this information will be stored in Storage Account.
Read the events from starting everytime your app connects to it: In this case, your application will keep on reading the event from very beginning whenever it will start.
So, the storage account is required to store the offset value while consuming the events from event hub in case if you want to read event only once.
Why does it needs to be created only when creating a consumer and not
when creating the event hub itself?
As it depends upon the scenario, whether you want to read your events only once or every time your app starts and that's why storage account is not required while creating event hub.
What is 'hostNamePrefix' and why is it required?
As the name states 'hostNamePrefix', this is name for your host. The host means the application which is consuming the events. And it's a good practice to make use of GUID as a hostNamePrefix. HostNamePrefix is required by the event hub to manage the connection with the host. In case, if you have 32 partitions, and you have deployed 4 instances of your same application then 8 partition each will be assigned to your 4 different instances and that's where the host name helps the event hub to manage the information about the connection of the respective partitions to their host.
I will suggest you to read this article on event hub for clear picture of the event processor host.

Related

Issue with Azure Event Hubs creation of 1 Hub

I have an issue when I am trying to create a new event hub in azure.
Should be able to create only 1 instance with 1 partition only. However, the process does not let me proceed.
Two issues:
-Impossible to create one hub with one partition (min 2 proposed)
-Impossible to generate de hub (error of ID and charge in relation)
Did you ever get the same issue
New EventHub 2 partitions min
ID problem when commit
I tried with an account having more credentials and it worked. The free version of azure does not allow to create an event hub it seems...

Azure Storage Queue message to Azure Blob Storage

I have access to a Azure Storage Queue using a connection string which was provided to me (not my created queue). The messages are sent once every minute. I want to take all the messages and place them in Azure Blob Storage.
My issue is that I haven't been succesful in getting the message from the attached Storage Queue. What is the "easiest" way of doing this data storage?
I've tried accessing the external queue using Logic Apps and then tried to place it in my own queue before moving it to Blob Storage, however without luck.
If you want to access and external storage in the logic app, you will need the name of the storage account and the Key.
You have to choose the trigger for an azure queues and then click in the "Manually enter connection information".
And in the next step you will be able to choose the queue you want to listen for.
I recomend you to use and azure function, something like in this article:
https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-blob-output?tabs=csharp
Firts you can try only reading the messages, and then add the output that create your blob:
[FunctionName("GetMessagesFromQueue")]
public IActionResult GetMessagesFromQueue(
[QueueTrigger("%ExternalStorage.QueueName%", Connection = "ExternalStorage.StorageConnection")ModelMessage modelmessage,
[Blob("%YourStorage.ContainerName%/{id}", FileAccess.Write, Connection = "YourStorage.StorageConnection")] Stream myBlob)
{
//put the modelmessage into the stream
}
You can bind to a lot of types not only Stream. In the link you have all the information.
I hope I've helped

How to pull Microsoft Azure Event Hub instance details like event hub connection string ,from our application

I am using Microsoft Azure Resource Manager API
https://learn.microsoft.com/en-us/rest/api/resources/resources/listbyresourcegroup
I am able to pull all event hub namespaces under particular
Subscription & resource group . But I am unable to pull event hub instances details like , event hub connection string and name of availabile event hub instances.
Seams that there is no such option in the rest api you mentioned.
You should use the following apis for fetching event hub connection string and name of event hub instances respectively.
For names of event hub instances, use this api, you need to specify the namespce:
GET
https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.EventHub/namespaces/{namespaceName}/eventhubs?api-version=2017-04-01
For connection strings, use this api. You should specify namespace / event hub name / connection string name:
POST
https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.EventHub/namespaces/{namespaceName}/eventhubs/{eventHubName}/authorizationRules/{authorizationRuleName}/ListKeys?api-version=2017-04-01
Response for connection string of event hub:
Hope it helps.

Executing Same Azure Function but for different input values in App Settings

I have a Azure Function scenario as follows:
1) If product = 123, then use Service Bus Topic1
2) If product = 456, then use Service Bus Topic2
I think there are 2 options to solve this:
OPTION 1: Have same Azure Function deploy 2 times (2 diff names) and each having different input/output mapping
OPTION 2: Have only 1 Azure Function but in Application Setting specify the input/output mapping. **From my understanding Application Setting is Key Value. Is this correct? if not, how can I specify complex value in this parameter **.
What is the best way to have this?
What I am thinking about is deploy same Azure Functions 2 times with different settings as follows:
Azure Function 1 with Application Settings as "productid" = 123 and "sbTopic" = topic1
Azure Function 2 with Application Settings as "productid" = 456 and "sbTopic" = topic2
I am wondering if there is a better way such that same Azure Function can run for any of my input/output mapping. If so, where and how do I specify my input (productid) and output (sbTopic) mapping?
EDIT 1: this is CosmosDB Trigger. Whenever we get products in Cosmos DB, i want to send to correct SB Topic
EDIT 2: I have something similar as follows:
Cosmos DB Trigger --> Azure Function --> Service Bus Topic for id=123
I am debating if I should have as follows
Cosmos DB Trigger --> Azure Function1 --> Service Bus Topic for id=123
Cosmos DB Trigger --> Azure Function2 --> Service Bus Topic for id=456
Cosmos DB Trigger --> Azure Function3 --> Service Bus Topic for id=789
which means I would have 3 AF duplicated
etc
or
Cosmos DB Trigger --> 1 Azure Function. Specify mappings (product id and sb topic) in App Settings and --> Add logic in AF such that: if id=123 send message to topic1 ; if id=456 send message to topic 2 etc.
You should have your Pub/Sub model driven by Topic based on the Subscription Rules. The following screen snippet shows an example of this model:
In the above Pub/Sub model, the Publisher fire a Topic with the application properties for additional details such as productid, type, etc. The Topic on the Service Bus can have more multiple Subscriptions entities for specific filters (Rules).
The Subscriber (in this example, the ServiceBusTrigger Function) can be triggered on the TopicName and SubscriptionName configured in the app settings.
In other words, the Subscription entity (Rules) can decided which subscriber will consume the event message. Also advantage of this Pub/Sub model is for continuous deployment, where each environment such as dev, stage, QA, pre-production, production has the same model (topicName, subscriptionName, Rules, etc.) and only connection string will specify which azure subscription (environment) is a current one.
The option suggested by #RomanKiss makes sense, and that's a "canonic" use of Service Bus topics & subscriptions.
But if you need to do what you are asking for, you can achieve that with a single Azure Function and imperative bindings.
First, do NOT declare your output Service Bus binding in function.json.
Implement your Function like this:
public static void Run([CosmosDBTrigger("...", "...")] Product product, Binder binder)
{
var topicName = product.Id == "123" ? "topic1" : "topic2";
var attribute = new ServiceBusAttribute(topicName);
var collector = binder.Bind<ICollector<string>>(attribute);
collector.Add("Your message");
}
By the way, if you deploy your function two times, you will have to put them on different lease collections, otherwise they will compete to each other, not do the processing twice. And you'll have to take care of NOT sending the message to Topic 1 when product is 123 and vice versa.

Routing Event Hub messages to Azure funtion by message type

I've create an Azure Function to listen to an Azure IoT Hub instance. When I post message to the IoT Hub I set a property like so:
Message iotMessage = new Message([myMessage]);
iotMessage.Properties.Add("Type", [MessageType]);
At the Azure Function end I want the Azure function only to receive/process messages that have a Type property and where the Type property equals "MessageType1".
I cannot see a way to do this in an Azure function. Can someone advise if this is possible?
EDIT: this appears to be what you're looking for:
https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-csharp-csharp-process-d2c#add-a-queue-to-your-iot-hub-and-route-messages-to-it
Your condition would be Type=MessageType1 and you would have the function trigger off of the output queue.
What type of input binding are you using for your Azure Function?
AFAIK, this isn't currently possible in one step. However, there are a few options you have:
connect IoT Hub to a ServiceBus Topic/Subscription, which allows you to do some filtering based on properties. Trigger on the subscription which filters by MessageType1
have a function dedicated to filtering IoT Hub messages. When it matches a MessageType1 message, put that message into a queue. This queue triggers another function which only processes matched messages

Resources