Is there a way to send a D2C message to a specific Consumer Group instead of Default Consumer group in Azure IoT Hub?
Message to the Event Hub endpoint are always delivered to all consumer groups. The same is true for the actual Event Hub.
But it sounds like you want to take a look at message routing: https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-messages-read-custom
Routing allows you to filter messages and/or send them to specifc endpoints. If you need more than one Event Hub endpoint, apart from the built-in one, add Event Hubs as custom endpoints: https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-messages-read-custom
Related
I want to send an event that is sent to Azure Service Bus topic to an event hub. Is this possible?
Details:
I am working with a different team in my company that receives third party events (via webhook) to Azure Service Bus topic and this is further used in different application.
My team wants to now listen/subscribe to this topic using our existing event hub and using azure capture store these events to a storage account.
I did the following:
I created a subscription to their topic in their Azure Service Bus.
I created an event hub in my Event hub namespace.
I am not sure, how to now connect the azure service bus topic subscription to send those events to my event hub.
Thanks for your time.
Service bus operates with the receivers having to pull messages from it. This is opposite to Eventgrid which pushes the events to its subscribers. Eventhub does not pull messages from the source, we need to push messages into it. So you cannot achieve your requirement without an extra component between Service Bus and Eventhub.
One of the possible components would be a service bus topic triggered azure function LINK which writes into the eventhub using output binding LINK or the SDK LINK.
You will need to choose your service plan carefully depending on the volume of messages expected but usually Consumption plan will suit this purpose.
I have a Azure service bus Topic with two subscriptions.
I want to send a message to topic from logic app using send message connector. How to send the message to a specific subscription.
Now it takes only topic name and does not have property to accept subscription name, how can i implement the same.
thanks in advance.
Unfortunately, that is not possible (just not with logic apps, but in general)
This is how a topic and subscription works.
A Service Bus topic provides an endpoint for sender applications to
send messages.
Each subscription of a topic gets a copy of the message sent to the topic.
Topics and subscriptions provide a one-to-many form of communication.
Having, said that you can configure filters at the Subscription end. This will facilitate receiving only those messages meeting the criteria from the central pool. When you want a specific subscription to receive it. You could send the message such a way that it matches the filter condition.
So, something like this :
Image Source
100 messages are sent to the topic, but are split to each subscription as 30,45,25 based on the filter rule. the messages that did not meet filter are not made available to the subscription.
In your case, you need set filters for both the subscription. Trigger the message such a way that it matches only for one of the subscription.
Alternatively, if it is going 1:1 - you could make use of the Queue.
References to set up filters at subscription level :
Filters Service Bus
Filtering the Service Bus
Stackthread on the implementation
A subscription in Service Bus is an isolated view into the messages of a Topic, essentially a copy of the messages private to the subscription. This allows multiple consumers to process topic messages without competing with one another.
You can't publish messages messages directly to a subscription, only to the topic that the subscription is associated with. All subscriptions associated with the topic will have access to the message.
If you are looking to send messages for a single consumer (or a set of competing consumers), a Service Bus queue may be a better fit for your scenario.
I have an Azure IoTHub with thousands of devices registered. These devices communicate through a Telco provider who sends messages through an Azure Storage Queue. This Storage Queue triggers an Azure Function which needs to parse the messages and Send an Event to the IoTHub as below.
Currently, we use the Azure IoTHub SDK to create a DeviceClient for each payload and we send the event. Because the DeviceClient represents a device in the IoTHub and is carrying the context of the source of the events, we are having to recreate a device client for each event. This quickly exceeds the threshold of the number of Connections allowed on Azure Functions.
We have tried using the IoTHub Output bindings for Azure Functions, but could not get to work and I do not think it would work because we need to make sure that the events get to the IoTHub with the right context (messages are sent by the right device).
What's the right way to solve this? Can the connections to the IoTHub be reused? Should we abandon Azure Function in favour of something else?
I assume that Telco is some kind of custom device management solution(vendor lock solution), that can also communicate with the device and receive the device telemetry, and eventually forward it to the specified endpoint, correct?
If I may ask and if my assumption is correct, why do you need to deliver the events to IoT Hub, if you are not managing Telco devices through IoT Hub(the arrows on your diagram are only in one direction)?
Using the IoT Hub just as a message broker for essentially cloud-to-cloud communication is not beneficial if that is the only purpose. Also conceptually what you described is cloud-to-cloud communication, and IoT Hub is intended to be used for devices.
Here is what I would do. Setup the API Management(or http triggered Azure Function) as a front door for Telco and pass the messages to the Event Hub.
You can choose here to pass request body for example where your telemetry data is - I assume again.
Keep the IoT Hub, and setup the routing to previously created Event Hub.
Now, in case you have devices that are not vendor locked and that can talk directly to IoT Hub, messages will be re-routed to Event Hub. Also Telco device messages will be routed to exactly the same Event Hub.
Now you can have for example Azure Stream Analytics that can analyze data stream just from the Event Hub, and for both, Telco devices and potentially non-Telco devices.
After trying a few things, I ended up moving away from using the SDK for pushing messages to IoT Hub. This is because the SDK uses AMQP, and creating a DeviceClient for each payload is not viable.
We switched to using HTTPS instead to push the messages to IoT Hub and using HttpClientFactory, we are able to do connection pooling.
I thought I would put this here in case someone has the same issue.
Here is an example of the Http request to send message to IoT Hub
Host: https://<iothubname>.azure-devices.net/devices/<deviceId>/messages/events?api-version=2018-06-30
Authorization: SharedAccessSignature sr=<iothubname>.azure-devices.net&sig=abc123;12344iweoippweruea=iothubowner&se=1570574220
Body: <normal Interval or alarms payloads> // example {"deviceid": "abc", "hello": "world"}
Lastly, thanks #kgalic for the answer but your suggestion would not work. This is not pure B2B integration. Our implementation have to allow for both devices connecting directly to the IoT Hub and devices connecting through the Telco. This is why every device needs to have its own identity and digital twin.
I have two devices registered in IoT Hub. say device1 and device2. I have two azure functions say azurefunction1 and azurefunction2.
When a message is sent to Iot Hub with device1 I want azurefunction1 to get triggered but not azurefunction2.
Similarly When a message is sent to Iot Hub with device2 I want azurefunction2 to get triggered but not azurefunction1.
I tried using consumer groups but both azure functions are getting triggered.
Any help appreciated.
This is not directly possible. Events from both devices go to the same Event Hub, and Event Hub is the level of granularity for Functions trigger.
Consumer groups are for independent consumers, which both will get all events.
I would suggest you having a single Function to receive events from all devices. This Function could then route the request based on Device ID to other Functions if needed, e.g. via HTTP calls or via separate Storage Queues.
Another option is to use IoT Hub routing. You can route events to different IoT Hub endpoints based in the deviceid. You can then have azurefunction1 connected to the endpoint for device1, and azurefunction2 connected to the endpoint for device2.
The Routing Messages tutorial describes how to configure and use routing.
I would like to get some recommendation, for designing a routing of IoT messages in Azure.
I have following scenario:
Senors sending messages to Azure IoT Hub in Google Protobuf format. Depending of the type of a message, I want to route the message to different applications inside a service fabric.
My current approach is to use a service fabric application to receive all messages from the IoT hub, parse the protobuf message, send the message depending on their type (attribute inside the protobuf) to an type-specific Azure event hub. Now the applications fetches the messages from their "own" event hub and process the messages.
I'm not sure if this is the best approach. I don't like the fact, to have one event hub for each type of message. Service Bus Topics are probably not an option, because I have a lot of messages (~30k per second).
Do I realy need a event hub, to decoupling this process, or does it make sense, to send the messages from the "routing application" direct to the different "type applications"?
What do you think?
Regards,
Markus
If you really need high performance you should take a look at IoT Hub and Event Hubs. Azure Event Hubs is a highly scalable data streaming platform and event ingestion service capable of receiving and processing millions of events per second. Event Hubs can process and store events, data, or telemetry produced by distributed software and devices. Data sent to an event hub can be transformed and stored using any real-time analytics provider or batching/storage adapters.
In other hand if you need only 30k messages per second you can go with Premium Messaging.
Comparison of Azure IoT Hub and Azure Event Hubs
Premium Messaging: How fast is it?
What is Event Hubs?