I have a backend device with MQTT client connected to opensource MQTT broker (Mosquitto). On the other hand I have many frontend devices (PC, Tablet, Mobile) with GUI application also connected to the broker. So here Mosquitto works as a communication point between backend device and frontend devices and just forwards messages between them. Sometimes amount of data transferred can be quite high (e.g. 1 MB / min).
One backend device + many frontend devices is one installation. I need to prepare infrastructure for thousands of such installations working simultaneously. So my service needs to be very scallable. My company uses Azure cloud solutions, so I've started learning of this solution and I must admit that I am a little bit confused. I've read that I need to use IoT Hub, but it would need MQTT Gateway to be able to speak with MQTT devices. On the other hand, if I understand it well, the Gateway needs to be running on some VM, so here I lose scalability of my solution. Am I right? Now if I will need to support 100k or 500k devices then I will need another VM? One more thing is that I need to integrate all of this with some webservice (for management of backend and frontend devices), so I need some connection between webservice and the MQTT broker...
Before I started to play with Azure, I imagined, that I will simply start an MQTT broker service and magically it will be highly scallable and will be able to provide service for thousands of devices.
Can anybody explain me how to bite that?
Azure IoT Hub now talks MQTT natively. A protocol gateway is no longer required.
https://azure.microsoft.com/en-us/documentation/articles/iot-hub-mqtt-support/
This is going to help you a lot if you've just spent the last hour trying to form the MQTT username and password:
https://github.com/Azure/azure-content/blob/master/articles/iot-hub/iot-hub-devguide.md#example
Example:
Username (DeviceId is case sensitive): iothubname.azure-devices.net/DeviceId
Password (Generate SAS with Device Explorer): SharedAccessSignature sr=iothubname.azure-devices.net%2fdevices%2fDeviceId&sig=kPszxZZZZZZZZZZZZZZZZZAhLT%2bV7o%3d&se=1487709501
Tested with Paho and MQTT.fx on Windows. I could not make it authenticate with mosquitto, and i've put in reasonable effort, even tried using stunnel just in case mosquitto's TLS support wasn't cutting it. Mosquitto probably doesn't handle the long password correctly or something along those lines. It throws an authentication error. Escaping % and & didn't help.
If someone gets Mosquitto to work with Azure IoT Hub, please open my eyes.
...and someone did (Thank you Timothy in the comments)
Mosquitto_pub works, I verified by monitoring with Device Explorer Twin. Example:
mosquitto_pub -h IOTHubACMxxx.azure-devices.net
-p 8883
--cafile "C:\Users\jlaird\Documents\dev\azureca.crt"
-t devices/eACM1/messages/events/
-m "john says hello to azure from mosquitto"
-i eACM1
-u IOTHubACMxxx.azure-devices.net/eACM1/?api-version=2018-06-30
-P "SharedAccessSignature sr=IOTHubACMxxx.azure-devices.net&sig=obfuscate&se=1593013589&skn=device"
Today there isn't an official support for MQTT protocol in Azure but only the public preview of IoT Hub that supports AMQP and HTTP.
To connect MQTT devices to the IoT Hub, Microsoft provides a "framework" named IoT Protocol Gateway (https://github.com/Azure/azure-iot-protocol-gateway) that executes a protocol translation between MQTT and AMQP.
The IoT Protocol Gateway can be installed on premise or in the cloud as an Azure Worker Role. In the second scenario you have the scalability offered by Azure and related to worker role instances.
This solution is absolutely new due to the short life of IoT Hub (still in public preview) and the IoT Protocol Gateway itself.
Your first solution is based on using a third-party MQTT broker (like mosquitto) that you should install in a VM. AFAIK mosquitto doesn't support clustering like HiveMQ broker (see another reply here : Cluster forming with Mosquitto broker).
Last thing about the connection between your web service and the MQTT broker.
In this case the web service should translate calls to him (from front end) to published message on the MQTT broker using an MQTT client that you need to include inside the web service itself.
Even if using AWS, the following link could be useful too :
https://groups.google.com/forum/#!topic/mqtt/19jqofoPLro
Paolo.
Related
I want to Forward messages from Cloud MQTT broker to Azure IOT Hub. I need the settings examples to create the bridge for multiple devices which can be accessible with Shared access key of IOT Hub rather than SAS for specific device. and can we use the single bridge for multiple devices or it needs single bridge for each device?
Note that IoT Hub is not a full-featured MQTT broker and does not support all the behaviors specified in the MQTT v3.1.1 standard.
A device can use the MQTT protocol to connect to an IoT hub using any of the following options:
Libraries in the Azure IoT SDKs.
The MQTT protocol directly.
can we use the single bridge for multiple devices or it needs single
bridge for each device?
AFAIK, Using the same connection string is not supported and you will see unexpected errors. Only one device can use the same connection string at a time. For example if you have two devices connecting with the same ID, then it would be difficult to identify where to send messages. You can use Modules Instead.
To answer your specific query, If your business scenario requires only sending a telemetry data (D2C messages), you can use a connection less device protocol such as the https using the REST API, with a sasToken for authorization header.
Also, see Using the MQTT protocol directly (as a device)
I'm trying to use an MQTT client to send device telemetry to an IoT Central instance. To do this, I think I need to know the hostname of the underlying IoTHub instance. I have a hostname of the form iotc-{a bunch of letters and numbers}.azure-devices.net, but I can't remember where I got it from or how to generate it again. I get a "connection refused: server unavailable" error when trying to connect to port 8883. I have tried both SAS and x509 certificate authentication, so either I'm getting the authentication wrong with both methods, or I'm getting the hostname wrong. Does anyone have any suggestions for retrieving the underlying IoT Hub hostname?
IoT Central uses the Device Provisioning Service to handle device provisioning - this approach replaces the use of connection strings as described in the previous answer.
You can use MQTT to interact with IoT Central's DPS instance and retrieve device connection details. This article provides the information you need: https://learn.microsoft.com/azure/iot-dps/iot-dps-mqtt-support
Alternatively, you could run a separate script to generate a connection string for your device to use. For an example of the type of code the script should run, see https://github.com/Azure/azure-iot-sdk-node/blob/main/device/samples/javascript/pnp_simple_thermostat.js. In particular the provisionDevice function.
Such feature is not available anymore. It was possible to get hostname of IoT Hub used in IoT Central some time ago, but from Jan 31 2020 it was depreciated.
https://github.com/Azure/dps-keygen#retrieve-hub-connection-string-deprecated
You can use Azure IoT Central Device Bridge for your case https://learn.microsoft.com/en-us/azure/iot-central/core/howto-build-iotc-device-bridge
But you'll need to have some additional services running on Azure.
Am working on MQTT with GSM Module previously done with HiveMq Server which is working perfectly by ref of doc. I have done Publish and Subscribe through broker.hivemq.com Port 1883.
Now Presently I am on Azure. So many Queries are there in these Proposal
As MQTT Packet is Unique, I guess i can connect in similar way as like hivemq but am unable to find IP address for Azure where i found by my guess is XXXXXX.azure-devices.net and port 8883. Is Correct?
I got Device ID key in this format dF8vP17DxCwG8IRccXXXXXXkIrCVfWIC7qmcM. Is it Password and should I provide Cert. also if yes what are they.
Is Username is {iothubname}?
After getting required Parameters above, Can I send MQTT packet Directly following this document. If not Can you Please describe
Am having Connection string {} and Endpoint what are they?
To create Topic messages/events/TOPIC. Is this correct?
Really Confused with bunch of Doubts to communicate my device (GSM Module) with Azure.
Thanks & Regards
kishore D
From Using the MQTT protocol directly:
For the ClientId field, use the deviceId.
For the Username field, use {iothubhostname}/{device_id}/api-version=2016-11-14, where {iothubhostname} is the full CName of the IoT hub.
For example, if the name of your IoT hub is contoso.azure-devices.net and if the name of your device is MyDevice01, the full Username field should contain:
contoso.azure-devices.net/MyDevice01/api-version=2016-11-14
For the Password field, use a SAS token. The format of the SAS token is the same as for both the HTTPS and AMQP protocols:
SharedAccessSignature sig={signature-string}&se={expiry}&sr={URL-encoded-resourceURI}
Azure IoT Hub is much more than an MQTT broker and doesn't really give you complete control over the protocol. A Mosquitto Docker container running on Azure Container Instances may be a better solution if protocol control is key and we're looking at a hobbyist project. For the cold path you can store telemetry data in Azure Cosmos DB.
I’m new to Microsoft’s Azure IoT Hub and MQTT/AMQP communication in general and I am trying to establish simple communication between my raspberry pi and a Ubunutu VM running on my computer. I intended on using my pi by as a client to publish messages to a service running on my VM that simply echoes the message contents to std out. After reading all the literature on the Azure website/github page I’m having trouble figuring out the best way to do this. Is it possible to host a service on your local network? Are the iotHub service client samples on the SDK meant to be used as services or as clients intended to communicate with existing services on Azure (SQL databases, webserver, etc)? I was able to send messages to my IoT Hub portal with azure-iot-sdk-c/iothub_client/samples/iothub_client_sample_mqtt.c sample but couldn’t figure out where to include the topic my message was getting published to. I would really appreciate any help/input.
Thanks
The Azure IoT device SDKs are meant to build device applications that will connect and communicate with the Azure IoT Hub service which lives in the Azure Cloud.
The Azure IoT service SDKs are meant to build applications that will be used to configure, monitor and interact with an instance of the Azure IoT Hub service running in the Azure Cloud.
If you are looking for a simple solution for establishing a local device to device or device to server communication leveraging MQTT, you can look into MQTT broker solutions out there.
Now if you want to leverage an IoT Cloud to manage devices, easily secure and authenticate them, upload telemetry to the Cloud at a high throughput and take advantage of advanced analytics services in the Cloud (Big Data, Machine Learning,...), then Azure IoT Hub is for you.
Azure IoT Hub is just a message channel, and it does not handle messages.
For device side (raspberry pi), you use a device SDK to send D2C messages with MQTT, and it works. For service side (Ubuntu VM), you need Event Hub SDK to receive D2C messages sent by raspberry pi (Why Event Hub? https://blogs.msdn.microsoft.com/zhqqitest/2017/03/18/do-not-make-it-a-mess-why-therere-so-many-endpoints-in-iot-hub/).
Also, IoT Hub supports MQTT on device side ONLY, that is to say, you cannot connect to IoT Hub with MQTT on service side (Ubuntu VM), but you need AMQP.
You may have another question, then what is the AMQP topic to listen for D2C messages on service side? It's /<compatibleName>/ConsumerGroups/<ConsumerGroups>/Partitions/<PartitionID>.
compatibleName: IoT Hub Event Hub-compatible name (what a mess :-S), you can find it from Azure portal (Endpoints – Built-in endpoints – Events)
ConsumerGroups: You can find it from Azure portal, $Default by default
PartitionID: Partition ID, you can get it when you connect to the Event Hub with AMQP, it's a number, ususally start from 0
If you don't want to call AMQP directly, you can simply use Event Hub SDK on Ubuntu VM with IoT Hub connection string (NOT Event Hub-compatible endpoint or Event Hub-compatible name, the Event Hub SDK will do that for you).
I have a virtual machine on Azure which will listen to messages over the servicebus of Azure. And another developer needs to connect to this servicebus to send messages to my service. To do so, we need to come up with some protocol for this communication system. And I was thinking about using WSDL to make the server something webservice-like, but instead of listening to the standard HTTP ports it would connect to the service bus and within it a topic with subscription, or whatever. I'm still not sure what would be best.
So, is it possible? Has anyone done something similar before? Are there some examples?
Service Bus provides both Brokered and Relayed messaging models. With using Topics/Queues you are essentially sending and receiving message to/from the broker. So you can use APIs/protocols that the broker supports. As an example you can use the NetMessagingBinding from your service/client to send and receive messages through the queue. There is no contract here other than what you put in the BrokeredMessage body and properties values. The application on either end can decode/decipher that info as needed.
In the case of Relayed messaging however you are making a end-to-end connection between the client and the service with Service Bus serving as the proxy in-between for auth, location-transparency and even load-balancing. So here you can use from a variety of RelayBindings such as NetTcpRelayBinding or webHttpRelayBinding but then both the client and the service need to be available at the same time. Here you need the service and clients to be using the same contract.