I did followings successfully:
Creating a hono tenant and registering a device for it.
Connecting a simple python based edge-device to hono.
Connecting hono to ditto.
Creating a twin for the above edge-device.
Sending telemtery data from the edge-device to ditto through hono works perfectly.
I also send a pseudo event every second form edge-device to ditto through hono as follows:
# sending an event from edge-device to ditto through hono-mqtt-adapter
correlation_id = ''.join(random.choices(string.ascii_uppercase + string.digits, k=10))
data = {
"topic": "de.iot1/dev1/things/live/messages/fire",
"headers": {
"content-type": "text/plain",
"correlation-id": correlation_id
},
"path": "/outbox/messages/fire",
"value": "Fire detected"
}
payload = json.dumps(data)
mqtt_client.publish("event", payload=payload, qos=1)
On the other side I wrote a simple ditto-amqp-client which just receives all dittos incomming messages. I receive all incoming telemetry messages as their correct interval - i.e. every second. In the case of events messages they seems to be buffered by ditto and sent to amqp-client every couple of seconds and not at the time they are send from the device! Why?
As far as I understood from the ditto-doc, ditto offers two communication channels. twin-channel for communicating with the twin through commands and events and a live-channel to communicate with the device directly through messages. But in the section protocol-topics
the channel can be either twin or live also for events or commands, which is confusing.
I would like to know what is the recommended way to send an event form the device to ditto?
Should one send it through live channels using events or messages (outbox)?
Is it better to define the event as a feature in the twin and send normal command/modify to its value?
Thanks in advance for any suggestion!
I receive all incoming telemetry messages as their correct interval - i.e. every second. In the case of events messages they seems to be buffered by ditto and sent to amqp-client every couple of seconds and not at the time they are send from the device! Why?
As Ditto does not do a "store-and-forward", but publishes applied events immediately, I can't really explain from Ditto side. Maybe by sending events (which in Hono are persisted using an AMQP broker), those are consumed with some delay (after persisted by the broker), however I can't explain the "every couple of seconds".
You could enable DEBUG logging in Ditto's connectivity service by setting the environment variable LOG_LEVEL_APPLICATION to DEBUG - that way you'll see when the messages are received and when they are forwarded again to your AMQP endpoint.
I would like to know what is the recommended way to send an event form the device to ditto?
Should one send it through live channels using events or messages (outbox)?
When you talk about the "fire detected" event (containing a payload which shall not be stored in a twin managed by Ditto), I would send it as live message from the device (using Hono event channel with "QoS 1" - at least once - semantics).
However, an application connected to Ditto (e.g. via Websocket or via HTTP webhook) must be consuming the live message and acknowledging that it received the message.
Events in Ditto are the entity which is persisted as part of the CQRS architecture style. After persisting, interested parties can be notified about them.
So live events are meant to be in the same Ditto event format and must not be confused with Hono events.
Is it better to define the event as a feature in the twin and send normal command/modify to its value?
It depends on your requirements.
You could also make the "fireDetected" a boolean property in a feature and subscribe to changes of this property. But then this should be only settable via the device (using an appropriate Ditto Policy) - and the "QoS 1" guarantees you get by combining Hono and Ditto cannot be used any longer.
Related
Intro
We're developing a system to support multiple real-time messages (chat) and updates (activity notifications).
That is, user A can receive via Web Socket messages for :
receiving new chat messages
receiving updates for some activity, for example if someone like their photo.
and more.
We use one single WebSocket connection to send all these different messages to the client.
However, we also support multiple applications/clients to be open by the user at the same time.
(i.e - user A connect on their web browser, and also from their mobile app, at the same time).
Architecture
We have a "Hub" that stores a map of UserId to a list of active websocket sessions.
(user:123 -> listOf(session#1, session#2))
Each client, once websocket connection is established, has its own Consumer which subscribes to a pulsar topic "userId" (e.g - user:123 topic).
If user A connected on both mobile and web, each client has its own Consumer to topic user:A.
When user A sends a new message from session #1 to user B, the flow is :
user makes a REST POST request to send a message.
service stores a new message to DB.
service sends a Pulsar message to topic user:B and user:A.
return 200 status code + created Message response.
Problem
If user A has two sessions open (two clients/websockets), and they send a message from session #1, how can we make sure only session #2 gets the message ?
Since user A has already received the 200 response with the created message in session #1, there's no need to send the message to him again by sending a message to his Consumer.
I'm not sure if it's a Pulsar configuration, or perhaps our architecture is wrong.
how can we make sure only session #2 gets the message ?
I'm going to address this at the app level.
Prepend a unique nonce (e.g. a guid) to each message sent.
Maintain a short list of recently sent nonces,
aging them out so we never have more than, say, half a dozen.
Upon receiving a message,
check to see if we sent it.
That is, check to see if its nonce is in the list.
If so, silently discard it.
Equivalently, name each connection.
You could roll a guid just once when a new websocket is opened.
Or you could incorporate some of the websocket's addressing
bits into the name.
Prepend the connection name to each outbound message.
Discard any received message which has "sender" of "self".
With this de-dup'ing approach
there's still some wasted network bandwidth.
We can quibble about it if you wish.
When the K-th websocket is created,
we could create K topics,
each excluding a different endpoint.
Sounds like more work than it's worth!
The use case is to receive device connected event as soon as the device is connected to the iothub.
There is a device connected/disconnected event that can be captured and routed to eventhub/servicebus/eventgrid, but, this event isn’t triggered in Amqp if the message is sent by .net program i.e it is inconsistent.
Is there any mechanism available to get the recent device connected event in Iothub! Or heartbeat pattern is the only mechanism available or is most efficient way?
Your understanding is correct. Suggestion is to use the Device heartbeat pattern.
The IoT Hub identity registry contains a field called connectionState. Only use the connectionState field during development and debugging. IoT solutions should not query the field at run time. For example, do not query the connectionState field to check if a device is connected before you send a cloud-to-device message or an SMS. We recommend subscribing to the device disconnected event on Event Grid to get alerts and monitor the device connection state. Use this tutorial to learn how to integrate Device Connected and Device Disconnected events from IoT Hub in your IoT solution.
If your IoT solution needs to know if a device is connected, you can implement the heartbeat pattern. In the heartbeat pattern, the device sends device-to-cloud messages at least once every fixed amount of time (for example, at least once every hour). Therefore, even if a device does not have any data to send, it still sends an empty device-to-cloud message (usually with a property that identifies it as a heartbeat). On the service side, the solution maintains a map with the last heartbeat received for each device. If the solution does not receive a heartbeat message within the expected time from the device, it assumes that there is a problem with the device.
I'm using the stomp-client library and i want to know if it is possible to know if the message was delivered to the queue. Because im implementing a java service to do the dequeue of the messages and an node js to send the messages to the queue. the code bellow shows how I send the message to the queue.
this._stompClient.publish('/queue/MessagesQueue', messageToPublish, { })
When you send a SEND frame (i.e. publish a message) you can add a receipt header and then when you receive the RECEIPT frame from the broker you know it has successfully received the message. The STOMP specification says this about the receipt header:
Any client frame other than CONNECT MAY specify a receipt header with an arbitrary value. This will cause the server to acknowledge receipt of the frame with a RECEIPT frame which contains the value of this header as the value of the receipt-id header in the RECEIPT frame.
However, looking at the documentation for stomp-client I don't see any mention of how to receive RECEIPT frames. I actually would expect the ability to specify a callback on the publish method which was called when the RECEIPT frame is received. It doesn't appear that stomp-client supports working with receipts. Unfortunately that means there's no real way to confirm the message was received by the broker.
I recommend you find a more mature STOMP client implementation that supports receipts. For example stomp-js supports receipts.
I'm trying to use the request reply pattern as described in the microsoft docs (https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-sessions#request-response-pattern)
"Multiple applications can send their requests to a single request queue, with a specific header parameter set to uniquely identify the sender application. The receiver application can process the requests coming in the queue and send replies on the session enabled queue, setting the session ID to the unique identifier the sender had sent on the request message. The application that sent the request can then receive messages on the specific session ID and correctly process the replies."
As I understand it, it should be possible to send a message from multiple applications, have the receiver handle the message and send a response that will only be picked up by the initial sender.
Maybe I'm wrong, but a bit like this.
This doesn't seem to be documented (only using sessions for ordered message handling) and I have no luck finding how to implement this.
Does anybody have an idea/experience with this?
I am using .net core 3.1 with the microsoft azure servicebus package (4.1.2)
Ok, it took some time figuring out but I think I was able to achieve the setup from the diagram.
Here is the process as it may help others:
I have one normal queue (PostNL queue) and one shared 'applications' queue that is sessions enabled
An application (e.g App1) sends a message to the PostNL queue using a QueueClient and setting a unique SessionId
The receiver handles the incoming messages through QueueClient.RegisterMessageHandler
The receiver processes the message and sends a reply to the applications queue using QueueClient.SendAsync (the replymessage has the SessionId set to UniqueSessionId)
The sender uses a session = SessionClient.AcceptMessageSessionAsync("UniqueSessionId")
The sender can start receiving messages in this session using session.ReceiveAsync
(all the other applications listening on the applications queue will not compete for these reply messages as long as they use other session Ids)
Is there any way to implement request-response pattern with mosca MQTT to "check reply from the client and re publish if i dont receive expected reply within expected time".
I believe this is possible in Mqtt 5, but as of now, I have to use Mosca broker with QoS 1(which support until Mqtt 3.1.1)
I am looking for a Node js workaround to achieve this.
As per my comment you can implement a request-response pattern with any MQTT broker but, prior to v5, you need to implement this yourself (either have a single reply-to topic and a message ID, or include a specific reply-to topic within each message).
Because MQTT 3.11 itself does not provide this functionality directly and there is no standard format for the MQTT payload (just some bytes!) its not possible to come up with a generic implementation (a unique id of some kind is needed within the request). This is resolved in MQTT v5 through the ability to include properties including Response Topic and Correlation Data. For earlier versions you are stuck with adding some extra information into the payload (using whatever encoding mechanism you choose).
There are a few Stack Overflow questions that might provide some insight:
MQTT topic names for request/response
RPC style request with MQTT
Other articles:
Eclipse Kura
Stock Explorer
IoT Application Development Using Request-Response Pattern with MQTT (Academic article - purchase needed to read whole thing).
Amazon device shadow MQTT topics (e.g. send message to $aws/things/thingName/shadow/get and AWS IoT responds on /get/accepted or /get/rejected).
Here are a few node packages (note: these have not been updated for some time and I have not reviewed the code):
replyer
resmetry
Even with MQTT v5 you would need to implement the idle timeout bit yourself. If you are using QOS 1/2 then the broker will take care of resending the message (until it receives a PUBACK/PUBCOMP) so resending the message may be counterproductive (lots of identical messages queued up while the comms link is down)
The summary of the workflow i have done
Adding "Correlation Id" for each message
The expected reply is stored in Redis as the Request Payload(Request with the
Correlation Id as a key) to compare response from the client.
The entry will be removed from Redis if the expected message is
equivalent to the expected response topic and payload.
Time out uses node cron jobs for each response from the client to
Server.