I'm looking for a stable AMQP 1.0 client for NodeJS so that I can consume events from Azure Event Hubs.
I have tried using https://github.com/noodlefrenzy/node-amqp10, but it chokes sporadically on certain payloads with messages like malformed payload: Unknown code prefix: 0x30 and malformed payload: Unable to decode bytes from message body: 005375b00002810c5b7b226576656e7474797065223 (...).
I would appreciate any direction people can offer for a stable AMQP 1.0 client in NodeJS. I am surprised that I have not been able to find one so far, since otherwise it seems Azure is failing to support an entire developer base.
I'm the (co-)author of https://github.com/noodlefrenzy/node-amqp10. The error you were seeing was due to large messages being split across multiple transfer frames - something we hadn't encountered before, so hadn't implemented. We just added support for this in v2.1.0, so please check out our library again.
My co-author uses this in production against QPID and has been for months, and we both actively support it as we have time, so if there are any instabilities or missing features, we will try our best to address them. Often they are just issues where we haven't needed a feature of the (incredibly large) spec or have misinterpreted the (incredibly scattered) spec details.
You can also use the official JavaScript SDK for Azure Event Hubs, #azure/event-hubs which internally uses rhea
Related
I get the following error messages:
'Listen' claim(s) are required to perform this operation. Resource: 'sb:/XXXXX.servicebus.windows.net/queue
proton._utils.LinkDetached: receiver 21881c3f-a71a-41fc-92c5-4a8d82956cdf from queue closed due to: Condition('amqp:unauthorized-access', "Unauthorized access.
'Listen' claim(s) are required to perform this operation.'
I have listen claims setup. Not sure what I am doing wrong. The same python code when run from a separate vm works fine. I can send a message to the queue.
However only when I am running the code from an app service I hit the Listen Claims error. I am using the proton qpid python library. I am passing the keyname and key to the connection string...
Without more detail this may be hard to gander a precise fix, but some thoughts come to mind, because, candidly, I trust that error (especially if it worked from another machine.)
You've checked that the queue and key bits are making it into the app service intact?
You've checked the connection string is formatted correctly, since you mention "passing the keyname to the connection string"; making me wonder if you built it by hand rather than copying from the portal. (and if so, if it could have gotten mangled)
(at this point I'm totally guessing) Perhaps qpid is picky about talking to entity. vs namespace level connection strings?
I can't even really expect firewall/network failures, that would likely manifest TOTALLY differently.
My next debugging steps would involve trying to validate the above, and trying to connect without the qpid intermediary, as I've seen other SO posts asking the same question (re Listen failure) but no followups, so perhaps there is an anomaly there; but my lack of familiarity with qpid makes it hard to do more than guess.
Don't hesitate to reach out on our github if you run into SDK related trouble in the future, full disclosure, am a maintainer for the python library.
Is there any way to implement request-response pattern with mosca MQTT to "check reply from the client and re publish if i dont receive expected reply within expected time".
I believe this is possible in Mqtt 5, but as of now, I have to use Mosca broker with QoS 1(which support until Mqtt 3.1.1)
I am looking for a Node js workaround to achieve this.
As per my comment you can implement a request-response pattern with any MQTT broker but, prior to v5, you need to implement this yourself (either have a single reply-to topic and a message ID, or include a specific reply-to topic within each message).
Because MQTT 3.11 itself does not provide this functionality directly and there is no standard format for the MQTT payload (just some bytes!) its not possible to come up with a generic implementation (a unique id of some kind is needed within the request). This is resolved in MQTT v5 through the ability to include properties including Response Topic and Correlation Data. For earlier versions you are stuck with adding some extra information into the payload (using whatever encoding mechanism you choose).
There are a few Stack Overflow questions that might provide some insight:
MQTT topic names for request/response
RPC style request with MQTT
Other articles:
Eclipse Kura
Stock Explorer
IoT Application Development Using Request-Response Pattern with MQTT (Academic article - purchase needed to read whole thing).
Amazon device shadow MQTT topics (e.g. send message to $aws/things/thingName/shadow/get and AWS IoT responds on /get/accepted or /get/rejected).
Here are a few node packages (note: these have not been updated for some time and I have not reviewed the code):
replyer
resmetry
Even with MQTT v5 you would need to implement the idle timeout bit yourself. If you are using QOS 1/2 then the broker will take care of resending the message (until it receives a PUBACK/PUBCOMP) so resending the message may be counterproductive (lots of identical messages queued up while the comms link is down)
The summary of the workflow i have done
Adding "Correlation Id" for each message
The expected reply is stored in Redis as the Request Payload(Request with the
Correlation Id as a key) to compare response from the client.
The entry will be removed from Redis if the expected message is
equivalent to the expected response topic and payload.
Time out uses node cron jobs for each response from the client to
Server.
I have an Azure application that connects to a large installed base of devices. Everything has been working fine for years until today when everything has stopped working. What I think has happened is Azure is now segmenting the small (87 byte payload) message and this has exposed a bug in my TCP handler.
Does anyone know if there is a way of forcing Azure not to segment small TCP messages?
Follow up - I think this is because the HTTP message is 'chunked' and send as 2 TCP segments. There is a bug in my code that does not handle chunks which as only now surfaced.
Can I turn off chunking in Azure?
If your question is whether there is a way to prevent a HTTP response from using HTTP Chunked Transfer Encoding, then perhaps, depending on the server-side APIs you are using. Setting a Content-Length header usually suffices.
But I think your question is whether you can force Azure to buffer IP packets in a way that avoids the bug in your client, which is assuming that a single socket Read() will return a complete message. In which case, no. It may not even be within Azure's control, as any intermediate router could cause a delay in the delivery of packets, which, in turn, will cause the client's Read() to return a partial message.
I found a solution here Disable chunking in Asp.Net Core
In summary:
response.Headers["Content-Encoding"] = "identity";
response.Headers["Transfer-Encoding"] = "identity";
This seems to disable unnecessary chunking. I have no idea how
I'm developing a web and mobile app using spring mvc and I'm having a problem understanding the different protocols and tools to make instant messaging (like facebook messaging in the website or whatsapp instant notifications). one functionality besides chatting in the project is for a user to send a request and waiting for the other user to respond with notifying them instantly).
However I'm lost cause some say to use GCM or FCM for the mobile and others say there other protocols like STOMP and AQMP and some others. I don't know what to see and use in my rest api so that it works for both browsers and mobiles while taking performance and other issues into consideration and how to consume these messages from client (I mean does the consumption method varies based on the chosen protocol?). should I use multiple protocols and tools based on the source of the request (i.e. if mobile and Android --> GCM or FCM, if browser STOMP for example, if iOS --> don't know what to use).
I know it looks like a general question but I really got lost specially that I don't know what are the right things to choose these days to start with.
Messaging stack consists of multiple components. One of them is message transport - used to pas messages between the server and the client. FCM/GCM/APNS in this context is the transport protocol.
GCM is deprecated in favor of FCM. FCM is going to work for Android, the latest versions of iOS, and even some browsers. However if the user disables notifications for your app, the messaging will stop working. There is also some unpredictable latency with push notifications, particularly if you would like to send high-volume messages like typing notifications.
Message format is another component of the stack. For instance, STOMP is the message format protocol. It's defined for any serial transport, i.e. can be used over FCM or TCP or websocket.
Given the questions you ask it looks completely impractical for you to write your own messaging stack. Just pick something off the shelf, like one of the million XMPP servers or a more modern one like Tinode. Google it.
I have successfully connected my ESP8266 Arduino project to the Azure IoT hub by following the steps outlined here:
https://azure.microsoft.com/en-gb/documentation/articles/iot-hub-mqtt-support/
The MQTT library that I am using is the following library:
https://github.com/Imroy/pubsubclient
Using the code I have been able to publish events to azure. The problem starts when trying to read the messages that are passed to the device.
Here is the code:
http://pastebin.com/7AVLQfFh
When I sue the Azure Device Explorer to send a message to the device the I get the following results:
As you can see the payload length is the same as the number of characters in the message but the characters are not the same. And also if I vary the string sent then the vales in the result are the same.
As a sanity check I have tried the code against a different MQTT service (Adafruit IO) and the results are as to be expected (here the message is 'ON' or 'OFF'):
The Azure MQTT service must be doing something different when it sends the messages to the device, what I needs to know is what it does differently.
Just a couple of hours ago I managed to get the messages flowing from Azure IoT hub to my ESP8266 using the original PubSubClient - link here. You are using the fork of that library that has been made more object oriented and probably has more enhancements.. but if you aren't very hooked on using the newer version I'd recommend getting the original one and trying it out.
I have my code on github if you need some reference: link
One important mention: the PubSubClient.h needs changing the #defineMQTT_MAX_PACKET_SIZE 128 to #defineMQTT_MAX_PACKET_SIZE 256 or more if you work with big packets
Also, I haven't tried to send any messages so I don't know how that'll work.
Update: I've also written a short blog post on how to set it up: link
According to this documentation page, property_bag is RFC2396 encoded.
RFC 2396-encoded(<PropertyName1>)=RFC 2396-encoded(<PropertyValue1>)&RFC 2396-encoded(<PropertyName2>)=RFC 2396-encoded(<PropertyValue2>)…
But isn't ABCD the same in plain text vs URI encoded you ask... Well, yes.
Install stunnel (https://www.stunnel.org/index.html), open a TLS 1.2 connection to IoT Hub on 8883/TCP and then point MQTT.fx or Paho to it and sniff the traffic with Wireshark, see what you get.
I was able to read all messages in plain ASCII text while talking to IoT Hub over REST, so whatever this is i think it's MQTT specific.
If you haven't done so already you should open an issue on their GitHub. They seem to be quite responsive: https://github.com/Azure/azure-iot-sdks/issues