I am building a specific device based on Node, Cylon and am publishing events to a MQTT broker. I'd like to know how to perform a certain action once a certain MQTT message comes to the device. Can anybody point me in the right direction? I'm a bit lost in the matter ;)
I use this to publish data:
mqtt.publish(thingTopic, JSON.stringify(data));
I'd like to create something like this:
if certain message arrives at broker -> do a post or get request to internal url.
The question is a bit vague, i must admit...
You would probably need to build your own custom MQTT broker to achieve what you are looking for which is not really the point of the pub/sub message paradigm. Instead of customizing the MQTT broker, look into creating your own subscribing application that will react to messages being received from the MQTT Broker.
Hopefully the following sequence diagram will help understand.
Related
I want to make an app which lets users comment and send messages. However, the notifications for these events will have to come instantly, just like any other social-media or chat application. This is what I'm thinking of:
Web-frontend: Angular, mobile: Ioinc with Angular
Backend: Node, Mongo
Now, this is how I was thinking I'd implement real-time notification.
There's a constant socket connection between the frontend (web & mobile-app) and the backend.
Whenever a message arrives, targeted to a specific user, I'll use some kind of a Mongo-hook to send the notification to the frontend via the socket connection.
Now, the confusion with this approach is:
Would millions of socket connections work at scale, at all? If not, what is the way to implement this pub-sub kind of system? I need to do it from scratch, not using Firebase.
What if a user is offline when he receives the message in the backend? If the socket is not on, how would he get the message? Is there a way to do it using Kafka? Please explain if you have some ideas on this.
Is this the correct approach? If not, can you suggest what would be appropriate?
Would millions of socket connections work at scale, at all? If not, what is the way to implement this pub-sub kind of system? I need to do it from scratch, not using Firebase.
Yes, it can work at scale just you have to made an architecture like that. You might find this useful
Scalable architecture for socket.io
https://socket.io/docs/v3/using-multiple-nodes/
What if a user is offline when he receives the message in the backend? If the socket is not on, how would he get the message?
If he the socket is not on or user is offline, then client Socket will be disconnected. At this point, notification will not be received and whenever the user comes online you'll have make an API call to get the notifications and connect again to the socket for further operations.
Is there a way to do it using Kafka?
Yes, you can also do it with Kafka. You'll need Consumer API(Subscriber) and Producer API(Publisher)
https://kafka.apache.org/documentation/#api
https://www.npmjs.com/package/kafka-node
Sending Apache Kafka data on web page
What do you use Apache Kafka for?
Real time notification with Kafka and NodeJS
Is there any way to implement request-response pattern with mosca MQTT to "check reply from the client and re publish if i dont receive expected reply within expected time".
I believe this is possible in Mqtt 5, but as of now, I have to use Mosca broker with QoS 1(which support until Mqtt 3.1.1)
I am looking for a Node js workaround to achieve this.
As per my comment you can implement a request-response pattern with any MQTT broker but, prior to v5, you need to implement this yourself (either have a single reply-to topic and a message ID, or include a specific reply-to topic within each message).
Because MQTT 3.11 itself does not provide this functionality directly and there is no standard format for the MQTT payload (just some bytes!) its not possible to come up with a generic implementation (a unique id of some kind is needed within the request). This is resolved in MQTT v5 through the ability to include properties including Response Topic and Correlation Data. For earlier versions you are stuck with adding some extra information into the payload (using whatever encoding mechanism you choose).
There are a few Stack Overflow questions that might provide some insight:
MQTT topic names for request/response
RPC style request with MQTT
Other articles:
Eclipse Kura
Stock Explorer
IoT Application Development Using Request-Response Pattern with MQTT (Academic article - purchase needed to read whole thing).
Amazon device shadow MQTT topics (e.g. send message to $aws/things/thingName/shadow/get and AWS IoT responds on /get/accepted or /get/rejected).
Here are a few node packages (note: these have not been updated for some time and I have not reviewed the code):
replyer
resmetry
Even with MQTT v5 you would need to implement the idle timeout bit yourself. If you are using QOS 1/2 then the broker will take care of resending the message (until it receives a PUBACK/PUBCOMP) so resending the message may be counterproductive (lots of identical messages queued up while the comms link is down)
The summary of the workflow i have done
Adding "Correlation Id" for each message
The expected reply is stored in Redis as the Request Payload(Request with the
Correlation Id as a key) to compare response from the client.
The entry will be removed from Redis if the expected message is
equivalent to the expected response topic and payload.
Time out uses node cron jobs for each response from the client to
Server.
I need to listen to RabbitMQ messages, process each message just a little bit, and submit it to another exchange. Each example I have seen so far includes either this:
reader_connection.ioloop.start()
or this:
writer_connection.ioloop.start()
Because I need to both receive and send messages, I probably need to run both loops at the same time. Is there a way I could accomplish that?
There is no difference between publishers and subscribers. You can publish and subscribe using same connection or different(then you will need to start ioloop for both of them)
You can find some examples here: https://pika.readthedocs.org/en/0.10.0/examples/connecting_async.html
I am using Node.js + ZeroMQ for subscribing to a certain feed using the PUB/SUB pattern.
How could i detect the condition where my publisher is disconnected? (I am connected as a subscriber)
Another thing: is there a way to get automatically messages from the past when i first connected to the publisher?
Thanks in advance
You could publish a heartbeat and if your subscriber misses one or more in-a-row you can assume that you lost the connection and try to reconnect.
To get the messages from the past you need to use a different pattern, like REQuesting those missing messages. In this case you need a way to identify which messages are missing.
In ZeroMQ's default pubsub model, there's no way for the subscriber to get messages from the past. See the ZeroMQ documentation, where you find statements like
If you start the SUB socket (i.e., establish a connection to a PUB
socket) after the PUB socket has started sending out data, you will
lose whatever it published before the connection was made. If this is
a problem, set up your architecture so the SUB socket starts first,
then the PUB socket starts publishing.
and
Pub-sub is like a radio broadcast; you miss everything before you
join, and then how much information you get depends on the quality of
your reception.
I am trying to build a generic publish/subscribe server with nodejs and node_redis that receives requests from a browser with a channel name and responds with any data that has been published too that channel. To do this, I am using long polling requests from the browser and dealing with these requests by sending a response when a message is received on a channel.
For each new request, an obect is created for subscribing to the channel (if and only if it does not already exist).
clients = {};
//when request comes in,
clients[channel] = redis.createClient();
clients[channel].subscribe(channel);
Is this the best way to deal with the subscribtion channels, or is there some other more intuitive way?
I don't know what's your design, but you can subscribe with one redis client on multiple channels (after you subscribe with client, then you can only subscribe to other channel or unsubscribe within this connection: http://redis.io/commands/subscribe), because after you receive message, you have full information which channel this message comes from. Then you can distribute this message to all interested clients.
This helped me a little, because I could put type of message in channel name and then dynamically choose action for each message from small function, instead of generating separate subscription for each channel with separate logic.
Inside my node.js server I have only 2 redis clients:
simple client for all standard actions - lpush, sadd and so on
subscribe client - which listens for messages over subscribed channels, then this messages are distribute to all sessions (stored as sets for each channel type) using first redis client.
I would like to point you out to my post about pubsub using socket.io together with redis. Socket.io is a very good library =>
How to use redis PUBLISH/SUBSCRIBE with nodejs to notify clients when data values change?
I think the design is very simple and it should also be very scalable.
That seems like a pretty reasonable solution to me. What don't you like about it?
Something to keep in mind is that you can have multiple subscriptions on each Redis connection. This might end up complicating your logic, which is the opposite of what you are asking for. However, at scale this might be necessary. Each Redis connection is relatively inexpensive, but it does require a file descriptor and some memory.
Complete Redis Pub/Sub Example (Real-time Chat using Hapi.js & Socket.io)
We were trying to understand Redis Publish/Subscribe ("Pub/Sub") and all the existing examples were either outdated, too simple or had no tests.
So we wrote a Complete Real-time Chat using Hapi.js + Socket.io + Redis Pub/Sub Example with End-to-End Tests!
https://github.com/dwyl/hapi-socketio-redis-chat-example
The Pub/Sub component is only a few lines of node.js code:
https://github.com/dwyl/hapi-socketio-redis-chat-example/blob/master/lib/chat.js#L33-L40
Rather than pasting it here (without any context) we encourage you to checkout/try the example.
We built it using Hapi.js but the chat.js file is de-coupled from Hapi and can easily be used with a basic node.js http server or express (etc.)