I'm following the very good tutorial
www.hivemq.com/mqtt-essentials-part-6-mqtt-quality-of-service-levels/
but I can't figure out to implement
Of course your application must be tolerating duplicates and process
them accordingly.
Can to give me a simple example, please ?
for instance now I've like
(I set up my deploy following https://medium.com/#lelylan/how-to-build-an-high-availability-mqtt-cluster-for-the-internet-of-things-8011a06bd000)
module.exports.authorizePublish = function(client, topic, payload, callback) {
var chunks = topic.split('/');
if(chunks.length === 4) {
Debug('AUTHORIZING SUBSCRIBE', client.device_id == chunks[1]);
Debug('NICKNAME', chunks[1]);
Debug('CHANNEL', chunks[3]);
Debug('TOPIC', chunks[2]);
Debug('PAYLOAD', payload.toString('utf8'));
var data = {
deviceNickname:chunks[1],
channel:chunks[3],
topic:chunks[2],
payload:payload.toString('utf8')
};
Message.insert(data, function (err, message) {
if (err){
Debug(err);
return;
}
Debug(message);
});
}
callback(null, client.device_id === chunks[1]);
}
I'm still rather a learner of MQTT than an expert, but my understanding of handling of duplicate messages with QoS 1 is following:
Suppose you have an application that for whatever reason needs to count messages received from a broker. However, you don't want to take duplicate messages (sent when your client didn't ACK the message in time) to be taken into account.
I use Java Paho client, so the code for it would be:
int counter = 0;
public void messageArrived(String topic, MqttMessage message) throws MqttException {
if (message.isDuplicate() == false) {
counter++;
}
}
Related
I'm trying to write a function to use callbacks to send a message in Facebook messenger. I need to do this because I'm having problems sending text from an array. The messages are sent, but not in the correct order. I THINK this is because Nodejs is looping over the elements faster than it can send the text. See my question about this here.
So now I am trying to rewrite my send functions using callbacks, in the vain hope that I can somehow FORCE NodeJS to actually WAIT before jumping to the next element!
So far I have the following code:
Main send function:
sendWithCallback: function(messageData, callback) {
request({
uri: 'https://graph.facebook.com/v2.6/me/messages',
qs: {
access_token: config.FB_PAGE_TOKEN
},
method: 'POST',
json: messageData
}, function (error, response, body) {
if (!error && response.statusCode === 200) {
let recipientId = body.recipient_id;
console.log("Message sent to recipient '%s'", recipientId);
callback(true);
} else {
console.error("Could not send message: ", response.statusCode, response.statusMessage, body.error)
callback(false);
}
}
);
},
Function for sending a "multi part" message (i.e. an Array of text):
sendMultipartMessage: function(recipientId, textArray) {
let messageData, msgPart, msgLength, count = 0;
msgLength = textArray.length;
while (count < msgLength) {
msgPart = textArray[count];
messageData = {
recipient: {
id: recipientId
},
message: {
text: msgPart
}
};
}
self.sendWithCallback(messageData, function(sent) {
if (sent) {
count++;
console.log("Message part %s sent.", msgPart);
}
else {
console.log("Couldn't send message");
}
});
},
In my head, this code works properly! It sends the text (taken from the array), then increments the count until it is equal to messageLength. But in reality it DOESN'T do that. Instead, it just goes into an infinite loop (which I can't see happening in my logs) and then crashes the app.
WHAT am I doing wrong?
If we simplify your loop it essentially becomes this:
let count = 0;
while (count < msgLength) {
messageData = data;
}
You never increment count.
I think that you intend to move the self.sendWithCallback call inside of the while loop. However, this still won't do what you want and will run forever. Even if it did do what you wanted, it wouldn't solve the problem of sending messages out in order.
JavaScript's concurrency model uses an event loop with "run-to-completion." You can pass messages to the event queue using something like request which you call by sendWithCallback. This only adds a message to the queue, but that message is not processed until the current running block completes. That means that your while loop actually has to complete before any of your requests start running. We can construct a simpler example with setTimeout:
let count = 0;
while (count < 1) {
setTimeout(() => {
count++;
}, 1000);
}
console.log('while loop completed');
In the above the while loop never completes because count never gets incremented in the same block (console.log will never be called). It needs to complete before it can start processing the asynchronous messages you are creating with setTimeout.
You could actually just rewrite it like this:
textArray.forEach(msgPart => self.sendWithCallback(msgPart, sent => {
if (!sent) console.error('Could not send message');
});
However this doesn't guarantee the order that the messages were sent and it will send messages even if one of the messages triggers an error. If you want to send them in order you will have to recursively call sendWithCallback within the callback for the next message once the previous one completes. That might look something like this:
let count = 0;
const sendMessage = (textArray, count) => {
self.sendMessageWithCallback(textArray[count], sent => {
count++;
if (sent && count < textArray.length) {
sendMessages(textArray, count);
}
});
}
sendMessages(textArray, 0);
If you were using promises and async/await you could write this much more simply as something like:
for (count = 0; count < msgLength; count++) {
await self.sendMessageAsync(textArray[count]);
}
However this would require a larger rewrite of the surrounding code and using something like request-promise instead of just request.
I'm developing an app with the following node.js stack: Express/Socket.IO + React. In React I have DataTables, wherein you can search and with every keystroke the data gets dynamically updated! :)
I use Socket.IO for data-fetching, so on every keystroke the client socket emits some parameters and the server calls then the callback to return data. This works like a charm, but it is not garanteed that the returned data comes back in the same order as the client sent it.
To simulate: So when I type in 'a', the server responds with this same 'a' and so for every character.
I found the async module for node.js and tried to use the queue to return tasks in the same order it received it. For simplicity I delayed the second incoming task with setTimeout to simulate a slow performing database-query:
Declaration:
const async = require('async');
var queue = async.queue(function(task, callback) {
if(task.count == 1) {
setTimeout(function() {
callback();
}, 3000);
} else {
callback();
}
}, 10);
Usage:
socket.on('result', function(data, fn) {
var filter = data.filter;
if(filter.length === 1) { // TEST SYNCHRONOUSLY
queue.push({name: filter, count: 1}, function(err) {
fn(filter);
// console.log('finished processing slow');
});
} else {
// add some items to the queue
queue.push({name: filter, count: filter.length}, function(err) {
fn(data.filter);
// console.log('finished processing fast');
});
}
});
But the way I receive it in the client console, when I search for abc is as follows:
ab -> abc -> a(after 3 sec)
I want it to return it like this: a(after 3sec) -> ab -> abc
My thought is that the queue runs the setTimeout and then goes further and eventually the setTimeout gets fired somewhere on the event loop later on. This resulting in returning later search filters earlier then the slow performing one.
How can i solve this problem?
First a few comments, which might help clear up your understanding of async calls:
Using "timeout" to try and align async calls is a bad idea, that is not the idea about async calls. You will never know how long an async call will take, so you can never set the appropriate timeout.
I believe you are misunderstanding the usage of queue from async library you described. The documentation for the queue can be found here.
Copy pasting the documentation in here, in-case things are changed or down:
Creates a queue object with the specified concurrency. Tasks added to the queue are processed in parallel (up to the concurrency limit). If all workers are in progress, the task is queued until one becomes available. Once a worker completes a task, that task's callback is called.
The above means that the queue can simply be used to priorities the async task a given worker can perform. The different async tasks can still be finished at different times.
Potential solutions
There are a few solutions to your problem, depending on your requirements.
You can only send one async call at a time and wait for the first one to finish before sending the next one
You store the results and only display the results to the user when all calls have finished
You disregard all calls except for the latest async call
In your case I would pick solution 3 as your are searching for something. Why would you use care about the results for "a" if they are already searching for "abc" before they get the response for "a"?
This can be done by giving each request a timestamp and then sort based on the timestamp taking the latest.
SOLUTION:
Server:
exports = module.exports = function(io){
io.sockets.on('connection', function (socket) {
socket.on('result', function(data, fn) {
var filter = data.filter;
var counter = data.counter;
if(filter.length === 1 || filter.length === 5) { // TEST SYNCHRONOUSLY
setTimeout(function() {
fn({ filter: filter, counter: counter}); // return to client
}, 3000);
} else {
fn({ filter: filter, counter: counter}); // return to client
}
});
});
}
Client:
export class FilterableDataTable extends Component {
constructor(props) {
super();
this.state = {
endpoint: "http://localhost:3001",
filters: {},
counter: 0
};
this.onLazyLoad = this.onLazyLoad.bind(this);
}
onLazyLoad(event) {
var offset = event.first;
if(offset === null) {
offset = 0;
}
var filter = ''; // filter is the search character
if(event.filters.result2 != undefined) {
filter = event.filters.result2.value;
}
var returnedData = null;
this.state.counter++;
this.socket.emit('result', {
offset: offset,
limit: 20,
filter: filter,
counter: this.state.counter
}, function(data) {
returnedData = data;
console.log(returnedData);
if(returnedData.counter === this.state.counter) {
console.log('DATA: ' + JSON.stringify(returnedData));
}
}
This however does send unneeded data to the client, which in return ignores it. Somebody any idea's for further optimizing this kind of communication? For example a method to keep old data at the server and only send the latest?
Edit: I know JS is asynchronous, I have looked over the How to Return thread. The issue I'm having is that going from "foo" examples to something specific = I'm not quite sure where to re-format this.
Also here is some context: https://github.com/sharkwheels/beanballs/blob/master/bean-to-osc-two.js
I have a question about returns in node. It might be a dumb question, but here goes. I have a function that connects to a socket, and gets OSC messages from processing:
var sock = dgram.createSocket("udp4", function(msg, rinfo) {
try {
// get at all that info being sent out from Processing.
//console.log(osc.fromBuffer(msg));
var getMsg = osc.fromBuffer(msg);
var isMsg = getMsg.args[0].value;
var isName = getMsg.args[1].value;
var isAdd = getMsg.address;
var isType = getMsg.oscType;
// make an array out of it
var isAll = [];
isAll.push(isName);
isAll.push(isMsg);
isAll.push(isAdd);
isAll.push(isType);
// return the array
console.log(isAll);
return isAll;
} catch (error) {
console.log(error);
}
});
Below I have the start of another function, to write some of that array to a BLE device. It needs name and characteristics from a different function. How do I get the below function to use isAll AND two existing parameters?
var writeToChars = function (name, characteristics) { // this is passing values from the BLE setup function
// i need to get isAll to here.
// eventually this will write some values from isAll into a scratch bank.
}
Thanks.
async call in this case be written something like this. state can be maintained in the variables in closure if required. In this particular case - you can do without any state (isAll) as well.
var isAll;
var soc = dgram.createSocket('udp4', oncreatesocket);
function oncreatesocket(msg, rinfo)
{
isAll = parseMessage(msg);
writeData(isAll);
}
function parseMessage(msg) {
...
// code to parse msg and return isAll
}
function writeData() {}
if the writeData is small enough function. It can be inside oncreatesocket without impacting the readability of the code.
Alright. So I figured out what to do, at least in this scenario. I'm sure there is a better way to do this, but for now, this works.
I'm mapping an existing global array of peripherals into the write function, while passing the OSC message to it as a parameter. This solved my issue of "how do I get two pieces of information to the same place". It figures out which peripheral is which and writes a different value to each scratch bank of each peripheral accordingly. Leaving here for future reference.
var writeToBean = function(passThrough){
var passThrough = passThrough;
console.log("in Write to bean: ", passThrough);
_.map(beanArray, function(n){
if(n.advertisement.localName === passThrough.name){
//var name = n.advertisement.localName;
n.discoverSomeServicesAndCharacteristics(['a495ff20c5b14b44b5121370f02d74de'], [scratchThr], function(error, services, characteristics){
var service = services[0];
var characteristic = characteristics[0];
var toSend = passThrough.msg;
console.log("service", service);
console.log("characteristic", characteristic);
if (toSend != null) {
characteristic.write(new Buffer([toSend]), false, function(error) {
if (error) { console.log(error); }
console.log("wrote " + toSend + " to scratch bank 3");
});
}
// not sure how to make the program resume, it stops here. No error, just stops processing.
});
}
});
}
Im using amqp.node library to integrate rabbitmq into my system.
But in consumer i want to process just one message at the time, then acknowledge the message then consume the next message from the queue.
The current code is:
// Consumer
open.then(function(conn) {
var ok = conn.createChannel();
ok = ok.then(function(ch) {
ch.assertQueue(q);
ch.consume(q, function(msg) {
if (msg !== null) {
othermodule.processMessage(msg, function(error, response){
console.log(msg.content.toString());
ch.ack(msg);
});
}
});
});
return ok;
}).then(null, console.warn);
The ch.consume will process all messages in the channel at one time and the function of the module call it here othermodule will not be executed in the same time line.
I want to wait for the othermodule function to finish before consume the next message in the queue.
At this moment (2018), I think RabbitMQ team has an option to do that:
https://www.rabbitmq.com/tutorials/tutorial-two-javascript.html
ch.prefetch(1);
In order to defeat that we can use the prefetch method with the value
of 1. This tells RabbitMQ not to give more than one message to a
worker at a time. Or, in other words, don't dispatch a new message to
a worker until it has processed and acknowledged the previous one.
Instead, it will dispatch it to the next worker that is not still
busy.
Follow up the example here :
https://www.npmjs.com/package/amqplib
// Consumer
function consumer(conn) {
var ok = conn.createChannel(on_open);
function on_open(err, ch) {
if (err != null) bail(err);
ch.assertQueue(q);
// IMPORTANT
ch.prefetch(1);
ch.consume(q, function(msg) {
if (msg !== null) {
console.log(msg.content.toString());
ch.ack(msg);
}
});
}
}
Refs: http://www.squaremobius.net/amqp.node/channel_api.html#channel_prefetch
You need to set a prefetch value as shown in this example:
https://github.com/squaremo/amqp.node/blob/master/examples/tutorials/rpc_server.js#L22
When you create the model you need to set the QOS on it. Here is how we would do it in C#:
var _model = rabbitConnection.CreateModel();
// Configure the Quality of service for the model. Below is how what each setting means.
// BasicQos(0="Dont send me a new message untill I’ve finshed", _fetchSize = "Send me N messages at a time", false ="Apply to this Model only")
_model.BasicQos(0, _fetchSize, false);
var consumerTag = _model.BasicConsume(rabbitQueue.QueueName, false, _consumerName, queueingConsumer);
You have to set QoS = 1.
ch = ...
ch.qos(1);
ch.consume(q, msg => { ... });
(javascript)
I have code like that
var subscribeNewMessages = require("redis").createClient(config.redis.port, config.redis.host);
subscribeNewMessages.subscribe('new-messages');
io.of('/new-messages').on('connection', function (client) {
subscribeNewMessages.on("message", function(channel, message) {
var obj = JSON.parse(message);
if (client.userId == obj.toUserId || client.guestId == obj.toUserId) {
client.send(message);
}
obj = null;
});
})
And how can I optimize it? Because this code parses string json for each new messages.
Also when I try to publish to redis chanel I need to JSON.stringify
redis1.publish(channelPrefix, JSON.stringify(clientData));
There isn't going to be a way to avoid JSON.parse()/JSON.stringify() as long as you're storing js objects. You could use a different serialization format like msgpack, but you're still calling functions to serialize/unserialize your data (also JSON.parse()/JSON.stringify() are already pretty hard to beat performance-wise in node).
I think the only real performance adjustment you could make with the code you've provided is to only parse the JSON once for all clients instead of for each client. Example:
var subscribeNewMessages = require('redis').createClient(config.redis.port, config.redis.host);
subscribeNewMessages.subscribe('new-messages');
var nsNewMsgs = io.of('/new-messages');
subscribeNewMessages.on('message', function(channel, message) {
var obj = JSON.parse(message),
clients = nsNewMsgs.connected,
ids = Object.keys(clients);
for (var i = 0, len = ids.length, client; i < len; ++i) {
client = clients[ids[i]];
if (client.userId == obj.toUserId || client.guestId == obj.toUserId)
client.send(message);
}
});
Depending on your application, you might even be able to avoid the for-loop entirely if you can store the socket.id values in your published messages, then you can simply look up clients[obj.userSockId] and clients[obj.guestSockId] because the connected is keyed on socket.id.