Read specific messages using Apache Kafka and NodeJS - node.js

Question

I want to build an API with NodeJS and Kafka, which can take an offset and a topic as an input and output the first 10 messages starting from the offset. I tried this approach with No-Kafka and Kafka-Node.
The consumer API provided by them allows consuming messages from a particular offset. I want to stop consuming the messages once I have read around 10 messages. But both the API calls will continue to fetch the messages till the last message. How can I stop doing that?
Here is my EDITED FULL CODE
var Kafka = require('no-kafka');
var express = require("express");
var app = express();
var producer = new Kafka.Producer();
producer.init().then(function() {
console.log("Producer Ready");
});
var consumer = new Kafka.SimpleConsumer();
consumer.init().then(function() {
console.log("Consumer Ready");
});
app.get('/produce/:topic/:msg', function(req, res) {
producer.send({
topic: req.params.topic,
partition: 0,
message: {
value: req.params.msg
}
});
res.send("Added: " + req.params.msg + " to topic: " + req.params.topic);
});
app.get('/consume/:topic/:off', function(req, res) {
console.log("Request for topic: " + req.params.topic + " Offset: " + req.params.off);
consumer.subscribe(req.params.topic, 0, {
offset: req.params.off,
maxBytes: 1000
}, function(messageSet, topic, partition) {
var msg = "";
var size = messageSet.length;
//console.log(messageSet);
messageSet.some(function(m) {
msg += m.message.value.toString('utf8') + " ";
if (parseInt(m.offset, 10) > parseInt(req.params.off, 10) + 10) {
return true;
}
});
res.send("Thank you " + size + " " + req.params.off + " " + msg);
});
});
app.listen(process.env.PORT);
Any response in this regard is appreciated.

Answer 1

You can't really stop consuming from Kafka so abruptly, for a couple of different reasons. For one thing, Kafka consumers, whether JavaScript or something else, don't read a message at a time -- they fetch batches of messages. I know with kafka-node it seems like they come in one at a time, since you get an EventEmitter event for each message. But under the hood, the client fetches them in batches.
The best you can do is to to keep track of your offsets as you go, and when you get outside of the range you want just to ignore them, and then unsubsribe the topic or close the consumer to stop listening.
This definitely gets trickier with partitions -- you have to keep track of offsets relative to all of your partitions. I don't do the same thing you do -- my typical use case is to read from a point in time up to the current offset for each partition. So I have not optimized my partition reads to die out as soon as they hit their last offset. I do addTopics and add all the partions at once. You, on the other hand, probably need to add the partitions one at a time -- i.e. do addTopic for a specific partition, read that partition until you find your offset, then ignore messages and removeTopic on the partition.
I believe I played around with that flow, and you might even have to stand up a new consumer for each partition, not to mention a whole new client.

Answer 2

I was also working on a similar project. But what I did is, I set a timeout and made an agreement that, pass me the offset and number of records u want to get. I will fetch records from the offset you send but I cannot guarantee the number of records we give you. It might fetch less messages if timeout happens. But we give you offset of the last read record so that you can call again with that offset.
(As #David Griffin said, the problem is we have to create new client each time for each partition. OR store your data in only one partition and get it from that partition.

Answer 3

you can reduce the maxBytes by 10 or less depends on the size of the message or you can do something like this:
if(req.body.off>= req.body.off+10){//return res..}