Node Kakfa receiving BrokerNotAvailable errors - node.js

Question

I'm trying to get started with NodeJS and Kafka. I have one Kafka server running in a docker container with one Zookeeper server.
I can successfully connect a listener to my topic and when my code behaves I see messages passed from Producer to Consumer.
However - more often than not, my code fails with an error. Here are two consecutive attempts to Produce a message into the Kafka topic
2018-01-09T22:14:29.061Z - info: KafkaProducer: client is ready
err: null
data: [object Object]
err: BrokerNotAvailableError: Broker not available
data: undefined
2018-01-09T22:15:03.770Z - info: KafkaProducer: client is ready
The code that I'm using to publish the message is here:
'use strict';
const log = require('./Logger');
const kafka = require('kafka-node');
const Producer = kafka.Producer;
const KeyedMessage = kafka.KeyedMessage;
const Client = kafka.Client;
const Promise = require('bluebird');
const uuid = require('uuid');
const client = new Client('192.168.99.100:2181', 'test', {
sessionTimeout: 300,
spinDelay: 100,
retries: 2
});
const producer = new Producer(client, {requireAcks: 1});
producer.on('ready', () => {
log.info('KafkaProducer: client is ready');
});
producer.on('error', (err) => {
log.error(`KafkaProducer: ${err}`);
});
class KafkaProducer {
constructor() {}
sendRecord(type, userId, sessionId, data, callback) {
if (!userId) {
return callback(new Error('A userId must be provided.'));
}
const event = {
id: uuid.v4(),
timestamp: Date.now(),
userId: userId,
sessionId: sessionId,
type: type,
data: data
};
const buffer = new Buffer.from(JSON.stringify(event));
// Create a new payload
const record = [
{
topic: 'datastore',
messages: buffer,
attributes: 1 /* Use GZip compression for the payload */
}
];
//Send record to Kafka and log result/error
producer.send(record, callback);
}
}
module.exports = KafkaProducer;
Not particularly elegant code, but I'm just trying to get a simple use case working before integrating Kafka into my application.
If it is relevant this are my Dockerfiles for Kafka and Zookeeper
FROM wurstmeister/kafka:1.0.0
ENV KAFKA_ADVERTISED_HOST_NAME service-kafka
ENV KAFKA_PORT 9092
ENV KAFKA_HOST_NAME service-kafka
ENV KAFKA_ZOOKEEPER_CONNECT service-zookeeper:2181
ENV KAFKA_CREATE_TOPICS "auth:1:1,datastore:1:1,transactions:1:1"
and
FROM zookeeper:latest
This comment in a GitHub issue seems to suggest that it is the applications responsibility to catch connection errors and retry the message, but this happens so frequently there is something else wrong that I need some guidance with. Thanks