How to prevent Python Kafka Producer fail due to use of logging - python-3.x

Question

For some reason when I add any logging code before using a Kafka producer the connection fails. It took me some time to isolate the problem because the error logs pertained to the connection.
Without these two logging lines the code works just fine:
logging.basicConfig(level=logging.INFO)
here is the faulty code:
#!/usr/bin/env python
import traceback
import logging
from kafka import KafkaProducer
from kafka.errors import KafkaError
def produce_test(listen=False):
topics = ['topic1', 'topic2']
messages = [['key1', 'logging hell'], ['key1', 'silly defaults']]
encoded = []
for m in messages:
encoded.append((str(m[0]).encode('utf-8'), str(m[1]).encode('utf-8')))
logging.info(f"something I want to log")
producer = KafkaProducer(
bootstrap_servers='localhost:9092',
max_request_size=15048576,
)
for t in topics:
for m in encoded:
# pt = producer.partitions_for(t)
# print(f'partitions for {t}: {pt}')
print(f"sending: {m} to topic: {t}")
future = producer.send(topic=t, key=m[0], value=m[1])
producer.flush()
if listen:
try:
record_metadata = future.get(timeout=10)
# print(f'record_metadata: {record_metadata}')
except KafkaError:
# Decide what to do if produce request failed...
print(traceback.format_exc())
result = 'Fail'
finally:
producer.close()
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)
print('in producer calzone di Napoli')
produce_test()
here is the error log:
in producer calzone di Napoli
INFO:root:something I want to log
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-0 host=localhost:9092 <connecting> [IPv6 ('::1', 9092, 0, 0)]>: connecting to localhost:9092 [('::1', 9092, 0, 0) IPv6]
INFO:kafka.conn:Probing node bootstrap-0 broker version
ERROR:kafka.conn:Connect attempt to <BrokerConnection node_id=bootstrap-0 host=localhost:9092 <connecting> [IPv6 ('::1', 9092, 0, 0)]> returned error 61. Disconnecting.
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-0 host=localhost:9092 <connecting> [IPv6 ('::1', 9092, 0, 0)]>: Closing connection. KafkaConnectionError: 61 ECONNREFUSED
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-0 host=localhost:9092 <connecting> [IPv6 ('::1', 9092, 0, 0)]>: connecting to localhost:9092 [('::1', 9092, 0, 0) IPv6]
ERROR:kafka.conn:Connect attempt to <BrokerConnection node_id=bootstrap-0 host=localhost:9092 <connecting> [IPv6 ('::1', 9092, 0, 0)]> returned error 61. Disconnecting.
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-0 host=localhost:9092 <connecting> [IPv6 ('::1', 9092, 0, 0)]>: Closing connection. KafkaConnectionError: 61 ECONNREFUSED
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-0 host=localhost:9092 <connecting> [IPv4 ('127.0.0.1', 9092)]>: connecting to localhost:9092 [('127.0.0.1', 9092) IPv4]
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-0 host=localhost:9092 <connecting> [IPv4 ('127.0.0.1', 9092)]>: Connection complete.
sending: (b'key1', b'logging hell') to topic: topic1
INFO:kafka.conn:Broker version identifed as 1.0.0
INFO:kafka.conn:Set configuration api_version=(1, 0, 0) to skip auto check_version requests on startup
INFO:kafka.conn:<BrokerConnection node_id=0 host=localhost:9092 <connecting> [IPv6 ('::1', 9092, 0, 0)]>: connecting to localhost:9092 [('::1', 9092, 0, 0) IPv6]
ERROR:kafka.conn:Connect attempt to <BrokerConnection node_id=0 host=localhost:9092 <connecting> [IPv6 ('::1', 9092, 0, 0)]> returned error 61. Disconnecting.
INFO:kafka.conn:<BrokerConnection node_id=0 host=localhost:9092 <connecting> [IPv6 ('::1', 9092, 0, 0)]>: Closing connection. KafkaConnectionError: 61 ECONNREFUSED
WARNING:kafka.client:Node 0 connection failed -- refreshing metadata
INFO:kafka.conn:<BrokerConnection node_id=0 host=localhost:9092 <connecting> [IPv4 ('127.0.0.1', 9092)]>: connecting to localhost:9092 [('127.0.0.1', 9092) IPv4]
INFO:kafka.conn:<BrokerConnection node_id=0 host=localhost:9092 <connecting> [IPv4 ('127.0.0.1', 9092)]>: Connection complete.
INFO:kafka.conn:<BrokerConnection node_id=bootstrap-0 host=localhost:9092 <connected> [IPv4 ('127.0.0.1', 9092)]>: Closing connection.
sending: (b'key1', b'silly defaults') to topic: topic1
sending: (b'key1', b'logging hell') to topic: topic2
sending: (b'key1', b'silly defaults') to topic: topic2
INFO:kafka.producer.kafka:Closing the Kafka producer with 0 secs timeout.
INFO:kafka.producer.kafka:Proceeding to force close the producer since pending requests could not be completed within timeout 0.
INFO:kafka.conn:<BrokerConnection node_id=0 host=localhost:9092 <connected> [IPv4 ('127.0.0.1', 9092)]>: Closing connection.
Process finished with exit code 0
I'm guessing the logger interferes with the Kafka library logger, but I just switched to confluent_kafka and bypassed the problem.

Answer 1

The logs you pasted show a producer successfully sending messages. Your application is working!
The couple of errors probably also happen when you disable logging, you just don't see them. Enabling logging does not break your application.
These errors happen because you're using localhost as a bootstrap server. The Kafka client has to resolve that name to an IP in order to connect to it. On your machine, it looks like localhost resolves to 2 IPs:
::1 on IPv6: The client tries to connect to that and fails
127.0.0.1 on IPv4: The client tries to connect to that and succeeds