Connect to kafka running in Azure Container Instance from outside - azure

Question

I have a kafka instance running in Azure Container instance. I want to connect to it (send messages) from outside the container (from application running on external server/local computer or another container).
After searching the internet, I understand that we need to provide the external IpAddress to kafka listener which would be listening from outside to connect.
Eg: KAFKA_ADVERTISED_LISTENERS: PLAINTEXT_INTERNAL://kafkaserver:29092,PLAINTEXT://<ip-address>:9092
But since azure container instance gets ip address after it has spin up how can we connect in this case?
docker-compose.yaml
version: '3.9'
services:
zookeeper:
image: confluentinc/cp-zookeeper:7.0.1
container_name: zookeeper
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
KAFKA_JMX_PORT: 39999
volumes:
- ../zookeeper_data:/var/lib/zookeeper/data
- ../zookeeper_log:/var/lib/zookeeper/log
networks:
- app_net
#*************kafka***************
kafkaserver:
image: confluentinc/cp-kafka:7.0.1
container_name: kafkaserver
ports:
# To learn about configuring Kafka for access across networks see
# https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
- "9092:9092"
depends_on:
- zookeeper
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT_INTERNAL://kafkaserver:29092,PLAINTEXT://localhost:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_JMX_PORT: 49999
volumes:
- ../kafka_data:/var/lib/kafka/data
networks:
- app_net
networks:
app_net:
driver: bridge

Answer 1

You could create an EventHubs cluster with Kafka support instead...
But if you want to run Kafka in Docker, the Confluent image would need extended with your own Dockerfile that would inject your own shell script between these lines which would use some shell command to fetch the external listener defined at runtime.
e.g. Create aci-run file with this section
echo "===> Configuring for ACI networking ..."
/etc/confluent/docker/aci-override
echo "===> Configuring ..."
/etc/confluent/docker/configure
echo "===> Running preflight checks ... "
/etc/confluent/docker/ensure
(Might need source /etc/confluent/docker/aci-override ... I haven't tested this)
Create a Dockerfile like so and build/push to your registry
ARG CONFLUENT_VERSION=7.0.1
FROM confluentinc/cp-kafka:${CONFLUENT_VERSION}
COPY aci-override /etc/confluent/docker/aci-override
COPY aci-run /etc/confluent/docker/run # override this file
In aci-override
#!/bin/bash
ACI_IP=...
ACI_EXTERNAL_PORT=...
ACI_SERVICE_NAME=...
export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://${ACI_IP}:${ACI_EXTERNAL_PORT}
You can remove localhost listener since you want to connect externally.
Then update the YAML to run that image.
I know Heroku, Apache Mesos, Kubernetes, etc all set some PORT environment variable within the container when it starts. I'm not sure what that is for ACI, but if you can exec into a simple running container and run env, you might see it.