Hi I'm trying to run a Kafka server on a RedHat server:
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: RedHatEnterpriseServer
Description: Red Hat Enterprise Linux Server release 6.6 (Santiago)
Release: 6.6
Codename: Santiago
The server has java installed:
java version "1.7.0_79"
OpenJDK Runtime Environment (suse-2.5.5.3.el6_6-x86_64 u79-b14)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
I start zookeeper without any problem but when trying to start kafka bin/kafka-server-start.sh config/server.properties It prompts an error:
[2015-08-27 11:03:01,542] INFO Verifying properties (kafka.utils.VerifiablePrope rties)
[2015-08-27 11:03:01,585] INFO Property broker.id is overridden to 0 (kafka.util s.VerifiableProperties)
[2015-08-27 11:03:01,585] INFO Property log.cleaner.enable is overridden to fals e (kafka.utils.VerifiableProperties)
[2015-08-27 11:03:01,585] INFO Property log.dirs is overridden to /tmp/kafka-log s (kafka.utils.VerifiableProperties)
[2015-08-27 11:03:01,586] INFO Property log.retention.check.interval.ms is overr idden to 300000 (kafka.utils.VerifiableProperties)
[2015-08-27 11:03:01,586] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)
[2015-08-27 11:03:01,586] INFO Property log.segment.bytes is overridden to 10737 41824 (kafka.utils.VerifiableProperties)
[2015-08-27 11:03:01,586] INFO Property num.io.threads is overridden to 8 (kafka .utils.VerifiableProperties)
[2015-08-27 11:03:01,586] INFO Property num.network.threads is overridden to 3 ( kafka.utils.VerifiableProperties)
[2015-08-27 11:03:01,587] INFO Property num.partitions is overridden to 1 (kafka .utils.VerifiableProperties)
[2015-08-27 11:03:01,587] INFO Property num.recovery.threads.per.data.dir is ove rridden to 1 (kafka.utils.VerifiableProperties)
[2015-08-27 11:03:01,587] INFO Property port is overridden to 9092 (kafka.utils. VerifiableProperties)
[2015-08-27 11:03:01,587] INFO Property socket.receive.buffer.bytes is overridde n to 102400 (kafka.utils.VerifiableProperties)
[2015-08-27 11:03:01,587] INFO Property socket.request.max.bytes is overridden t o 104857600 (kafka.utils.VerifiableProperties)
[2015-08-27 11:03:01,588] INFO Property socket.send.buffer.bytes is overridden t o 102400 (kafka.utils.VerifiableProperties)
[2015-08-27 11:03:01,588] INFO Property zookeeper.connect is overridden to local host:2181 (kafka.utils.VerifiableProperties)
[2015-08-27 11:03:01,588] INFO Property zookeeper.connection.timeout.ms is overr idden to 6000 (kafka.utils.VerifiableProperties)
[2015-08-27 11:03:01,656] INFO [Kafka Server 0], starting (kafka.server.KafkaSer ver)
[2015-08-27 11:03:01,658] INFO [Kafka Server 0], Connecting to zookeeper on loca lhost:2181 (kafka.server.KafkaServer)
[2015-08-27 11:03:01,668] INFO Starting ZkClient event thread. (org.I0Itec.zkcli ent.ZkEventThread)
[2015-08-27 11:03:01,678] INFO Client environment:zookeeper.version=3.4.6-156996 5, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
[2015-08-27 11:03:01,678] INFO Client environment:host.name=<NA> (org.apache.zoo keeper.ZooKeeper)
[2015-08-27 11:03:01,678] INFO Client environment:java.version=1.7.0_79 (org.apa che.zookeeper.ZooKeeper)
[2015-08-27 11:03:01,678] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2015-08-27 11:03:01,678] INFO Client environment:java.home=/usr/lib/jvm/java-1. 7.0-openjdk-1.7.0.79.x86_64/jre (org.apache.zookeeper.ZooKeeper)
[2015-08-27 11:03:01,679] INFO Client environment:java.class.path=:/opt/kafka_2. 9.1-0.8.2.1/bin/../core/build/dependant-libs-2.10.4*/*.jar:/opt/kafka_2.9.1-0.8. 2.1/bin/../examples/build/libs//kafka-examples*.jar:/opt/kafka_2.9.1-0.8.2.1/bin /../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/opt/kafka_2. 9.1-0.8.2.1/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.ja r:/opt/kafka_2.9.1-0.8.2.1/bin/../clients/build/libs/kafka-clients*.jar:/opt/kaf ka_2.9.1-0.8.2.1/bin/../libs/jopt-simple-3.2.jar:/opt/kafka_2.9.1-0.8.2.1/bin/.. /libs/kafka_2.9.1-0.8.2.1.jar:/opt/kafka_2.9.1-0.8.2.1/bin/../libs/kafka_2.9.1-0 .8.2.1-javadoc.jar:/opt/kafka_2.9.1-0.8.2.1/bin/../libs/kafka_2.9.1-0.8.2.1-scal adoc.jar:/opt/kafka_2.9.1-0.8.2.1/bin/../libs/kafka_2.9.1-0.8.2.1-sources.jar:/o pt/kafka_2.9.1-0.8.2.1/bin/../libs/kafka_2.9.1-0.8.2.1-test.jar:/opt/kafka_2.9.1 -0.8.2.1/bin/../libs/kafka-clients-0.8.2.1.jar:/opt/kafka_2.9.1-0.8.2.1/bin/../l ibs/log4j-1.2.16.jar:/opt/kafka_2.9.1-0.8.2.1/bin/../libs/lz4-1.2.0.jar:/opt/kaf ka_2.9.1-0.8.2.1/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka_2.9.1-0.8.2.1/bin /../libs/scala-library-2.9.1.jar:/opt/kafka_2.9.1-0.8.2.1/bin/../libs/slf4j-api- 1.7.6.jar:/opt/kafka_2.9.1-0.8.2.1/bin/../libs/slf4j-log4j12-1.6.1.jar:/opt/kafk a_2.9.1-0.8.2.1/bin/../libs/snappy-java-1.1.1.6.jar:/opt/kafka_2.9.1-0.8.2.1/bin /../libs/zkclient-0.3.jar:/opt/kafka_2.9.1-0.8.2.1/bin/../libs/zookeeper-3.4.6.j ar:/opt/kafka_2.9.1-0.8.2.1/bin/../core/build/libs/kafka_2.10*.jar (org.apache.z ookeeper.ZooKeeper)
[2015-08-27 11:03:01,679] INFO Client environment:java.library.path=/usr/java/pa ckages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper )
[2015-08-27 11:03:01,679] INFO Client environment:java.io.tmpdir=/tmp (org.apach e.zookeeper.ZooKeeper)
[2015-08-27 11:03:01,679] INFO Client environment:java.compiler=<NA> (org.apache .zookeeper.ZooKeeper)
[2015-08-27 11:03:01,679] INFO Client environment:os.name=Linux (org.apache.zook eeper.ZooKeeper)
[2015-08-27 11:03:01,679] INFO Client environment:os.arch=amd64 (org.apache.zook eeper.ZooKeeper)
[2015-08-27 11:03:01,679] INFO Client environment:os.version=2.6.32-504.23.4.el6 .x86_64 (org.apache.zookeeper.ZooKeeper)
[2015-08-27 11:03:01,679] INFO Client environment:user.name=root (org.apache.zoo keeper.ZooKeeper)
[2015-08-27 11:03:01,679] INFO Client environment:user.home=/root (org.apache.zo okeeper.ZooKeeper)
[2015-08-27 11:03:01,679] INFO Client environment:user.dir=/opt/kafka_2.9.1-0.8. 2.1 (org.apache.zookeeper.ZooKeeper)
[2015-08-27 11:03:01,680] INFO Initiating client connection, connectString=local host:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient#1d88518f (org .apache.zookeeper.ZooKeeper)
[2015-08-27 11:03:01,700] INFO Opening socket connection to server localhost/127 .0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.ap ache.zookeeper.ClientCnxn)
[2015-08-27 11:03:01,705] INFO Socket connection established to localhost/127.0. 0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2015-08-27 11:03:01,728] INFO Session establishment complete on server localhos t/127.0.0.1:2181, sessionid = 0x14f6e6478700000, negotiated timeout = 6000 (org. apache.zookeeper.ClientCnxn)
[2015-08-27 11:03:01,730] INFO zookeeper state changed (SyncConnected) (org.I0It ec.zkclient.ZkClient)
[2015-08-27 11:03:01,828] INFO Log directory '/tmp/kafka-logs' not found, creati ng it. (kafka.log.LogManager)
[2015-08-27 11:03:01,839] INFO Loading logs. (kafka.log.LogManager)
[2015-08-27 11:03:01,848] INFO Logs loading complete. (kafka.log.LogManager)
[2015-08-27 11:03:01,849] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2015-08-27 11:03:01,853] INFO Starting log flusher with a default period of 922 3372036854775807 ms. (kafka.log.LogManager)
[2015-08-27 11:03:01,886] INFO Awaiting socket connections on 0.0.0.0:9092. (kaf ka.network.Acceptor)
[2015-08-27 11:03:01,887] INFO [Socket Server on Broker 0], Started (kafka.netwo rk.SocketServer)
[2015-08-27 11:03:01,964] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2015-08-27 11:03:02,007] INFO 0 successfully elected as leader (kafka.server.Zo okeeperLeaderElector)
[2015-08-27 11:03:02,081] FATAL [Kafka Server 0], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.net.UnknownHostException: SRV101004013: SRV101004013
at java.net.InetAddress.getLocalHost(InetAddress.java:1473)
at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:54)
at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:45)
at kafka.server.KafkaServer.startup(KafkaServer.scala:124)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala: 29)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
Caused by: java.net.UnknownHostException: SRV101004013
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:129 3)
at java.net.InetAddress.getLocalHost(InetAddress.java:1469)
... 6 more
[2015-08-27 11:03:02,084] INFO [Kafka Server 0], shutting down (kafka.server.Kaf kaServer)
[2015-08-27 11:03:02,086] INFO [Socket Server on Broker 0], Shutting down (kafka .network.SocketServer)
[2015-08-27 11:03:02,091] INFO [Socket Server on Broker 0], Shutdown completed ( kafka.network.SocketServer)
[2015-08-27 11:03:02,092] INFO [Kafka Request Handler on Broker 0], shutting dow n (kafka.server.KafkaRequestHandlerPool)
[2015-08-27 11:03:02,128] INFO [Kafka Request Handler on Broker 0], shut down co mpletely (kafka.server.KafkaRequestHandlerPool)
[2015-08-27 11:03:02,164] INFO New leader is 0 (kafka.server.ZookeeperLeaderElec tor$LeaderChangeListener)
[2015-08-27 11:03:02,346] INFO [Replica Manager on Broker 0]: Shut down (kafka.s erver.ReplicaManager)
[2015-08-27 11:03:02,347] INFO [ReplicaFetcherManager on broker 0] shutting down (kafka.server.ReplicaFetcherManager)
[2015-08-27 11:03:02,348] INFO [ReplicaFetcherManager on broker 0] shutdown comp leted (kafka.server.ReplicaFetcherManager)
[2015-08-27 11:03:02,352] INFO [Replica Manager on Broker 0]: Shut down complete ly (kafka.server.ReplicaManager)
[2015-08-27 11:03:02,352] INFO Shutting down. (kafka.log.LogManager)
[2015-08-27 11:03:02,362] INFO Shutdown complete. (kafka.log.LogManager)
[2015-08-27 11:03:02,366] INFO Terminate ZkClient event thread. (org.I0Itec.zkcl ient.ZkEventThread)
[2015-08-27 11:03:02,368] INFO Session: 0x14f6e6478700000 closed (org.apache.zoo keeper.ZooKeeper)
[2015-08-27 11:03:02,368] INFO EventThread shut down (org.apache.zookeeper.Clien tCnxn)
[2015-08-27 11:03:02,368] INFO [Kafka Server 0], shut down completed (kafka.serv er.KafkaServer)
[2015-08-27 11:03:02,369] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.net.UnknownHostException: SRV101004013: SRV101004013
at java.net.InetAddress.getLocalHost(InetAddress.java:1473)
at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:54)
at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:45)
at kafka.server.KafkaServer.startup(KafkaServer.scala:124)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala: 29)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
Caused by: java.net.UnknownHostException: SRV101004013
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:129 3)
at java.net.InetAddress.getLocalHost(InetAddress.java:1469)
... 6 more
[2015-08-27 11:03:02,372] INFO [Kafka Server 0], shutting down (kafka.server.Kaf kaServer)
I guess the problem has to be with this opened issue but I can't be sure
Does anyone found the same problem? Does anyone know how to fix it?
Well it cannot resolve the host SRV101004013. The one question is did you write this host to some Kafka or ZK config? Can you resolve the hostname using nslookup or simple ping? If yes do you have the host in /etc/hosts? Because InetAddress.getLocalHost() ignores the /etc/resolv.conf but only looks at the /etc/hosts file.
Related
I'm using a rasbery pi with Rasbian. I want to use Kafka to stream data from a camera to my phone. I downloaded this package from the Kafka website that contains Zookeeper and Kafka:
https://www.apache.org/dyn/closer.cgi?path=/kafka/2.4.1/kafka_2.12-2.4.1.tgz
First I started zookeeper with the zookeeper-server-start.sh located in the bin directory with:
"sudo bin/zookeeper-server-start.sh config/zookeeper.properties" in the terminal. I got back:
[2020-04-07 17:56:44,843] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:44,854] WARN config/zookeeper.properties is relative. Prepend ./ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:44,936] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:44,937] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:44,975] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2020-04-07 17:56:44,976] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2020-04-07 17:56:44,978] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2020-04-07 17:56:44,979] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2020-04-07 17:56:45,010] INFO Log4j found with jmx enabled. (org.apache.zookeeper.jmx.ManagedUtil)
[2020-04-07 17:56:45,263] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:45,264] WARN config/zookeeper.properties is relative. Prepend ./ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:45,268] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:45,269] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:45,271] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
[2020-04-07 17:56:45,319] INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2020-04-07 17:56:45,483] INFO Server environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,484] INFO Server environment:host.name=Rupert (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,485] INFO Server environment:java.version=11.0.6 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,486] INFO Server environment:java.vendor=Raspbian (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,487] INFO Server environment:java.home=/usr/lib/jvm/java-11-openjdk-armhf (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,488] INFO Server environment:java.class.path=/usr/local/kafka/bin/../libs/activation-1.1.1.jar:/usr/local/kafka/bin/../libs/aopalliance-repackaged-2.5.0.jar:/usr/local/kafka/bin/../libs/argparse4j-0.7.0.jar:/usr/local/kafka/bin/../libs/audience-annotations-0.5.0.jar:/usr/local/kafka/bin/../libs/commons-cli-1.4.jar:/usr/local/kafka/bin/../libs/commons-lang3-3.8.1.jar:/usr/local/kafka/bin/../libs/connect-api-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-basic-auth-extension-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-file-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-json-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-mirror-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-mirror-client-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-runtime-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-transforms-2.4.1.jar:/usr/local/kafka/bin/../libs/guava-20.0.jar:/usr/local/kafka/bin/../libs/hk2-api-2.5.0.jar:/usr/local/kafka/bin/../libs/hk2-locator-2.5.0.jar:/usr/local/kafka/bin/../libs/hk2-utils-2.5.0.jar:/usr/local/kafka/bin/../libs/jackson-annotations-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-core-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-databind-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-dataformat-csv-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-datatype-jdk8-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-jaxrs-base-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-module-paranamer-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-module-scala_2.12-2.10.0.jar:/usr/local/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/usr/local/kafka/bin/../libs/jakarta.annotation-api-1.3.4.jar:/usr/local/kafka/bin/../libs/jakarta.inject-2.5.0.jar:/usr/local/kafka/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/usr/local/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/usr/local/kafka/bin/../libs/javassist-3.22.0-CR2.jar:/usr/local/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/usr/local/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/usr/local/kafka/bin/../libs/jaxb-api-2.3.0.jar:/usr/local/kafka/bin/../libs/jersey-client-2.28.jar:/usr/local/kafka/bin/../libs/jersey-common-2.28.jar:/usr/local/kafka/bin/../libs/jersey-container-servlet-2.28.jar:/usr/local/kafka/bin/../libs/jersey-container-servlet-core-2.28.jar:/usr/local/kafka/bin/../libs/jersey-hk2-2.28.jar:/usr/local/kafka/bin/../libs/jersey-media-jaxb-2.28.jar:/usr/local/kafka/bin/../libs/jersey-server-2.28.jar:/usr/local/kafka/bin/../libs/jetty-client-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-continuation-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-http-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-io-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-security-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-server-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-servlet-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-servlets-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-util-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jopt-simple-5.0.4.jar:/usr/local/kafka/bin/../libs/kafka_2.12-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka_2.12-2.4.1-sources.jar:/usr/local/kafka/bin/../libs/kafka-clients-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-log4j-appender-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-examples-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-scala_2.12-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-test-utils-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-tools-2.4.1.jar:/usr/local/kafka/bin/../libs/log4j-1.2.17.jar:/usr/local/kafka/bin/../libs/lz4-java-1.6.0.jar:/usr/local/kafka/bin/../libs/maven-artifact-3.6.1.jar:/usr/local/kafka/bin/../libs/metrics-core-2.2.0.jar:/usr/local/kafka/bin/../libs/netty-buffer-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-codec-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-common-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-handler-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-resolver-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-transport-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-transport-native-epoll-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-transport-native-unix-common-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/usr/local/kafka/bin/../libs/paranamer-2.8.jar:/usr/local/kafka/bin/../libs/plexus-utils-3.2.0.jar:/usr/local/kafka/bin/../libs/reflections-0.9.11.jar:/usr/local/kafka/bin/../libs/rocksdbjni-5.18.3.jar:/usr/local/kafka/bin/../libs/scala-collection-compat_2.12-2.1.2.jar:/usr/local/kafka/bin/../libs/scala-java8-compat_2.12-0.9.0.jar:/usr/local/kafka/bin/../libs/scala-library-2.12.10.jar:/usr/local/kafka/bin/../libs/scala-logging_2.12-3.9.2.jar:/usr/local/kafka/bin/../libs/scala-reflect-2.12.10.jar:/usr/local/kafka/bin/../libs/slf4j-api-1.7.28.jar:/usr/local/kafka/bin/../libs/slf4j-log4j12-1.7.28.jar:/usr/local/kafka/bin/../libs/snappy-java-1.1.7.3.jar:/usr/local/kafka/bin/../libs/validation-api-2.0.1.Final.jar:/usr/local/kafka/bin/../libs/zookeeper-3.5.7.jar:/usr/local/kafka/bin/../libs/zookeeper-jute-3.5.7.jar:/usr/local/kafka/bin/../libs/zstd-jni-1.4.3-1.jar (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,498] INFO Server environment:java.library.path=/usr/java/packages/lib:/usr/lib/arm-linux-gnueabihf/jni:/lib/arm-linux-gnueabihf:/usr/lib/arm-linux-gnueabihf:/usr/lib/jni:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,499] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,500] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,501] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,502] INFO Server environment:os.arch=arm (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,503] INFO Server environment:os.version=4.19.66-v7+ (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,504] INFO Server environment:user.name=root (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,506] INFO Server environment:user.home=/root (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,507] INFO Server environment:user.dir=/usr/local/kafka (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,508] INFO Server environment:os.memory.free=493MB (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,509] INFO Server environment:os.memory.max=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,510] INFO Server environment:os.memory.total=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,529] INFO minSessionTimeout set to 6000 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,531] INFO maxSessionTimeout set to 60000 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,539] INFO Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /tmp/zookeeper/version-2 snapdir /tmp/zookeeper/version-2 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,661] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2020-04-07 17:56:45,699] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers. (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2020-04-07 17:56:45,773] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2020-04-07 17:56:45,965] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2020-04-07 17:56:45,994] INFO Reading snapshot /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileSnap)
[2020-04-07 17:56:46,076] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2020-04-07 17:56:46,377] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.apache.zookeeper.server.ContainerManager)
I believe that his is correct, but please do bring anything to my attention.
Next I tried to run Kafka using "sudo bin/kafka-server-start.sh config/server.properties" and after running for 10 seconds it returned:
[2020-04-07 17:49:40,577] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2020-04-07 17:49:47,215] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2020-04-07 17:49:47,222] INFO starting (kafka.server.KafkaServer)
[2020-04-07 17:49:47,233] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2020-04-07 17:49:47,558] INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2020-04-07 17:49:47,679] INFO Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,681] INFO Client environment:host.name=Rupert (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,682] INFO Client environment:java.version=11.0.6 (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,682] INFO Client environment:java.vendor=Raspbian (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,683] INFO Client environment:java.home=/usr/lib/jvm/java-11-openjdk-armhf (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,684] INFO Client environment:java.class.path=/usr/local/kafka/bin/../libs/activation-1.1.1.jar:/usr/local/kafka/bin/../libs/aopalliance-repackaged-2.5.0.jar:/usr/local/kafka/bin/../libs/argparse4j-0.7.0.jar:/usr/local/kafka/bin/../libs/audience-annotations-0.5.0.jar:/usr/local/kafka/bin/../libs/commons-cli-1.4.jar:/usr/local/kafka/bin/../libs/commons-lang3-3.8.1.jar:/usr/local/kafka/bin/../libs/connect-api-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-basic-auth-extension-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-file-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-json-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-mirror-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-mirror-client-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-runtime-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-transforms-2.4.1.jar:/usr/local/kafka/bin/../libs/guava-20.0.jar:/usr/local/kafka/bin/../libs/hk2-api-2.5.0.jar:/usr/local/kafka/bin/../libs/hk2-locator-2.5.0.jar:/usr/local/kafka/bin/../libs/hk2-utils-2.5.0.jar:/usr/local/kafka/bin/../libs/jackson-annotations-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-core-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-databind-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-dataformat-csv-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-datatype-jdk8-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-jaxrs-base-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-module-paranamer-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-module-scala_2.12-2.10.0.jar:/usr/local/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/usr/local/kafka/bin/../libs/jakarta.annotation-api-1.3.4.jar:/usr/local/kafka/bin/../libs/jakarta.inject-2.5.0.jar:/usr/local/kafka/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/usr/local/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/usr/local/kafka/bin/../libs/javassist-3.22.0-CR2.jar:/usr/local/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/usr/local/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/usr/local/kafka/bin/../libs/jaxb-api-2.3.0.jar:/usr/local/kafka/bin/../libs/jersey-client-2.28.jar:/usr/local/kafka/bin/../libs/jersey-common-2.28.jar:/usr/local/kafka/bin/../libs/jersey-container-servlet-2.28.jar:/usr/local/kafka/bin/../libs/jersey-container-servlet-core-2.28.jar:/usr/local/kafka/bin/../libs/jersey-hk2-2.28.jar:/usr/local/kafka/bin/../libs/jersey-media-jaxb-2.28.jar:/usr/local/kafka/bin/../libs/jersey-server-2.28.jar:/usr/local/kafka/bin/../libs/jetty-client-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-continuation-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-http-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-io-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-security-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-server-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-servlet-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-servlets-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-util-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jopt-simple-5.0.4.jar:/usr/local/kafka/bin/../libs/kafka_2.12-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka_2.12-2.4.1-sources.jar:/usr/local/kafka/bin/../libs/kafka-clients-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-log4j-appender-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-examples-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-scala_2.12-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-test-utils-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-tools-2.4.1.jar:/usr/local/kafka/bin/../libs/log4j-1.2.17.jar:/usr/local/kafka/bin/../libs/lz4-java-1.6.0.jar:/usr/local/kafka/bin/../libs/maven-artifact-3.6.1.jar:/usr/local/kafka/bin/../libs/metrics-core-2.2.0.jar:/usr/local/kafka/bin/../libs/netty-buffer-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-codec-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-common-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-handler-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-resolver-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-transport-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-transport-native-epoll-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-transport-native-unix-common-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/usr/local/kafka/bin/../libs/paranamer-2.8.jar:/usr/local/kafka/bin/../libs/plexus-utils-3.2.0.jar:/usr/local/kafka/bin/../libs/reflections-0.9.11.jar:/usr/local/kafka/bin/../libs/rocksdbjni-5.18.3.jar:/usr/local/kafka/bin/../libs/scala-collection-compat_2.12-2.1.2.jar:/usr/local/kafka/bin/../libs/scala-java8-compat_2.12-0.9.0.jar:/usr/local/kafka/bin/../libs/scala-library-2.12.10.jar:/usr/local/kafka/bin/../libs/scala-logging_2.12-3.9.2.jar:/usr/local/kafka/bin/../libs/scala-reflect-2.12.10.jar:/usr/local/kafka/bin/../libs/slf4j-api-1.7.28.jar:/usr/local/kafka/bin/../libs/slf4j-log4j12-1.7.28.jar:/usr/local/kafka/bin/../libs/snappy-java-1.1.7.3.jar:/usr/local/kafka/bin/../libs/validation-api-2.0.1.Final.jar:/usr/local/kafka/bin/../libs/zookeeper-3.5.7.jar:/usr/local/kafka/bin/../libs/zookeeper-jute-3.5.7.jar:/usr/local/kafka/bin/../libs/zstd-jni-1.4.3-1.jar (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,693] INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/arm-linux-gnueabihf/jni:/lib/arm-linux-gnueabihf:/usr/lib/arm-linux-gnueabihf:/usr/lib/jni:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,695] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,696] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,697] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,698] INFO Client environment:os.arch=arm (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,699] INFO Client environment:os.version=4.19.66-v7+ (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,700] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,701] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,702] INFO Client environment:user.dir=/usr/local/kafka (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,703] INFO Client environment:os.memory.free=975MB (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,704] INFO Client environment:os.memory.max=1024MB (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,704] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,769] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$#114918a (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,887] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2020-04-07 17:49:48,017] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2020-04-07 17:49:48,130] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:48,313] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2020-04-07 17:49:48,449] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:48,557] INFO Socket error occurred: localhost/0:0:0:0:0:0:0:1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:49,678] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:49,682] INFO Socket error occurred: localhost/0:0:0:0:0:0:0:1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:50,784] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:50,787] INFO Socket error occurred: localhost/0:0:0:0:0:0:0:1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:51,890] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:51,894] INFO Socket error occurred: localhost/127.0.0.1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:52,997] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:53,000] INFO Socket error occurred: localhost/0:0:0:0:0:0:0:1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:54,114] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:54,118] INFO Socket error occurred: localhost/127.0.0.1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:54,353] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2020-04-07 17:49:55,221] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:55,375] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:55,396] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:55,412] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2020-04-07 17:49:55,449] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:259)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:255)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:113)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1858)
at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:375)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:399)
at kafka.server.KafkaServer.startup(KafkaServer.scala:207)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:84)
at kafka.Kafka.main(Kafka.scala)
[2020-04-07 17:49:55,480] INFO shutting down (kafka.server.KafkaServer)
[2020-04-07 17:49:55,577] INFO shut down completed (kafka.server.KafkaServer)
[2020-04-07 17:49:55,583] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2020-04-07 17:49:55,612] INFO shutting down (kafka.server.KafkaServer)
My assumption is that Kafka can't reach zookeeper but I really have no clue.
Is there a way I can test if zookeeper is working?
Thanks to anyone who helps, stay healthy.
When I pressed ctrl+shift+c to get rid of the text that showed up afterwards it also undid what needed to happen.
This happens even when the nodes are in the same subnet.
I am using the Docker-Flink project in:
https://github.com/apache/flink/tree/master/flink-contrib/docker-flink
I am creating the services with the following commands:
docker network create -d overlay overlay
docker service create --name jobmanager --env JOB_MANAGER_RPC_ADDRESS=jobmanager -p 8081:8081 --network overlay --constraint 'node.hostname == ubuntu-swarm-manager' flink jobmanager
docker service create --name taskmanager --env JOB_MANAGER_RPC_ADDRESS=jobmanager --network overlay --constraint 'node.hostname != ubuntu-swarm-manager' flink taskmanager
This is the error I get:
- Trying to register at JobManager akka.tcp://flink#jobmanager:6123/ user/jobmanager (attempt 4, timeout: 4000 milliseconds)
These are my environment configurations:
node: ubuntu-swarm-master Azure VM Standard D4s v3 (4 vcpus, 16 GB
memory) Docker version 17.03.1-ce, build c6d412e
node: azure-swarm-worker-1 Azure VM Standard D2 v2 Promo (2 vcpus, 7
GB memory) Docker version 17.09.0-ce, build afdb6d4
Flink: using image 1.3.2-hadoop2-scala_2.10
This is from the log of the container running TaskManager:
Starts ok...
Starting Task Manager
config file:
jobmanager.rpc.address: jobmanager
jobmanager.rpc.port: 6123
jobmanager.heap.mb: 1024
taskmanager.heap.mb: 1024
taskmanager.numberOfTaskSlots: 2
taskmanager.memory.preallocate: false
parallelism.default: 1
jobmanager.web.port: 8081
blob.server.port: 6124
query.server.port: 6125
Starting taskmanager as a console application on host 00afd4130a94.
Then there are some errors (scroll right):
2017-11-02 14:06:51,064 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - Trying to select the network interface and address to use by connecting to the leading JobManager.
2017-11-02 14:06:51,065 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils - TaskManager will try to connect for 10000 milliseconds before falling back to heuristics
2017-11-02 14:06:51,067 INFO org.apache.flink.runtime.net.ConnectionUtils - Retrieved new target address jobmanager/10.0.0.2:6123.
2017-11-02 14:06:54,578 INFO org.apache.flink.runtime.net.ConnectionUtils - Trying to connect to address jobmanager/10.0.0.2:6123
2017-11-02 14:06:54,779 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '00afd4130a94/10.0.0.5': connect timed out
2017-11-02 14:06:54,829 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/10.0.0.5': connect timed out
2017-11-02 14:06:54,880 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/10.0.0.4': connect timed out
2017-11-02 14:06:54,931 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/172.18.0.3': connect timed out
2017-11-02 14:06:54,981 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/10.0.0.5': connect timed out
2017-11-02 14:06:55,031 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/10.0.0.4': connect timed out
2017-11-02 14:06:55,032 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/127.0.0.1': Invalid argument (connect failed)
2017-11-02 14:06:56,034 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/172.18.0.3': connect timed out
2017-11-02 14:06:57,036 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/10.0.0.5': connect timed out
2017-11-02 14:06:58,037 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/10.0.0.4': connect timed out
2017-11-02 14:06:58,038 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/127.0.0.1': Invalid argument (connect failed)
2017-11-02 14:06:58,138 INFO org.apache.flink.runtime.net.ConnectionUtils - Trying to connect to address jobmanager/10.0.0.2:6123
2017-11-02 14:06:58,339 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '00afd4130a94/10.0.0.5': connect timed out
2017-11-02 14:06:58,389 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/10.0.0.5': connect timed out
2017-11-02 14:06:58,439 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/10.0.0.4': connect timed out
2017-11-02 14:06:58,490 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/172.18.0.3': connect timed out
2017-11-02 14:06:58,541 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/10.0.0.5': connect timed out
2017-11-02 14:06:58,592 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/10.0.0.4': connect timed out
2017-11-02 14:06:58,592 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/127.0.0.1': Invalid argument (connect failed)
2017-11-02 14:06:59,593 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/172.18.0.3': connect timed out
2017-11-02 14:07:00,595 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/10.0.0.5': connect timed out
2017-11-02 14:07:01,599 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/10.0.0.4': connect timed out
2017-11-02 14:07:01,599 INFO org.apache.flink.runtime.net.ConnectionUtils - Failed to connect from address '/127.0.0.1': Invalid argument (connect failed)
2017-11-02 14:07:01,600 WARN org.apache.flink.runtime.net.ConnectionUtils - Could not connect to jobmanager/10.0.0.2:6123. Selecting a local address using heuristics.
2017-11-02 14:07:01,601 INFO org.apache.flink.runtime.taskmanager.TaskManager - TaskManager will use hostname/address '00afd4130a94' (10.0.0.5) for communication.
2017-11-02 14:07:01,601 INFO org.apache.flink.runtime.taskmanager.TaskManager - Starting TaskManager
2017-11-02 14:07:01,601 INFO org.apache.flink.runtime.taskmanager.TaskManager - Starting TaskManager actor system at 00afd4130a94:0.
2017-11-02 14:07:01,947 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2017-11-02 14:07:01,978 INFO Remoting - Starting remoting
2017-11-02 14:07:02,168 INFO Remoting - Remoting started; listening on addresses :[akka.tcp://flink#00afd4130a94:33881]
2017-11-02 14:07:02,174 INFO org.apache.flink.runtime.taskmanager.TaskManager - Starting TaskManager actor
2017-11-02 14:07:02,192 INFO org.apache.flink.runtime.io.network.netty.NettyConfig - NettyConfig [server address: 00afd4130a94/10.0.0.5, server port: 0, ssl enabled: false, memory segment size (bytes): 32768, transport type: NIO, number of server threads: 2 (manual), number of client threads: 2 (manual), server connect backlog: 0 (use Netty's default), client connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)]
2017-11-02 14:07:02,199 INFO org.apache.flink.runtime.taskexecutor.TaskManagerConfiguration - Messages have a max timeout of 10000 ms
2017-11-02 14:07:02,201 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file directory '/tmp': total 29 GB, usable 25 GB (86.21% usable)
2017-11-02 14:07:02,286 INFO org.apache.flink.runtime.io.network.buffer.NetworkBufferPool - Allocated 101 MB for network buffer pool (number of memory segments: 3260, bytes per segment: 32768).
2017-11-02 14:07:02,393 INFO org.apache.flink.runtime.io.network.NetworkEnvironment - Starting the network environment and its components.
2017-11-02 14:07:02,400 INFO org.apache.flink.runtime.io.network.netty.NettyClient - Successful initialization (took 2 ms).
2017-11-02 14:07:02,434 INFO org.apache.flink.runtime.io.network.netty.NettyServer - Successful initialization (took 32 ms). Listening on SocketAddress /10.0.0.5:42921.
2017-11-02 14:07:02,493 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices - Limiting managed memory to 0.7 of the currently free heap space (640 MB), memory will be allocated lazily.
2017-11-02 14:07:02,498 INFO org.apache.flink.runtime.io.disk.iomanager.IOManager - I/O manager uses directory /tmp/flink-io-e57d51fa-2269-4df0-9910-0fe26c6042bd for spill files.
2017-11-02 14:07:02,501 INFO org.apache.flink.runtime.metrics.MetricRegistry - No metrics reporter configured, no metrics will be exposed/reported.
2017-11-02 14:07:02,553 INFO org.apache.flink.runtime.filecache.FileCache - User file cache uses directory /tmp/flink-dist-cache-2c0c063f-464e-48f1-9fb8-fcfa48868e3a
2017-11-02 14:07:02,564 INFO org.apache.flink.runtime.filecache.FileCache - User file cache uses directory /tmp/flink-dist-cache-0c5e2b25-70a2-4964-9eec-24b0e79d560e
2017-11-02 14:07:02,572 INFO org.apache.flink.runtime.taskmanager.TaskManager - Starting TaskManager actor at akka://flink/user/taskmanager#1719715507.
2017-11-02 14:07:02,572 INFO org.apache.flink.runtime.taskmanager.TaskManager - TaskManager data connection information: df5992297d269fa16a5e945e1dce0451 # 00afd4130a94 (dataPort=42921)
2017-11-02 14:07:02,573 INFO org.apache.flink.runtime.taskmanager.TaskManager - TaskManager has 2 task slot(s).
2017-11-02 14:07:02,574 INFO org.apache.flink.runtime.taskmanager.TaskManager - Memory usage stats: [HEAP: 113/1024/1024 MB, NON HEAP: 33/33/-1 MB (used/committed/max)]
2017-11-02 14:07:02,576 INFO org.apache.flink.runtime.taskmanager.TaskManager - Trying to register at JobManager akka.tcp://flink#jobmanager:6123/user/jobmanager (attempt 1, timeout: 500 milliseconds)
2017-11-02 14:07:03,106 INFO org.apache.flink.runtime.taskmanager.TaskManager - Trying to register at JobManager akka.tcp://flink#jobmanager:6123/user/jobmanager (attempt 2, timeout: 1000 milliseconds)
2017-11-02 14:07:04,126 INFO org.apache.flink.runtime.taskmanager.TaskManager - Trying to register at JobManager akka.tcp://flink#jobmanager:6123/user/jobmanager (attempt 3, timeout: 2000 milliseconds)
Here is the log from the container running JobManager:
Starting Job Manager
config file:
jobmanager.rpc.address: jobmanager
jobmanager.rpc.port: 6123
jobmanager.heap.mb: 1024
taskmanager.heap.mb: 1024
taskmanager.numberOfTaskSlots: 1
taskmanager.memory.preallocate: false
parallelism.default: 1
jobmanager.web.port: 8081
blob.server.port: 6124
query.server.port: 6125
Starting jobmanager as a console application on host c30e0fe7b765.
2017-11-02 13:42:33,721 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-11-02 13:42:33,796 INFO org.apache.flink.runtime.jobmanager.JobManager - --------------------------------------------------------------------------------
2017-11-02 13:42:33,796 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager (Version: 1.3.2, Rev:0399bee, Date:03.08.2017 # 10:23:11 UTC)
2017-11-02 13:42:33,796 INFO org.apache.flink.runtime.jobmanager.JobManager - Current user: flink
2017-11-02 13:42:33,796 INFO org.apache.flink.runtime.jobmanager.JobManager - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.141-b15
2017-11-02 13:42:33,796 INFO org.apache.flink.runtime.jobmanager.JobManager - Maximum heap size: 981 MiBytes
2017-11-02 13:42:33,796 INFO org.apache.flink.runtime.jobmanager.JobManager - JAVA_HOME: /docker-java-home/jre
2017-11-02 13:42:33,799 INFO org.apache.flink.runtime.jobmanager.JobManager - Hadoop version: 2.7.2
2017-11-02 13:42:33,800 INFO org.apache.flink.runtime.jobmanager.JobManager - JVM Options:
2017-11-02 13:42:33,800 INFO org.apache.flink.runtime.jobmanager.JobManager - -Xms1024m
2017-11-02 13:42:33,800 INFO org.apache.flink.runtime.jobmanager.JobManager - -Xmx1024m
2017-11-02 13:42:33,800 INFO org.apache.flink.runtime.jobmanager.JobManager - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2017-11-02 13:42:33,800 INFO org.apache.flink.runtime.jobmanager.JobManager - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
2017-11-02 13:42:33,800 INFO org.apache.flink.runtime.jobmanager.JobManager - Program Arguments:
2017-11-02 13:42:33,800 INFO org.apache.flink.runtime.jobmanager.JobManager - --configDir
2017-11-02 13:42:33,800 INFO org.apache.flink.runtime.jobmanager.JobManager - /opt/flink/conf
2017-11-02 13:42:33,800 INFO org.apache.flink.runtime.jobmanager.JobManager - --executionMode
2017-11-02 13:42:33,800 INFO org.apache.flink.runtime.jobmanager.JobManager - cluster
2017-11-02 13:42:33,800 INFO org.apache.flink.runtime.jobmanager.JobManager - Classpath: /opt/flink/lib/flink-python_2.11-1.3.2.jar:/opt/flink/lib/flink-shaded-hadoop2-uber-1.3.2.jar:/opt/flink/lib/log4j-1.2.17.jar:/opt/flink/lib/slf4j-log4j12-1.7.7.jar:/opt/flink/lib/flink-dist_2.11-1.3.2.jar:::
2017-11-02 13:42:33,801 INFO org.apache.flink.runtime.jobmanager.JobManager - --------------------------------------------------------------------------------
2017-11-02 13:42:33,801 INFO org.apache.flink.runtime.jobmanager.JobManager - Registered UNIX signal handlers for [TERM, HUP, INT]
2017-11-02 13:42:33,911 INFO org.apache.flink.runtime.jobmanager.JobManager - Loading configuration from /opt/flink/conf
2017-11-02 13:42:33,914 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, jobmanager
2017-11-02 13:42:33,915 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123
2017-11-02 13:42:33,915 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 1024
2017-11-02 13:42:33,915 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.mb, 1024
2017-11-02 13:42:33,915 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2017-11-02 13:42:33,915 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.preallocate, false
2017-11-02 13:42:33,916 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2017-11-02 13:42:33,916 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.web.port, 8081
2017-11-02 13:42:33,917 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: blob.server.port, 6124
2017-11-02 13:42:33,917 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125
2017-11-02 13:42:33,924 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager without high-availability
2017-11-02 13:42:33,926 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager on jobmanager:6123 with execution mode CLUSTER
2017-11-02 13:42:33,934 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, jobmanager
2017-11-02 13:42:33,934 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123
2017-11-02 13:42:33,934 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 1024
2017-11-02 13:42:33,934 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.heap.mb, 1024
2017-11-02 13:42:33,935 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2017-11-02 13:42:33,935 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.preallocate, false
2017-11-02 13:42:33,935 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: parallelism.default, 1
2017-11-02 13:42:33,935 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.web.port, 8081
2017-11-02 13:42:33,936 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: blob.server.port, 6124
2017-11-02 13:42:33,936 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: query.server.port, 6125
2017-11-02 13:42:33,962 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to flink (auth:SIMPLE)
2017-11-02 13:42:34,026 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager actor system reachable at jobmanager:6123
2017-11-02 13:42:34,290 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started
2017-11-02 13:42:34,327 INFO Remoting - Starting remoting
2017-11-02 13:42:34,505 INFO Remoting - Remoting started; listening on addresses :[akka.tcp://flink#jobmanager:6123]
2017-11-02 13:42:34,524 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager web frontend
2017-11-02 13:42:34,532 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - Log file environment variable 'log.file' is not set.
2017-11-02 13:42:34,532 WARN org.apache.flink.runtime.webmonitor.WebMonitorUtils - JobManager log files are unavailable in the web dashboard. Log file location not found in environment variable 'log.file' or configuration key 'jobmanager.web.log.path'.
2017-11-02 13:42:34,532 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using directory /tmp/flink-web-9f0ba581-3488-4086-a79c-53e17b56352c for the web interface files
2017-11-02 13:42:34,533 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using directory /tmp/flink-web-17a58ccf-7d8b-475e-b727-4a7935a19c0f for web frontend JAR file uploads
2017-11-02 13:42:34,741 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Web frontend listening at 0:0:0:0:0:0:0:0:8081
2017-11-02 13:42:34,741 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager actor
2017-11-02 13:42:34,751 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-d10b620a-73ae-40af-bd23-aad5211fe1cc
2017-11-02 13:42:34,752 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:6124 - max concurrent requests: 50 - max backlog: 1000
2017-11-02 13:42:34,763 INFO org.apache.flink.runtime.metrics.MetricRegistry - No metrics reporter configured, no metrics will be exposed/reported.
2017-11-02 13:42:34,769 INFO org.apache.flink.runtime.jobmanager.MemoryArchivist - Started memory archivist akka://flink/user/archive
2017-11-02 13:42:34,774 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Starting with JobManager akka.tcp://flink#jobmanager:6123/user/jobmanager on port 8081
2017-11-02 13:42:34,774 INFO org.apache.flink.runtime.webmonitor.JobManagerRetriever - New leader reachable under akka.tcp://flink#jobmanager:6123/user/jobmanager:00000000-0000-0000-0000-000000000000.
2017-11-02 13:42:34,776 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager at akka.tcp://flink#jobmanager:6123/user/jobmanager.
2017-11-02 13:42:34,785 INFO org.apache.flink.runtime.clusterframework.standalone.StandaloneResourceManager - Trying to associate with JobManager leader akka.tcp://flink#jobmanager:6123/user/jobmanager
2017-11-02 13:42:34,801 INFO org.apache.flink.runtime.jobmanager.JobManager - JobManager akka.tcp://flink#jobmanager:6123/user/jobmanager was granted leadership with leader session ID Some(00000000-0000-0000-0000-000000000000).
2017-11-02 13:42:34,814 INFO org.apache.flink.runtime.clusterframework.standalone.StandaloneResourceManager - Resource Manager associating with leading JobManager Actor[akka://flink/user/jobmanager#844712453] - leader session 00000000-0000-0000-0000-000000000000
Why can't the TaskManagers talk to JobManager? I wonder if there's some configuration missing. Any help will be much appreciated. Thank you very much!
This is a snippet from the system log while shutting down:
INFO [RMI TCP Connection(12)-127.0.0.1] 2016-07-27 22:28:50,995 StorageService.java:3788 - Announcing that I have left the ring for 30000ms
INFO [RMI TCP Connection(12)-127.0.0.1] 2016-07-27 22:29:20,995 ThriftServer.java:142 - Stop listening to thrift clients
INFO [RMI TCP Connection(12)-127.0.0.1] 2016-07-27 22:29:20,997 Server.java:182 - Stop listening for CQL clients
WARN [RMI TCP Connection(12)-127.0.0.1] 2016-07-27 22:29:20,997 Gossiper.java:1508 - No local state or state is in silent shutdown, not announcing shutdown
INFO [RMI TCP Connection(12)-127.0.0.1] 2016-07-27 22:29:20,997 MessagingService.java:786 - Waiting for messaging service to quiesce
INFO [ACCEPT-sysengplayl0127.bio-iad.ea.com/10.72.194.229] 2016-07-27 22:29:20,998 MessagingService.java:1133 - MessagingService has terminated the accept() thread
INFO [RMI TCP Connection(12)-127.0.0.1] 2016-07-27 22:29:21,022 StorageService.java:1411 - DECOMMISSIONED
INFO [main] 2016-07-27 22:32:17,534 YamlConfigurationLoader.java:89 - Configuration location: file:/opt/cassandra/product/apache-cassandra-3.7/conf/cassandra.yaml
And then while starting up:
INFO [main] 2016-07-27 22:32:20,316 StorageService.java:630 - Cassandra version: 3.7
INFO [main] 2016-07-27 22:32:20,316 StorageService.java:631 - Thrift API version: 20.1.0
INFO [main] 2016-07-27 22:32:20,316 StorageService.java:632 - CQL supported versions: 3.4.2 (default: 3.4.2)
INFO [main] 2016-07-27 22:32:20,351 IndexSummaryManager.java:85 - Initializing index summary manager with a memory pool size of 397 MB and a resize interval of 60 minutes
ERROR [main] 2016-07-27 22:32:20,357 CassandraDaemon.java:731 - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: This node was decommissioned and will not rejoin the ring unless cassandra.override_decommission=true has been set, or all existing data is removed and the node is bootstrapped again
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:815) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:725) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:625) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:370) [apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:585) [apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:714) [apache-cassandra-3.7.jar:3.7]
WARN [StorageServiceShutdownHook] 2016-07-27 22:32:20,358 Gossiper.java:1508 - No local state or state is in silent shutdown, not announcing shutdown
INFO [StorageServiceShutdownHook] 2016-07-27 22:32:20,359 MessagingService.java:786 - Waiting for messaging service to quiesce
Is there something wrong with the configuration?
I had faced same issue.
Posting the answer so that it might help others.
As the log suggests, the property "cassandra.override_decommission" should be overridden.
start cassandra with the syntax:
cassandra -Dcassandra.override_decommission=true
This should add the node back to the cluster.
I am trying to configure a two node cluster with cassandra in windows r2 2008 So i installed cassandra community version in one server (10.xxx.0.1,10.xxx.0.2) And then I stopped the service and then edited the configuraton.yaml file in the conf folder.
The changes are:
cluster_name
commented the num_tokens
gave the tokens in initial_token,
seeds as 10.xxx.0.1,10.xxx.0.2,
listen_addresses are their respective ip addresses which are 10.xxx.0.1,10.xxx.0.2,
rpc_addresses are same as listen_address,
endpointsnitch as gossip
I also changed the cassandra rackdc.properties file to dc=DC1 rack=RAC1.
and changed the cassandra-env file - uncommented: JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=10.xxx.0.1
I then saved and started back the service and ran nodetool status and got
Datacenter: DC1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Owns (effective) Host ID
Token Rack
UN 10.104.0.15 46.65 KB 100.0% bc41a884-baaf-4a52-85f3-f3270c2ec9
57 -9223372036854775808 RAC1
and opened the cqlsh, but it is not connecting. Below is the error:
ERROR [Initialization] 2015-10-12 17:10:21,353 Error connecting via JMX: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
java.net.ConnectException: Connection refused: connect]
INFO [main] 2015-10-12 17:10:24,612 Reconnecting to a backup OpsCenter instance
INFO [main] 2015-10-12 17:10:24,615 SSL communication is disabled
INFO [main] 2015-10-12 17:10:24,615 Creating stomp connection to 127.0.0.1:61620
INFO [main] 2015-10-12 17:10:24,628 Starting Jetty server: {:join? false, :ssl? false, :host nil, :port 61621}
INFO [Initialization] 2015-10-12 17:10:24,632 Sleeping for 2s before trying to determine IP over JMX again
INFO [StompConnection receiver] 2015-10-12 17:10:24,640 Reconnecting in 0s.
INFO [main] 2015-10-12 18:25:56,347 Waiting for the config from OpsCenter
INFO [main] 2015-10-12 18:25:56,349 Attempting to determine Cassandra's broadcast address through JMX
INFO [main] 2015-10-12 18:25:56,350 Starting Stomp
INFO [main] 2015-10-12 18:25:56,350 Starting up agent communcation with OpsCenter.
INFO [Initialization] 2015-10-12 18:25:56,356 New JMX connection (127.0.0.1:7199)
ERROR [Initialization] 2015-10-12 18:25:57,652 Error connecting via JMX: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
java.net.ConnectException: Connection refused: connect]
INFO [main] 2015-10-12 18:26:00,768 Reconnecting to a backup OpsCenter instance
INFO [main] 2015-10-12 18:26:00,770 SSL communication is disabled
INFO [main] 2015-10-12 18:26:00,770 Creating stomp connection to 127.0.0.1:61620
INFO [main] 2015-10-12 18:26:00,779 Starting Jetty server: {:join? false, :ssl? false, :host nil, :port 61621}
INFO [Initialization] 2015-10-12 18:26:00,782 Sleeping for 2s before trying to determine IP over JMX again
INFO [StompConnection receiver] 2015-10-12 18:26:00,945 Reconnecting in 0s.
INFO [StompConnection receiver] 2015-10-12 18:26:01,136 Connected to 127.0.0.1:61620
I'm using kafka-node to send messages between the services in my application. I'm using the wurstmeister/kafka and wurstmeister/zookeeper containers to run Kafka and Zookeeper (for simplicity's sake).
Every time I try to send() a message to the broker, I get a LeaderNotAvailable error. I used kafka-web-console to inspect the broker, and it shows the two topics that I had set up, but it shows no leader (see http://cl.ly/image/1f3g0K2S1J0F).
Neither from reading the container logs (below), nor any documentation, nor trying to search for this problem can I work out why it's not working (or how to fix it).
Update sending seems to work if I destroy the containers and start them up again (rather than just restarting them using docker-compose up).
zookeeper_1 | JMX enabled by default
zookeeper_1 | Using config: /opt/zookeeper-3.4.6/bin/../conf/zoo.cfg
zookeeper_1 | 2015-09-01 10:44:15,108 [myid:] - INFO [main:QuorumPeerConfig#103] - Reading configuration from: /opt/zookeeper-3.4.6/bin/../conf/zoo.cfg
zookeeper_1 | 2015-09-01 10:44:15,113 [myid:] - INFO [main:DatadirCleanupManager#78] - autopurge.snapRetainCount set to 3
zookeeper_1 | 2015-09-01 10:44:15,113 [myid:] - INFO [main:DatadirCleanupManager#79] - autopurge.purgeInterval set to 1
zookeeper_1 | 2015-09-01 10:44:15,119 [myid:] - WARN [main:QuorumPeerMain#113] - Either no config or no quorum defined in config, running in standalone mode
zookeeper_1 | 2015-09-01 10:44:15,119 [myid:] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask#138] - Purge task started.
zookeeper_1 | 2015-09-01 10:44:15,133 [myid:] - INFO [PurgeTask:DatadirCleanupManager$PurgeTask#144] - Purge task completed.
zookeeper_1 | 2015-09-01 10:44:15,135 [myid:] - INFO [main:QuorumPeerConfig#103] - Reading configuration from: /opt/zookeeper-3.4.6/bin/../conf/zoo.cfg
zookeeper_1 | 2015-09-01 10:44:15,135 [myid:] - INFO [main:ZooKeeperServerMain#95] - Starting server
zookeeper_1 | 2015-09-01 10:44:15,143 [myid:] - INFO [main:Environment#100] - Server environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
zookeeper_1 | 2015-09-01 10:44:15,143 [myid:] - INFO [main:Environment#100] - Server environment:host.name=abd919b6a5a2
zookeeper_1 | 2015-09-01 10:44:15,144 [myid:] - INFO [main:Environment#100] - Server environment:java.version=1.7.0_65
zookeeper_1 | 2015-09-01 10:44:15,144 [myid:] - INFO [main:Environment#100] - Server environment:java.vendor=Oracle Corporation
zookeeper_1 | 2015-09-01 10:44:15,144 [myid:] - INFO [main:Environment#100] - Server environment:java.home=/usr/lib/jvm/java-7-openjdk-amd64/jre
zookeeper_1 | 2015-09-01 10:44:15,144 [myid:] - INFO [main:Environment#100] - Server environment:java.class.path=/opt/zookeeper-3.4.6/bin/../build/classes:/opt/zookeeper-3.4.6/bin/../build/lib/*.jar:/opt/zookeeper-3.4.6/bin/../lib/slf4j-log4j12-1.6.1.jar:/opt/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/opt/zookeeper-3.4.6/bin/../lib/netty-3.7.0.Final.jar:/opt/zookeeper-3.4.6/bin/../lib/log4j-1.2.16.jar:/opt/zookeeper-3.4.6/bin/../lib/jline-0.9.94.jar:/opt/zookeeper-3.4.6/bin/../zookeeper-3.4.6.jar:/opt/zookeeper-3.4.6/bin/../src/java/lib/*.jar:/opt/zookeeper-3.4.6/bin/../conf:
zookeeper_1 | 2015-09-01 10:44:15,144 [myid:] - INFO [main:Environment#100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
zookeeper_1 | 2015-09-01 10:44:15,144 [myid:] - INFO [main:Environment#100] - Server environment:java.io.tmpdir=/tmp
zookeeper_1 | 2015-09-01 10:44:15,146 [myid:] - INFO [main:Environment#100] - Server environment:java.compiler=<NA>
zookeeper_1 | 2015-09-01 10:44:15,146 [myid:] - INFO [main:Environment#100] - Server environment:os.name=Linux
zookeeper_1 | 2015-09-01 10:44:15,146 [myid:] - INFO [main:Environment#100] - Server environment:os.arch=amd64
zookeeper_1 | 2015-09-01 10:44:15,146 [myid:] - INFO [main:Environment#100] - Server environment:os.version=3.18.11-tinycore64
zookeeper_1 | 2015-09-01 10:44:15,147 [myid:] - INFO [main:Environment#100] - Server environment:user.name=root
zookeeper_1 | 2015-09-01 10:44:15,147 [myid:] - INFO [main:Environment#100] - Server environment:user.home=/root
zookeeper_1 | 2015-09-01 10:44:15,147 [myid:] - INFO [main:Environment#100] - Server environment:user.dir=/opt/zookeeper-3.4.6
zookeeper_1 | 2015-09-01 10:44:15,148 [myid:] - INFO [main:ZooKeeperServer#755] - tickTime set to 2000
zookeeper_1 | 2015-09-01 10:44:15,148 [myid:] - INFO [main:ZooKeeperServer#764] - minSessionTimeout set to -1
zookeeper_1 | 2015-09-01 10:44:15,148 [myid:] - INFO [main:ZooKeeperServer#773] - maxSessionTimeout set to -1
zookeeper_1 | 2015-09-01 10:44:15,161 [myid:] - INFO [main:NIOServerCnxnFactory#94] - binding to port 0.0.0.0/0.0.0.0:2181
kafka_1 | [2015-09-01 10:44:16,754] INFO Verifying properties (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,782] INFO Property advertised.host.name is overridden to 192.168.59.103 (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,782] INFO Property advertised.port is overridden to 32773 (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,782] INFO Property broker.id is overridden to 32773 (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,782] INFO Property log.cleaner.enable is overridden to false (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,783] INFO Property log.dirs is overridden to /kafka/kafka-logs-32773 (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,783] INFO Property log.retention.check.interval.ms is overridden to 300000 (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,783] INFO Property log.retention.hours is overridden to 168 (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,783] INFO Property log.segment.bytes is overridden to 1073741824 (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,783] INFO Property num.io.threads is overridden to 8 (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,783] INFO Property num.network.threads is overridden to 3 (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,784] INFO Property num.partitions is overridden to 1 (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,784] INFO Property num.recovery.threads.per.data.dir is overridden to 1 (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,785] INFO Property port is overridden to 9092 (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,785] INFO Property socket.receive.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,785] INFO Property socket.request.max.bytes is overridden to 104857600 (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,785] INFO Property socket.send.buffer.bytes is overridden to 102400 (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,785] WARN Property version is not valid (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,786] INFO Property zookeeper.connect is overridden to 172.17.0.10:2181 (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,786] INFO Property zookeeper.connection.timeout.ms is overridden to 6000 (kafka.utils.VerifiableProperties)
kafka_1 | [2015-09-01 10:44:16,817] INFO [Kafka Server 32773], starting (kafka.server.KafkaServer)
kafka_1 | [2015-09-01 10:44:16,820] INFO [Kafka Server 32773], Connecting to zookeeper on 172.17.0.10:2181 (kafka.server.KafkaServer)
kafka_1 | [2015-09-01 10:44:16,828] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
kafka_1 | [2015-09-01 10:44:16,837] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
kafka_1 | [2015-09-01 10:44:16,837] INFO Client environment:host.name=34ff07e953a2 (org.apache.zookeeper.ZooKeeper)
kafka_1 | [2015-09-01 10:44:16,837] INFO Client environment:java.version=1.6.0_34 (org.apache.zookeeper.ZooKeeper)
kafka_1 | [2015-09-01 10:44:16,837] INFO Client environment:java.vendor=Sun Microsystems Inc. (org.apache.zookeeper.ZooKeeper)
kafka_1 | [2015-09-01 10:44:16,837] INFO Client environment:java.home=/usr/lib/jvm/java-6-openjdk-amd64/jre (org.apache.zookeeper.ZooKeeper)
kafka_1 | [2015-09-01 10:44:16,838] INFO Client environment:java.class.path=:/opt/kafka_2.10-0.8.2.1/bin/../core/build/dependant-libs-2.10*/*.jar:/opt/kafka_2.10-0.8.2.1/bin/../examples/build/libs//kafka-examples*.jar:/opt/kafka_2.10-0.8.2.1/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/opt/kafka_2.10-0.8.2.1/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/opt/kafka_2.10-0.8.2.1/bin/../clients/build/libs/kafka-clients*.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/jopt-simple-3.2.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka-clients-0.8.2.1.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-javadoc.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-scaladoc.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-sources.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-test.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/log4j-1.2.16.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/lz4-1.2.0.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/scala-library-2.10.4.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/slf4j-api-1.7.6.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/slf4j-log4j12-1.6.1.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/snappy-java-1.1.1.6.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/zkclient-0.3.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/zookeeper-3.4.6.jar:/opt/kafka_2.10-0.8.2.1/bin/../core/build/libs/kafka_2.10*.jar (org.apache.zookeeper.ZooKeeper)
kafka_1 | [2015-09-01 10:44:16,838] INFO Client environment:java.library.path=/usr/lib/jvm/java-6-openjdk-amd64/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk-amd64/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk-amd64/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
kafka_1 | [2015-09-01 10:44:16,838] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
kafka_1 | [2015-09-01 10:44:16,838] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
kafka_1 | [2015-09-01 10:44:16,838] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
kafka_1 | [2015-09-01 10:44:16,838] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
kafka_1 | [2015-09-01 10:44:16,838] INFO Client environment:os.version=3.18.11-tinycore64 (org.apache.zookeeper.ZooKeeper)
kafka_1 | [2015-09-01 10:44:16,838] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
kafka_1 | [2015-09-01 10:44:16,838] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
kafka_1 | [2015-09-01 10:44:16,838] INFO Client environment:user.dir=/ (org.apache.zookeeper.ZooKeeper)
kafka_1 | [2015-09-01 10:44:16,839] INFO Initiating client connection, connectString=172.17.0.10:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient#4b704006 (org.apache.zookeeper.ZooKeeper)
kafka_1 | [2015-09-01 10:44:16,860] INFO Opening socket connection to server 172.17.0.10/172.17.0.10:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
kafka_1 | [2015-09-01 10:44:16,865] INFO Socket connection established to 172.17.0.10/172.17.0.10:2181, initiating session (org.apache.zookeeper.ClientCnxn)
zookeeper_1 | 2015-09-01 10:44:16,866 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory#197] - Accepted socket connection from /172.17.0.12:53850
zookeeper_1 | 2015-09-01 10:44:16,872 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer#868] - Client attempting to establish new session at /172.17.0.12:53850
zookeeper_1 | 2015-09-01 10:44:16,873 [myid:] - INFO [SyncThread:0:FileTxnLog#199] - Creating new log file: log.52
zookeeper_1 | 2015-09-01 10:44:16,883 [myid:] - INFO [SyncThread:0:ZooKeeperServer#617] - Established session 0x14f8881d8da0000 with negotiated timeout 6000 for client /172.17.0.12:53850
kafka_1 | [2015-09-01 10:44:16,886] INFO Session establishment complete on server 172.17.0.10/172.17.0.10:2181, sessionid = 0x14f8881d8da0000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
kafka_1 | [2015-09-01 10:44:16,887] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
kafka_1 | [2015-09-01 10:44:17,030] INFO Log directory '/kafka/kafka-logs-32773' not found, creating it. (kafka.log.LogManager)
kafka_1 | [2015-09-01 10:44:17,037] INFO Loading logs. (kafka.log.LogManager)
kafka_1 | [2015-09-01 10:44:17,042] INFO Logs loading complete. (kafka.log.LogManager)
kafka_1 | [2015-09-01 10:44:17,043] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
kafka_1 | [2015-09-01 10:44:17,047] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
kafka_1 | [2015-09-01 10:44:17,074] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
kafka_1 | [2015-09-01 10:44:17,075] INFO [Socket Server on Broker 32773], Started (kafka.network.SocketServer)
kafka_1 | [2015-09-01 10:44:17,129] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
kafka_1 | [2015-09-01 10:44:17,162] INFO 32773 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
zookeeper_1 | 2015-09-01 10:44:17,344 [myid:] - INFO [ProcessThread(sid:0 cport:-1)::PrepRequestProcessor#645] - Got user-level KeeperException when processing sessionid:0x14f8881d8da0000 type:delete cxid:0x27 zxid:0x55 txntype:-1 reqpath:n/a Error Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for /admin/preferred_replica_election
kafka_1 | [2015-09-01 10:44:17,358] INFO New leader is 32773 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
kafka_1 | [2015-09-01 10:44:17,360] INFO Registered broker 32773 at path /brokers/ids/32773 with address 192.168.59.103:32773. (kafka.utils.ZkUtils$)
kafka_1 | [2015-09-01 10:44:17,371] INFO [Kafka Server 32773], started (kafka.server.KafkaServer)
zookeeper_1 | 2015-09-01 10:49:01,422 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory#197] - Accepted socket connection from /192.168.59.3:50220
zookeeper_1 | 2015-09-01 10:49:01,426 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer#822] - Connection request from old client /192.168.59.3:50220; will be dropped if server is in r-o mode
zookeeper_1 | 2015-09-01 10:49:01,426 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer#868] - Client attempting to establish new session at /192.168.59.3:50220
zookeeper_1 | 2015-09-01 10:49:01,428 [myid:] - INFO [SyncThread:0:ZooKeeperServer#617] - Established session 0x14f8881d8da0001 with negotiated timeout 30000 for client /192.168.59.3:50220
zookeeper_1 | 2015-09-01 10:52:20,886 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory#197] - Accepted socket connection from /172.17.42.1:50639
zookeeper_1 | 2015-09-01 10:52:20,890 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer#822] - Connection request from old client /172.17.42.1:50639; will be dropped if server is in r-o mode
zookeeper_1 | 2015-09-01 10:52:20,890 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer#868] - Client attempting to establish new session at /172.17.42.1:50639
zookeeper_1 | 2015-09-01 10:52:20,893 [myid:] - INFO [SyncThread:0:ZooKeeperServer#617] - Established session 0x14f8881d8da0002 with negotiated timeout 6000 for client /172.17.42.1:50639
This nonsense cost me more than it should have. For anyone else that comes across this - removing the image worked for me as well. Here's some copy pasta:
# Stop all containers
docker stop $(docker ps -a -q)
# Get the id of the container you want to remove, looks like: 8e8683455c7c
docker ps -a
# Remove it
docker rm 8e8683455c7c
# Get the id of the image you want to remove, looks like: f82f62b5876b
docker images
# Delete image
docker rmi f82f62b5876b
# Pull the slot machine lever one more time
docker-compose up
Also also - source: https://gist.github.com/JeffBelback/5687bb02f3618965ca8f