So I have my master node on my Mac and I can check on the webserver my master url as spark://private_ip_address:7077. Then I try to connect a slave node on a remote server.
So I run start-slaves.sh from the master and it creates a log on the remote server. So I assume that the ssh connection is ok. However the connection with the master is not possible as shown by the log content.
21/01/12 13:06:36 INFO ResourceUtils: ==============================================================
21/01/12 13:06:36 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
21/01/12 13:06:36 INFO WorkerWebUI: Bound WorkerWebUI to 127.0.0.0, and started at http://127.0.0.0:8081
21/01/12 13:06:36 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:06:48 INFO Worker: Retrying connection to master (attempt # 1)
21/01/12 13:06:48 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:00 INFO Worker: Retrying connection to master (attempt # 2)
21/01/12 13:07:00 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:12 INFO Worker: Retrying connection to master (attempt # 3)
21/01/12 13:07:12 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:24 INFO Worker: Retrying connection to master (attempt # 4)
21/01/12 13:07:24 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:36 INFO Worker: Retrying connection to master (attempt # 5)
21/01/12 13:07:36 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:48 INFO Worker: Retrying connection to master (attempt # 6)
21/01/12 13:07:48 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:08:36 ERROR RpcOutboxMessage: Ask terminated before connecting successfully
21/01/12 13:08:36 WARN NettyRpcEnv: Ignored failure: java.io.IOException: Connecting to /master_node_private_ip_address:7077 timed out (120000 ms)
21/01/12 13:08:36 WARN Worker: Failed to connect to master master_node_private_ip_address:7077
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:302)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)
at org.apache.spark.deploy.worker.Worker$$anon$1.run(Worker.scala:277)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Connecting to /master_node_private_ip_address:7077 timed out (120000 ms)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:251)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:195)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)
... 4 more
I would be so grateful for some help to understand why this does not connect properly and show on the webserver the slave node.
I'm using a rasbery pi with Rasbian. I want to use Kafka to stream data from a camera to my phone. I downloaded this package from the Kafka website that contains Zookeeper and Kafka:
https://www.apache.org/dyn/closer.cgi?path=/kafka/2.4.1/kafka_2.12-2.4.1.tgz
First I started zookeeper with the zookeeper-server-start.sh located in the bin directory with:
"sudo bin/zookeeper-server-start.sh config/zookeeper.properties" in the terminal. I got back:
[2020-04-07 17:56:44,843] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:44,854] WARN config/zookeeper.properties is relative. Prepend ./ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:44,936] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:44,937] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:44,975] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2020-04-07 17:56:44,976] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2020-04-07 17:56:44,978] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2020-04-07 17:56:44,979] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2020-04-07 17:56:45,010] INFO Log4j found with jmx enabled. (org.apache.zookeeper.jmx.ManagedUtil)
[2020-04-07 17:56:45,263] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:45,264] WARN config/zookeeper.properties is relative. Prepend ./ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:45,268] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:45,269] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:45,271] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
[2020-04-07 17:56:45,319] INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2020-04-07 17:56:45,483] INFO Server environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,484] INFO Server environment:host.name=Rupert (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,485] INFO Server environment:java.version=11.0.6 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,486] INFO Server environment:java.vendor=Raspbian (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,487] INFO Server environment:java.home=/usr/lib/jvm/java-11-openjdk-armhf (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,488] INFO Server environment:java.class.path=/usr/local/kafka/bin/../libs/activation-1.1.1.jar:/usr/local/kafka/bin/../libs/aopalliance-repackaged-2.5.0.jar:/usr/local/kafka/bin/../libs/argparse4j-0.7.0.jar:/usr/local/kafka/bin/../libs/audience-annotations-0.5.0.jar:/usr/local/kafka/bin/../libs/commons-cli-1.4.jar:/usr/local/kafka/bin/../libs/commons-lang3-3.8.1.jar:/usr/local/kafka/bin/../libs/connect-api-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-basic-auth-extension-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-file-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-json-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-mirror-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-mirror-client-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-runtime-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-transforms-2.4.1.jar:/usr/local/kafka/bin/../libs/guava-20.0.jar:/usr/local/kafka/bin/../libs/hk2-api-2.5.0.jar:/usr/local/kafka/bin/../libs/hk2-locator-2.5.0.jar:/usr/local/kafka/bin/../libs/hk2-utils-2.5.0.jar:/usr/local/kafka/bin/../libs/jackson-annotations-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-core-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-databind-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-dataformat-csv-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-datatype-jdk8-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-jaxrs-base-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-module-paranamer-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-module-scala_2.12-2.10.0.jar:/usr/local/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/usr/local/kafka/bin/../libs/jakarta.annotation-api-1.3.4.jar:/usr/local/kafka/bin/../libs/jakarta.inject-2.5.0.jar:/usr/local/kafka/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/usr/local/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/usr/local/kafka/bin/../libs/javassist-3.22.0-CR2.jar:/usr/local/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/usr/local/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/usr/local/kafka/bin/../libs/jaxb-api-2.3.0.jar:/usr/local/kafka/bin/../libs/jersey-client-2.28.jar:/usr/local/kafka/bin/../libs/jersey-common-2.28.jar:/usr/local/kafka/bin/../libs/jersey-container-servlet-2.28.jar:/usr/local/kafka/bin/../libs/jersey-container-servlet-core-2.28.jar:/usr/local/kafka/bin/../libs/jersey-hk2-2.28.jar:/usr/local/kafka/bin/../libs/jersey-media-jaxb-2.28.jar:/usr/local/kafka/bin/../libs/jersey-server-2.28.jar:/usr/local/kafka/bin/../libs/jetty-client-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-continuation-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-http-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-io-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-security-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-server-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-servlet-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-servlets-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-util-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jopt-simple-5.0.4.jar:/usr/local/kafka/bin/../libs/kafka_2.12-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka_2.12-2.4.1-sources.jar:/usr/local/kafka/bin/../libs/kafka-clients-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-log4j-appender-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-examples-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-scala_2.12-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-test-utils-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-tools-2.4.1.jar:/usr/local/kafka/bin/../libs/log4j-1.2.17.jar:/usr/local/kafka/bin/../libs/lz4-java-1.6.0.jar:/usr/local/kafka/bin/../libs/maven-artifact-3.6.1.jar:/usr/local/kafka/bin/../libs/metrics-core-2.2.0.jar:/usr/local/kafka/bin/../libs/netty-buffer-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-codec-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-common-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-handler-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-resolver-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-transport-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-transport-native-epoll-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-transport-native-unix-common-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/usr/local/kafka/bin/../libs/paranamer-2.8.jar:/usr/local/kafka/bin/../libs/plexus-utils-3.2.0.jar:/usr/local/kafka/bin/../libs/reflections-0.9.11.jar:/usr/local/kafka/bin/../libs/rocksdbjni-5.18.3.jar:/usr/local/kafka/bin/../libs/scala-collection-compat_2.12-2.1.2.jar:/usr/local/kafka/bin/../libs/scala-java8-compat_2.12-0.9.0.jar:/usr/local/kafka/bin/../libs/scala-library-2.12.10.jar:/usr/local/kafka/bin/../libs/scala-logging_2.12-3.9.2.jar:/usr/local/kafka/bin/../libs/scala-reflect-2.12.10.jar:/usr/local/kafka/bin/../libs/slf4j-api-1.7.28.jar:/usr/local/kafka/bin/../libs/slf4j-log4j12-1.7.28.jar:/usr/local/kafka/bin/../libs/snappy-java-1.1.7.3.jar:/usr/local/kafka/bin/../libs/validation-api-2.0.1.Final.jar:/usr/local/kafka/bin/../libs/zookeeper-3.5.7.jar:/usr/local/kafka/bin/../libs/zookeeper-jute-3.5.7.jar:/usr/local/kafka/bin/../libs/zstd-jni-1.4.3-1.jar (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,498] INFO Server environment:java.library.path=/usr/java/packages/lib:/usr/lib/arm-linux-gnueabihf/jni:/lib/arm-linux-gnueabihf:/usr/lib/arm-linux-gnueabihf:/usr/lib/jni:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,499] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,500] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,501] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,502] INFO Server environment:os.arch=arm (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,503] INFO Server environment:os.version=4.19.66-v7+ (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,504] INFO Server environment:user.name=root (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,506] INFO Server environment:user.home=/root (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,507] INFO Server environment:user.dir=/usr/local/kafka (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,508] INFO Server environment:os.memory.free=493MB (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,509] INFO Server environment:os.memory.max=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,510] INFO Server environment:os.memory.total=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,529] INFO minSessionTimeout set to 6000 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,531] INFO maxSessionTimeout set to 60000 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,539] INFO Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /tmp/zookeeper/version-2 snapdir /tmp/zookeeper/version-2 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,661] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2020-04-07 17:56:45,699] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers. (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2020-04-07 17:56:45,773] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2020-04-07 17:56:45,965] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2020-04-07 17:56:45,994] INFO Reading snapshot /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileSnap)
[2020-04-07 17:56:46,076] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2020-04-07 17:56:46,377] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.apache.zookeeper.server.ContainerManager)
I believe that his is correct, but please do bring anything to my attention.
Next I tried to run Kafka using "sudo bin/kafka-server-start.sh config/server.properties" and after running for 10 seconds it returned:
[2020-04-07 17:49:40,577] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2020-04-07 17:49:47,215] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2020-04-07 17:49:47,222] INFO starting (kafka.server.KafkaServer)
[2020-04-07 17:49:47,233] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2020-04-07 17:49:47,558] INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2020-04-07 17:49:47,679] INFO Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,681] INFO Client environment:host.name=Rupert (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,682] INFO Client environment:java.version=11.0.6 (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,682] INFO Client environment:java.vendor=Raspbian (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,683] INFO Client environment:java.home=/usr/lib/jvm/java-11-openjdk-armhf (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,684] INFO Client environment:java.class.path=/usr/local/kafka/bin/../libs/activation-1.1.1.jar:/usr/local/kafka/bin/../libs/aopalliance-repackaged-2.5.0.jar:/usr/local/kafka/bin/../libs/argparse4j-0.7.0.jar:/usr/local/kafka/bin/../libs/audience-annotations-0.5.0.jar:/usr/local/kafka/bin/../libs/commons-cli-1.4.jar:/usr/local/kafka/bin/../libs/commons-lang3-3.8.1.jar:/usr/local/kafka/bin/../libs/connect-api-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-basic-auth-extension-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-file-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-json-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-mirror-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-mirror-client-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-runtime-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-transforms-2.4.1.jar:/usr/local/kafka/bin/../libs/guava-20.0.jar:/usr/local/kafka/bin/../libs/hk2-api-2.5.0.jar:/usr/local/kafka/bin/../libs/hk2-locator-2.5.0.jar:/usr/local/kafka/bin/../libs/hk2-utils-2.5.0.jar:/usr/local/kafka/bin/../libs/jackson-annotations-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-core-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-databind-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-dataformat-csv-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-datatype-jdk8-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-jaxrs-base-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-module-paranamer-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-module-scala_2.12-2.10.0.jar:/usr/local/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/usr/local/kafka/bin/../libs/jakarta.annotation-api-1.3.4.jar:/usr/local/kafka/bin/../libs/jakarta.inject-2.5.0.jar:/usr/local/kafka/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/usr/local/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/usr/local/kafka/bin/../libs/javassist-3.22.0-CR2.jar:/usr/local/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/usr/local/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/usr/local/kafka/bin/../libs/jaxb-api-2.3.0.jar:/usr/local/kafka/bin/../libs/jersey-client-2.28.jar:/usr/local/kafka/bin/../libs/jersey-common-2.28.jar:/usr/local/kafka/bin/../libs/jersey-container-servlet-2.28.jar:/usr/local/kafka/bin/../libs/jersey-container-servlet-core-2.28.jar:/usr/local/kafka/bin/../libs/jersey-hk2-2.28.jar:/usr/local/kafka/bin/../libs/jersey-media-jaxb-2.28.jar:/usr/local/kafka/bin/../libs/jersey-server-2.28.jar:/usr/local/kafka/bin/../libs/jetty-client-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-continuation-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-http-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-io-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-security-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-server-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-servlet-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-servlets-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-util-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jopt-simple-5.0.4.jar:/usr/local/kafka/bin/../libs/kafka_2.12-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka_2.12-2.4.1-sources.jar:/usr/local/kafka/bin/../libs/kafka-clients-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-log4j-appender-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-examples-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-scala_2.12-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-test-utils-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-tools-2.4.1.jar:/usr/local/kafka/bin/../libs/log4j-1.2.17.jar:/usr/local/kafka/bin/../libs/lz4-java-1.6.0.jar:/usr/local/kafka/bin/../libs/maven-artifact-3.6.1.jar:/usr/local/kafka/bin/../libs/metrics-core-2.2.0.jar:/usr/local/kafka/bin/../libs/netty-buffer-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-codec-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-common-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-handler-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-resolver-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-transport-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-transport-native-epoll-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-transport-native-unix-common-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/usr/local/kafka/bin/../libs/paranamer-2.8.jar:/usr/local/kafka/bin/../libs/plexus-utils-3.2.0.jar:/usr/local/kafka/bin/../libs/reflections-0.9.11.jar:/usr/local/kafka/bin/../libs/rocksdbjni-5.18.3.jar:/usr/local/kafka/bin/../libs/scala-collection-compat_2.12-2.1.2.jar:/usr/local/kafka/bin/../libs/scala-java8-compat_2.12-0.9.0.jar:/usr/local/kafka/bin/../libs/scala-library-2.12.10.jar:/usr/local/kafka/bin/../libs/scala-logging_2.12-3.9.2.jar:/usr/local/kafka/bin/../libs/scala-reflect-2.12.10.jar:/usr/local/kafka/bin/../libs/slf4j-api-1.7.28.jar:/usr/local/kafka/bin/../libs/slf4j-log4j12-1.7.28.jar:/usr/local/kafka/bin/../libs/snappy-java-1.1.7.3.jar:/usr/local/kafka/bin/../libs/validation-api-2.0.1.Final.jar:/usr/local/kafka/bin/../libs/zookeeper-3.5.7.jar:/usr/local/kafka/bin/../libs/zookeeper-jute-3.5.7.jar:/usr/local/kafka/bin/../libs/zstd-jni-1.4.3-1.jar (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,693] INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/arm-linux-gnueabihf/jni:/lib/arm-linux-gnueabihf:/usr/lib/arm-linux-gnueabihf:/usr/lib/jni:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,695] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,696] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,697] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,698] INFO Client environment:os.arch=arm (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,699] INFO Client environment:os.version=4.19.66-v7+ (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,700] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,701] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,702] INFO Client environment:user.dir=/usr/local/kafka (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,703] INFO Client environment:os.memory.free=975MB (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,704] INFO Client environment:os.memory.max=1024MB (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,704] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,769] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$#114918a (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,887] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2020-04-07 17:49:48,017] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2020-04-07 17:49:48,130] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:48,313] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2020-04-07 17:49:48,449] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:48,557] INFO Socket error occurred: localhost/0:0:0:0:0:0:0:1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:49,678] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:49,682] INFO Socket error occurred: localhost/0:0:0:0:0:0:0:1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:50,784] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:50,787] INFO Socket error occurred: localhost/0:0:0:0:0:0:0:1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:51,890] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:51,894] INFO Socket error occurred: localhost/127.0.0.1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:52,997] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:53,000] INFO Socket error occurred: localhost/0:0:0:0:0:0:0:1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:54,114] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:54,118] INFO Socket error occurred: localhost/127.0.0.1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:54,353] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2020-04-07 17:49:55,221] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:55,375] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:55,396] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:55,412] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2020-04-07 17:49:55,449] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:259)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:255)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:113)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1858)
at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:375)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:399)
at kafka.server.KafkaServer.startup(KafkaServer.scala:207)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:84)
at kafka.Kafka.main(Kafka.scala)
[2020-04-07 17:49:55,480] INFO shutting down (kafka.server.KafkaServer)
[2020-04-07 17:49:55,577] INFO shut down completed (kafka.server.KafkaServer)
[2020-04-07 17:49:55,583] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2020-04-07 17:49:55,612] INFO shutting down (kafka.server.KafkaServer)
My assumption is that Kafka can't reach zookeeper but I really have no clue.
Is there a way I can test if zookeeper is working?
Thanks to anyone who helps, stay healthy.
When I pressed ctrl+shift+c to get rid of the text that showed up afterwards it also undid what needed to happen.
This is a snippet from the system log while shutting down:
INFO [RMI TCP Connection(12)-127.0.0.1] 2016-07-27 22:28:50,995 StorageService.java:3788 - Announcing that I have left the ring for 30000ms
INFO [RMI TCP Connection(12)-127.0.0.1] 2016-07-27 22:29:20,995 ThriftServer.java:142 - Stop listening to thrift clients
INFO [RMI TCP Connection(12)-127.0.0.1] 2016-07-27 22:29:20,997 Server.java:182 - Stop listening for CQL clients
WARN [RMI TCP Connection(12)-127.0.0.1] 2016-07-27 22:29:20,997 Gossiper.java:1508 - No local state or state is in silent shutdown, not announcing shutdown
INFO [RMI TCP Connection(12)-127.0.0.1] 2016-07-27 22:29:20,997 MessagingService.java:786 - Waiting for messaging service to quiesce
INFO [ACCEPT-sysengplayl0127.bio-iad.ea.com/10.72.194.229] 2016-07-27 22:29:20,998 MessagingService.java:1133 - MessagingService has terminated the accept() thread
INFO [RMI TCP Connection(12)-127.0.0.1] 2016-07-27 22:29:21,022 StorageService.java:1411 - DECOMMISSIONED
INFO [main] 2016-07-27 22:32:17,534 YamlConfigurationLoader.java:89 - Configuration location: file:/opt/cassandra/product/apache-cassandra-3.7/conf/cassandra.yaml
And then while starting up:
INFO [main] 2016-07-27 22:32:20,316 StorageService.java:630 - Cassandra version: 3.7
INFO [main] 2016-07-27 22:32:20,316 StorageService.java:631 - Thrift API version: 20.1.0
INFO [main] 2016-07-27 22:32:20,316 StorageService.java:632 - CQL supported versions: 3.4.2 (default: 3.4.2)
INFO [main] 2016-07-27 22:32:20,351 IndexSummaryManager.java:85 - Initializing index summary manager with a memory pool size of 397 MB and a resize interval of 60 minutes
ERROR [main] 2016-07-27 22:32:20,357 CassandraDaemon.java:731 - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: This node was decommissioned and will not rejoin the ring unless cassandra.override_decommission=true has been set, or all existing data is removed and the node is bootstrapped again
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:815) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:725) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:625) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:370) [apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:585) [apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:714) [apache-cassandra-3.7.jar:3.7]
WARN [StorageServiceShutdownHook] 2016-07-27 22:32:20,358 Gossiper.java:1508 - No local state or state is in silent shutdown, not announcing shutdown
INFO [StorageServiceShutdownHook] 2016-07-27 22:32:20,359 MessagingService.java:786 - Waiting for messaging service to quiesce
Is there something wrong with the configuration?
I had faced same issue.
Posting the answer so that it might help others.
As the log suggests, the property "cassandra.override_decommission" should be overridden.
start cassandra with the syntax:
cassandra -Dcassandra.override_decommission=true
This should add the node back to the cluster.
I am trying to configure a two node cluster with cassandra in windows r2 2008 So i installed cassandra community version in one server (10.xxx.0.1,10.xxx.0.2) And then I stopped the service and then edited the configuraton.yaml file in the conf folder.
The changes are:
cluster_name
commented the num_tokens
gave the tokens in initial_token,
seeds as 10.xxx.0.1,10.xxx.0.2,
listen_addresses are their respective ip addresses which are 10.xxx.0.1,10.xxx.0.2,
rpc_addresses are same as listen_address,
endpointsnitch as gossip
I also changed the cassandra rackdc.properties file to dc=DC1 rack=RAC1.
and changed the cassandra-env file - uncommented: JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=10.xxx.0.1
I then saved and started back the service and ran nodetool status and got
Datacenter: DC1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Owns (effective) Host ID
Token Rack
UN 10.104.0.15 46.65 KB 100.0% bc41a884-baaf-4a52-85f3-f3270c2ec9
57 -9223372036854775808 RAC1
and opened the cqlsh, but it is not connecting. Below is the error:
ERROR [Initialization] 2015-10-12 17:10:21,353 Error connecting via JMX: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
java.net.ConnectException: Connection refused: connect]
INFO [main] 2015-10-12 17:10:24,612 Reconnecting to a backup OpsCenter instance
INFO [main] 2015-10-12 17:10:24,615 SSL communication is disabled
INFO [main] 2015-10-12 17:10:24,615 Creating stomp connection to 127.0.0.1:61620
INFO [main] 2015-10-12 17:10:24,628 Starting Jetty server: {:join? false, :ssl? false, :host nil, :port 61621}
INFO [Initialization] 2015-10-12 17:10:24,632 Sleeping for 2s before trying to determine IP over JMX again
INFO [StompConnection receiver] 2015-10-12 17:10:24,640 Reconnecting in 0s.
INFO [main] 2015-10-12 18:25:56,347 Waiting for the config from OpsCenter
INFO [main] 2015-10-12 18:25:56,349 Attempting to determine Cassandra's broadcast address through JMX
INFO [main] 2015-10-12 18:25:56,350 Starting Stomp
INFO [main] 2015-10-12 18:25:56,350 Starting up agent communcation with OpsCenter.
INFO [Initialization] 2015-10-12 18:25:56,356 New JMX connection (127.0.0.1:7199)
ERROR [Initialization] 2015-10-12 18:25:57,652 Error connecting via JMX: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
java.net.ConnectException: Connection refused: connect]
INFO [main] 2015-10-12 18:26:00,768 Reconnecting to a backup OpsCenter instance
INFO [main] 2015-10-12 18:26:00,770 SSL communication is disabled
INFO [main] 2015-10-12 18:26:00,770 Creating stomp connection to 127.0.0.1:61620
INFO [main] 2015-10-12 18:26:00,779 Starting Jetty server: {:join? false, :ssl? false, :host nil, :port 61621}
INFO [Initialization] 2015-10-12 18:26:00,782 Sleeping for 2s before trying to determine IP over JMX again
INFO [StompConnection receiver] 2015-10-12 18:26:00,945 Reconnecting in 0s.
INFO [StompConnection receiver] 2015-10-12 18:26:01,136 Connected to 127.0.0.1:61620
I tried to create the folder in file system of HDFS by means of a command
./hadoop fs -mkdir /user/hadoop
and as a result I received the following messages
13/02/17 09:45:50 INFO ipc.Client: Retrying connect to server: one/192.168.1.8:9000. Already tried 0 time(s).
13/02/17 09:45:51 INFO ipc.Client: Retrying connect to server: one/192.168.1.8:9000. Already tried 1 time(s).
13/02/17 09:45:52 INFO ipc.Client: Retrying connect to server: one/192.168.1.8:9000. Already tried 2 time(s).
13/02/17 09:45:53 INFO ipc.Client: Retrying connect to server: one/192.168.1.8:9000. Already tried 3 time(s).
13/02/17 09:45:54 INFO ipc.Client: Retrying connect to server: one/192.168.1.8:9000. Already tried 4 time(s).
13/02/17 09:45:55 INFO ipc.Client: Retrying connect to server: one/192.168.1.8:9000. Already tried 5 time(s).
13/02/17 09:45:56 INFO ipc.Client: Retrying connect to server: one/192.168.1.8:9000. Already tried 6 time(s).
13/02/17 09:45:57 INFO ipc.Client: Retrying connect to server: one/192.168.1.8:9000. Already tried 7 time(s).
13/02/17 09:45:58 INFO ipc.Client: Retrying connect to server: one/192.168.1.8:9000. Already tried 8 time(s).
13/02/17 09:45:59 INFO ipc.Client: Retrying connect to server: one/192.168.1.8:9000. Already tried 9 time(s).
Bad connection to FS. command aborted. exception: Call to one/192.168.1.8:9000 failed on connection exception: java.net.ConnectException: Connection refused
In the /export/hadoop-1.0.1/conf/core-site.xml file the following is specified:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.1.8:9000</value>
</property>
</configuration>
Wanted to specify - whether correct I specified port? If isn't present, tell what it is necessary?
seems like hadoop daemons are not running. check with jps