I am trying to configure a two node cluster with cassandra in windows r2 2008 So i installed cassandra community version in one server (10.xxx.0.1,10.xxx.0.2) And then I stopped the service and then edited the configuraton.yaml file in the conf folder.
The changes are:
cluster_name
commented the num_tokens
gave the tokens in initial_token,
seeds as 10.xxx.0.1,10.xxx.0.2,
listen_addresses are their respective ip addresses which are 10.xxx.0.1,10.xxx.0.2,
rpc_addresses are same as listen_address,
endpointsnitch as gossip
I also changed the cassandra rackdc.properties file to dc=DC1 rack=RAC1.
and changed the cassandra-env file - uncommented: JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname=10.xxx.0.1
I then saved and started back the service and ran nodetool status and got
Datacenter: DC1
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Owns (effective) Host ID
Token Rack
UN 10.104.0.15 46.65 KB 100.0% bc41a884-baaf-4a52-85f3-f3270c2ec9
57 -9223372036854775808 RAC1
and opened the cqlsh, but it is not connecting. Below is the error:
ERROR [Initialization] 2015-10-12 17:10:21,353 Error connecting via JMX: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
java.net.ConnectException: Connection refused: connect]
INFO [main] 2015-10-12 17:10:24,612 Reconnecting to a backup OpsCenter instance
INFO [main] 2015-10-12 17:10:24,615 SSL communication is disabled
INFO [main] 2015-10-12 17:10:24,615 Creating stomp connection to 127.0.0.1:61620
INFO [main] 2015-10-12 17:10:24,628 Starting Jetty server: {:join? false, :ssl? false, :host nil, :port 61621}
INFO [Initialization] 2015-10-12 17:10:24,632 Sleeping for 2s before trying to determine IP over JMX again
INFO [StompConnection receiver] 2015-10-12 17:10:24,640 Reconnecting in 0s.
INFO [main] 2015-10-12 18:25:56,347 Waiting for the config from OpsCenter
INFO [main] 2015-10-12 18:25:56,349 Attempting to determine Cassandra's broadcast address through JMX
INFO [main] 2015-10-12 18:25:56,350 Starting Stomp
INFO [main] 2015-10-12 18:25:56,350 Starting up agent communcation with OpsCenter.
INFO [Initialization] 2015-10-12 18:25:56,356 New JMX connection (127.0.0.1:7199)
ERROR [Initialization] 2015-10-12 18:25:57,652 Error connecting via JMX: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0.0.1; nested exception is:
java.net.ConnectException: Connection refused: connect]
INFO [main] 2015-10-12 18:26:00,768 Reconnecting to a backup OpsCenter instance
INFO [main] 2015-10-12 18:26:00,770 SSL communication is disabled
INFO [main] 2015-10-12 18:26:00,770 Creating stomp connection to 127.0.0.1:61620
INFO [main] 2015-10-12 18:26:00,779 Starting Jetty server: {:join? false, :ssl? false, :host nil, :port 61621}
INFO [Initialization] 2015-10-12 18:26:00,782 Sleeping for 2s before trying to determine IP over JMX again
INFO [StompConnection receiver] 2015-10-12 18:26:00,945 Reconnecting in 0s.
INFO [StompConnection receiver] 2015-10-12 18:26:01,136 Connected to 127.0.0.1:61620
Related
So I have my master node on my Mac and I can check on the webserver my master url as spark://private_ip_address:7077. Then I try to connect a slave node on a remote server.
So I run start-slaves.sh from the master and it creates a log on the remote server. So I assume that the ssh connection is ok. However the connection with the master is not possible as shown by the log content.
21/01/12 13:06:36 INFO ResourceUtils: ==============================================================
21/01/12 13:06:36 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
21/01/12 13:06:36 INFO WorkerWebUI: Bound WorkerWebUI to 127.0.0.0, and started at http://127.0.0.0:8081
21/01/12 13:06:36 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:06:48 INFO Worker: Retrying connection to master (attempt # 1)
21/01/12 13:06:48 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:00 INFO Worker: Retrying connection to master (attempt # 2)
21/01/12 13:07:00 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:12 INFO Worker: Retrying connection to master (attempt # 3)
21/01/12 13:07:12 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:24 INFO Worker: Retrying connection to master (attempt # 4)
21/01/12 13:07:24 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:36 INFO Worker: Retrying connection to master (attempt # 5)
21/01/12 13:07:36 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:48 INFO Worker: Retrying connection to master (attempt # 6)
21/01/12 13:07:48 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:08:36 ERROR RpcOutboxMessage: Ask terminated before connecting successfully
21/01/12 13:08:36 WARN NettyRpcEnv: Ignored failure: java.io.IOException: Connecting to /master_node_private_ip_address:7077 timed out (120000 ms)
21/01/12 13:08:36 WARN Worker: Failed to connect to master master_node_private_ip_address:7077
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:302)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)
at org.apache.spark.deploy.worker.Worker$$anon$1.run(Worker.scala:277)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Connecting to /master_node_private_ip_address:7077 timed out (120000 ms)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:251)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:195)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)
... 4 more
I would be so grateful for some help to understand why this does not connect properly and show on the webserver the slave node.
I'm using a rasbery pi with Rasbian. I want to use Kafka to stream data from a camera to my phone. I downloaded this package from the Kafka website that contains Zookeeper and Kafka:
https://www.apache.org/dyn/closer.cgi?path=/kafka/2.4.1/kafka_2.12-2.4.1.tgz
First I started zookeeper with the zookeeper-server-start.sh located in the bin directory with:
"sudo bin/zookeeper-server-start.sh config/zookeeper.properties" in the terminal. I got back:
[2020-04-07 17:56:44,843] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:44,854] WARN config/zookeeper.properties is relative. Prepend ./ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:44,936] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:44,937] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:44,975] INFO autopurge.snapRetainCount set to 3 (org.apache.zookeeper.server.DatadirCleanupManager)
[2020-04-07 17:56:44,976] INFO autopurge.purgeInterval set to 0 (org.apache.zookeeper.server.DatadirCleanupManager)
[2020-04-07 17:56:44,978] INFO Purge task is not scheduled. (org.apache.zookeeper.server.DatadirCleanupManager)
[2020-04-07 17:56:44,979] WARN Either no config or no quorum defined in config, running in standalone mode (org.apache.zookeeper.server.quorum.QuorumPeerMain)
[2020-04-07 17:56:45,010] INFO Log4j found with jmx enabled. (org.apache.zookeeper.jmx.ManagedUtil)
[2020-04-07 17:56:45,263] INFO Reading configuration from: config/zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:45,264] WARN config/zookeeper.properties is relative. Prepend ./ to indicate that you're sure! (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:45,268] INFO clientPortAddress is 0.0.0.0:2181 (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:45,269] INFO secureClientPort is not set (org.apache.zookeeper.server.quorum.QuorumPeerConfig)
[2020-04-07 17:56:45,271] INFO Starting server (org.apache.zookeeper.server.ZooKeeperServerMain)
[2020-04-07 17:56:45,319] INFO zookeeper.snapshot.trust.empty : false (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2020-04-07 17:56:45,483] INFO Server environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,484] INFO Server environment:host.name=Rupert (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,485] INFO Server environment:java.version=11.0.6 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,486] INFO Server environment:java.vendor=Raspbian (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,487] INFO Server environment:java.home=/usr/lib/jvm/java-11-openjdk-armhf (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,488] INFO Server environment:java.class.path=/usr/local/kafka/bin/../libs/activation-1.1.1.jar:/usr/local/kafka/bin/../libs/aopalliance-repackaged-2.5.0.jar:/usr/local/kafka/bin/../libs/argparse4j-0.7.0.jar:/usr/local/kafka/bin/../libs/audience-annotations-0.5.0.jar:/usr/local/kafka/bin/../libs/commons-cli-1.4.jar:/usr/local/kafka/bin/../libs/commons-lang3-3.8.1.jar:/usr/local/kafka/bin/../libs/connect-api-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-basic-auth-extension-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-file-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-json-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-mirror-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-mirror-client-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-runtime-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-transforms-2.4.1.jar:/usr/local/kafka/bin/../libs/guava-20.0.jar:/usr/local/kafka/bin/../libs/hk2-api-2.5.0.jar:/usr/local/kafka/bin/../libs/hk2-locator-2.5.0.jar:/usr/local/kafka/bin/../libs/hk2-utils-2.5.0.jar:/usr/local/kafka/bin/../libs/jackson-annotations-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-core-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-databind-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-dataformat-csv-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-datatype-jdk8-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-jaxrs-base-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-module-paranamer-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-module-scala_2.12-2.10.0.jar:/usr/local/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/usr/local/kafka/bin/../libs/jakarta.annotation-api-1.3.4.jar:/usr/local/kafka/bin/../libs/jakarta.inject-2.5.0.jar:/usr/local/kafka/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/usr/local/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/usr/local/kafka/bin/../libs/javassist-3.22.0-CR2.jar:/usr/local/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/usr/local/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/usr/local/kafka/bin/../libs/jaxb-api-2.3.0.jar:/usr/local/kafka/bin/../libs/jersey-client-2.28.jar:/usr/local/kafka/bin/../libs/jersey-common-2.28.jar:/usr/local/kafka/bin/../libs/jersey-container-servlet-2.28.jar:/usr/local/kafka/bin/../libs/jersey-container-servlet-core-2.28.jar:/usr/local/kafka/bin/../libs/jersey-hk2-2.28.jar:/usr/local/kafka/bin/../libs/jersey-media-jaxb-2.28.jar:/usr/local/kafka/bin/../libs/jersey-server-2.28.jar:/usr/local/kafka/bin/../libs/jetty-client-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-continuation-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-http-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-io-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-security-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-server-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-servlet-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-servlets-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-util-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jopt-simple-5.0.4.jar:/usr/local/kafka/bin/../libs/kafka_2.12-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka_2.12-2.4.1-sources.jar:/usr/local/kafka/bin/../libs/kafka-clients-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-log4j-appender-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-examples-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-scala_2.12-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-test-utils-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-tools-2.4.1.jar:/usr/local/kafka/bin/../libs/log4j-1.2.17.jar:/usr/local/kafka/bin/../libs/lz4-java-1.6.0.jar:/usr/local/kafka/bin/../libs/maven-artifact-3.6.1.jar:/usr/local/kafka/bin/../libs/metrics-core-2.2.0.jar:/usr/local/kafka/bin/../libs/netty-buffer-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-codec-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-common-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-handler-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-resolver-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-transport-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-transport-native-epoll-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-transport-native-unix-common-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/usr/local/kafka/bin/../libs/paranamer-2.8.jar:/usr/local/kafka/bin/../libs/plexus-utils-3.2.0.jar:/usr/local/kafka/bin/../libs/reflections-0.9.11.jar:/usr/local/kafka/bin/../libs/rocksdbjni-5.18.3.jar:/usr/local/kafka/bin/../libs/scala-collection-compat_2.12-2.1.2.jar:/usr/local/kafka/bin/../libs/scala-java8-compat_2.12-0.9.0.jar:/usr/local/kafka/bin/../libs/scala-library-2.12.10.jar:/usr/local/kafka/bin/../libs/scala-logging_2.12-3.9.2.jar:/usr/local/kafka/bin/../libs/scala-reflect-2.12.10.jar:/usr/local/kafka/bin/../libs/slf4j-api-1.7.28.jar:/usr/local/kafka/bin/../libs/slf4j-log4j12-1.7.28.jar:/usr/local/kafka/bin/../libs/snappy-java-1.1.7.3.jar:/usr/local/kafka/bin/../libs/validation-api-2.0.1.Final.jar:/usr/local/kafka/bin/../libs/zookeeper-3.5.7.jar:/usr/local/kafka/bin/../libs/zookeeper-jute-3.5.7.jar:/usr/local/kafka/bin/../libs/zstd-jni-1.4.3-1.jar (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,498] INFO Server environment:java.library.path=/usr/java/packages/lib:/usr/lib/arm-linux-gnueabihf/jni:/lib/arm-linux-gnueabihf:/usr/lib/arm-linux-gnueabihf:/usr/lib/jni:/lib:/usr/lib (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,499] INFO Server environment:java.io.tmpdir=/tmp (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,500] INFO Server environment:java.compiler=<NA> (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,501] INFO Server environment:os.name=Linux (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,502] INFO Server environment:os.arch=arm (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,503] INFO Server environment:os.version=4.19.66-v7+ (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,504] INFO Server environment:user.name=root (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,506] INFO Server environment:user.home=/root (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,507] INFO Server environment:user.dir=/usr/local/kafka (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,508] INFO Server environment:os.memory.free=493MB (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,509] INFO Server environment:os.memory.max=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,510] INFO Server environment:os.memory.total=512MB (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,529] INFO minSessionTimeout set to 6000 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,531] INFO maxSessionTimeout set to 60000 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,539] INFO Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 60000 datadir /tmp/zookeeper/version-2 snapdir /tmp/zookeeper/version-2 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-04-07 17:56:45,661] INFO Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory (org.apache.zookeeper.server.ServerCnxnFactory)
[2020-04-07 17:56:45,699] INFO Configuring NIO connection handler with 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads, and 64 kB direct buffers. (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2020-04-07 17:56:45,773] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2020-04-07 17:56:45,965] INFO zookeeper.snapshotSizeFactor = 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2020-04-07 17:56:45,994] INFO Reading snapshot /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileSnap)
[2020-04-07 17:56:46,076] INFO Snapshotting: 0x0 to /tmp/zookeeper/version-2/snapshot.0 (org.apache.zookeeper.server.persistence.FileTxnSnapLog)
[2020-04-07 17:56:46,377] INFO Using checkIntervalMs=60000 maxPerMinute=10000 (org.apache.zookeeper.server.ContainerManager)
I believe that his is correct, but please do bring anything to my attention.
Next I tried to run Kafka using "sudo bin/kafka-server-start.sh config/server.properties" and after running for 10 seconds it returned:
[2020-04-07 17:49:40,577] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2020-04-07 17:49:47,215] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2020-04-07 17:49:47,222] INFO starting (kafka.server.KafkaServer)
[2020-04-07 17:49:47,233] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2020-04-07 17:49:47,558] INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2020-04-07 17:49:47,679] INFO Client environment:zookeeper.version=3.5.7-f0fdd52973d373ffd9c86b81d99842dc2c7f660e, built on 02/10/2020 11:30 GMT (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,681] INFO Client environment:host.name=Rupert (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,682] INFO Client environment:java.version=11.0.6 (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,682] INFO Client environment:java.vendor=Raspbian (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,683] INFO Client environment:java.home=/usr/lib/jvm/java-11-openjdk-armhf (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,684] INFO Client environment:java.class.path=/usr/local/kafka/bin/../libs/activation-1.1.1.jar:/usr/local/kafka/bin/../libs/aopalliance-repackaged-2.5.0.jar:/usr/local/kafka/bin/../libs/argparse4j-0.7.0.jar:/usr/local/kafka/bin/../libs/audience-annotations-0.5.0.jar:/usr/local/kafka/bin/../libs/commons-cli-1.4.jar:/usr/local/kafka/bin/../libs/commons-lang3-3.8.1.jar:/usr/local/kafka/bin/../libs/connect-api-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-basic-auth-extension-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-file-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-json-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-mirror-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-mirror-client-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-runtime-2.4.1.jar:/usr/local/kafka/bin/../libs/connect-transforms-2.4.1.jar:/usr/local/kafka/bin/../libs/guava-20.0.jar:/usr/local/kafka/bin/../libs/hk2-api-2.5.0.jar:/usr/local/kafka/bin/../libs/hk2-locator-2.5.0.jar:/usr/local/kafka/bin/../libs/hk2-utils-2.5.0.jar:/usr/local/kafka/bin/../libs/jackson-annotations-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-core-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-databind-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-dataformat-csv-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-datatype-jdk8-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-jaxrs-base-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-jaxrs-json-provider-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-module-jaxb-annotations-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-module-paranamer-2.10.0.jar:/usr/local/kafka/bin/../libs/jackson-module-scala_2.12-2.10.0.jar:/usr/local/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/usr/local/kafka/bin/../libs/jakarta.annotation-api-1.3.4.jar:/usr/local/kafka/bin/../libs/jakarta.inject-2.5.0.jar:/usr/local/kafka/bin/../libs/jakarta.ws.rs-api-2.1.5.jar:/usr/local/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/usr/local/kafka/bin/../libs/javassist-3.22.0-CR2.jar:/usr/local/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/usr/local/kafka/bin/../libs/javax.ws.rs-api-2.1.1.jar:/usr/local/kafka/bin/../libs/jaxb-api-2.3.0.jar:/usr/local/kafka/bin/../libs/jersey-client-2.28.jar:/usr/local/kafka/bin/../libs/jersey-common-2.28.jar:/usr/local/kafka/bin/../libs/jersey-container-servlet-2.28.jar:/usr/local/kafka/bin/../libs/jersey-container-servlet-core-2.28.jar:/usr/local/kafka/bin/../libs/jersey-hk2-2.28.jar:/usr/local/kafka/bin/../libs/jersey-media-jaxb-2.28.jar:/usr/local/kafka/bin/../libs/jersey-server-2.28.jar:/usr/local/kafka/bin/../libs/jetty-client-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-continuation-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-http-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-io-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-security-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-server-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-servlet-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-servlets-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jetty-util-9.4.20.v20190813.jar:/usr/local/kafka/bin/../libs/jopt-simple-5.0.4.jar:/usr/local/kafka/bin/../libs/kafka_2.12-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka_2.12-2.4.1-sources.jar:/usr/local/kafka/bin/../libs/kafka-clients-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-log4j-appender-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-examples-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-scala_2.12-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-streams-test-utils-2.4.1.jar:/usr/local/kafka/bin/../libs/kafka-tools-2.4.1.jar:/usr/local/kafka/bin/../libs/log4j-1.2.17.jar:/usr/local/kafka/bin/../libs/lz4-java-1.6.0.jar:/usr/local/kafka/bin/../libs/maven-artifact-3.6.1.jar:/usr/local/kafka/bin/../libs/metrics-core-2.2.0.jar:/usr/local/kafka/bin/../libs/netty-buffer-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-codec-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-common-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-handler-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-resolver-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-transport-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-transport-native-epoll-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/netty-transport-native-unix-common-4.1.45.Final.jar:/usr/local/kafka/bin/../libs/osgi-resource-locator-1.0.1.jar:/usr/local/kafka/bin/../libs/paranamer-2.8.jar:/usr/local/kafka/bin/../libs/plexus-utils-3.2.0.jar:/usr/local/kafka/bin/../libs/reflections-0.9.11.jar:/usr/local/kafka/bin/../libs/rocksdbjni-5.18.3.jar:/usr/local/kafka/bin/../libs/scala-collection-compat_2.12-2.1.2.jar:/usr/local/kafka/bin/../libs/scala-java8-compat_2.12-0.9.0.jar:/usr/local/kafka/bin/../libs/scala-library-2.12.10.jar:/usr/local/kafka/bin/../libs/scala-logging_2.12-3.9.2.jar:/usr/local/kafka/bin/../libs/scala-reflect-2.12.10.jar:/usr/local/kafka/bin/../libs/slf4j-api-1.7.28.jar:/usr/local/kafka/bin/../libs/slf4j-log4j12-1.7.28.jar:/usr/local/kafka/bin/../libs/snappy-java-1.1.7.3.jar:/usr/local/kafka/bin/../libs/validation-api-2.0.1.Final.jar:/usr/local/kafka/bin/../libs/zookeeper-3.5.7.jar:/usr/local/kafka/bin/../libs/zookeeper-jute-3.5.7.jar:/usr/local/kafka/bin/../libs/zstd-jni-1.4.3-1.jar (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,693] INFO Client environment:java.library.path=/usr/java/packages/lib:/usr/lib/arm-linux-gnueabihf/jni:/lib/arm-linux-gnueabihf:/usr/lib/arm-linux-gnueabihf:/usr/lib/jni:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,695] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,696] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,697] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,698] INFO Client environment:os.arch=arm (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,699] INFO Client environment:os.version=4.19.66-v7+ (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,700] INFO Client environment:user.name=root (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,701] INFO Client environment:user.home=/root (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,702] INFO Client environment:user.dir=/usr/local/kafka (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,703] INFO Client environment:os.memory.free=975MB (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,704] INFO Client environment:os.memory.max=1024MB (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,704] INFO Client environment:os.memory.total=1024MB (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,769] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$#114918a (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:47,887] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util)
[2020-04-07 17:49:48,017] INFO jute.maxbuffer value is 4194304 Bytes (org.apache.zookeeper.ClientCnxnSocket)
[2020-04-07 17:49:48,130] INFO zookeeper.request.timeout value is 0. feature enabled= (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:48,313] INFO [ZooKeeperClient Kafka server] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2020-04-07 17:49:48,449] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:48,557] INFO Socket error occurred: localhost/0:0:0:0:0:0:0:1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:49,678] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:49,682] INFO Socket error occurred: localhost/0:0:0:0:0:0:0:1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:50,784] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:50,787] INFO Socket error occurred: localhost/0:0:0:0:0:0:0:1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:51,890] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:51,894] INFO Socket error occurred: localhost/127.0.0.1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:52,997] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:53,000] INFO Socket error occurred: localhost/0:0:0:0:0:0:0:1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:54,114] INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:54,118] INFO Socket error occurred: localhost/127.0.0.1:2181: Connection refused (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:54,353] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2020-04-07 17:49:55,221] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:55,375] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2020-04-07 17:49:55,396] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2020-04-07 17:49:55,412] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2020-04-07 17:49:55,449] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:259)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:255)
at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:113)
at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1858)
at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:375)
at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:399)
at kafka.server.KafkaServer.startup(KafkaServer.scala:207)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:44)
at kafka.Kafka$.main(Kafka.scala:84)
at kafka.Kafka.main(Kafka.scala)
[2020-04-07 17:49:55,480] INFO shutting down (kafka.server.KafkaServer)
[2020-04-07 17:49:55,577] INFO shut down completed (kafka.server.KafkaServer)
[2020-04-07 17:49:55,583] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2020-04-07 17:49:55,612] INFO shutting down (kafka.server.KafkaServer)
My assumption is that Kafka can't reach zookeeper but I really have no clue.
Is there a way I can test if zookeeper is working?
Thanks to anyone who helps, stay healthy.
When I pressed ctrl+shift+c to get rid of the text that showed up afterwards it also undid what needed to happen.
This is a snippet from the system log while shutting down:
INFO [RMI TCP Connection(12)-127.0.0.1] 2016-07-27 22:28:50,995 StorageService.java:3788 - Announcing that I have left the ring for 30000ms
INFO [RMI TCP Connection(12)-127.0.0.1] 2016-07-27 22:29:20,995 ThriftServer.java:142 - Stop listening to thrift clients
INFO [RMI TCP Connection(12)-127.0.0.1] 2016-07-27 22:29:20,997 Server.java:182 - Stop listening for CQL clients
WARN [RMI TCP Connection(12)-127.0.0.1] 2016-07-27 22:29:20,997 Gossiper.java:1508 - No local state or state is in silent shutdown, not announcing shutdown
INFO [RMI TCP Connection(12)-127.0.0.1] 2016-07-27 22:29:20,997 MessagingService.java:786 - Waiting for messaging service to quiesce
INFO [ACCEPT-sysengplayl0127.bio-iad.ea.com/10.72.194.229] 2016-07-27 22:29:20,998 MessagingService.java:1133 - MessagingService has terminated the accept() thread
INFO [RMI TCP Connection(12)-127.0.0.1] 2016-07-27 22:29:21,022 StorageService.java:1411 - DECOMMISSIONED
INFO [main] 2016-07-27 22:32:17,534 YamlConfigurationLoader.java:89 - Configuration location: file:/opt/cassandra/product/apache-cassandra-3.7/conf/cassandra.yaml
And then while starting up:
INFO [main] 2016-07-27 22:32:20,316 StorageService.java:630 - Cassandra version: 3.7
INFO [main] 2016-07-27 22:32:20,316 StorageService.java:631 - Thrift API version: 20.1.0
INFO [main] 2016-07-27 22:32:20,316 StorageService.java:632 - CQL supported versions: 3.4.2 (default: 3.4.2)
INFO [main] 2016-07-27 22:32:20,351 IndexSummaryManager.java:85 - Initializing index summary manager with a memory pool size of 397 MB and a resize interval of 60 minutes
ERROR [main] 2016-07-27 22:32:20,357 CassandraDaemon.java:731 - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: This node was decommissioned and will not rejoin the ring unless cassandra.override_decommission=true has been set, or all existing data is removed and the node is bootstrapped again
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:815) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:725) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:625) ~[apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:370) [apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:585) [apache-cassandra-3.7.jar:3.7]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:714) [apache-cassandra-3.7.jar:3.7]
WARN [StorageServiceShutdownHook] 2016-07-27 22:32:20,358 Gossiper.java:1508 - No local state or state is in silent shutdown, not announcing shutdown
INFO [StorageServiceShutdownHook] 2016-07-27 22:32:20,359 MessagingService.java:786 - Waiting for messaging service to quiesce
Is there something wrong with the configuration?
I had faced same issue.
Posting the answer so that it might help others.
As the log suggests, the property "cassandra.override_decommission" should be overridden.
start cassandra with the syntax:
cassandra -Dcassandra.override_decommission=true
This should add the node back to the cluster.
I have installed hadoop 2.6.0 multi-node cluster in EC2 instance(ubuntu 14.04 64 bit). All demons(NameNode,SecondaryNameNode,ResourceManager) in master is up,but in slave machine only DataNode is up NodeManager is shutting down due to connection refuse.
Kindly help me in this regard. Thanks in advance
The log file of my NodeManager is below:
2015-09-08 07:59:36,606 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: NodeManager configured with 8 G physical memory allocated to containers, which is more than 80% of the total physical memory available (992.5 M). Thrashing might happen.
2015-09-08 07:59:36,613 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Initialized nodemanager for null: physical-memory=8192 virtual-memory=17204 virtual-cores=8
2015-09-08 07:59:36,646 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2015-09-08 07:59:36,666 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 53949
2015-09-08 07:59:36,688 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.api.ContainerManagementProtocolPB to the server
2015-09-08 07:59:36,688 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Blocking new container-requests as container manager rpc server is still starting.
2015-09-08 07:59:36,691 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2015-09-08 07:59:36,692 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 53949: starting
2015-09-08 07:59:36,707 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager: Updating node address : ec2-52-88-167-9.us-west-2.compute.amazonaws.com:53949
2015-09-08 07:59:36,713 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
2015-09-08 07:59:36,713 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 8040
2015-09-08 07:59:36,716 INFO org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.yarn.server.nodemanager.api.LocalizationProtocolPB to the server
2015-09-08 07:59:36,717 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2015-09-08 07:59:36,717 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 8040: starting
2015-09-08 07:59:36,717 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Localizer started on port 8040
2015-09-08 07:59:36,719 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: ContainerManager started at ec2-52-88-167-9.us-west-2.compute.amazonaws.com/172.31.29.154:53949
2015-09-08 07:59:36,719 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: ContainerManager bound to 0.0.0.0/0.0.0.0:0
2015-09-08 07:59:36,719 INFO org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer: Instantiating NMWebApp at 0.0.0.0:8042
2015-09-08 07:59:36,790 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2015-09-08 07:59:36,793 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.nodemanager is not defined
2015-09-08 07:59:36,805 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2015-09-08 07:59:36,806 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context node
2015-09-08 07:59:36,806 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2015-09-08 07:59:36,807 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2015-09-08 07:59:36,812 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /node/*
2015-09-08 07:59:36,812 INFO org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
2015-09-08 07:59:36,820 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 8042
2015-09-08 07:59:36,820 INFO org.mortbay.log: jetty-6.1.26
2015-09-08 07:59:36,863 INFO org.mortbay.log: Extract jar:file:/home/ubuntu/hadoop/hadoop-2.6.0/share/hadoop/yarn/hadoop-yarn-common-2.6.0.jar!/webapps/node to /tmp/Jetty_0_0_0_0_8042_node____19tj0x/webapp
2015-09-08 07:59:37,358 INFO org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup#0.0.0.0:8042
2015-09-08 07:59:37,359 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /node started at 8042
2015-09-08 07:59:37,879 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
2015-09-08 07:59:37,885 INFO org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8031
2015-09-08 07:59:37,913 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out 0 NM container statuses: []
2015-09-08 07:59:37,917 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registering with RM using containers :[]
**2015-09-08 07:59:38,951 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-09-08 07:59:39,956 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-09-08 07:59:40,957 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-09-08 07:59:41,957 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-09-08 07:59:42,958 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8031. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)**
2015-09-08 08:19:48,256 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl failed in state STARTED; cause: **org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.ConnectException: Call From ec2-52-88-167-9.us-west-2.compute.amazonaws.com/172.31.29.154 to 0.0.0.0:8031 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.ConnectException: Call From ec2-52-88-167-9.us-west-2.compute.amazonaws.com/172.31.29.154 to 0.0.0.0:8031 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused**
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:197)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:264)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:463)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:509)
Caused by: java.net.ConnectException: Call From ec2-52-88-167-9.us-west-2.compute.amazonaws.com/172.31.29.154 to 0.0.0.0:8031 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor8.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy27.registerNodeManager(Unknown Source)
at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:68)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy28.registerNodeManager(Unknown Source)
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:257)
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:191)
... 6 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 18 more
2015-09-08 08:19:48,257 INFO org.apache.hadoop.service.AbstractService: Service NodeManager failed in state STARTED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.ConnectException: Call From ec2-52-88-167-9.us-west-2.compute.amazonaws.com/172.31.29.154 to 0.0.0.0:8031 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.ConnectException: Call From ec2-52-88-167-9.us-west-2.compute.amazonaws.com/172.31.29.154 to 0.0.0.0:8031 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:197)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:264)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:463)
at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:509)
Caused by: java.net.ConnectException: Call From ec2-52-88-167-9.us-west-2.compute.amazonaws.com/172.31.29.154 to 0.0.0.0:8031 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor8.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
at org.apache.hadoop.ipc.Client.call(Client.java:1472)
at org.apache.hadoop.ipc.Client.call(Client.java:1399)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
at com.sun.proxy.$Proxy27.registerNodeManager(Unknown Source)
at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:68)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy28.registerNodeManager(Unknown Source)
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:257)
at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:191)
... 6 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
at org.apache.hadoop.ipc.Client.call(Client.java:1438)
... 18 more
2015-09-08 08:19:48,263 INFO org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup#0.0.0.0:8042
2015-09-08 08:19:48,264 INFO org.apache.hadoop.ipc.Server: Stopping server on 53949
2015-09-08 08:19:48,266 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 53949
2015-09-08 08:19:48,267 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2015-09-08 08:19:48,267 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl is interrupted. Exiting.
2015-09-08 08:19:48,267 INFO org.apache.hadoop.ipc.Server: Stopping server on 8040
2015-09-08 08:19:48,268 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8040
2015-09-08 08:19:48,268 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2015-09-08 08:19:48,269 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService: Public cache exiting
2015-09-08 08:19:48,269 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NodeManager metrics system...
2015-09-08 08:19:48,270 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system stopped.
2015-09-08 08:19:48,270 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NodeManager metrics system shutdown complete.
2015-09-08 08:19:48,270 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
core-site.xml:
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://ec2-52-26-161-203.us-west-2.compute.amazonaws.com:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/ubuntu/hdfstmp</value>
</property>
</configuration>
mapred-site.xml:
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hdfs://ec2-52-26-161-203.us-west-2.compute.amazonaws.com:8021</value>
</property>
</configuration>
hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>
Master machine:
ubuntu#ec2-52-26-161-203:~$ vim /etc/hosts
172.31.23.167 ec2-52-26-161-203.us-west-2.compute.amazonaws.com
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
ubuntu#ec2-52-26-161-203:~$ vim /etc/hadoop/masters
ec2-52-26-161-203.us-west-2.compute.amazonaws.com
ubuntu#ec2-52-26-161-203:~$ vim /etc/hadoop/slaves
ec2-52-88-167-9.us-west-2.compute.amazonaws.com
Slave Machine :
ubuntu#ec2-52-88-167-9:~ vim /etc/hosts
172.31.29.154 ec2-52-88-167-9.us-west-2.compute.amazonaws.com
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
ubuntu#ec2-52-88-167-9:~ vim /etc/hadoop/slaves
ec2-52-88-167-9.us-west-2.compute.amazonaws.com
ubuntu#ec2-52-26-161-203:~$ sudo netstat -lpten | grep java
tcp 0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 1000 569904 19910/java
tcp 0 0 0.0.0.0:50090 0.0.0.0:* LISTEN 1000 570916 20136/java
tcp 0 0 172.31.23.167:8020 0.0.0.0:* LISTEN 1000 569911 19910/java
tcp6 0 0 :::8088 :::* LISTEN 1000 571699 20278/java
tcp6 0 0 :::8030 :::* LISTEN 1000 571690 20278/java
tcp6 0 0 :::8031 :::* LISTEN 1000 571683 20278/java
tcp6 0 0 :::8032 :::* LISTEN 1000 571695 20278/java
tcp6 0 0 :::8033 :::* LISTEN 1000 571702 20278/java
Telnet command:
ubuntu#ec2-52-26-161-203:~$ telnet localhost 8031
Trying ::1...
Connected to localhost.
Escape character is '^]'.
How is it taking 8031 port for resource manager ? I haven't giving in my hadoop configuration files(coresite.xml,mapred-site.xml,hdfs-site.xml) which is above.
I have made modifications in mapred-site.xml and yarn-site.xml which solved my issue.Since I haven't mentioned the host name property value for resource manager in yarn-site.xml it was trying to connect with the address 0.0.0.0 which was the cause for connection refused exception.
mapred-site.xml
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
yarn-site.xml
<property>
<name>yarn.resourcemanager.hostname</name>
<value>ec2-52-26-161-203.us-west-2.compute.amazonaws.com</value>
</property>
The hadoop documentation http://wiki.apache.org/hadoop/ConnectionRefused
clearly says:
Make sure the destination address in the exception isn't 0.0.0.0 -this
means that you haven't actually configured the client with the real
address for that
Could you please try adding master ip entry in slave machine's host and slave's ip entry to master as well. Also comment out all the entries in the hosts file, if not needed.
opscenter agent can't connet to opscenter.
opscenter's agent.log show error like this. ( replace IP to XX)
INFO [pdp-loader] 2014-11-28 12:03:53,517 Attempting to load stored metric values.
ERROR [StompConnection receiver] 2014-11-28 12:03:54,814 failed connecting to <X.X.X.X>:61620:java.net.UnknownHostException: <X.X.X.X>
INFO [StompConnection receiver] 2014-11-28 12:03:54,814 Reconnecting in 6s.
ERROR [StompConnection receiver] 2014-11-28 12:04:00,814 failed connecting to <X.X.X.X>:61620:java.net.UnknownHostException: <X.X.X.X>
INFO [StompConnection receiver] 2014-11-28 12:04:00,814 Reconnecting in 14s.
ERROR [StompConnection receiver] 2014-11-28 12:04:14,818 failed connecting to <X.X.X.X>:61620:java.net.UnknownHostException: <X.X.X.X>
INFO [StompConnection receiver] 2014-11-28 12:04:14,818 Reconnecting in 30s.
ERROR [StompConnection receiver] 2014-11-28 12:04:44,822 failed connecting to <X.X.X.X>:61620:java.net.UnknownHostException: <X.X.X.X>
INFO [StompConnection receiver] 2014-11-28 12:04:44,822 Reconnecting in 62s.
ERROR [StompConnection receiver] 2014-11-28 12:05:46,826 failed connecting to <X.X.X.X>:61620:java.net.UnknownHostException: <X.X.X.X>
INFO [StompConnection receiver] 2014-11-28 12:05:46,826 Reconnecting in 60s.
ERROR [StompConnection receiver] 2014-11-28 12:06:46,830 failed connecting to <X.X.X.X>:61620:java.net.UnknownHostException: <X.X.X.X>
and opscenterd.log has nothing special.
my config below.
opscenter config.
$cat opscenter-5.0.1/conf/opscenterd.conf
[webserver]
port = 8888
interface = 0.0.0.0
[logging]
[authentication]
enabled = False
[stat_reporter]
[agents]
use_ssl = False
agent config.
$ cat datastax-agent-5.0.1/conf/address.yaml
stomp_interface: <X.X.X.X>
use_ssl: 0
so I chekc port.
$netstat -an | grep 61620
tcp 0 0 0.0.0.0:61620 0.0.0.0:* LISTEN
$ telnet X.X.X.X 61620
Trying X.X.X.X...
Connected to X.X.X.X.
Escape character is '^]'.
it seems ok.
but opscenter agent show me a error .. again and again and.....
INFO [StompConnection receiver] 2014-11-28 12:05:46,826 Reconnecting in 60s.
ERROR [StompConnection receiver] 2014-11-28 12:06:46,830 failed connecting to <X.X.X.X>:61620:java.net.UnknownHostException: <X.X.X.X>
is addition ....
I'm running cassandra 2.1.2( 3 replica) on CentOS release 6.5 (Final)
and kernel: 2.6.32-431.23.3.el6.x86_64
anyone can help me?
So I had a slightly different message in the OpsCenter diagnostic output:
ProcessingError while calling CreateClusterConfController: Unable to connect to cluster
However, I'm running Cassandra 2.1.2 and OpsCenter 5.0.1 as well, and I had trouble connecting to an existing cluster until I saw this in cassandra.yaml
listen_address
# You _must_ change this if you want multiple nodes to be able to communicate!
# Leaving it blank leaves it up to InetAddress.getLocalHost(). This
# will always do the Right Thing _if_ the node is properly configured
# (hostname, name resolution, etc), and the Right Thing is to use the
# address associated with the hostname (it might not be).
It seems this includes OpsCenter. So I commented it out. I also commented out rpc_address, and let it auto-configure itself as well.
Please verify these settings, restart cassandra, and try connecting again through OpsCenter.