Spark Worker Nodes starting but not showing in WebUI - apache-spark

Question

I am trying to setup a small 3 node Spark cluster using some Raspberry Pi's and my main desktop but can't seem to get the Pi's to talk to my master node (the desktop). I have the network configured properly as I am also running Cassandra (Open source not DSE) on all three nodes. If I go to the web UI it only shows my main computer. I can put in the web ui address for each of the worker nodes and get their individual web ui page. They don't seem to know about my master node. I have each of the slave nodes in my slaves file. I feel like I am missing just one small thing to get this to work. Any suggestions would be much appreciated. Below are some logs and any other information I could think of that might be helpful , while trying to keep this fairly short and concise.
My spark-env.shon all the nodes is as follows (except Local IP is adjusted appropriately)
export SPARK_WORKER_CORES=6
export SPARK_MASTER_HOST=192.168.0.106
export SPARK_LOCAL_IP=192.168.0.201
Log from a Worker Node:
Spark Command: /usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/bin/java -cp /home/spark/spark/conf/:/home/spark/spark/jars/* -Xmx1g org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://Palehorse:7077
========================================
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/07/05 03:22:40 INFO Worker: Started daemon with process name: 11065#PiCamp1
17/07/05 03:22:40 INFO SignalUtils: Registered signal handler for TERM
17/07/05 03:22:40 INFO SignalUtils: Registered signal handler for HUP
17/07/05 03:22:40 INFO SignalUtils: Registered signal handler for INT
17/07/05 03:22:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/07/05 03:22:42 INFO SecurityManager: Changing view acls to: spark
17/07/05 03:22:42 INFO SecurityManager: Changing modify acls to: spark
17/07/05 03:22:42 INFO SecurityManager: Changing view acls groups to:
17/07/05 03:22:42 INFO SecurityManager: Changing modify acls groups to:
17/07/05 03:22:42 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); groups with view permissions: Set(); users with modify permissions: Set(spark); groups with modify permissions: Set()
17/07/05 03:22:43 INFO Utils: Successfully started service 'sparkWorker' on port 35342.
17/07/05 03:22:44 INFO Worker: Starting Spark worker 192.168.0.201:35342 with 6 cores, 1024.0 MB RAM
17/07/05 03:22:44 INFO Worker: Running Spark version 2.1.1
17/07/05 03:22:44 INFO Worker: Spark home: /home/spark/spark
17/07/05 03:22:45 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
17/07/05 03:22:45 INFO WorkerWebUI: Bound WorkerWebUI to 192.168.0.201, and started at http://192.168.0.201:8081
17/07/05 03:22:45 INFO Worker: Connecting to master Palehorse:7077...
17/07/05 03:22:51 INFO Worker: Retrying connection to master (attempt # 1)
17/07/05 03:22:51 INFO Worker: Connecting to master Palehorse:7077...
17/07/05 03:22:57 INFO Worker: Retrying connection to master (attempt # 2)
17/07/05 03:22:57 INFO Worker: Connecting to master Palehorse:7077...
17/07/05 03:23:03 INFO Worker: Retrying connection to master (attempt # 3)
17/07/05 03:23:03 INFO Worker: Connecting to master Palehorse:7077...
17/07/05 03:23:09 INFO Worker: Retrying connection to master (attempt # 4)
17/07/05 03:23:09 INFO Worker: Connecting to master Palehorse:7077...
17/07/05 03:23:15 INFO Worker: Retrying connection to master (attempt # 5)
17/07/05 03:23:15 INFO Worker: Connecting to master Palehorse:7077...
17/07/05 03:23:21 INFO Worker: Retrying connection to master (attempt # 6)
17/07/05 03:23:21 INFO Worker: Connecting to master Palehorse:7077...
17/07/05 03:23:57 INFO Worker: Retrying connection to master (attempt # 7)
17/07/05 03:23:57 INFO Worker: Connecting to master Palehorse:7077...
17/07/05 03:24:33 INFO Worker: Retrying connection to master (attempt # 8)
17/07/05 03:24:33 INFO Worker: Connecting to master Palehorse:7077...
17/07/05 03:24:45 ERROR RpcOutboxMessage: Ask timeout before connecting successfully
17/07/05 03:24:45 WARN NettyRpcEnv: Ignored failure: java.io.IOException: Connecting to Palehorse/198.105.254.63:7077 timed out (120000 ms)
17/07/05 03:24:45 WARN Worker: Failed to connect to master Palehorse:7077
org.apache.spark.SparkException: Exception thrown in awaitResult
at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:100)
at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:108)
at org.apache.spark.deploy.worker.Worker$$anonfun$org$apache$spark$deploy$worker$Worker$$tryRegisterAllMasters$1$$anon$1.run(Worker.scala:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Connecting to Palehorse/198.105.254.63:7077 timed out (120000 ms)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:229)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:182)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:197)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
... 4 more
17/07/05 03:25:09 INFO Worker: Retrying connection to master (attempt # 9)
17/07/05 03:25:09 INFO Worker: Connecting to master Palehorse:7077...
17/07/05 03:25:45 INFO Worker: Retrying connection to master (attempt # 10)
17/07/05 03:25:45 INFO Worker: Connecting to master Palehorse:7077...
17/07/05 03:26:21 INFO Worker: Retrying connection to master (attempt # 11)
17/07/05 03:26:21 INFO Worker: Connecting to master Palehorse:7077...
17/07/05 03:26:57 INFO Worker: Retrying connection to master (attempt # 12)
17/07/05 03:26:57 INFO Worker: Connecting to master Palehorse:7077...
17/07/05 03:27:09 ERROR RpcOutboxMessage: Ask timeout before connecting successfully
17/07/05 03:27:09 WARN NettyRpcEnv: Ignored failure: java.io.IOException: Connecting to Palehorse/198.105.254.63:7077 timed out (120000 ms)
17/07/05 03:27:09 WARN Worker: Failed to connect to master Palehorse:7077
org.apache.spark.SparkException: Exception thrown in awaitResult
at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:100)
at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:108)
at org.apache.spark.deploy.worker.Worker$$anonfun$org$apache$spark$deploy$worker$Worker$$tryRegisterAllMasters$1$$anon$1.run(Worker.scala:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Connecting to Palehorse/198.105.254.63:7077 timed out (120000 ms)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:229)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:182)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:197)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
... 4 more
17/07/05 03:27:33 INFO Worker: Retrying connection to master (attempt # 13)
17/07/05 03:27:33 INFO Worker: Connecting to master Palehorse:7077...
17/07/05 03:28:09 INFO Worker: Retrying connection to master (attempt # 14)
17/07/05 03:28:09 INFO Worker: Connecting to master Palehorse:7077...
17/07/05 03:28:45 INFO Worker: Retrying connection to master (attempt # 15)
17/07/05 03:28:45 INFO Worker: Connecting to master Palehorse:7077...
17/07/05 03:29:21 INFO Worker: Retrying connection to master (attempt # 16)
17/07/05 03:29:21 INFO Worker: Connecting to master Palehorse:7077...
17/07/05 03:29:33 ERROR RpcOutboxMessage: Ask timeout before connecting successfully
17/07/05 03:29:33 WARN NettyRpcEnv: Ignored failure: java.io.IOException: Connecting to Palehorse/198.105.254.63:7077 timed out (120000 ms)
17/07/05 03:29:33 WARN Worker: Failed to connect to master Palehorse:7077
org.apache.spark.SparkException: Exception thrown in awaitResult
at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:100)
at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:108)
at org.apache.spark.deploy.worker.Worker$$anonfun$org$apache$spark$deploy$worker$Worker$$tryRegisterAllMasters$1$$anon$1.run(Worker.scala:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Connecting to Palehorse/198.105.254.63:7077 timed out (120000 ms)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:229)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:182)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:197)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
... 4 more
17/07/05 03:29:57 ERROR Worker: All masters are unresponsive! Giving up.

Answer 1

I was able to get the slaves to talk to the master finally. Seemed to be a combination of things, one was a problem with my /etc/hosts file having the master name being set to the 127.0.1.1 address and the other problem did seem to be a problem with the start-all.sh solved by running the start-slave.sh spark://<master ip address>:7077

Answer 2

I was also able to solve this problem by commenting out the weird IP address 127.0.0.1 and 127.0.1.1