Connect to Spark running on VM - apache-spark

Question

I have a Spark enviroment running on Ubuntu 16.2 over VirtualBox. Its configured to run locally and when I start Spark with
./start-all
I can access to it on VM via web-ui using the URL: http://localhost:8080
From the host machine (windows), I can access it too using the VM IP: http://192.168.x.x:8080.
The problem appears when I try to create a context from my host machine. I have a project in eclipse that uses maven, and I try to run the following code:
ConfigLoader.masterEndpoint = "spark://192.168.1.132:7077"
val conf = new SparkConf().setMaster(ConfigLoader.masterEndpoint).setAppName("SimpleApp")
val sc = new SparkContext(conf)
I got this error:
16/12/21 00:52:05 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://192.168.1.132:8080...
16/12/21 00:52:06 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master 192.168.1.132:8080
org.apache.spark.SparkException: Exception thrown in awaitResult
at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
I've tried changing the URL for:
ConfigLoader.masterEndpoint = "spark://192.168.1.132:7077"
Unsuccessfully.
Also, if I try to access directly to the master URL via web (http://localhost:7077 in VM), I don't get anything. I don't know if its normal.
What am I missing?

Answer 1

In your VM go to spark-2.0.2-bin-hadoop2.7/conf directory and create spark-env.sh file using below command.
cp spark-env.sh.template spark-env.sh
Open spark-env.sh file in vi editor and add below line.
SPARK_MASTER_HOST=192.168.1.132
Stop and start Spark using stop-all.sh and start-all.sh. Now in your program you can set the master like below.
val spark = SparkSession.builder()
.appName("SparkSample")
.master("spark://192.168.1.132:7077")
.getOrCreate()