we have installed a spark-jobserver that is launched on a spark cluster using server_start.sh, however no matter how we go about it we cannot make it launch on multiple workers. We can manage to get it to run with several cores and more memory, but not over several nodes.
The commands we have tried are as follows:
./server_start.sh --master spark://IP:PORT --deploy-mode cluster --total-executor cores 6
./server_start.sh --master spark://IP:PORT --deploy-mode cluster --total-executor cores 4 --executor-cores 2
./server_start.sh --master spark://IP:PORT --deploy-mode cluster --conf spark.driver.cores=4 --conf spark.driver.memory=7g
./server_start.sh --master spark://IP:PORT --deploy-mode cluster --conf spark.driver.cores=6 --conf spark.driver.memory=7g
The first two commands launched and showed one worker using one core and 1GB, while the third shows one worker using 4 cores and 7g. The fourth command shows 6 cores to be used, but state SUBMITTED.
We have verified that it does work to launch and application on multiple workers by launching the spark shell with the following command, which shows up as running driver with 2 workers and a total of 6 cores.
./spark-shell --master spark://IP:PORT --total-executor cores 6
Would appreciate any help.
Spark jobserver is actually spark driver not an application as such. I don't you can distribute the load among workers.
Related
I read this: Running Spark on Kubernetes.
I want to know more details about the interaction between Kubernetes Controller/Scheduler and Spark runtime when launching a Spark job on K8s.
Specially, assuming we launch an Spark app by :
bin/spark-submit \
--master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=5 \
--..............
My question is: the K8s may not be able to allocate 5 executors (or called containers/pods) immediately due to unavailability of cluster resources at the moment the Spark app is launched. Which way does Spark app take? (1) Spark starts running tasks as soon as possible when there is at least one executor is allocated. (2) Spark won't launch any tasks until all of the 5 executors have been allocated.
If you know Hadoop YARN, it would be great if you could also answer the question in the scenario of running Spark app on Hadoop YARN(DynamicAllocation Disabled) and point out the difference.
I am running Spark on YARN in client mode, so I expect that YARN will allocate containers only for the executors. Yet, from what I am seeing, it seems like a container is also allocated for the driver, and I don't get as many executors as I was expecting.
I am running spark submit on the master node. Parameters are as follows:
sudo spark-submit --class ... \
--conf spark.master=yarn \
--conf spark.submit.deployMode=client \
--conf spark.yarn.am.cores=2 \
--conf spark.yarn.am.memory=8G \
--conf spark.executor.instances=5 \
--conf spark.executor.cores=3 \
--conf spark.executor.memory=10G \
--conf spark.dynamicAllocation.enabled=false \
While running this application, Spark UI's Executors page shows 1 driver and 4 executors (5 entries in total). I would expect 5, not 4 executors.
At the same time, YARN UI's Nodes tab shows that on the node that isn't actually used (at least according to Spark UI's Executors page...) there's a container allocated, using 9GB of memory. The rest of the nodes have containers running on them, 11GB of memory each.
Because in my Spark Submit the driver has 2GB less memory than executors, I think that the 9GB container allocated by YARN is for the driver.
Why is this extra container allocated? How can i prevent this?
Spark UI:
YARN UI:
Update after answer by Igor Dvorzhak
I was falsely assuming that the AM will run on the master node, and that it will contain the driver app (so setting spark.yarn.am.* settings will relate to the driver process).
So I've made the following changes:
set the spark.yarn.am.* settings to defaults (512m of memory, 1 core)
set the driver memory through spark.driver.memory to 8g
did not try to set driver cores at all, since it is only valid for cluster mode
Because AM on default settings takes up 512m + 384m of overhead, its container fits into the spare 1GB of free memory on a worker node.
Spark gets the 5 executors it requested, and the driver memory is appropriate to the 8g setting. All works as expected now.
Spark UI:
YARN UI:
Extra container is allocated for YARN application master:
In client mode, the driver runs in the client process, and the application master is only used for requesting resources from YARN.
Even though in client mode driver runs in the client process, YARN application master is still running on YARN and requires container allocation.
There are no way to prevent container allocation for YARN application master.
For reference, similar question asked time ago: Resource Allocation with Spark and Yarn.
You can specify the driver memory and number of executors in spark submit as below.
spark-submit --jars..... --master yarn --deploy-mode cluster --driver-memory 2g --driver-cores 4 --num-executors 5 --executor-memory 10G --executor-cores 3
Hope it helps you.
I am new to spark. When I try to run spark-submit in client mode with 3 executors , I expect 3 java processes (since there are 3 executors ) to show up when I execute ps -ef
$SPARK_HOME/bin/spark-submit --num-executors 3 --class AverageCalculation --master local[1] /home/customer/SimpleETL/target/SimpleETL-0.1.jar hdfs://node1:9000/home/customer/SimpleETL/standard_input.csv
But, I dont see 3 java processes. My undrstanding is that each executor process is a java process. Please advise. Thanks.
Because you use local mode (--master local[1]) executor settings are not applicable. In this case, spark starts only a single JVM to emulate all components, and allocates number of threads specified in local definition (1) as executor threads.
In other modes, exectuors are separate JVM instances.
Each executors are a java process. Each executors comprises a jvm.
jps
Number of java process is same as the number of executors. If the executors are distributed across the worker nodes. Need to check the process the corresponding worker nodes. We can get the information about executors and where it has been launched from spark history server web UI.
In Spark, there are master nodes and worker nodes. Executors run on worker nodes in their own java processes.
In your spark-submit you can add --deploy-mode cluster and see that executors are running on worker nodes in their own JVM instances.
You can check this answer for detailed workflow of Apache Spark.
/home/spark/spark-2.2.1-bin-hadoop2.7/bin/spark-submit --class org.apache.spark.examples.SparkPi \
--num-executors 1000 \
--master yarn --deploy-mode cluster --driver-memory 4g --executor-memory 2g --executor-cores 1 \
--queue default /home/spark/spark-2.2.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.2.1.jar
~
I executed the command above. and checked ps -ef | grep java. But I dont see a lot of java processes . Any easy way to identify the executors ?
I have setup a Spark standalone cluster, where I can submit jobs with spark-submit:
spark-submit \
--class blah.blah.MyClass \
--master spark://myaddress:6066 \
--executor-memory 8G \
--deploy-mode cluster \
--total-executor-cores 12 \
/path/to/jar/myjar.jar
Problem is when I send multiple jobs at the same time, say over 20 in one go, the first few finished successfully. All the others are now stuck waiting for resources. I noticed all the available memory has gone to the drivers, so in the drivers section they are all running but in the running application section they all are in WAITING state.
How can I tell spark stand alone to first allocate memory to the WAITING executors instead of the SUBMITTED drivers?
thank you
Below is an extract of my spark-defaults.conf
spark.master spark://address:7077
spark.eventLog.enabled true
spark.eventLog.dir /path/tmp/sparkEventLog
spark.driver.memory 5g
spark.local.dir /path/tmp
spark.ui.port xxx
I have set up a hadoop cluster with 3 machines one master and 2 slave
In the master i have installed spark
SPARK_HADOOP_VERSION=2.4.0 SPARK_YARN=true sbt/sbt clean assembly
Added HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop spark-env.sh
Then i ran SPARK_JAR=./assembly/target/scala-2.10/spark-assembly-1.0.0-SNAPSHOT-hadoop2.4.0.jar HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop ./bin/spark-submit --master yarn --deploy-mode cluster --class org.apache.spark.examples.SparkPi --num-executors 3 --driver-memory 4g --executor-memory 2g --executor-cores 1 examples/target/scala-2.10/spark-examples-1.0.0-SNAPSHOT-hadoop2.4.0.jar
I checked localhost:8088 and i saw application SparkPi running..
Is it just this or i should install spark in the 2 slave machines..
How can i get all the machine started?
Is there any help doc out there.. I feel like i am missing something..
In spark standalone more we start the master and worker
./bin/spark-class org.apache.spark.deploy.worker.Worker spark://IP:PORT
i also wanted to know how to get more than one worked running in this case as well
and i know we can can configure slaves in conf/slave but can anyone share an example
Please help i am stuck
Assuming you're using Spark 1.1.0, as it says in the documentation (http://spark.apache.org/docs/1.1.0/submitting-applications.html#master-urls), for the master parameter you can use values yarn-cluster or yarn-client. You do not need to use deploy-mode parameter in that case.
You do not have to install Spark on all the YARN nodes. That is what YARN is for: to distribute your application (in this case Spark) over a Hadoop cluster.