spark cluster has 2 worker nodes.
Node 1: 64 GB, 8 cores.
Node 2: 64 GB, 8 cores.
Now if i submit a spark job using spark-submit in cluster mode with
2 executors and each executor memory as 32 GB, 4 cores/executor.
Now my question is, as the above configuration can be accommodated in a single node itself, will spark run it using 2 worker nodes or just in one node?
Also, if a configuration doesn't have a multiple of cores as the executors then how many cores allocated for each executor?
Example: if num of cores in a node available after excluding one core for yarn deamon are 7. since 2 nodes, 2*7=14 (total cores available)and as HDFS give good throughput if num of cores per executor were 5..
Now 14/5 to find the num of executors. should i consider 14/5 as 2 or 3 exeutors? then how these cores are equally distributed?
It is more of a resource manager question then a Spark question, but in your case the 2 executors cant run in a single machine cause the OS has an overhead that uses at least 1 core and 1GB RAM , even if you will set the ram to 30 GB and 3 cores/executor. they will run on different nodes because Spark tries to get the best data locality it can so obviously it wont use the same node for 2 executors.
Related
Say I have a total of 4 cores,
What happens if I define num of executors as 8..
Can we share a single core among 2 executors ?
Can the num of cores for a executor be a fraction ?
What is the impact on performance with this kind of config.
This is what I observed in spark standalone mode:
The total cores of my system are 4
if I execute spark-shell command with spark.executor.cores=2
Then 2 executors will be created with 2 core each.
But if I configure the no of executors more than available cores,
Then only one executor will be created, with the max core of the system.
The number of the core will never be of fraction value.
If you assign fraction value in the configuration, you will end up with exception:
Feel free to edit/correct the post if anything is wrong.
how to configure the number of cores to SPARK_WORKER_CORES and SPARK_EXECUTOR_CORES, when using stand alone cluster manager.
As per the spark documentation here
SPARK_WORKER_CORES : Total number of cores to allow Spark applications to use on the machine (default: all available cores).
for example: if you have a cluster of 5 nodes(1 master and 4 slavenode) and each node configuration is 8 core and 32GB RAM... so by using the SPARK_WORKER_CORES we can configure how many cores spark can use per worker(i.e node). The default value is to use all cores. (hint: we need to also keep aside 2 cores for OS and 1 core for NM and 1 for Spark Daemons)
SPARK_EXECUTOR_CORES : This property is to control cores in the executor level. spark can launch many executors per each worker(i.e, node) based on the resource availability. this condition should always satisfy(SPARK_EXECUTOR_CORES < SPARK_WORKER_CORES)
if you configure SPARK_WORKER_CORES = 5 cores and SPARK_EXECUTOR_CORES=1 then spark can run 5 executors in each machine.
note: SPARK_WORKER_CORES property only makes sense when you are running spark on Spark Standalone Mode.
I have installed Spark on master and 2 workers. The original core number per worker is 8. When I start the master, the workers are work properly without any problem, but the problem is in Spark GUI each worker has only 2 cores assigned.
Kindly, how can I increase the number of the cores in which each worker works with 8 cores?
The setting which controls cores per executor is spark.executor.cores. See doc. It can be set either via spark-submit cmd argument or in spark-defaults.conf. The file is usually located in /etc/spark/conf (ymmv). YOu can search for the conf file with find / -type f -name spark-defaults.conf
spark.executor.cores 8
However the setting does not guarantee that each executor will always get all the available cores. This depends on your workload.
If you schedule tasks on a dataframe or rdd, spark will run a parallel task for each partition of the dataframe. A task will be scheduled to an executor (separate jvm) and the executor can run multiple tasks in parallel in jvm threads on each core.
Also an exeucutor will not necessarily run on a separate worker. If there is enough memory, 2 executors can share a worker node.
In order to use all the cores the setup in your case could look as follows:
given you have 10 gig of memory on each node
spark.default.parallelism 14
spark.executor.instances 2
spark.executor.cores 7
spark.executor.memory 9g
Setting memory to 9g will make sure, each executor is assigned to a separate node. Each executor will have 7 cores available. And each dataframe operation will be scheduled to 14 concurrent tasks, which will be distributed x 7 to each executor. You can also repartition a dataframe, instead of setting default.parallelism. One core and 1gig of memory is left for the operating system.
Let's say I have 5 worker nodes in a cluster and each node has 48 cores and 256 GB RAM.
Then what are the maximum number of executors possible in the clusters?
will cluster have 5*48 = 240 executors or only 5 executors?
Or there are some other factors that will decide the number of executors in a cluster, then what are they?
Thanks.
The number of executors is related to the amount of parallelism your application need. You can create 5*48 executor with 1 core each, but there's others processes that should be considered, like memory overhead, cluster management process, scheduler, so you may need to reserve 2-5 cores/node to management processes.
I don't know what architecture your cluster use, but this article is a good start if you are using hadoop , https://spoddutur.github.io/spark-notes/distribution_of_executors_cores_and_memory_for_spark_application.html
I am running Spark over Yarn on a 4 Node Cluster. The configuration of each machine in the node is 128GB Memory, 24 Core CPU per node. I run Spark on using this command
spark-shell --master yarn --num-executors 19 --executor-memory 18g --executor-cores 4 --driver-memory 4g
But Spark only launches 16 executors maximum. I have maximum-vcore allocation in yarn set to 80 (out of the 94 cores i have). So i was under the impression that this will launch 19 executors but it can only go upto 16 executors. Also I don't think even these executors are using the allocated VCores completely.
These are my questions
Why isn't spark creating 19 executors. Is there a computation behind
the scenes that's limiting it?
What is the optimal configuration to run spark-shell given my cluster configuration, if I wanted to get the best possible spark performance
driver-core is set to 1 by default. Will increasing it improve performance.
Here is my Yarn Config
yarn.nodemanager.resource.memory-mb: 106496
yarn..minimum-allocation-mb: 3584
yarn..maximum-allocation-mb: 106496
yarn..minimum-allocation-vcores: 1
yarn..maximum-allocation-vcores: 20
yarn.nodemanager.resource.cpu-vcores: 20
Ok so going by your configurations we have:
(I am also a newbie at Spark but below is what I speculate in this scenario)
24 cores and 128GB ram per node and we have 4 nodes in the cluster.
We allocate 1 core and 1 GB memory for overhead and considering you're running your cluster in YARN-Client mode.
We have 127GB Ram and 23 Cores left with us in 4 nodes.
As mentioned in Cloudera blog YARN runs at optimal performance when 5 cores are allocated per executor at max.
So, 23X4 = 92 Cores.
If we allocated 5 cores per executor then 18 executor have 5 cores and 1 executor has 2 cores or likewise.
So lets assume we have 18 executor in our application and 5 cores per executor.
Spark distributes these 18 executors across 4 nodes. suppose its distributed as:
1st node : 4 executors
2nd node : 4 executors
3rd node : 5 executors
4th node : 5 executors
Now, as 'yarn.nodemanager.resource.memory-mb: 106496' is set as 104GB in your configurations, each node can have max 104 GB memory allocated (I would suggest increasing this parameter).
For nodes with 4 executors: 104/4 - 26GB per executor
For nodes with 5 executors: 104/5 ~ 21GB per executor.
Now leaving out 7% memory for overhead we get 24GB and 20GB.
So i would suggest using following configurations:-
--num-executors : 18
--executor-memory : 20G
--executor-cores : 5
Also, This is considering that you're running your cluster in client mode but if you run your cluster in Yarn-cluster mode 1 node will be allocated fir driver program and the calculations will need to be done differently.
I still cannot comment, so it will be as an answer.
See this question. Could you please decrease executor memory and try run this again?