Currently, I am running Spark job through EMR and working on Spark tuning. I read about number of executor per instance and memory calculation etc, and I got confused based on the current setup.
So currently it uses spark.dynamicAllocation.enabled as true by default from EMR and spark.executor.cores as 4 (not set by me, I assume it is by default). Also use one r6.xlarge (32 GiB of Memory, 4 vCPUs) for master and two for cores.
In this case, based on the formula: Number of executors per instance = (total number of virtual cores per instance - 1)/ spark.executors.cores , (4 - 1) / 4 = 0. Would it be correct?
When I check spark UI, it added many executors. What information did I miss from this..?
Related
In our Application ,We have submitted Spark job with the following configuration values:
'--num-executors' (or) 'spark.executor.instances' - Not set
'spark.executor.cores' - Not set
'spark.dynamicAllocation.enabled' -'True'
'spark.executor.memory' - 1g
(No.of worker nodes available - 3 having 4 vCores each)
In 'Environment' page of Spark Web UI, following values are observed :
'spark.executor.instances' - '3'
'spark.executor.cores' - '4'
Can we assume that the above values shown for 'spark.executor.instances' (3) and
'spark.executor.cores' (4) are the initial values only ?
The reason for this assumption is ,
From the 'Executors' page it can be observed that total '14' executors are used.
From the 'Event Timeline' , it can be observed that at one moment, maximum '8' executors' are running .Since total number of cores available are '12' (3 x 4) only , it looks like the number of cores used per executor also will not be constant during runtime. i.e. Initially it starts with '4' but will reduce when the number of executors increase!
You post cover 2 questions:
Are the initial value of spark.executor.instances and spark.executor.cores 3 and 4 respectively? It depends on which mode are you using. Based on the configurations that you provide, which you set the spark.dynamicAllocation.enabled be True, and you mentioned that you have 3 nodes with 4 cores each, it will scales your number of executors based on workload. Also, If you're running your spark application on YARN, the default value of spark.executor.cores should be 1. As you didn't mention about your mode and number of spark application that you run at the same time, I assume you're only running a single spark job and you're not running in YARN mode. You can check the spark config based on the option you input: https://spark.apache.org/docs/latest/configuration.html
Will the number of cores and executors differ from what you config your in spark-submit if your number of executors increase? No, once you submitted your spark application and create the sparkContext, the number of executors and cores will not change, unless you create the new one.
I'm running a spark job on a Google Dataproc cluster (3 nodes n1-highmem-4 so 4 cores and 26GB each, same type for the master).
I have a few questions about informations displayed on the Hadoop and the spark UI:
When I check the Hadoop UI I get this:
My question here is : my total RAM is supposed to be 84 (3x26) so why only 60Gb displayed here ? Is 24GB used for something else ?
2)
This is the screen showing currently launched executors.
My questions are:
Why only 10 cores are used ? Shouldn't we be able to launch a 6th executor using the 2 remaining cores since we have 12, and 2 seem be used per executor ?
Why 2 cores per executor ? Does it change anything if we run 12 executor with 1 core each instead ?
What is the "Input" column ? The total volume each executor received to analyze ?
3)
This is a screenshot of the "Storage" panel. I see the dataframe I'm working on.
I don't understand the "size in memory" column. Is it the total RAM used to cache the dataframe ? It seems very low compared to the size of row files I load into the Dataframe ( 500GB+ ). Is it a wrong interpretation ?
Thanks to anyone who will read this !
If you can take a look at this answer, it mostly answers your question 1 and 2.
To sum up, the total memory is less because some memory are reserved to run OS and system daemons or Hadoop daemons itself, e.g.Namenode, NodeManager.
Similar to cores, in your case it would be 3 nodes and each node runs 2 executors and each executor uses up 2 cores, except for the application master. For the node that application master lives in, there will be only one executor and the cores left are given to master. That's why you see only 5 executor and 10 cores.
For your 3rd question, that number should be the memory used up by the partitions in that RDD, which is approximately equal to memory allocated to each executor in your case it's ~13G.
Note that Spark doesn't load your 500G data at once instead it loads in data in partitions, the number of concurrently loaded partitions depend on the number of cores you have available.
I have read that having 5 cores per Executor in Spark achieves the optimal read/write throughput - so setting spark.executor.cores = 5 is usually desired. And also that you should subtract one core per node to allow for the underlying daemon processes to run.
So, determining the number of executors per node follows this formula:
executors per node = (cores per node - 1) / 5 cores per executor
However, what is the best approach in a scenario where you have 8 cores in each node machine?
1.4 executors per node = (8 - 1) / 5
First question - will Spark/yarn have an executor spanning multiple nodes?
If not - then I need to round. Which way should I go? It seems my options are:
1.) round down to 1 - meaning I'd only have 1 executor per node. I could increase the cores per executor, though don't know if I would get any benefit to that.
2.) round up to 2 - that means I'd have to decrease the cores per executor to 3 (8 cores available, - 1 for the daemons, and can't have 1/2 a core), which could decrease their efficiency.
Here spark.executor.cores = 5 is not a hard-lined value. Thumb rule is # of cores equal to or less than 5.
We need 1 core for OS & other Hadoop daemons. We are left with 7 cores per node.
Remember we need 1 executor for YARN out of all the executors.
When spark.executor.cores = 4 we cannot leave 1 executor for YARN, so I suggest I not take up this value.
When spark.executor.cores = 3 or spark.executor.cores = 2 after leaving one node for YARN we will always be left out with 1 executor per node.
Now which one is efficient for your code. Well that cannot be interpreted , it depends on multiple other factors like the amount of data used, # of joins used etc.
This is based on my understanding. It provides a start to explore multiple other options.
NOTE: If you are using some outer Java libraries & Datasets in your code, you might need to have 1 core per executor for preserving the type safety.
Hope it helps ...
Below are configurations:
Hadoop-2x (1 master, 2 slaves)
yarn.nodemanager.resource.memory = 7096 m
yarn.scheduler.maximum-allocation= 2560 m
Spark - 1.5.1
spark/conf details in all three nodes :
spark.driver.memory 4g
spark.executor.memory 2g
spark.executor.instances 2
spark-sql>CREATE TABLE demo
USING org.apache.spark.sql.json
OPTIONS path
This path has 32 GB compressed data. It is taking 25 minutes to create table demo. Is there anyway to optimize and bring it down in few minutes? Am I missing something out here?
Most usually each executor should represent each core of your CPU. Also note that master is the most irrelevant of your all your machines, because it only assigns tasks to slaves, which do the actual data processing. Your setup is then correct if your slaves are single-core machines but in most cases you would do something like:
spark.driver.memory // this may be the whole memory of your master
spark.executor.instances // sum of all CPU cores that your slaves have
spark.executor.memory // (sum of all slaves memory) / (executor.instances)
That's the easiest formula and will work in vast majority of Spark jobs.
How to decide the number of workers on spark standalone cluster mode?
The duration will decreased when I added workers in standalone cluster mode.
For example, for my input data 3.5 G, it would take 3.8 min for WordCount. However, it would take 2.6 min after I added one worker with memory 4 G.
Is it fine to add workers for tuning spark? I am thinking about the risk on that.
My environment setting were as below,
Memory 128 G, 16 CPU for 9 VM
Centos
Hadoop 2.5.0-cdh5.2.0
Spark 1.1.0
Input data information
3.5 G data file from HDFS
You can tune both the executors (number of JVMs and their memory) as well as the number of tasks. if what you're doing can benefit from parallelism you can spin more executors by configuration and increase the number of tasks (by calling partitions/coalesce etc in your code).
When you set the parallelism take into account if you're doing mostly IO or computations etc. generally speaking Spark recommendation is for 2-3 tasks per CPU core