Whether I use dynamic allocation or explicitly specify executors (16) and executor cores (8), I have been losing executors even though the tasks outstanding are well beyond the current number of executors.
For example, I have a job (Spark SQL) running with over 27,000 tasks and 14,000 of them were complete, but executors "decayed" from 128 down to as few as 16 with thousands of tasks still outstanding. The log doesn't note any errors/exceptions precipitating these lost executors.
It is a Cloudera CDH 5.10 cluster running on AWS EC2 instances with 136 CPU cores and Spark 2.1.0 (from Cloudera).
17/05/23 18:54:17 INFO yarn.YarnAllocator: Driver requested a total number of 91 executor(s).
17/05/23 18:54:17 INFO yarn.YarnAllocator: Canceling requests for 1 executor container(s) to have a new desired total 91 executors.
It's a slow decay where every minute or so more executors are removed.
Some potentially relevant configuration options:
spark.dynamicAllocation.maxExecutors = 136
spark.dynamicAllocation.minExecutors = 1
spark.dynamicAllocation.initialExecutors = 1
yarn.nodemanager.resource.cpu-vcores = 8
yarn.scheduler.minimum-allocation-vcores = 1
yarn.scheduler.increment-allocation-vcores = 1
yarn.scheduler.maximum-allocation-vcores = 8
Why are the executors decaying away and how can I prevent it?
Related
I am running 20 spark applications on an emr cluster of 2 workers and 1 master node with c5.24xlarge instances, thus I have 192 cores in total and 1024 gb ram in total.
Each application is processing around 1.5gb data!
I am having dynamic allocation as enabled and other spark configuration as following!
spark.executor.memory = 9000M,
spark.executor.memoryoverhead = 1000M,
spark.executor.cores = 5,
spark.sql.shuffle.partitions = 40,
spark.dynamicallocation.initialExecutor = 2,
spark.dynamicallocation.maxExecutor = 10,
spark.dynamicallocation.minExecutor = 10,
spark.dynamicallocation.executorIdleTimeOut = 60s,
spark.dynamicallocation.SchedulerBacklogTimeout = 120s,
spark.driver.memory = 9000M,
spark.driver.memoryoverhead = 1000M,
spark.driver.cores = 9
And spark.default.parallelism =384 (This is decided by emr, not sure how it is decided)
These, I have set at the cluster level which means for all 20 applications this would be the spark confs properties.
having these configurations settings, I can see that only few applications are getting completed in around 20 mins, and around 10 applications keep running for more than 2 hours with only one task running and for some around 10.
Questions:
Why other applications are not getting completed like other
completed one?
I am giving partitions count as 40, then why around 350 tasks
are being added (is it because of parallelism) for each application.
the data for each task is showing like 1.4 gb for task 0 , then
1.3gb for task 1 and so on, is that how data is shown for task pane or data is not being divided properly? (Although I can see in event matrics that partitions seem to be of same size, execution timewise.)
Data to executor in executor summary tab is showing more
than for 1.4 gb, which is sum of all input for each task, but its
processing 1.5gb only (from spark sql tab), so it means that's how
data is shown in executor pane?
Thanks!
Can Spark change number of executors during runtime?
Example, In an Action(Job), Stage 1 runs with 4 executor * 5 partitions per executor = 20 partitions in parallel.
If I repartition with .repartition(100), Which is Stage 2 now (because of repartition shuffle), Can in any case Spark increases from 4 executors to 5 executors (or more)?
If I cache some data in Stage 1 which was running with 4 executors * 5 partition per executor = 20 partitions, then the cached data must be in the RAM of 4 machines. If I repartition with .repartition(2), in this case, definitely there will be <= 2 executor machines involved. Will spark shuffle my cached data to the active tasks?
My Spark job failed with the YARN error Container killed by YARN for exceeding memory limits 10.0 GB of 10 GB physical memory used.
Intuitively, I decreased the number of cores from 5 to 1 and the job ran successfully.
I did not increase the executor-memory because 10g was the max for my YARN cluster.
I just wanted to confirm if my intuition. Does reducing the number of executor-cores consume less executor-memory? If so, why?
spark.executor.cores = 5, spark.executor.memory=10G
This means an executor can run 5 tasks in parallel. That means 10 GB needs to be shared by 5 tasks.So effectively on an average - each task will have 2 GB available. If all the tasks consumes more than 2 GB, than overall JVM will end up consuming more than 10 GB and so YARN will kill the container.
spark.executor.cores = 1, spark.executor.memory=10G
This means an executor can run only 1 task. That means 10 GB is available to 1 task completely. So if the task uses more than 2 GB but less than 10 GB, it will work fine. That was the case in your Job and so it worked.
Yes, each executor uses an extra 7% of memoryOverhead.
This calculation will be created thinking that you have two nodes, so we have three executors in one node and two executors in the other node.
Memory per executor in the first node = 10GB/3 = 3,333GB
Counting off heap overhead = 7% of 3,333GB = 0,233GB.
So, your executor-memory should be 3,333GB - 0,233GB = 3,1GB per node
You can read another explanation here:
https://spoddutur.github.io/spark-notes/distribution_of_executors_cores_and_memory_for_spark_application.html
We have a cluster of 4 nodes with the characteristics above :
Spark jobs make a lot of times in processing, how could we optimize this time knowing that our jobs run from RStudio and we still have a lot of memory not utilized.
To add more context to the answer above, I would like to give explanation on how to set those parameters --num-executors, --executor-memory, --executor-cores appropriately.
The following answer covers the 3 main aspects mentioned in title - number of executors, executor memory and number of cores.
There may be other parameters like driver memory and others which I did not address as of this answer.
Case 1 Hardware - 6 Nodes, and Each node 16 cores, 64 GB RAM
Each executor is a JVM instance. So we can have multiple executors in a single Node
First 1 core and 1 GB is needed for OS and Hadoop Daemons, so available are 15 cores, 63 GB RAM for each node
Start with one by one how to choose these parameters.
Number of cores:
Number of cores = Concurrent tasks as executor can run
So we might think, more concurrent tasks for each executor will give better performance.
But research shows that any application with more than 5 concurrent tasks, would lead to bad show. So stick this to 5.
This number came from the ability of executor and not from how many cores a system has. So the number 5 stays same
even if you have double(32) cores in the CPU.
Number of executors:
Coming back to next step, with 5 as cores per executor, and 15 as total available cores in one Node(CPU) - we come to
3 executors per node.
So with 6 nodes, and 3 executors per node - we get 18 executors. Out of 18 we need 1 executor (java process) for AM in YARN we get 17 executors
This 17 is the number we give to spark using --num-executors while running from spark-submit shell command
Memory for each executor:
From above step, we have 3 executors per node. And available RAM is 63 GB
So memory for each executor is 63/3 = 21GB.
However small overhead memory is also needed to determine the full memory request to YARN for each executor.
Formula for that over head is max(384, .07 * spark.executor.memory)
Calculating that overhead - .07 * 21 (Here 21 is calculated as above 63/3)
= 1.47
Since 1.47 GB > 384 MB, the over head is 1.47.
Take the above from each 21 above => 21 - 1.47 ~ 19 GB
So executor memory - 19 GB
Final numbers - Executors - 17 per node, Cores 5 per executor, Executor Memory - 19 GB
This way, assigning the resources properly to the spark jobs in the cluster would speed up the jobs; efficiently using available resources.
I recommend you to have a look to these parameters :
--num-executors : controls how many executors will be allocated
--executor-memory : RAM for each executor
--executor-cores : cores for each executor
Where do you start to tune the above mentioned params. Do we start with executor memory and get number of executors, or we start with cores and get the executor number. I followed the link. However got a high level idea, but still not sure how or where to start and arrive to a final conclusion.
The following answer covers the 3 main aspects mentioned in title - number of executors, executor memory and number of cores. There may be other parameters like driver memory and others which I did not address as of this answer, but would like to add in near future.
Case 1 Hardware - 6 Nodes, and Each node 16 cores, 64 GB RAM
Each executor is a JVM instance. So we can have multiple executors in a single Node
First 1 core and 1 GB is needed for OS and Hadoop Daemons, so available are 15 cores, 63 GB RAM for each node
Start with how to choose number of cores:
Number of cores = Concurrent tasks as executor can run
So we might think, more concurrent tasks for each executor will give better performance. But research shows that
any application with more than 5 concurrent tasks, would lead to bad show. So stick this to 5.
This number came from the ability of executor and not from how many cores a system has. So the number 5 stays same
even if you have double(32) cores in the CPU.
Number of executors:
Coming back to next step, with 5 as cores per executor, and 15 as total available cores in one Node(CPU) - we come to
3 executors per node.
So with 6 nodes, and 3 executors per node - we get 18 executors. Out of 18 we need 1 executor (java process) for AM in YARN we get 17 executors
This 17 is the number we give to spark using --num-executors while running from spark-submit shell command
Memory for each executor:
From above step, we have 3 executors per node. And available RAM is 63 GB
So memory for each executor is 63/3 = 21GB.
However small overhead memory is also needed to determine the full memory request to YARN for each executor.
Formula for that over head is max(384, .07 * spark.executor.memory)
Calculating that overhead - .07 * 21 (Here 21 is calculated as above 63/3)
= 1.47
Since 1.47 GB > 384 MB, the over head is 1.47.
Take the above from each 21 above => 21 - 1.47 ~ 19 GB
So executor memory - 19 GB
Final numbers - Executors - 17, Cores 5, Executor Memory - 19 GB
Case 2 Hardware : Same 6 Node, 32 Cores, 64 GB
5 is same for good concurrency
Number of executors for each node = 32/5 ~ 6
So total executors = 6 * 6 Nodes = 36. Then final number is 36 - 1 for AM = 35
Executor memory is : 6 executors for each node. 63/6 ~ 10 . Over head is .07 * 10 = 700 MB. So rounding to 1GB as over head, we get 10-1 = 9 GB
Final numbers - Executors - 35, Cores 5, Executor Memory - 9 GB
Case 3
The above scenarios start with accepting number of cores as fixed and moving to # of executors and memory.
Now for first case, if we think we dont need 19 GB, and just 10 GB is sufficient, then following are the numbers:
cores 5
# of executors for each node = 3
At this stage, this would lead to 21, and then 19 as per our first calculation. But since we thought 10 is ok (assume little overhead), then we cant switch # of executors
per node to 6 (like 63/10). Becase with 6 executors per node and 5 cores it comes down to 30 cores per node, when we only have 16 cores. So we also need to change number of
cores for each executor.
So calculating again,
The magic number 5 comes to 3 (any number less than or equal to 5). So with 3 cores, and 15 available cores - we get 5 executors per node. So (5*6 -1) = 29 executors
So memory is 63/5 ~ 12. Over head is 12*.07=.84
So executor memory is 12 - 1 GB = 11 GB
Final Numbers are 29 executors, 3 cores, executor memory is 11 GB
Dynamic Allocation:
Note : Upper bound for the number of executors if dynamic allocation is enabled. So this says that spark application can eat away all the resources if needed. So in
a cluster where you have other applications are running and they also need cores to run the tasks, please make sure you do it at cluster level. I mean you can allocate
specific number of cores for YARN based on user access. So you can create spark_user may be and then give cores (min/max) for that user. These limits are for sharing between spark and other applications which run on YARN.
spark.dynamicAllocation.enabled - When this is set to true - We need not mention executors. The reason is below:
The static params number we give at spark-submit is for the entire job duration. However if dynamic allocation comes into picture, there would be different stages like
What to start with :
Initial number of executors (spark.dynamicAllocation.initialExecutors) to start with
How many :
Then based on load (tasks pending) how many to request. This would eventually be the numbers what we give at spark-submit in static way. So once the initial executor numbers are set, we go to min (spark.dynamicAllocation.minExecutors) and max (spark.dynamicAllocation.maxExecutors) numbers.
When to ask or give:
When do we request new executors (spark.dynamicAllocation.schedulerBacklogTimeout) - There have been pending tasks for this much duration. so request. number of executors requested in each round increases exponentially from the previous round. For instance, an application will add 1 executor in the first round, and then 2, 4, 8 and so on executors in the subsequent rounds. At a specific point, the above max comes into picture
when do we give away an executor (spark.dynamicAllocation.executorIdleTimeout) -
Please correct me if I missed anything. The above is my understanding based on the blog i shared in question and some online resources. Thank you.
References:
http://site.clairvoyantsoft.com/understanding-resource-allocation-configurations-spark-application/
http://spark.apache.org/docs/latest/configuration.html#dynamic-allocation
http://spark.apache.org/docs/latest/job-scheduling.html#resource-allocation-policy
Also, it depends on your use case, an important config parameter is:
spark.memory.fraction(Fraction of (heap space - 300MB) used for execution and storage) from http://spark.apache.org/docs/latest/configuration.html#memory-management.
If you dont use cache/persist, set it to 0.1 so you have all the memory for your program.
If you use cache/persist, you can check the memory taken by:
sc.getExecutorMemoryStatus.map(a => (a._2._1 - a._2._2)/(1024.0*1024*1024)).sum
Do you read data from HDFS or from HTTP?
Again, a tuning depend on your use case.