I have been prototyping Spark Streaming 1.6.1 using kafka receiver on a Mesos 0.28 cluster running with Coarse grained mode.
I have 6 mesos slaves each with 64GB RAM and 16 Cores.
My kafka topic has 3 partitions.
My goal is to launch 3 executors in all (each on a different mesos slave) with each executor having one kafka receiver reading from one kafka partition.
When I launch my spark application with spark.cores.max set to 24 and spark.executor.memory set to 8GB, I get two executors - with 16 cores on one slave and with 8 cores on another slave.
I am looking to get 3 executors with 8 cores each on three different slaves. Is that possible with mesos through resource reservation / isolation, constraints etc. ?
Only workaround that works for me now is to scale down each mesos slave node to only have 8 cores max. I don't want to use mesos in fine-grained mode for performance reasons and plus its support is going away soon.
Mesosphere has contributed the following patch to Spark: https://github.com/apache/spark/commit/80cb963ad963e26c3a7f8388bdd4ffd5e99aad1a. This improvement will land in Spark 2.0. Mesosphere has backported this and other improvements to Spark 1.6.1 and made it available in DC/OS (http://dcos.io).
This patch introduces a new "spark.executor.cores" config variable in course gain mode. When the "spark.executor.cores" config variable is set, executors will be sized with the specified number of cores.
If an offer arrives with a multiple of (spark.executor.memory, spark.executor.cores), multiple executors will be launched on that offer. This means there could be multiple, but seperate, Spark executors running on the same Mesos agent node.
There is no way (currently) to spread the executors across N Mesos agents. We briefly discussed adding the ability to spread Spark executors across N Mesos agents but concluded it doesn't buy much in terms of improved availability.
Can you help us understand your motivations for spreading Spark executors across 3 Mesos agents? It's likely we haven't considered all possibly use cases and advantages.
Keith
Related
I want to process a 250gb gzip(filename.json.gzip) file in Azure Hd Insight cluster with Spark. But I could not do it.
I guess because of a bad relationship between cores, ram, vCPU(s)so I would like to kwnow the better cluster to create and Spark configuration to send.
Currently I'm using this instance:
6 nodes of a cluster E8a v4 (8 Cores, 64 GB RAM)
And My Spark Configuration Are:
Driver Memory: 10Gb
Driver Cores: 7
Executor Memory: 10Gb
Executor Cores: 7
Num Executors: 7
So, there is a better choise in the Azure HDInsight Clusters (Link To the all avaiable clusters i can create) and in the Spark submition configuration?
The performance of your Apache Spark jobs depends on multiple factors. These performance factors include: how your data is stored, how the cluster is configured, and the operations that are used when processing the data.
Common challenges you might face include: memory constraints due to improperly sized executors, long-running operations, and tasks that result in cartesian operations.
There are also many optimizations that can help you overcome these challenges, such as caching, and allowing for data skew.
For more details, refer to Optimize Apache Spark jobs in HDInsight.
I would like to know if it is true that on mesos we can have only one executor per node?
Context
I am running a spark-submit (Spark 2.0.1) job on a cluster of 5 nodes (workers) each with 80 CPU and 512 GB memory in coarse-grained mode.
Official documentation of Spark Running Spark on Mesos in Mesos Run Modes section, says that in coarse-grained mode (default) I can set two parameters: spark.executor.memory, spark.executor.cores and that spark.cores.max/spark.executor.cores will give me number of executors.
Question Is this correct or not?
I have been playing with spark-submit setup for a week now and the maximum number of executors I was able to get on my cluster is 5 (1 for driver and 4 for actual work). This is based on the Executors tab in Spark UI.
I have seen this StackOverflow question: Understanding resource allocation for spark jobs on mesos Where it is said:
In coarse-grained mode, Spark launch only one executor per host
In Mastering Apache Spark Schedulers in Mesos section it says
In coarse-grained mode, there is a single Spark executor per Mesos executor with many Spark tasks.
Which I don't understand what it means. Is there always only one Mesos_executor per node, and that implies one Spark_executor per node?
If all of this is not true and I can have more executors.
Question Is there some mesos setting that limits number of executors?
It is not true (anymore). SPARK-5095 Support launching multiple mesos executors in coarse grained mesos mode has been resolved in Spark 2.0 and according to the merged PR:
This PR implements two high-level features. These two features are co-dependent, so they're implemented both here:
Mesos support for spark.executor.cores
Multiple executors per slave
I'm spinning up an EMR 5.4.0 cluster with Spark installed. I have a job for which performance really degrades if it's scheduled on executors which aren't available (eg on a cluster w/ 2 m3.xlarge core nodes there are about 16 executors available).
Is there any way for my app to discover this number? I can discover the hosts by doing this:
sc.range(1,100,1,100).pipe("hostname").distinct().count(), but I'm hoping there's a better way of getting an understanding of the cluster that Spark is running on.
I am trying to set DC/OS Spark-Kafka-Cassandra cluster using 1 master and 3 private AWS m3.xlarge instances (each having 4 processors, 15GB RAM).
I have questions regarding some strange behaviour I have incurred in the spike I did several days ago.
On each of the private nodes I have following fixed resources reserved (I speak about CPU usage, memory is not the issue)
0.5 CPUs for Cassandra on each node
0.3 - 0.5 CPUs for Kafka one each node
0.5 CPUs is the Mesos overhead (I simply see in DC/OS UI that it is occupied 0.5CPUs more than the summation of all the services that are running on a node -> this probably belongs to some sort of Mesos overhead)
rest of the resources I have available for running Spark jobs (around 2.5 CPUs)
Now, I want to run 2 streaming jobs, so that they run on every node of the cluster. This requires me to set in dcos spark run command that number of executors is 3 (although I have 3 nodes in the cluster), as well as that number of CPU cores is 3 (it is impossible to set 1 or 2,because as far as I see minimum CPUs per executor is 1). Of course, for each of the streaming jobs, 1 CPU in the cluster is occupied by the driver program.
First strange situation that I see is that instead of running 3 executors with 1 core each, Mesos launches 2 executors on 2 nodes where one has 2 CPUs, while the other has 1 CPU. There is nothing launched on the 3rd node even though there were enough resources. How to force Mesos to run 3 executors on the cluster?
Also, when I run 1 pipeline with 3 CPUs, I see that those CPUs are blocked, and cannot be reused by other streaming pipeline, even though they are not doing any workload. Why Mesos can not shift available resources between applications? Isn't that the main benefit of using Mesos? Or maybe simply there are not enough resources to be shifted?
EDITED
Also the question is can I assign less than one CPU per Executor?
Kindest regards,
Srdjan
I have a Mesos cluster with 1 Master and 3 slaves (with 2 cores and 4GB RAM each) that has a Spark application already up and running. I wanted to run another application on the same cluster, as the CPU and Memory utilization isn't high. Regardless, when I try to run the new Application, I get the error:
16/02/25 13:40:18 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory
I guess the new process is not getting any CPU as the old one occupies all 6.
I have tried enabling dynamic allocation, making the spark app Fine grained. Assigning numerous combinations of executor cores and number of executors. What I am missing here? Is it possible to run a Mesos Cluster with multiple Spark Frameworks at all?
You can try setting spark.cores.max to limit the number of CPUs used by each Spark driver, which will free up some resources.
Docs: https://spark.apache.org/docs/latest/configuration.html#scheduling