I am wondering if I change the settings in spark-env.sh for the worker nodes, will that be changed for that executor? I know SPARK_WORKER_INSTANCES is applied for each executor, I can see that in web UI, but what about the rest?
I have set this for one executor, i am not able to verify if this settings is indeed being applied for this worker node. I am using spark standalone in cluster mode.
SPARK_EXECUTOR_CORES=4
SPARK_EXECUTOR_MEMORY=6G
Is there any way of verifying the settings for each executor node? If not, is there a way I can apply different settings for different executor?
Thanks and Regards,
Sudip
Related
We are currently running parallel Spark jobs on an EMR cluster using HadoopActivity task from Datapipeline. By default, the newer versions of EMR clusters sets spark dynamic allocation to true which will increase/ reduce the number of executors required based on the load. So do we need to set any other property along with spark-submit e.g. number of cores, executor memory etc. or its best to have EMR cluster handle it dynamically?
This always depends of how you application is working. I can give you an good example of how I work here. For the Data Scientists in general they use the default configuration and it works pretty well due to they use Jupyter here to run their models. The only thing that we setup that can be useful for you is the conf spark.dynamicAllocation.minExecutors this allow to setup at least two or one worker for the job. To not be without any executor. That is what we do with the Data Scientists.
But, EMR has one specific type of configuration for each type of machine you choose. So in general it is optimized for the most common activities. But sometimes you need to change according your request, if you need more memory and less cores for skewed data that is better to change.
Apologies in advance as I am new to spark. I have created a spark cluster in standalone mode with 4 workers, and after successfully being able to configure worker properties, I wanted to know how to configure the master properties.
I am writing an application and connecting it to the cluster using SparkSession.builder (I do not want to submit it using spark-submit.)
I know that that the workers can be configured in the conf/spark-env.sh file and has parameters which can be set such as 'SPARK_WORKER_MEMORY' and 'SPARK_WORKER_CORES'
My question is: How do I configure the properties for the master? Because there is no 'SPARK_MASTER_CORES' or 'SPARK_MASTER_MEMORY' in this file.
I thought about setting this in the spark-defaults.conf file, however it seems that this is only used for spark-submit.
I thought about setting it in the application using SparkConf().set("spark.driver.cores", "XX") however this only specifies the number of cores for this application to use.
Any help would be greatly appreciated.
Thanks.
Three ways of setting the configurations of Spark Master node (Driver) and spark worker nodes. I will show examples of setting the memory of the master node. Other settings can be found here
1- Programatically through SpackConf class.
Example:
new SparkConf().set("spark.driver.memory","8g")
2- Using Spark-Submit: make sure not to set the same configuraiton in your code (Programatically like 1) and while doing spark submit. if you already configured settings programatically, every job configuration mentioned in spark-submit that overlap with (1) will be ignored.
example :
spark-submit --driver-memory 8g
3- through the Spark-defaults.conf:
In case none of the above is set this settings will be the defaults.
example :
spark.driver.memory 8g
Let's assume that I have 4 NM and I have configured spark in yarn-client mode. Then, I set dynamic allocation to true to automatically add or remove a executor based on workload. If I understand correctly, each Spark executor runs as a Yarn container.
So, if I add more NM will the number of executors increase ?
If I remove a NM while a Spark application is running, something will happen to that application?
Can I add/remove executors based on other metrics ? If the answer is yes, there is a function, preferably in python,that does that ?
If I understand correctly, each Spark executor runs as a Yarn container.
Yes. That's how it happens for any application deployed to YARN, Spark including. Spark is not in any way special to YARN.
So, if I add more NM will the number of executors increase ?
No. There's no relationship between the number of YARN NodeManagers and Spark's executors.
From Dynamic Resource Allocation:
Spark provides a mechanism to dynamically adjust the resources your application occupies based on the workload. This means that your application may give resources back to the cluster if they are no longer used and request them again later when there is demand.
As you may have guessed correctly by now, it is irrelevant how many NMs you have in your cluster and it's by the workload when Spark decides whether to request new executors or remove some.
If I remove a NM while a Spark application is running, something will happen to that application?
Yes, but only when Spark uses that NM for executors. After all, NodeManager gives resources (CPU and memory) to a YARN cluster manager that will in turn give them to applications like Spark applications. If you take them back, say by shutting the node down, the resource won't be available anymore and the process of a Spark executor simply dies (as any other process with no resources to run).
Can I add/remove executors based on other metrics ?
Yes, but usually it's Spark job (no pun intended) to do the calculation and requesting new executors.
You can use SparkContext to manage executors using killExecutors, requestExecutors and requestTotalExecutors methods.
killExecutor(executorId: String): Boolean Request that the cluster manager kill the specified executor.
requestExecutors(numAdditionalExecutors: Int): Boolean Request an additional number of executors from the cluster manager.
requestTotalExecutors(numExecutors: Int, localityAwareTasks: Int, hostToLocalTaskCount: Map[String, Int]): Boolean Update the cluster manager on our scheduling needs.
I did modify the configurations on the driver of spark cluster, such as the both files of spark-defaults.conf and spark-env.sh. Do we need do the same things on the workers. It seems to not do those, but I am not sure.
Spark Properties (spark-defaults.conf):
No. Properties are applications specific not a cluster wide so has to be set only in your Spark directory.
Environment variables:
Yes if you need custom settings. Environment variables are machine specific and don't depend on application.
I have pretty low configuration testing machine for my data pipelines developed in Spark. I will use only one AWS t2.large instance, which has only 2 CPUs and 8 GB of RAM.
I need to run 2 spark streaming jobs, as well as leave some memory and CPU power for occasionally testing batch jobs.
So I have master and one worker, which are on the same machine.
I have some general questions:
1) How many executors can run per one worker? I know that default is one, but does it make sense to change this?
2) Can one executor execute multiple applications, or one executor is dedicated only to one application?
3) Is a way to make this work, to set memory that application can use in configuration file, or when I create spark context?
Thank you
How many executors can run per one worker? I know that default is one, but does it make sense to change this?
It makes sense only in case you have enough resources. Say, on a machine with 24 GB and 12 cores it's possible to run 3 executors if you're sure that 8 GB is enough for one executor.
Can one executor execute multiple applications, or one executor is dedicated only to one application?
Nope, every application starts their own executors.
Is a way to make this work, to set memory that application can use in configuration file, or when I create spark context?
I'm not sure I understand the question, but there are 3 ways to provide configuration for applications
file spark-defaults.conf, but don't forget to turn on to read default properties, when you create new SparkConf instance.
providing system properties through -D, when you run the application or --conf if that's spark-submit or spark-shell. Although for memory options there are specific parameters like spark.executor.memory or spark.driver.memory and others to be used.
provides the same options through new SparkConf instance using its set methods.