I have a Spark 1.1.0 cluster with three machines of differing power. When I run the start-all.sh script and check the UI I have all slaves and the master listed. Each worker is listed (they have differing number of cores) with the number of cores listed correctly but the notice that zero are used.
cores
4 (0 Used)
2 (0 Used)
8 (8 Used)
Ssh is set up and working, hadoop seems fine too. The 8 core machine is the master so any submitted job runs only there. I see it being executed in the web UI but the other workers are never given work.
What might be happening here is that the Total_Input_File_Size could be less than MAX_SPLIT_SIZE. So, there will be only one mapper running which will be executing only on the master.
The number of mappers generated are Total_Input_File_Size/MAX_SPLIT_SIZE. So, if you have given a small file, try to give a large input file or lower the value of max_split_size.
Let me know if the problem is anything else.
Have you set --deploy-mode cluster in your spark-submit command?
if you empty this option, the application will not travel to other workers.
Related
I installed the following spark benchmark:
https://github.com/BBVA/spark-benchmarks
I run Spark on top of YARN on 8 workers but I only get 2 running executors during the job (TestDFSIO).
I also set executor-cores to be 9 but only 2 are running.
Why would that happen?
I think the problem is coming from YARN because I get a similar (almost) issue with TestDFSIO on Hadoop. In fact, at the beginning of the job, only two nodes run, but then all the nodes execute the application in parallel!
Note that I am not using HDFS for storage!
I solved this issue. What I've done is that I set the number of cores per executor to 5 (--executor-cores) and the total number of executors to 23 (--num-executors) which was at first 2 by default.
Why does Spark UI show only 6 cores available per worker (not the number of cores used) while I have 16 on each of my 3 machines (8 sockets * 2 cores/socket) or even 32 if take in account the number of threads per core (2). I tried to set SPARK_WORKER_CORES in spark-env.sh file but it changes nothing (I made the changes on all 3 workers). I also comment the line to see if it changes something: number of cores available is always stuck at 6.
I'm using Spark 2.2.0 in standalone cluster:
pyspark --master spark://myurl:7077
result of lscpu command:
I've found that I simply had to stop the master and slaves and restart them so the parameter SPARK_WORKER_CORES is refreshed.
TL;DR
Spark UI shows different number of cores and memory than what I'm asking it when using spark-submit
more details:
I'm running Spark 1.6 in standalone mode.
When I run spark-submit I pass it 1 executor instance with 1 core for the executor and also 1 core for the driver.
What I would expect to happen is that my application will be ran with 2 cores total.
When I check the environment tab on the UI I see that it received the correct parameters I gave it, however it still seems like its using a different number of cores. You can see it here:
This is my spark-defaults.conf that I'm using:
spark.executor.memory 5g
spark.executor.cores 1
spark.executor.instances 1
spark.driver.cores 1
Checking the environment tab on the Spark UI shows that these are indeed the received parameters but the UI still shows something else
Does anyone have any idea on what might cause Spark to use different number of cores than what I want I pass it? I obviously tried googling it but didn't find anything useful on that topic
Thanks in advance
TL;DR
Use spark.cores.max instead to define the total number of cores available, and thus limit the number of executors.
In standalone mode, a greedy strategy is used and as many executors will be created as there are cores and memory available on your worker.
In your case, you specified 1 core and 5GB of memory per executor.
The following will be calculated by Spark :
As there are 8 cores available, it will try to create 8 executors.
However, as there is only 30GB of memory available, it can only create 6 executors : each executor will have 5GB of memory, which adds up to 30GB.
Therefore, 6 executors will be created, and a total of 6 cores will be used with 30GB of memory.
Spark basically fulfilled what you asked for. In order to achieve what you want, you can make use of the spark.cores.max option documented here and specify the exact number of cores you need.
A few side-notes :
spark.executor.instances is a YARN-only configuration
spark.driver.memory defaults to 1 core already
I am also working on easing the notion of the number of executors in standalone mode, this might get integrated into a next release of Spark and hopefully help figuring out exactly the number of executors you are going to have, without having to calculate it on the go.
I want to run Spark with only 1 thread. But whatever option I tried, Spark always used all 8 cores in my CPU.
I tried various mixtures of --master local, --master local[1], --executor-cores 1, --total-executor-cores 1, --conf spark.max.cores=1 options but nothing worked. When I see the top result on my Ubuntu 14.04, CPU usage is always about 600% (approximately 75% * 8 cores).
My goal is to compare running time of Spark tasks by varying number of cores used. Please help!
** Added
I'm working on the code from https://github.com/amplab/SparkNet/blob/master/src/main/scala/apps/CifarApp.scala . Sincerely thank you for everybody's help.
First of all you're mixing options which belong to different deployment modes. Parameters like spark.cores.max (not spark.max.cores) or spark.executor.cores are meaningful only in a Standalone Mode (not the same as local) and on Yarn.
In case of a local mode the only thing that really matters is the parameter n passed with master definition (local[n]). It doesn't mean that local[1] will run only using one thread. Spark alone is using a number of different threads (20 or so if I remember correctly) for bookkeeping, supervising, shuffles, UI and other stuff.
What is limited is the number of the executor threads. It still doesn't mean that a single executor cannot start more than one thread which is most likely the case here. You're using libraries which are designed for parallel execution. If you don't use GPU then computations are most likely executed in parallel on the CPUs. All of that is independent and not controlled by Spark itself. If you want a full control you should execute your application in a restricted environment like VM or container.
The code you referred to uses SparkContext.parallelize(...) without setting the numPartitions argument. This means that the value of spark.default.parallelism (see docs) is used to decide the number of parts == cores to use.
From the docs, this parameter defaults to:
For operations like parallelize with no parent RDDs, it depends on the cluster manager:
Local mode: number of cores on the local machine
So, adding --conf spark.default.parallelism=1 to your command should make these RDDs use a single partition, thus use a single core.
I'm running an EMR cluster (version emr-4.2.0) for Spark using the Amazon specific maximizeResourceAllocation flag as documented here. According to those docs, "this option calculates the maximum compute and memory resources available for an executor on a node in the core node group and sets the corresponding spark-defaults settings with this information".
I'm running the cluster using m3.2xlarge instances for the worker nodes. I'm using a single m3.xlarge for the YARN master - the smallest m3 instance I can get it to run on, since it doesn't do much.
The situation is this: When I run a Spark job, the number of requested cores for each executor is 8. (I only got this after configuring "yarn.scheduler.capacity.resource-calculator": "org.apache.hadoop.yarn.util.resource.DominantResourceCalculator" which isn't actually in the documentation, but I digress). This seems to make sense, because according to these docs an m3.2xlarge has 8 "vCPUs". However, on the actual instances themselves, in /etc/hadoop/conf/yarn-site.xml, each node is configured to have yarn.nodemanager.resource.cpu-vcores set to 16. I would (at a guess) think that must be due to hyperthreading or perhaps some other hardware fanciness.
So the problem is this: when I use maximizeResourceAllocation, I get the number of "vCPUs" that the Amazon Instance type has, which seems to be only half of the number of configured "VCores" that YARN has running on the node; as a result, the executor is using only half of the actual compute resources on the instance.
Is this a bug in Amazon EMR? Are other people experiencing the same problem? Is there some other magic undocumented configuration that I am missing?
Okay, after a lot of experimentation, I was able to track down the problem. I'm going to report my findings here to help people avoid frustration in the future.
While there is a discrepancy between the 8 cores asked for and the 16 VCores that YARN knows about, this doesn't seem to make a difference. YARN isn't using cgroups or anything fancy to actually limit how many CPUs the executor can actually use.
"Cores" on the executor is actually a bit of a misnomer. It is actually how many concurrent tasks the executor will willingly run at any one time; essentially boils down to how many threads are doing "work" on each executor.
When maximizeResourceAllocation is set, when you run a Spark program, it sets the property spark.default.parallelism to be the number of instance cores (or "vCPUs") for all the non-master instances that were in the cluster at the time of creation. This is probably too small even in normal cases; I've heard that it is recommended to set this at 4x the number of cores you will have to run your jobs. This will help make sure that there are enough tasks available during any given stage to keep the CPUs busy on all executors.
When you have data that comes from different runs of different spark programs, your data (in RDD or Parquet format or whatever) is quite likely to be saved with varying number of partitions. When running a Spark program, make sure you repartition data either at load time or before a particularly CPU intensive task. Since you have access to the spark.default.parallelism setting at runtime, this can be a convenient number to repartition to.
TL;DR
maximizeResourceAllocation will do almost everything for you correctly except...
You probably want to explicitly set spark.default.parallelism to 4x number of instance cores you want the job to run on on a per "step" (in EMR speak)/"application" (in YARN speak) basis, i.e. set it every time and...
Make sure within your program that your data is appropriately partitioned (i.e. want many partitions) to allow Spark to parallelize it properly
With this setting you should get 1 executor on each instance (except the master), each with 8 cores and about 30GB of RAM.
Is the Spark UI at http://:8088/ not showing that allocation?
I'm not sure that setting is really a lot of value compared to the other one mentioned on that page, "Enabling Dynamic Allocation of Executors". That'll let Spark manage it's own number of instances for a job, and if you launch a task with 2 CPU cores and 3G of RAM per executor you'll get a pretty good ratio of CPU to memory for EMR's instance sizes.
in the EMR version 3.x, this maximizeResourceAllocation was implemented with a reference table: https://github.com/awslabs/emr-bootstrap-actions/blob/master/spark/vcorereference.tsv
it used by a shell script: maximize-spark-default-config, in the same repo, you can take a look how they implemented this.
maybe in the new EMR version 4, this reference table was somehow wrong... i believe you can find all this AWS script in your EC2 instance of EMR, should be located in /usr/lib/spark or /opt/aws or something like this.
anyway, at least, you can write your own bootstrap action scripts for this in EMR 4, with a correct reference table, similar to the implementation in EMR 3.x
moreover, since we are going to use STUPS infrastructure, worth take a look the STUPS appliance for Spark: https://github.com/zalando/spark-appliance
you can explicitly specify the number of cores by setting senza parameter DefaultCores when you deploy your spark cluster
some of highlight of this appliance comparing to EMR are:
able to use it with even t2 instance type, auto-scalable based on roles like other STUPS appliance, etc.
and you can easily deploy your cluster in HA mode with zookeeper, so no SPOF on master node, HA mode in EMR is currently still not possible, and i believe EMR is mainly designed for "large clusters temporarily for ad-hoc analysis jobs", not for "dedicated cluster that is permanently on", so HA mode will not be possible in near further with EMR.