Is there a spark config parameter we can pass while submitting jobs through spark-submit, that will kill/fail job if it does not gets containers in given time?
For example if job requested for 8 yarn containers, which could not be allocated for 2 hours, then the job kill itself.
EDIT: We have scripts launching spark or MR jobs on a cluster. This issue is not a major one for MR jobs as they can start even if 1 container is available. Also MR jobs need much less memory so their containers can be smaller, hence more containers are available in cluster.
I've just finished the SLURM installation in our cluster. How do I run programs in node2 (slave) from node1 (head/controller node)? Is there a special configuration for this kind of setup?
Make sure that UIDs are uniform across the cluster nodes, and that a common filesystem (for instance NFS) is mounted on all nodes.
Then you run programs typically by submitting jobs with the sbatch command.
You can refer to this set of slides for a quick start.
We have a moderately large PySpark program that we run on a Mesos cluster.
We run the program with spark.executor.cores=8 and spark.cores.max=24. Each Mesos node has 12 vcpu, so that only 1 executor is started on each node.
The program runs flawlessly, with correct results.
However, the issue is that each executor consumes much more CPU than 8. CPU load frequently reaches 25 or more.
With the htop program, we see that 8 python processes are started, as expected. However, each Python spawn several threads, so each python process can go up to 300% CPU.
This behavior is annoying in a shared cluster deployment.
Can someone explain this behavior ?
What are these 3 additional threads that pyspark starts ?
Additional infos:
The functions we use in our Spark operations are not multithreaded
We have the same behavior in local mode, outside of Mesos
We use Spark 2.1.1 and Python 3.5
Nothing else runs on the Mesos nodes, excepted the usual base services
In our test platform, Mesos nodes are actually OpenStack VM
In Spark 1.0.0 Standalone mode with multiple worker nodes, I'm trying to run a Spark shell from two different computers (same Linux user).
In the documentation, it says "By default, applications submitted to the standalone mode cluster will run in FIFO (first-in-first-out) order, and each application will try to use all available nodes."
The number of cores per worker is set to 4 with 8 being available (via SPARK_JAVA_OPTS="-Dspark.cores.max=4"). Memory is also limited such that enough should be available for both.
However, when looking at the Spark Master WebUI, the shell application that was started later will always remain in state "WAITING" until the first one is exited. The number of cores assigned to it is 0, the Memory per node 10G (same as the one that is already running)
Is there a way to have both shells running at the same time without using Mesos?
Before a shell will start processing on a spark standalone cluster, there has to be sufficient cores and memory. You must specify from each spark shell the number of cores you want, or it will use them all. If you specify 5 cores, with executor memory=10G (the amount of memory you allocated for the executors), and the second spark shell to run with 2 cores, and 10G of memory, the second one will still not start, because the first shell is using both executors, and is using all of the memory on both. If you specify 5G of executor memory for each spark shell, then they can concurrently run.
Essentially you want to have multiple jobs running on a standalone cluster -- unfortunately, it is really not designed to handle this case well. If you want to do that you should use either mesos or yarn.
One workaround to this is to restrict the number of cores per spark shell using total-executor-cores. For example to restrict it to 16 cores, launch it like this:
bin/spark-shell --total-executor-cores 16 --master spark://$MASTER:7077
In this case each shell will use only 16 cores, so you can have two shells running on your 32 cores cluster. They can then run simultaneously but never use more than 16 cores each :(
This solution is far from ideal, I know. You depend on users to restrict themselves, to shut down their shells, and resources are wasted when a user is not running code. I have created a request to fix this on JIRA, which you can vote for.
The application ends when your shell dies. So, you cannot run concurrently two spark-shells on two laptops. What you can do is launch one spark-shell, launch the other, and have the second start when the first one dies.
Contrarily to spark-shell, spark-submit does terminate once computation is over. So you can spark-submit one app, launch a spark-shell, and have the shell take over the moment the application is done.
Or you can run two apps sequentially (one after the other) with two spark-submit launches.
I have a condor pool which consist of 4 dedicated machine one is set as a centeral manager, submitting, and executing node while the other three is set to be executing nodes I used CentOS 5.4 as an OS for all the machines. My problem is when I submitted a job from the central manager it works just on the central manager so when I specify in the JDL file that the job should run in any machine apart from the central manager the job stay in hold and does not run. When I type condor_status all nodes appear. I keep the daemon MASTER, STARTD in the daemon list for the executing nodes. Does any one come across this problem?
There's not enough information to answer your question, but the first thing to do is to run condor_q -analyze <jobid> and see what it tells you. See the Condor manual Section 2.6.5: Why is the job not running?
One possible cause is that you're not telling Condor to transfer your input/output files for you, and your nodes have different "filesystem domains", so Condor is unable to find a host which shares a common filesystem with your submit host.