Spark not using all nodes after resizing - apache-spark

I've got an EMR cluster that I'm trying to use to execute large text processing jobs, and I had it running on a smaller cluster, however after resizing the master keeps running the jobs locally and crashing due to memory issues.
This is the current configuration I have for my cluster:
[
{
"classification":"capacity-scheduler",
"properties":
{
"yarn.scheduler.capacity.resource-calculator":"org.apache.hadoop.yarn.util.resource.DominantResourceCalculator"
},
"configurations":[]
},
{
"classification":"spark",
"properties":
{
"maximizeResourceAllocation":"true"
},
"configurations":[]
},
{
"classification":"spark-defaults",
"properties":
{
"spark.executor.instances":"0",
"spark.dynamicAllocation.enabled":"true"
},
"configurations":[]
}
]
This was a potential solution I saw from this question, and it did work before I resized.
Now whenever I attempt to submit a spark job like this spark-submit mytask.py
I see tons of log entries where it doesn't seem to leave the master host, like so:
17/08/14 23:49:23 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0,localhost, executor driver, partition 0, PROCESS_LOCAL, 405141 bytes)
I've tried different parameters, like setting --deploy-mode cluster and --master yarn, since yarn is running on the master node, but still seeing all the work being done by the master host, while the core nodes sit idle.
Is there another configuration I'm missing, preferably one that doesn't require rebuilding the cluster?

Related

Error reading from large Cassandra table with Spark, getting "Remote RPC client disassociated"

I set stand alone spark cluster (with cassandra) and i did it but when i read data i get error.My cluster has 3 nodes and each node has 64 GB ram and 20 cores. I'm sharing some Spark-env.sh configuration like spark_executor_cores: 5, spark_executor_memory:5G, spark_worker_cores:20 and spark_worker_memory:45g.
I want to give another information, when i read small table there is no problem but when i read big table i get error. Error description at below. Also when i start pyspark i use this command:
$ ./pyspark --master spark://10.0.0.100:7077
--packages com.datastax.spark:spark-cassandra-connector_2.12:3.1.0
--conf spark.driver.extraJavaOptions=-Xss1024m
--conf spark.driver.port:36605
--conf spark.driver.blockManager.port=42365
Thanks for your interest
ERROR TaskSchedulerImpl: Lost executor 5 on 10.0.0.10: Remote RPC client disassociated. likely due to containers exceeding threshold, or network issues. Chec driver logs for WARN messages
WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0) (10.0.0.10 executor 5): ExecutorLostFailure (executor 5 exited caused by one of the runnning task) reason: remote RPC client disassociated.
WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1) (10.0.0.11 executor 2):Java.lang.StackOverflowError
at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1094)
at java.base/java.nio.HeapByteBuffer.get(HeapByteBuffer.java:184)
at org.apache.spark.util.ByteBufferInputStream.read(ObjectInputStream.scala:49)
at java.base/java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2887)
at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2903)
at java.base/java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3678)
at java.base/java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3678)
at java.base/java.io.ObjectInputStream.readString(ObjectInputStream.java:2058)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1663)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2490)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2384)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2222)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1681)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2490)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2384)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2222)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1681)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2490)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2384)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2222)
at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1681)
at java.base/java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2490)
at java.base/java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2384)
at java.base/java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2222)
The problem you're running into is most likely a networking issue.
It's highly unusual that you need to pin the driver ports with:
--conf spark.driver.port:36605
--conf spark.driver.blockManager.port=42365
You'll need to provide background information on why you're doing this.
Also as I previously advised you on another question last week, you need to provide the minimal code + minimal configuration that replicates the problem. Otherwise, there isn't enough information for others to be able to help you. Cheers!

Databricks is always auto starting

I can't put Databricks to idle because something is auto starting it. I have no idea how to check it. Here are some of the details I found so far...
Event Log
STARTING
2020-12-28 13:07:42 PST
Started by b161878e-ff38-4fca-924a-b5c5ce4c3193.
Running Applications
Application ID
- app-20201228051029-0000
Name
- Databricks Shell
Cores
4
Memory per Executor
- 7.1 GiB
Resources Per Executor
-
Submitted Time
- 2020/12/28 05:10:29
User
- root
State
- RUNNING
Duration
- 27 s
Tried killing this but got Method Not Allowed.
No running Jobs
No attached Notebooks
I have limited Databricks knowledge, and I don't know how to check who started it. Any help is very much appreciated.
Added event logs details.
Event during termination because of inactivity.
{
"reason": {
"code": "INACTIVITY",
"type": "SUCCESS",
"parameters": {
"inactivity_duration_min": "60"
}
}
}
After a few seconds, it's going to start again.
{
"user": "b161878e-ff38-4fca-924a-b5c5ce4c3193"
}

Controlling the number of spark drivers running on Mesos

We have a cluster with 6 EC2 nodes in AWS (16 cpu, 64 gb per node - 1 node running mesos master and 5 nodes running mesos slave). The mesos dispatcher is running on the master node. We use this cluster exclusively to run spark jobs. Our spark configuration is 5 cpu and 10 gb per executor (same for driver). I
In one of our scheduled jobs, we have a scenario where we need to do a few hundred spark-submits at the same time. When this happens, the dispatchers starts drivers for all these spark-submits leaving no room for any executors in the cluster.
I'm looking at a few options. I would like to get some pointers from members in the spark/mesos community.
Some options which I don't want to get into are : increasing the cluster size, asking the analysts to change their job structure to combine all spark-submits into a single one, switching to YARN/EMR (actually I tried this and got into some messy queue problems there)
Option 1 : Using Mesos roles
I found some documentation on the use of quotas and roles to solve this. But I'm not sure about the following :
How to create a mesos role and update the resources to be made available to
this role ?
How to set up separate roles for spark drivers and spark executors ?
By default all resources are in the * role. There is a flag called spark.mesos.role that I can set while doing spark-submit but not sure how to create this role and ensure this role is used only for executors ?
Option 2 : Modifying the mesos cluster scheduler
When spark-submit happens to mesos dispatcher, it adds the driver request to a WaitingQueue. When drivers fail while executing and if supervise mode is available, they are sent to a PendingRetryQueue with custom retry schedule settings. When resources are available from mesos, these drivers from the PendingRetryQueue are scheduled first and WaitingQueue are scheduled next.I was thinking of keeping the WaitingQueue with size 5 (spark.mesos.maxDrivers) and when there are more spark-submits than the queue size, I would add these drivers to the PendingRetryQueue and schedule them to run later. Currently, as per my understanding, when there are more that 200 drivers in the WaitingQueue mesos rest server sends a failed message and doesn't add it to the PendingRetryQueue.
Any help on implementing either of the options would be very helpful to me. Thanks in advance.
Update : Just saw that by when I give spark-submit with a role, it runs only executors in that role and drivers run in the default * role. I think this should solve this issue for me. Once I test this, I'll post my update here and close this. Thanks
As mentioned in the update, by default the mesos runs spark drivers in default role (*) and executors in the role provided by 'spark.mesos.role' parameter. To control the resources available for each role, we can use quotas , guarantees or reservations. We went ahead with static reservations since it suited our requirements.Thanks.
Option 1 is the good one.
First set dispatcher quota by creating a file like dispatcher-quota.json
cat dispatcher-quota.json
{
"role": "dispatcher",
"guarantee": [
{
"name": "cpus",
"type": "SCALAR",
"scalar": { "value": 5.0 }
},
{
"name": "mem",
"type": "SCALAR",
"scalar": { "value": 5120.0 }
}
]
}
Then push it to you're mesos master (leader) with
curl -d #dispatcher-quota.json -X POST http://<master>:5050/quota
So now you will have a quota for driver
Ensure you dispatcher is running with the right service role set if needed ajust it. If in DC/OS use
$ cat options.json
{
"service": {
"role": "dispatcher"
}
}
$ dcos package install spark --options=options.json
Otherwise feel free to share how you've deployed you dispatcher. I will provide you a how to guide.
That's ok for drivers. Now let's work with executor folowing the same way
$ cat executor-quota.json
{
"role": "executor",
"guarantee": [
{
"name": "cpus",
"type": "SCALAR",
"scalar": { "value": 100.0 }
},
{
"name": "mem",
"type": "SCALAR",
"scalar": { "value": 409600.0 }
}
]
}
$ curl -d #executor-quota.json -X POST http://<master>:5050/quota
Adapt values to you requirements
Then ensure to launch executor on with the correct role by providing
--conf spark.mesos.role=executor \
Source of my explanation came from https://github.com/mesosphere/spark-build/blob/master/docs/job-scheduling.md don't hesistate if it's not enought.
This should do the work

Why spark executor are not dying

Here is my setup:
Kubernetes cluster running airflow, which submits the spark job to Kubernetes cluster, job runs fine but the container are suppose to die once the job is done but they are still hanging there.
Airflow Setup comes up on K8S cluster.
Dag is baked in the airflow docker image because somehow I am not able to sync the dags from s3. For some reason the cron wont run.
Submits the spark job to K8S Cluster and job runs fine.
But now instead of dying post execution and completion of job it still hangs around.
Here is my SparkSubmitOperator function
spark_submit_task = SparkSubmitOperator(
task_id='spark_submit_job_from_airflow',
conn_id='k8s_spark',
java_class='com.dom.rom.mainclass',
application='s3a://some-bucket/jars/demo-jar-with-dependencies.jar',
application_args=['300000'],
total_executor_cores='8',
executor_memory='20g',
num_executors='9',
name='mainclass',
verbose=True,
driver_memory='10g',
conf={
'spark.hadoop.fs.s3a.aws.credentials.provider': 'com.amazonaws.auth.InstanceProfileCredentialsProvider',
'spark.rpc.message.maxSize': '1024',
'spark.hadoop.fs.s3a.impl': 'org.apache.hadoop.fs.s3a.S3AFileSystem',
'spark.kubernetes.container.image': 'dockerhub/spark-image:v0.1',
'spark.kubernetes.namespace' : 'random',
'spark.kubernetes.container.image.pullPolicy': 'IfNotPresent',
'spark.kubernetes.authenticate.driver.serviceAccountName': 'airflow-spark'
},
dag=dag,
)
Figured the problem it was my mistake I wasn't closing the spark session, added the following
session.stop();

how to tell mesos frameworks which launched by different commands/parameters

I am building metrics collector to collect the running status about all the Spark Jobs running on it. The mesos API http://masterip/frameworks return a lot of details about all the frameworks and then I run http://slaveip/slave(1)/monitor/statistics to get each frameworks detail info from each slave, then correlate them.
This works fine for most of the jobs, but I have some jobs which behave different according to different parameters when submitting. They are shown as same framework name in Mesos GUI and I can not tell each other.
Is there a way to get the detail full commands which launches the job? Or any other idea about how to tell them?
You can find there are multiple instances with same framework name. As they are different spark job instances.
When I connect to Mesos slave, the monitor/statistics doesn't show the full command with all the parameters, so I can not tell which framework correlate to which Spark job instance.
{
"executor_id": "0",
"executor_name": "Command Executor (Task: 0) (Command: sh -c '
\"/usr/local...')",
"framework_id": "06ba8de8-7fc3-422d-9ee3-17dd9ddcb2ca-3157",
"source": "0",
"statistics": {
"cpus_limit": 2.1,
"cpus_system_time_secs": 848.689999999,
"cpus_user_time_secs": 5128.78,
"mem_limit_bytes": 4757389312,
"mem_rss_bytes": 2243149824,
"timestamp": 1522858776.20098
}
},
Thanks

Resources