I am loading data from 1200 MS SQL Server tables into BigQuery with a spark job. It's all part of an orchestrated ETL process where the spark job consists of scala code which receives messages from PubSub. So 1200 messages are being received over a period of around an hour. Each message triggers code (async) which reads data from a table, with minor transformations, and writes to BigQuery. The process itself works fine. My problem is that the number of active jobs in spark never goes above 5, in spite of a lot of "jobs" waiting and plenty of resources being available.
I've tried upping the spark.driver.cores to 30, but there is no change. Also, this setting, while visible in the Google Console, doesn't seem to make it through to the actual spark job (when viewed in the spark UI). Here is the spark job running in the console:
And here are the spark job properties:
It's a pretty big cluster, with plenty of resources to spare:
Here is the command line for creating the cluster:
gcloud dataproc clusters create odsengine-cluster \
--properties dataproc:dataproc.conscrypt.provider.enable=false,spark:spark.executor.userClassPathFirst=true,spark:spark.driver.userClassPathFirst=true \
--project=xxx \
--region europe-north1 \
--zone europe-north1-a \
--subnet xxx \
--master-machine-type n1-standard-4 \
--worker-machine-type m1-ultramem-40 \
--master-boot-disk-size 30GB \
--worker-boot-disk-size 2000GB \
--image-version 1.4 \
--master-boot-disk-type=pd-ssd \
--worker-boot-disk-type=pd-ssd \
--num-workers=2 \
--scopes cloud-platform \
--initialization-actions gs://xxx/cluster_init/init_actions.sh
And the command line for submitting the spark job:
gcloud dataproc jobs submit spark \
--project=velliv-dwh-development \
--cluster odsengine-cluster \
--region europe-north1 \
--jars gs://velliv-dwh-dev-bu-dcaods/OdsEngine_2.11-0.1.jar \
--class Main \
--properties \
spark.executor.memory=35g,\
spark.executor.cores=2,\
spark.executor.memoryOverhead=2g,\
spark.dynamicAllocation.enabled=true,\
spark.shuffle.service.enabled=true,\
spark.driver.cores=30\
-- yarn
I am aware that I could look into using partitioning to spread out the load of large individual tables, and I've also had that working in another scenario with success, but in this case I just want to load many tables at once without partitioning each table.
Regarding "a lot of jobs waiting and plenty of resources being available", I'd suggest you check Spark log, YARN web UI and log to see if there are pending applications and why. It also helps to check the cluster web UI's monitoring tab for YARN resource utilization.
Regarding the spark.driver.cores problem, it is effective only in cluster mode, see this doc:
Number of cores to use for the driver process, only in cluster mode
Spark driver runs in client mode by default in Dataproc, which means the driver runs on the master node outside of YARN. You can run the driver in cluster mode as a YARN container with property spark.submit.deployMode=cluster.
Related
I am able to successfully build the example project from https://github.com/datastax/SparkBuildExamples/tree/master/scala/sbt/dse/src/main/scala/com/datastax/spark/example
I am also successful in submitting dse spark-submit. The program runs fine and results are good as expected
dse spark-submit --class com.datastax.spark.example.WriteRead target/writeRead-0.1.jar
I now wish to submit it the above job to an existing pool as configured in dse.yaml
resource_manager_options:
worker_options:
cores_total: 6
memory_total: 32G
workpools:
- name: alwayson_sql
cores: 2
memory: 4G
- name: pool_1
cores: 2
memory: 16G
I am unable to determine how/what changes in code or spark-submit that I should do in order to submit the application to the pool "pool_1"
The application is submitted to the default pool and I am unable to submit it to "pool_1".
Please help.
After some additional research I figured out the correct way to dse spark-submit to use the pool "pool_1"
bin/dse spark-submit \
--master dse://?workpool=pool_1 \
--conf spark.network.timeout=500 \
--class com.datastax.spark.example.WriteRead target/writeRead-0.1.jar
(Per input from Alex)DSE Documentation:
Documentation link
I'm submitting Spark jobs in Kubernetes running locally (Docker desktop). I'm able to submit the jobs and see their final output in the screen.
However, even if they're completed, the driver and executor pods are still in a RUNNING state.
The base images used to submit the Spark jobs to kubernetes are the ones that come with Spark, as described in the docs.
This is what my spark-submit command looks like:
~/spark-2.4.3-bin-hadoop2.7/bin/spark-submit \
--master k8s://https://kubernetes.docker.internal:6443 \
--deploy-mode cluster \
--name my-spark-job \
--conf spark.kubernetes.container.image=my-spark-job \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.submission.waitAppCompletion=false \
local:///opt/spark/work-dir/my-spark-job.py
And this is what kubectl get pods returns:
NAME READY STATUS RESTARTS AGE
my-spark-job-1568669908677-driver 1/1 Running 0 11m
my-spark-job-1568669908677-exec-1 1/1 Running 0 10m
my-spark-job-1568669908677-exec-2 1/1 Running 0 10m
Figured it out. I forgot to stop the Spark Context. My script looks like this now, and at completion, the driver goes into Completed status and the drivers get deleted.
sc = SparkContext()
sqlContext = SQLContext(sc)
# code
sc.stop()
I read this: Running Spark on Kubernetes.
I want to know more details about the interaction between Kubernetes Controller/Scheduler and Spark runtime when launching a Spark job on K8s.
Specially, assuming we launch an Spark app by :
bin/spark-submit \
--master k8s://https://<k8s-apiserver-host>:<k8s-apiserver-port> \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=5 \
--..............
My question is: the K8s may not be able to allocate 5 executors (or called containers/pods) immediately due to unavailability of cluster resources at the moment the Spark app is launched. Which way does Spark app take? (1) Spark starts running tasks as soon as possible when there is at least one executor is allocated. (2) Spark won't launch any tasks until all of the 5 executors have been allocated.
If you know Hadoop YARN, it would be great if you could also answer the question in the scenario of running Spark app on Hadoop YARN(DynamicAllocation Disabled) and point out the difference.
I am going to run spark job on mesos, and I want to limit resource of specified role.
I try to run 5 executors in my job
spark-shell \
--master mesos://zk://host1:2181,host2:2181,host3:2181/mesos \
--conf spark.executor.cores=1 \
--conf spark.cores.max=5 \
--conf spark.mesos.role=myrole
It works well that I can get many resource offers to distribute executors when quota setting is disable.
18/01/25 13:35:49 DEBUG MesosCoarseGrainedSchedulerBackend: Received 4 resource offers.
If I enable quota setting(http://mesos.apache.org/documentation/latest/quota/), then I always get only 1 resource offer.
18/01/25 13:36:31 DEBUG MesosCoarseGrainedSchedulerBackend: Received 1 resource offers.
I have no idea what happened there
My environment:
spark 2.2
mesos 1.4.1 (master*3, slave*5)
CentOS 7.3
I have setup a Spark standalone cluster, where I can submit jobs with spark-submit:
spark-submit \
--class blah.blah.MyClass \
--master spark://myaddress:6066 \
--executor-memory 8G \
--deploy-mode cluster \
--total-executor-cores 12 \
/path/to/jar/myjar.jar
Problem is when I send multiple jobs at the same time, say over 20 in one go, the first few finished successfully. All the others are now stuck waiting for resources. I noticed all the available memory has gone to the drivers, so in the drivers section they are all running but in the running application section they all are in WAITING state.
How can I tell spark stand alone to first allocate memory to the WAITING executors instead of the SUBMITTED drivers?
thank you
Below is an extract of my spark-defaults.conf
spark.master spark://address:7077
spark.eventLog.enabled true
spark.eventLog.dir /path/tmp/sparkEventLog
spark.driver.memory 5g
spark.local.dir /path/tmp
spark.ui.port xxx