Spark Structured Streaming + pyspark app returns "Initial job has not accepted any resources" - apache-spark

Question

RunCode
spark-submit --master spark://{SparkMasterIP}:7077
--deploy-mode cluster --packages org.apache.spark:spark-sql-kafka-0-10_2.12:3.1.2,
com.datastax.spark:spark-cassandra-connector_2.12:3.2.0,
com.github.jnr:jnr-posix:3.1.15
--conf spark.dynamicAllocation.enabled=false
--conf com.datastax.spark:spark.cassandra.connectiohost={SparkMasterIP==CassandraIP},
spark.sql.extensions=com.datastax.spark.connector.CassandraSparkExtensions test.py
Source Code
from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark.sql.types import *
from pyspark.sql import SQLContext
# Spark Bridge local to spark_master == Connect master
spark = SparkSession.builder \
.master("spark://{SparkMasterIP}:7077") \
.appName("Spark_Streaming+kafka+cassandra") \
.config('spark.cassandra.connection.host', '{SparkMasterIP==CassandraIP}') \
.config('spark.cassandra.connection.port', '9042') \
.getOrCreate()
# Parse Schema of json
schema = StructType() \
.add("col1", StringType()) \
.add("col2", StringType()) \
.add("col3", StringType()) \
.add("col4", StringType()) \
.add("col5", StringType()) \
.add("col6", StringType()) \
.add("col7", StringType())
# Read Stream From {TOPIC} at BootStrap
df = spark.readStream \
.format("kafka") \
.option("kafka.bootstrap.servers", "{KAFKAIP}:9092") \
.option('startingOffsets','earliest') \
.option("subscribe", "{TOPIC}") \
.load() \
.select(from_json(col("value").cast("String"), schema).alias("parsed_value")) \
.select("parsed_value.*")
df.printSchema()
# write Stream at cassandra
ds = df.writeStream \
.trigger(processingTime='15 seconds') \
.format("org.apache.spark.sql.cassandra") \
.option("checkpointLocation","./checkPoint") \
.options(table='{TABLE}',keyspace="{KEY}") \
.outputMode('append') \
.start()
ds.awaitTermination()
Error Code
Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
I was checked Spark UI, workers have no problem.
here is my Spark status
[![enter image description here][2]][2]
my plan is
kafka(DBIP)--readStream-->LOCAL(DriverIP)--writeStream-->Spark&kafka&casaandra(MasterIP)
DBIP, DriverIP, MasterIP is different IP.
LOCAL have no spark, so i use pyspark on python_virtualenv
Edit

Answer 1

You app can't run because there are no resources available in your Spark cluster.
If you look closely at the Spark UI screenshot you posted, all the cores are used on all the 3 workers. That means there are no cores left for any other apps so any new submitted app will have to wait until resources are available before it can be scheduled. Cheers!
👉 Please support the Apache Cassandra community by hovering over the cassandra tag above and click on Watch tag. 🙏 Thanks!