Spark-Submit cannot connect to ResourceManager - apache-spark

Question

I'm trying to start a example spark job (hadoop is 2.7 and spark is 3.3.1) on my hadoop cluster containing of namenode and datanode0.
Upon running start-dfs.sh, I can see the datanode within the UI, and running jps on datanode returns me with "Datanode" process.
When I try to run the spark-submit with example, I get following output:
spark#namenode:~$ spark-submit --deploy-mode client --class org.apache.spark.examples.SparkPi $SPARK_HOME/examples/jars/spark-examples_2.12-3.3.1.jar 10
23/02/07 21:23:08 INFO SparkContext: Running Spark version 3.3.1
23/02/07 21:23:08 INFO ResourceUtils: ==============================================================
23/02/07 21:23:08 INFO ResourceUtils: No custom resources configured for spark.driver.
23/02/07 21:23:08 INFO ResourceUtils: ==============================================================
23/02/07 21:23:08 INFO SparkContext: Submitted application: Spark Pi
23/02/07 21:23:09 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 512, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
23/02/07 21:23:09 INFO ResourceProfile: Limiting resource is cpus at 1 tasks per executor
23/02/07 21:23:09 INFO ResourceProfileManager: Added ResourceProfile id: 0
23/02/07 21:23:09 INFO SecurityManager: Changing view acls to: spark
23/02/07 21:23:09 INFO SecurityManager: Changing modify acls to: spark
23/02/07 21:23:09 INFO SecurityManager: Changing view acls groups to:
23/02/07 21:23:09 INFO SecurityManager: Changing modify acls groups to:
23/02/07 21:23:09 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark); groups with view permissions: Set(); users with modify permissions: Set(spark); groups with modify permissions: Set()
23/02/07 21:23:09 INFO Utils: Successfully started service 'sparkDriver' on port 45161.
23/02/07 21:23:09 INFO SparkEnv: Registering MapOutputTracker
23/02/07 21:23:09 INFO SparkEnv: Registering BlockManagerMaster
23/02/07 21:23:09 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
23/02/07 21:23:09 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
23/02/07 21:23:09 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
23/02/07 21:23:09 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-09c3077f-8d16-4496-9808-0626cefe1cc7
23/02/07 21:23:09 INFO MemoryStore: MemoryStore started with capacity 93.3 MiB
23/02/07 21:23:09 INFO SparkEnv: Registering OutputCommitCoordinator
23/02/07 21:23:10 INFO Utils: Successfully started service 'SparkUI' on port 4040.
23/02/07 21:23:10 INFO SparkContext: Added JAR file:/home/spark/spark/examples/jars/spark-examples_2.12-3.3.1.jar at spark://namenode:45161/jars/spark-examples_2.12-3.3.1.jar with timestamp 1675801388738
23/02/07 21:23:10 INFO RMProxy: Connecting to ResourceManager at namenode/192.168.1.17:8032
23/02/07 21:23:11 INFO Client: Retrying connect to server: namenode/192.168.1.17:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/02/07 21:23:12 INFO Client: Retrying connect to server: namenode/192.168.1.17:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/02/07 21:23:13 INFO Client: Retrying connect to server: namenode/192.168.1.17:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/02/07 21:23:14 INFO Client: Retrying connect to server: namenode/192.168.1.17:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/02/07 21:23:15 INFO Client: Retrying connect to server: namenode/192.168.1.17:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
23/02/07 21:23:16 INFO Client: Retrying connect to server: namenode/192.168.1.17:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
^C23/02/07 21:23:17 INFO DiskBlockManager: Shutdown hook called
23/02/07 21:23:17 INFO ShutdownHookManager: Shutdown hook called
23/02/07 21:23:17 INFO ShutdownHookManager: Deleting directory /tmp/spark-5a15bfc6-654d-4332-be86-3f20b4cf40f1/userFiles-049088cd-d541-4424-930b-dcaa18634860
23/02/07 21:23:17 INFO ShutdownHookManager: Deleting directory /tmp/spark-4d529cc6-0bc8-43a9-b2af-3671ee11f963
23/02/07 21:23:17 INFO ShutdownHookManager: Deleting directory /tmp/spark-5a15bfc6-654d-4332-be86-3f20b4cf40f1
Is it possible that I've messed up the yarn configuration?
Here's my etc hosts:
192.168.1.17 namenode
192.168.1.23 datanode0
This is my yarn-site.xml, maybe I've messed up the configuration?
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>namenode</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>${yarn.resourcemanager.hostname}:8030</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>${yarn.resourcemanager.hostname}:8032</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>${yarn.resourcemanager.hostname}:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>${yarn.resourcemanager.hostname}:8088</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>1536</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>1536</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>128</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
</configuration>