Spark submit on Kubernetes cloud engine using only one node, one cpu requested - apache-spark

Question

I have set up a cluster with 4 nodes each having 2 CPUs so 8 in total.
My code end up running only on one CPU and no matter the settings the requested CPUs is always 1 in the pod description and the execution time stays the same. Tried the spark examples and same thing applies.
Spark submit script I use:
./bin/spark-submit \
--master k8s://https://34.64.87.144 \
--deploy-mode cluster \
--name spark-counter \
--class DataCounter \
--driver-java-options "-Dlog4j.configuration=file:////opt/spark/data/log4j.properties" \
--conf spark.executor.cores=4 \
--conf spark.kubernetes.executor.request.cores=3.6 \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--conf spark.kubernetes.driver.pod.name=spark-counter \
--conf spark.kubernetes.container.image=asia.gcr.io/profound-media-298808/spark-base:latest \
local:///opt/spark/data/spark_counter-1.0.jar /opt/spark/data/input1
and the description of the pod
spark-role=driver
Annotations: <none>
Status: Succeeded
IP: 10.32.4.37
IPs:
IP: 10.32.4.37
Containers:
spark-kubernetes-driver:
Container ID: containerd://bd31b8112159145169ab1b6397af8bc2f10cee5429b11c8025f2359ab5194882
Image: asia.gcr.io/profound-media-298808/spark-base:latest
Image ID: asia.gcr.io/profound-media-298808/spark-base#sha256:6aaf817da5606a39bf2aeea769c4ec2d62c7986d06109cb4a38f4f7157702ff1
Ports: 7078/TCP, 7079/TCP, 4040/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Args:
driver
--properties-file
/opt/spark/conf/spark.properties
--class
DataCounter
spark-internal
/opt/spark/data/input1
State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 18 Dec 2020 07:13:45 +0000
Finished: Fri, 18 Dec 2020 07:14:28 +0000
Ready: False
Restart Count: 0
Limits:
memory: 1408Mi
Requests:
cpu: 1
memory: 1408Mi
Environment:
SPARK_DRIVER_BIND_ADDRESS: (v1:status.podIP)
SPARK_LOCAL_DIRS: /var/data/spark-f4832458-7450-4d96-b7ed-c672d5ec0eda
SPARK_CONF_DIR: /opt/spark/conf
Mounts:
/opt/spark/conf from spark-conf-volume (rw)
/var/data/spark-f4832458-7450-4d96-b7ed-c672d5ec0eda from spark-local-dir-1 (rw)
/var/run/secrets/kubernetes.io/serviceaccount from spark-token-5zk2c (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
spark-local-dir-1:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
spark-conf-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: spark-counter-1608275621712-driver-conf-map
Optional: false
spark-token-5zk2c:
Type: Secret (a volume populated by a Secret)
SecretName: spark-token-5zk2c
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 7m44s default-scheduler Successfully assigned default/spark-counter to gke-cluster-1-default-pool-f51b7df5-g048
Warning FailedMount 7m43s (x2 over 7m43s) kubelet MountVolume.SetUp failed for volume "spark-conf-volume" : configmap "spark-counter-1608275621712-driver-conf-map" not found
Normal Pulled 7m42s kubelet Container image "asia.gcr.io/profound-media-298808/spark-base:latest" already present on machine
Normal Created 7m42s kubelet Created container spark-kubernetes-driver
Normal Started 7m42s kubelet Started container spark-kubernetes-driver
I have no additional configuration files set up, using the spark image builder script as a base to build the image used only adding my own jar and data to it. Should I have something more?
How do I set up my cluster to utilize all nodes?