Spark job getting stuck forever - apache-spark

Question

I am trying to run it in the docker. But it is getting stuck after giving output. Am i missing something?
FROM bitnami/spark:3
USER root
RUN curl https://repo1.maven.org/maven2/com/amazonaws/aws-java-sdk-bundle/1.12.231/aws-java-sdk-bundle-1.12.231.jar --output /opt/bitnami/spark/jars/aws-java-sdk-bundle-1.12.231.jar RUN curl https://repo1.maven.org/maven2/net/java/dev/jets3t/jets3t/0.9.4/jets3t-0.9.4.jar --output /opt/bitnami/spark/jars/jets3t-0.9.4.jar RUN curl https://s3.amazonaws.com/redshift-downloads/drivers/jdbc/2.1.0.10/redshift-jdbc42-2.1.0.10.jar --output /opt/bitnami/spark/jars/redshift-jdbc42-2.1.0.10.jar RUN curl https://storage.googleapis.com/spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.28.0.jar --output /opt/bitnami/spark/jars/spark-bigquery-with-dependencies_2.12-0.28.0.jar
COPY ./requirements_copy.txt / RUN pip install -r /requirements_copy.txt
version: '3'
services:
spark:
image: spark-air:latest
environment:
- SPARK_MODE=master
- SPARK_RPC_AUTHENTICATION_ENABLED=no
- SPARK_RPC_ENCRYPTION_ENABLED=no
- SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
- SPARK_SSL_ENABLED=no
- AWS_ACCESS_KEY=${AWS_ACCESS_KEY}
- AWS_SECRET_KEY=${AWS_SECRET_KEY}
volumes:
- ./dags:/opt/bitnami/spark/dags/:rw
ports:
- '8090:8080'
spark-worker:
image: spark-air:latest
environment:
- SPARK_MODE=worker
- SPARK_MASTER_URL=spark://spark:7077
- SPARK_WORKER_MEMORY=1G
- SPARK_WORKER_CORES=1
- SPARK_RPC_AUTHENTICATION_ENABLED=no
- SPARK_RPC_ENCRYPTION_ENABLED=no
- SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no
- SPARK_SSL_ENABLED=no
- AWS_ACCESS_KEY=${AWS_ACCESS_KEY}
- AWS_SECRET_KEY=${AWS_SECRET_KEY}
volumes:
- ./dags:/opt/bitnami/spark/dags/:rw
from pyspark import SparkContext
sc = SparkContext("local", "First App")
data = [{"Category": 'A', "ID": 1, "Value": 121.44, "Truth": True},
{"Category": 'B', "ID": 2, "Value": 300.01, "Truth": False},
{"Category": 'C', "ID": 3, "Value": 10.99, "Truth": None},
{"Category": 'E', "ID": 4, "Value": 33.87, "Truth": True}]
df = sc.parallelize(data)
df = df.collect()
print(df)
23/02/15 05:32:58 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1090 bytes result sent to driver
23/02/15 05:32:58 DEBUG ExecutorMetricsPoller: stageTCMP: (0, 0) -> 0
23/02/15 05:32:58 DEBUG TaskSchedulerImpl: parentName: , name: TaskSet_0.0, runningTasks: 0
23/02/15 05:32:58 DEBUG TaskSetManager: No tasks for locality level NO_PREF, so moving to locality level ANY
23/02/15 05:32:58 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 1867 ms on d981929c6c2c (executor driver) (1/1)
23/02/15 05:32:58 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
23/02/15 05:32:58 INFO DAGScheduler: ResultStage 0 (collect at /opt/bitnami/spark/dags/test.py:12) finished in 3.275 s
23/02/15 05:32:58 DEBUG DAGScheduler: After removal of stage 0, remaining stages = 0
23/02/15 05:32:58 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
23/02/15 05:32:58 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
23/02/15 05:32:58 INFO DAGScheduler: Job 0 finished: collect at /opt/bitnami/spark/dags/test.py:12, took 3.794097 s
[{'Category': 'A', 'ID': 1, 'Value': 121.44, 'Truth': True}, {'Category': 'B', 'ID': 2, 'Value': 300.01, 'Truth': False}, {'Category': 'C', 'ID': 3, 'Value': 10.99, 'Truth': None}, {'Category': 'E', 'ID': 4, 'Value': 33.87, 'Truth': True}]
23/02/15 05:33:04 DEBUG ExecutorMetricsPoller: removing (0, 0) from stageTCMP
It is getting the expecting output but it is getting stuck after that for ever until i manually killing it. I want to integrate with airflow i can't manually kill it there.