Spark streaming with kafka(ssl enabled) - apache-spark

Question

I have seen multiple questions regarding this topic but unable to find an answer for it? I have tried to add comments and have not got any response yet.
Requirement -> spark-streaming with Kafka SSL enabled
Below are the version details
I am running spark locally.
Spark -> version 2.4.6
kafka version -> 2.2.1
code snippet:
import sys
import logging
from datetime import datetime
try:
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils
sc = SparkContext(appName="PythonStreamingDirectKafkaWordCount")
ssc = StreamingContext(sc, 20)
brokers = "b1:p,b2:p"
topic = "topic1"
kafkaParams = {"security.protocol":"SSL","metadata.broker.list": brokers}
kvs = KafkaUtils.createDirectStream(ssc, [topic],kafkaParams)
lines = kvs.map(lambda x: x[1])
print(lines)
counts = lines.flatMap(lambda line: line.split(" ")) \
.map(lambda word: (word, 1)) \
.reduceByKey(lambda a, b: a+b)
counts.pprint()
ssc.start()
ssc.awaitTermination()
except ImportError as e:
print("Error importing Spark Modules :", e)
sys.exit(1)
When I am submitting it as below.
spark-submit --packages org.apache.spark:spark-streaming-kafka-0-10_2.10:2.0.0 --master local pysparkKafka.py
This is due to the reference I took from this
Enabling SSL between Apache spark and Kafka broker
I am getting the below error
Spark Streaming's Kafka libraries not found in class path. Try one of the following.
1. Include the Kafka library and its dependencies with in the
spark-submit command as
$ bin/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8:2.4.6 ...
2. Download the JAR of the artifact from Maven Central http://search.maven.org/,
Group Id = org.apache.spark, Artifact Id = spark-streaming-kafka-0-8-assembly, Version = 2.4.6.
Then, include the jar in the spark-submit command as
$ bin/spark-submit --jars <spark-streaming-kafka-0-8-assembly.jar> ...
Now when I submit it as below. This is due to seeing the above error and picking a jar from maven.
spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.0.2 --master local pysparkKafka.py
I am getting the below error with the SSL port of the brokers.
20/07/24 11:58:46 INFO VerifiableProperties: Verifying properties
20/07/24 11:58:46 INFO VerifiableProperties: Property group.id is overridden to
20/07/24 11:58:46 WARN VerifiableProperties: Property security.protocol is not valid
20/07/24 11:58:46 INFO VerifiableProperties: Property zookeeper.connect is overridden to
20/07/24 11:58:46 INFO SimpleConsumer: Reconnect due to socket error: java.io.EOFException: Received -1 when reading from channel, socket has likely been closed.
20/07/24 11:58:47 INFO SimpleConsumer: Reconnect due to socket error: java.io.EOFException: Received -1 when reading from channel, socket has likely been closed.
Traceback (most recent call last):
Note -> The above submit works well without SSL. But my requirement is to enable SSL.
Any help is greatly appreciated. Thanks.