Pyspark Streaming with Kafka in PyCharm - apache-spark

Question

I have been recently trying to debug the pyspark.streaming.kafka class in Pycharm so that it is easier to troubleshoot compared to working on that on the linux box.
Here is my sample code:
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils, TopicAndPartition
sc = SparkContext(appName="sample app")
ssc = StreamingContext(sc, 1)
kafkaParams = {"metadata.broker.list": "{broker list}",
"auto.offset.reset": "smallest"}
kafka_stream = KafkaUtils.createDirectStream(ssc, {topic list}, kafkaParams)
However, i got the error below:
Traceback (most recent call last):
File "C:\Program Files (x86)\JetBrains\PyCharm 5.0.3\helpers\pydev\pydevd.py", line 2411, in <module>
globals = debugger.run(setup['file'], None, None, is_module)
File "C:\Program Files (x86)\JetBrains\PyCharm 5.0.3\helpers\pydev\pydevd.py", line 1802, in run
launch(file, globals, locals) # execute the script
File "{script path}", line 30, in <module> {topic}], kafkaParams)
File "C:\spark-1.6.0-bin- hadoop2.6\python\lib\pyspark.zip\pyspark\streaming\kafka.py", line 152, in createDirectStream
py4j.protocol.Py4JJavaError: An error occurred while calling o20.loadClass.
: java.lang.ClassNotFoundException: org.apache.spark.streaming.kafka.KafkaUtilsPythonHelper
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
at py4j.Gateway.invoke(Gateway.java:259)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Unknown Source)
16/02/22 11:45:49 INFO SparkContext: Invoking stop() from shutdown hook
I would appreciate if someone can provide some guidance on how to debug the PySpark Kafka streaming module in PyCharm

Answer 1

Kafka support depends on external spark-streaming-kafka JAR which is not shipped with Spark binaries. Typically this can be specified on submit with --packages argument.
For local development using PyCharm the simplest solution I can think off is to add it to $SPARK_HOME/conf/spark-defaults.conf. Assuming you use Spark 1.6.0 built with Scala 2.10:
spark.jars.packages org.apache.spark:spark-streaming-kafka_2.10:1.6.0
Keep in mind that you won't be able to use PyCharm debugger with Python worker process. See How can pyspark be called in debug mode?