Print RDD out to console in spark streaming - apache-spark

Question

I write a spark streaming application to receive data from Kafka by using KafkaUtils, and what I want to do is to print out data I received from Kafka. Here is my code(I use spark-submit to execute my spark streaming job):
val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topicSet)
messages.print()
When I run this, it works pretty fine. If the input is a,b,c in Kafka producer, I can get the result from Spark streaming as below:
Time: 1476481700000 ms
-------------------------------------------
(null,a)
(null,b)
(null,c)
But if I add one line to count the number of lines, messages.print() cannot work. Codes are shown below:
val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topicSet)
messages.print()
messages.count().print()
I am getting the following result:
-------------------------------------------
Time: 1476481800000 ms
-------------------------------------------
4
Only count number is getting printed out, and data cannot be printed out.
My question is why messages.print() would not be executed after I add messages.count.print().
Another question is what null stands for in the tuple (null, a)(null, b)(null, c).

Answer 1

There is no issue with print() and it will print both messages and count like below. Scroll and check your log.
-------------------------------------------
Time: 1476481700000 ms
-------------------------------------------
(null,a)
(null,b)
(null,c)
-------------------------------------------
Time: 1476481800000 ms
-------------------------------------------
4
KafkaUtils.createDirectStream method returns DStream of <Kafka topic, Kafka message>. Check this and this post related to topic is null.

Answer 2

Your code should be working but giving you an alternative.But this approach is only meant for testing or learning. Instead of performing two actions , you can achieve the end goal with just single action
val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topicSet)
//Cache your RDD before you perform any heavyweight operations.
messages.cache()
val result = messages.collect();
println(result.size + " size")
result.foreach { input => println(input) }