Spark streaming socket stream exemple not working - apache-spark

Question

I'm trying to use Spark Streaming but I'm stuck at the frist exemple:
import java.util.Arrays;
import org.apache.spark.*;
import org.apache.spark.api.java.function.*;
import org.apache.spark.streaming.*;
import org.apache.spark.streaming.api.java.*;
import scala.Tuple2;
public class NetworkWordCount {
public static void main(String[] args) {
// Create a local StreamingContext with two working thread and batch interval of 1 second
SparkConf conf = new SparkConf().setMaster("local[4]").setAppName("NetworkWordCount");
JavaStreamingContext jssc = new JavaStreamingContext(conf, Durations.seconds(1));
JavaReceiverInputDStream<String> lines = jssc.socketTextStream("localhost", 9999);
JavaDStream<String> words = lines.flatMap(
new FlatMapFunction<String, String>() {
#Override public Iterable<String> call(String x) {
return Arrays.asList(x.split(" "));
}
});
// Count each word in each batch
JavaPairDStream<String, Integer> pairs = words.mapToPair(
new PairFunction<String, String, Integer>() {
#Override public Tuple2<String, Integer> call(String s) {
return new Tuple2<String, Integer>(s, 1);
}
});
JavaPairDStream<String, Integer> wordCounts = pairs.reduceByKey(
new Function2<Integer, Integer, Integer>() {
#Override public Integer call(Integer i1, Integer i2) {
return i1 + i2;
}
});
// Print the first ten elements of each RDD generated in this DStream to the console
wordCounts.print();
jssc.start(); // Start the computation
jssc.awaitTermination();
}
}
This piece of code is literally a copy past of the documentation.
https://spark.apache.org/docs/1.6.3/streaming-programming-guide.html
I set up one netcat server like:
nc -lk 9999
and obne nc client like:
nc localhost 9999
In which I type sentence such as:
Hello world !
Hellow world ! \n
That appears properly on the netcat server.
And yet it's not working. At every batch I just have an empty print.
21/02/16 00:36:41 INFO SocketInputDStream: Removing blocks of RDD BlockRDD[137] at socketTextStream at NetworkWordCount.java:17 of time 1613432201000 ms
21/02/16 00:36:41 INFO ReceivedBlockTracker: Deleting batches ArrayBuffer(1613432199000 ms)
21/02/16 00:36:41 INFO InputInfoTracker: remove old batch metadata: 1613432199000 ms
-------------------------------------------
Time: 1613432201000 ms
-------------------------------------------
I tried different value for the setMaster such as local[4], local[2] and local[*] but the result is the same.
Also if I run the spark streaming code before my netcat client, I can't even see the strings on the nc server.

Answer 1

I found the solution for my problem.
In short you need to write the messages directly in the terminal running the TCP server, you don't need another netcat client.
There is no bug or missing configuration it was just a misunderstanding of how netcat is working.
I understood from the man nc than the -k option allow netcat to manage multiple connexion but it was partly wrong.
-k When a connection is completed, listen for another one. Requires -l. When used together with the -u option, the server socket is not connected and it can receive UDP datagrams from multiple
hosts.
But it indeed doesn't mean what I thought at all. If you setup a netcat server with the -k option then it will accept multiple connection but will still process one at a time.
Which means that if you have 2 nc client, and that if you type some text into both of the client, only the text from one of them will be received by the server until you close this first connection.