Is it good practice to open several Kafka streams in one Spark Context? - apache-spark

Question

We have several applications that follow the same logic and patterns, and would like to know if it's good practice to open several streams in one spark context. So the main application to submit, would have something of this sort;
SparkConf conf = new SparkConf().setMaster("local[*]").setAppName("test-app");
conf.set("log4j.configuration", "\\log4j.properties");
JavaStreamingContext ssc = new JavaStreamingContext(conf, new Duration(20));
// Iterate streams
for (RealtimeApplication app : realtimeApplications)
{
app.execute(ssc);
}
// Trigger!
ssc.start();
// Await stopping of the service...
ssc.awaitTermination();
Then, in the implementation of the abstract method execute(JavaStreamingContext ssc) you would have the following code...
JavaPairReceiverInputDStream<String, String> kafkaStream = KafkaUtils.createStream(ssc, this.getZkQuorum(), this.getSparkGroup(), topicsSet);
JavaDStream<String> lines = kafkaStream.map(new Function<Tuple2<String, String>, String>() {
#Override
public String call(Tuple2<String, String> tuple2) {
// Extract transation
String value = tuple2._2();
// Do something here...
String result = executeSomething(value);
return result;
}
});
Is this something to be considered wrong in Spark development?

Answer 1

I would rather share your logic through RDD like
JavaDStream<String> lines1 = kafkaStream.map(new Function<Tuple2<String, String>, String>() {...});
JavaDStream<String> lines2 = kafkaStream.map(new Function<Tuple2<String, String>, String>() {...});
JavaDStream<String> lines3 = kafkaStream.map(new Function<Tuple2<String, String>, String>() {...});
With one source stream