Publish Apache Spark result to another Application/Kafka - apache-spark

Question

I am currently designing a fast data aggregation module which receives events and publish them to Kafka cluster. Then we have an integration of Kafka and Spark Streaming. Spark Streaming reads stream from Kafka and executes some computation. When the computation is done, we need to send the result to another application. This application could be a web service or a Kafka cluster.
I am wondering how we can do this? From what I've read, Spark Stream pushes the data to downstream like Databases and file systems.
How would you go with design such an application? Should I replace Spark Stream with Storm to be able to publish the results to another application?

Answer 1

Please refer to dstream.foreachRDD, which is a powerful primitive that allows data to be sent out to external systems.
Design Patterns for using foreachRDD
Below is my kafka integration code for your reference(not optimized, just for POC, KafkaProducer object could be re-used in foreachRDD):
DStream.foreachRDD(rdd => {
rdd.foreachPartition { partitionOfRecords =>
val kafkaProps = new Properties()
kafkaProps.put("bootstrap.servers", props("bootstrap.servers"))
kafkaProps.put("client.id", "KafkaIntegration Producer");
kafkaProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
kafkaProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
val producer = new KafkaProducer[String, String](kafkaProps);
partitionOfRecords.foreach(record => {
val message = new ProducerRecord[String, String]("hdfs_log_test", record.asInstanceOf[String])
producer.send(message)
})
producer.close()
}
})

Answer 2

I am wondering how we can do this? From what I've read, Spark Stream pushes the data to downstream like Databases and file systems.
Spark is not limited to HDFS or Databases, you're free to initialize a connection to any external resource which is available. It can be back to Kafka, RabbitMQ or a WebService.
If you're doing simple transformation like map, filter, reduceByKey etc, then using DStream.foreachRDD will do fine. If you'll be doing stateful computations like DStream.mapWithState, then once you're done processing the state you can you can simply send the data to any external service.
For example, we're using Kafka as an input stream of data, and RabbitMQ and an output after doing some stateful computations.