Invoking a java function after each micro-batch in Spark streaming - apache-spark

Question

The below simple program reads from kafka stream and writes to CSV file every 5 mins, and its spark streaming. Is there a way I can invoke a Java function after micro-batch in the "DRIVER PROGRAM" ( not in executor ).
I agree its not a good practice to call the arbitrary code in the stream, but this is special case where we have low volume data. please adivse. Thanks.
public static void main(String[] args) throws Exception {
if (args.length == 0)
throw new Exception("Usage program configFilename");
String configFilename = args[0];
addShutdownHook();
ConfigLoader.loadConfig(configFilename);
sparkSession = SparkSession
.builder()
.appName(TestKafka.class.getName())
.master(ConfigLoader.getValue("master")).getOrCreate();
SparkContext context = sparkSession.sparkContext();
context.setLogLevel(ConfigLoader.getValue("logLevel"));
SQLContext sqlCtx = sparkSession.sqlContext();
System.out.println("Spark context established");
DataStreamReader kafkaDataStreamReader = sparkSession.readStream()
.format("kafka")
.option("kafka.bootstrap.servers", ConfigLoader.getValue("brokers"))
.option("group.id", ConfigLoader.getValue("groupId"))
.option("subscribe", ConfigLoader.getValue("topics"))
.option("failOnDataLoss", false);
Dataset<Row> rawDataSet = kafkaDataStreamReader.load();
rawDataSet.printSchema();
rawDataSet.createOrReplaceTempView("rawEventView1");
rawDataSet = rawDataSet.withColumn("rawEventValue", rawDataSet.col("value").cast("string"));
rawDataSet.printSchema();
rawDataSet.createOrReplaceTempView("eventView1");
sqlCtx.sql("select * from eventView1")
.writeStream()
.format("csv")
.option("header", "true")
.option("delimiter", "~")
.option("checkpointLocation", ConfigLoader.getValue("checkpointPath"))
.option("path", ConfigLoader.getValue("recordsPath"))
.outputMode(OutputMode.Append())
.trigger(ProcessingTime.create(Integer.parseInt(ConfigLoader.getValue("kafkaProcessingTime"))
, TimeUnit.SECONDS))
.start()
.awaitTermination();
}

Answer 1

You should be able to achieve this by something like this:
kafkaDataStreamReader.map{value -> mySideEffect(); value}
This will call the function mySideEffect every time a microbatch is received from kafka, how ever I would not recommend doing this, a better approach would be to watch the folder you persist the CSV it or simply checking the web ui, considering a micro batch happens every few seconds at most you will be swamped by emails. if you want to make sure the streaming application is up you can query the spark REST API every few seconds and make sure it's still up
https://spark.apache.org/docs/latest/monitoring.html