Spark Streaming for evaluating rules - apache-spark

Question

I am planning to implement Spark Streaming for evaluating rules in real time instead of doing it from a database.
I have data ingested into Kafka topic. Data ingested is about user and his actions. I have rules like identifying users doing more than transactions 100 in 10 mins, identify user doing 0 transaction in last 5 mins.
The time interval(window) and rules vary and to some extent. I have around 100 such rules. As we specify the window when we create a stream do we need to create as many stream as the number of rules. I am not sure is it a right approach and Spark Streaming fits for this use case. As I am new Spark, I am looking for inputs to solve this efficiently.
Please find the sample code below with rules. These rules will configured by Analyst and there can be 50 rules and the time period can vary based on analyst requirements from 1 minute to 1 hour. If i have 10 different time lines do i need to create 10 different sliding windows with this approach. Can you please let me know if this is an right approach.
package com.spark.play;
import java.util.Arrays;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Map;
import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.function.Function;
import org.apache.spark.sql.DataFrame;
import org.apache.spark.sql.SQLContext;
import org.apache.spark.streaming.Durations;
import org.apache.spark.streaming.StreamingContext;
import org.apache.spark.streaming.api.java.JavaDStream;
import org.apache.spark.streaming.api.java.JavaPairDStream;
import org.apache.spark.streaming.api.java.JavaPairInputDStream;
import org.apache.spark.streaming.api.java.JavaStreamingContext;
import org.apache.spark.streaming.kafka.KafkaUtils;
import kafka.serializer.StringDecoder;
import scala.Tuple2;
public class SparkUserStreaming {
public static void main(String args[]){
System.out.println("Start User streaming App for Policy...");
SparkConf sparkConf = new SparkConf().setAppName("Policy").setMaster("local[2]").setJars(
JavaStreamingContext.jarOfClass(SparkUserStreaming.class)).setSparkHome("SPARK_HOME");
SparkContext sc = new SparkContext(sparkConf);
StreamingContext streamContex = new StreamingContext(sc, Durations.seconds(60));
JavaStreamingContext jssc = new JavaStreamingContext(streamContex);
Map<String, Integer> topicMap = new HashMap<String, Integer>();
topicMap.put("user", 1);
String brokers = "vchenst:9092";
String topics = "user";
HashSet<String> topicsSet = new HashSet<>(
Arrays.asList(topics.split(",")));
HashMap<String, String> kafkaParams = new HashMap<>();
kafkaParams.put("metadata.broker.list", brokers);
JavaPairInputDStream<String, String> dstream =
KafkaUtils.createDirectStream(
jssc,
String.class,
String.class,
StringDecoder.class,
StringDecoder.class,
kafkaParams,
topicsSet
);
JavaPairDStream<String, String> mesages = dstream.window(Durations.seconds(300), Durations.seconds(60));
JavaDStream<User> streamData = mesages.map(new Function<Tuple2<String, String>, User>() {
public User call(Tuple2<String, String> tuple2) throws Exception {
User user = new User();
//System.out.println("Line ::" + tuple2._1());
String[] dataArray = tuple2._2().split(",");
//user:arul,product:mobile,country:india,state:tn,price:1000,txid:12300086,datetime:2009-01-16 16:47:08
user.setName(dataArray[0]);
user.setProduct(dataArray[1]);
user.setCountry(dataArray[2]);
user.setState(dataArray[3]);
user.setPrice(Double.parseDouble(dataArray[4]));
user.setTxId(dataArray[5]);
user.setDateTime(dataArray[6]);
return user;
}
});
final SQLContext sqlContext = new SQLContext(sc);
streamData.foreachRDD(new Function<JavaRDD<User>, Void>() {
public Void call(JavaRDD<User> rdd) {
DataFrame userDf = sqlContext.createDataFrame(rdd, User.class);
//userDf.show();
//Rules applied for Five Minutes
//Rule 1 - Print Users making more than 2 transactions
userDf.groupBy("txId").count()
.withColumnRenamed("count", "n")
.filter("n >= 2")
.show();
//Rule 2 - Print User making tx with cost of more than 1000
userDf.filter(userDf.col("price").gt(1000)).show();
//Rule 3 - Print State in which max transaction taken place in Last 5 mins
userDf.groupBy("state").count();
return null;
}
});
jssc.start();
jssc.awaitTermination();
}
}

Answer 1

try to broadcast the rules to every spark stream and evaluate data from those rules .
This might help , if still not clear then let me know ill explain the architecture.