Apache Kafka + Spark Integration (REST API is needed?) - apache-spark

Question

Got some fundamental problems, hope someone can clear them up.
So I want to use Apache Kafka and Apache spark for my application. I have gone through numerous tutorials and got the basic idea of what it is and how it will work.
Use case :
Data will be generated from a mobile device(multiple devices, lets say 1000) at an interval of 40 sec and I need to process that data and add values to the database which in turn will be reflected back in a dashboard.
What I wanted to do is to use Apache Streams and make a post request from android itself and then those data will be processed by the spark application and that's it.
Issues:
Apache Spark
I am following this tutorial to get it up and running.( Am using JAVA, not scala)
Link : https://www.santoshsrinivas.com/installing-apache-spark-on-ubuntu-16-04/
After everything is done, I execute spark-shell and it start. I have also installed zookeeper and kafka on my server and I have started the Kafka in the background, so that's not an issue.
When I run http://161.xxx.xxx.xxx:4040/jobs/ I get this page
In all the tutorial which I have gone through, there is a page like this : https://i.stack.imgur.com/gF1fN.png but I don't get this. Is it that spark is not properly installed?
Now when I want to deploy a standalone jar to spark, (Using this link : http://data-scientist-in-training.blogspot.in/2015/03/apache-spark-cluster-deployment-part-1.html ) am able to run it.
i.e with the command : spark-submit --class SimpleApp.SimpleApp --master spark://http://161.xxx.xxx.xxx:7077 --name "try" /opt/spark/bin/try-0.0.1-SNAPSHOT.jar , I get the output.
Do I need to submit the application everytime if I want to use it?
This is my Program :
package SimpleApp;
/* SimpleApp.java */
import org.apache.spark.api.java.*;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.function.Function;
public class SimpleApp {
public static void main(String[] args) {
String logFile = "/opt/spark/README.md"; // Should be some file on your system
SparkConf conf = new SparkConf().setAppName("Simple Application").setMaster("local[*]");
JavaSparkContext sc = new JavaSparkContext(conf);
//System.setProperty("hadoop.home.dir", "C:/winutil");
sc.setLogLevel("ERROR"); // Don't want the INFO stuff
JavaRDD<String> logData = sc.textFile(logFile).cache();
long numAs = logData.filter(new Function<String, Boolean>() {
public Boolean call(String s) { return s.contains("a"); }
}).count();
long numBs = logData.filter(new Function<String, Boolean>() {
public Boolean call(String s) { return s.contains("b"); }
}).count();
System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs);
System.out.println("word count : "+logData.first());
sc.stop();
}
}
Now how do I integrate Kafka into it?
How to configure the app in such a way that it get executed everytime
kafka receives a message?
Moreover, do I need to make a REST API through which I need to send
the data to kafka i.e the REST api will be used as producer?
Something like spark Java framework? http://sparkjava.com/
If yes, again the bottleneck will happen at REST api level i.e how
many request it can handle or not because everywhere I read that
Kafka has a very high throughput.
Is the final structure going to be like SPARK JAVA -> KAFKA -> APACHE
SPARK ?
Lastly how to do I set up the development structure on my local
device? I have kafka/apache spark installed. And am using Eclipse.
Thanks

Answer 1

Well,
You are facing some problems to understand how Spark works with Kafka.
First let's understand somethings:
Kafka is a Stream process platform for low latency and high throughput. This will allow you to store and read lot's of data really fast.
Spark has two types of processing, Spark Batch and Spark Streaming. What you are studying is batch, for your problem I suggest you to see apache streaming.
What is Streaming?
Streaming is a way to transport and transform your data in real time or near real time. It will not be necessary to create a process that you need to call every 10 minutes or every 10 seconds. You will start the job and it will consume the source and will post in the sink.
Kafka is a passive platform, so Kafka can be a source or a sink of a stream process.
In your case, what I suggest is:
Create a streaming producer for your Kafka, you will read the log of your mobile application in your web server. So, you need to plug something at your web server to start the consumption of the data. What I suggest you is FluentdIs a really strong application for streaming, this is in Ruby but is really easy to use. If you want something more robust and more focused in BigData I suggest Apache Nifi This is hard to work, that is not easy but you can create pipelines of data flow to transfer your information to your cluster. And something REALLY SIMPLE and that will solve your problem is Apache Flume.
Start your Kafka, you can use Docker to use it. This will hold your data for a period, and will allow you to take your data when you need really fast and with a lot of information. Please read the docs to understand how it works.
Spark Streaming - That will not make sense to use a Kafka if you don't have a stream process, your solution of Rest to produce the data at Kafka is slow and if is batch doesn't make sense. So if you are writing as streaming, you should analyse as streaming too. I suggest you to read about Spark Streaming here. And how integrate the Spark with Kafka here.
So, as you asked:
Do I need a REST API?
The answer is No.
The architecture will be like this:
Web Server -> Fluentd -> Apache Kafka -> Spark Streaming -> Output
I hope that will help