As I understand, Spark can analyze streams with Spark Streaming.
And Kafka can receive data from multiple sources.
What I don't understand is, if i have a Kafka cluster receiving data from multiple sources, will the data be send to a database with Spark Streaming running? Or is Spark Streaming running on a application server?
If you use Spark Streaming, you need to set up a Spark cluster and you will submit you Spark Streaming job to the cluster. Thus, you will have to 2 cluster: Kafka + Spark (or actually 3, as you also need a Zookeeper cluster for Kafka).
Related
I am trying to calculate Kafka lag on my spark structured streaming application.
I can get the current processed offset from my Kafka metadata which comes along with actual data.
Is there a way through which we can get the latest offsets of all partitions in a Kafka topic programmatically from spark interface ?
Can I use Apache Kafka admin classes or Kafka interfaces to get the latest offset information for each batch in my spark app ?
I want to send my data from kafka to Spark.
I have installed spark in my system and kafka is also working in my system in proper way.
You need to use a Kafka connector from Spark. Technically, Kafka won't send the data to Spark. In fact, Spark pull the data from Kafka.
Here the link from the documentation : https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html
I have a Spark 2.0.2 structured streaming job connecting to Apache Kafka data stream as the source. The job takes in Twitter data (JSON) from Kafka and uses CoreNLP to annotate the data with things like sentiment, parts of speech tagging etc.. It works well with a local[*] master. However, when I setup a stand alone Spark cluster, only one worker gets used to process the data. I have two workers with the same capability.
Is there something I need to set when submitting my job that I'm missing. I've tried setting the --num-executors in my spark-submit command but I have had no luck.
Thanks in advance for the pointer in the right direction.
I ended up creating the kafka source stream with more partitions. This seems to have sped up the processing part 9 folds. Spark and kafka have a lot of knobs. Lots to sift through... See Kafka topic partitions to Spark streaming
I've integrated kafka and spark streaming after downloading from the apache website. However, I wanted to use Datastax for my Big Data solution and I saw you can easily integrate Cassandra and Spark.
But I can't see any kafka modules in the latest version of Datastax enterprise. How to integrate kafka with spark streaming here?
What I want to do is basically:
Start necessary brokers and servers
Start kafka producer
Start kafka consumer
Connect spark streaming to kafka broker and receive the messages from there
However after a quick google search, I can't see anywhere that kafka has been incorporated with datastax enterprise.
How can I achieve this? I'm really new to datastax and kafka and all so I need some advice. Language preference- Python.
Thanks!
Good question. DSE does not incorporate Kafka out of the box, you must set up kafka yourself and then set up your spark streaming job to read from kafka. Since DSE does bundle spark, use DSE Spark to run your spark streaming job.
You can use either the direct kafka API or kafka receivers, more details here on the tradeoffs. TL;DR direct api does not require WAL or zookeeper for HA.
Here is an example of how you can configure Kafka to work with DSE by Cary Bourgeois:
https://github.com/CaryBourgeois/DSE-Spark-Streaming/tree/master
Has anyone been using apache spark on multi-region?
We are building an application that must be multi-region deployed. Our stack is basically Scala, Spark, Cassandra and Kafka. The main goal is to use Spark streaming with Kafka and insert it on Cassandra.
Reading the Spark documentation, Zookeeper is needed to keep high availability as well as in Kafka.
The question is: Should I consider keep a spark cluster on each region or should I use like cassandra? Since it depends on zookeeper to keep high availability on master nodes, how about that? The same applies to zookeeper or not?