I have to write data from Spark Structure streaming as JSON Array, I have tried using below code:
df.selectExpr("to_json(struct(*)) AS value").toJSON
which returns me DataSet[String], but unable to write as JSON Array.
Current Output:
{"name":"test","id":"id"}
{"name":"test1","id":"id1"}
Expected Output:
[{"name":"test","id":"id"},{"name":"test1","id":"id1"}]
Edit (moving comments into question):
After using proposed collect_list method I am getting
Exception in thread "main" org.apache.spark.sql.AnalysisException: Append output mode not supported when there are streaming aggregations on streaming DataFrames/DataSets without watermark;
Then I tried something like this -
withColumn("timestamp", unix_timestamp(col("event_epoch"), "MM/dd/yyyy hh:mm:ss aa")) .withWatermark("event_epoch", "1 minutes") .groupBy(col("event_epoch")) .agg(max(col("event_epoch")).alias("timestamp"))
But I don't want to add a new column.
You can use the SQL built-in function collect_list for this. This function collects and returns a set of non-unique elements (compared to collect_set which returns only unique elements).
From the source code for collect_list you will see that this is an aggregation function. Based on the requirements given in the Structured Streaming Programming Guide on Output Modes it is highlighted that the output modes "complete" and "updated" are supported for aggregations without a watermark.
As I understand from your comments, you do not wish to add watermark and new columns. Also, the error you are facing
Exception in thread "main" org.apache.spark.sql.AnalysisException: Append output mode not supported when there are streaming aggregations on streaming DataFrames/DataSets without watermark;
reminds you to not use the output mode "append".
In the comments, you have mentioned that you plan to produce the results into a Kafka message. One big JSON Array as one Kafka value. The complete code could look like
val df = spark.readStream
.[...] // in my test I am reading from Kafka source
.load()
.selectExpr("CAST(key AS STRING) as key", "CAST(value AS STRING) as value", "offset", "partition")
// do not forget to convert you data into a String before writing to Kafka
.selectExpr("CAST(collect_list(to_json(struct(*))) AS STRING) AS value")
df.writeStream
.format("kafka")
.outputMode("complete")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("topic", "test")
.option("checkpointLocation", "/path/to/sparkCheckpoint")
.trigger(Trigger.ProcessingTime(10000))
.start()
.awaitTermination()
Given the key/value pairs (k1,v1), (k2,v2), and (k3,v3) as inputs you will get a value in the Kafka topic that contains all selected data as a JSON Array:
[{"key":"k1","value":"v1","offset":7,"partition":0}, {"key":"k2","value":"v2","offset":8,"partition":0}, {"key":"k3","value":"v3","offset":9,"partition":0}]
Tested with Spark 3.0.1 and Kafka 2.5.0.
Related
Need to validate each row of Streaming Dataframe (consumed through readStream(kafka) - Getting error
Queries with streaming sources must be executed with writeStream.start()
as it is not allowing to validate row by row
I have created spark batch job to consume data from Kafka , validated each row against HBase data, another set of validations based on rowkey and created a dataframe out of it. But here I need to handle the Kafka offset manually in the code.
To avoid the offset handling, am trying to use spark structural Streaming but there am not able to validate row by row.
writestream.foreach (foreachwriter) is good to sink with any external datasource or writing to kafka.
But in my case, I need to validate each row and create a new dataframe based on my validation. foreachwriter - process is not allowing to collect the data using other external classes/list.
Errors:
Getting this error when I tried to access the streaming dataframe to validate
Queries with streaming sources must be executed with writeStream.start();
Task is not serializable when I tried to create a list out of foreach(foreachwriter extended object). Will update with more details (as I am trying other options)
I am trying to achieve spark batch using writerstream.trigger(Trigger.once) with checkpointlocation
Updating with Spark batch and Structural Streaming Code.
.read
.format("kafka")
.option("kafka.bootstrap.servers", kafkaBootStrap)
.option("subscribePattern", kafkaSubTopic)
.option("startingOffsets", "earliest")
//.option("endingOffsets", "latest")
.load()
rawData.collect.foreach(row => {
if (dValidate.dValidate(row)) {
validatedCandidates += (row.getString(0))
}
==================== in the above code I need to handle the offset manually for rerun so decided to use structural streaming.============
val rawData = spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", kafkaBootStrap)
.option("subscribe", kafkaSubTopic)
.option("enable.auto.commit", "true")
.option("startingOffsets","latest")
.option("minPartitions", "10")
.option("failOnDataLoss", "true")
.load()
val sInput = new SinkInput(validatedCandidates,dValidate)
rawData.writeStream
.foreach(sInput)
.outputMode(OutputMode.Append())
.option("truncate", "false")
.trigger(Trigger.Once())
.start()
am getting "Task not serialized" error in here.
with class SinkInput , I am trying to do the same collect operation with external dValidate instance
Unless I misunderstood your case, rawData is a streaming query (a streaming Dataset) and does not support collect. The following part of your code is not correct:
rawData.collect
That's not supported and hence the exception.
You should be using foreach or foreachBatch to access rows.
Do this instead:
rawData.write.foreach(...)
I want to write a structured spark streaming Kafka consumer which reads data from a one partition Kafka topic, repartitions the incoming data by "key" to 3 spark partitions while keeping the messages ordered per key, and writes them to another Kafka topic with 3 partitions.
I used Dataframe.repartition(3, $"key") which I believe uses HashPartitioner. Code is provided below.
When I executed the query with fixed-batch interval trigger type, I visually verified the output messages were in the expected order. My assumption is that order is not guaranteed on the resulting partition. I am looking to receive some affirmation or veto on my assumption in terms of code pointers in the spark code repo or documentation.
I also tried using Dataframe.sortWithinPartitions, however this does not seem to be supported on streaming data frame without aggregation.
One option I tried was to convert the Dataframe to RDD and apply repartitionAndSortWithinPartitions which repartitions the RDD according to the given partitioner and, within each resulting partition, sort records by their keys. However, then I cannot use this RDD in the query.writestream operation to write the result in the output Kafka topic.
Is there a data frame repartitioning API that helps sort the repartitioned data in the streaming context?
Are there any other alternatives?
Does the default trigger type or fixed-interval trigger type for micro-batch execution provide any sort of message ordering guarantees?
Incoming data:
case class KVOutput(key: String, ts: Long, value: String, spark_partition: Int)
val df = spark.readStream.format("kafka")
.option("kafka.bootstrap.servers", kafkaBrokers.get)
.option("subscribe", Array(kafkaInputTopic.get).mkString(","))
.option("maxOffsetsPerTrigger",30)
.load()
val inputDf = df.selectExpr("CAST(key AS STRING)","CAST(value AS STRING)")
val resDf = inputDf.repartition(3, $"key")
.select(from_json($"value", schema).as("kv"))
.selectExpr("kv.key", "kv.ts", "kv.value")
.withColumn("spark_partition", spark_partition_id())
.select($"key", $"ts", $"value", $"spark_partition").as[KVOutput]
.sortWithinPartitions($"ts", $"value")
.select($"key".cast(StringType).as("key"), to_json(struct($"*")).cast(StringType).as("value"))
val query = resDf.writeStream
.format("kafka")
.option("kafka.bootstrap.servers", kafkaBrokers.get)
.option("topic", kafkaOutputTopic.get)
.option("checkpointLocation", checkpointLocation.get)
.start()
When I submit this application, it fails with
8/11/08 22:13:20 ERROR ApplicationMaster: User class threw exception: org.apache.spark.sql.AnalysisException: Sorting is not supported on streaming DataFrames/Datasets, unless it is on aggregated DataFrame/Dataset in Complete output mode;;
This question already has an answer here:
How to read records in JSON format from Kafka using Structured Streaming?
(1 answer)
Closed 5 years ago.
I am using Spark Structured streaming to process data from Kafka. I transform each message to JSON. However, spark needs an explicit schema to obtain columns from JSON. Spark Streaming with DStreams allows doing following
spark.read.json(spark.createDataset(jsons))
where jsons is RDD[String].
In case of Spark Structured Streaming similar approach
df.sparkSession.read.json(jsons)
(jsons is DataSet[String])
results to the following exception
Exception in thread "main" org.apache.spark.sql.AnalysisException: Queries with streaming sources must be executed with writeStream.start();;
I assume that read triggers execution instead of start, but is there a way to bypass this?
To stream from JSON on Kafka to a DataFrame you need to do something like this:
case class Colour(red: Int, green: Int, blue: Int)
val colourSchema: StructType = new StructType()
.add("entity", "int")
.add("security", "int")
.add("client", "int")
val streamingColours: DataFrame = spark
.readStream
.format("kafka")
.load()
.selectExpr("CAST(value AS STRING)")
.select(from_json($"value", colourSchema))
streamingColours
.writeStream
.outputMode("complete")
.format("console")
.start()
This should create a streaming DataFrame, and show the results of reading from Kafka on the console.
I do not believe it is possible to use "infer schema" with streaming data sets. And this makes sense, since infer schema looks at a large set of data to work out what the types are etc. With streaming datasets the schema that could be inferred by processing the first message might be different to the schema of the second message, etc. And Spark needs one schema for all elements of the DataFrame.
What we have done in the past is to process a batch of JSON messages with Spark's batch processing and using infer schema. And then export that schema for use with streaming datasets.
I read Spark Structured Streaming doesn't support schema inference for reading Kafka messages as JSON. Is there a way to retrieve schema the same as Spark Streaming does:
val dataFrame = spark.read.json(rdd.map(_.value()))
dataFrame.printschema
Here is one possible way to do this:
Before you start streaming, get a small batch of the data from Kafka
Infer the schema from the small batch
Start streaming the data using the extracted schema.
The pseudo-code below illustrates this approach.
Step 1:
Extract a small (two records) batch from Kafka,
val smallBatch = spark.read.format("kafka")
.option("kafka.bootstrap.servers", "node:9092")
.option("subscribe", "topicName")
.option("startingOffsets", "earliest")
.option("endingOffsets", """{"topicName":{"0":2}}""")
.load()
.selectExpr("CAST(value AS STRING) as STRING").as[String].toDF()
Step 2:
Write the small batch to a file:
smallBatch.write.mode("overwrite").format("text").save("/batch")
This command writes the small batch into hdfs directory /batch. The name of the file that it creates is part-xyz*. So you first need to rename the file using hadoop FileSystem commands (see org.apache.hadoop.fs._ and org.apache.hadoop.conf.Configuration, here's an example https://stackoverflow.com/a/41990859) and then read the file as json:
val smallBatchSchema = spark.read.json("/batch/batchName.txt").schema
Here, batchName.txt is the new name of the file and smallBatchSchema contains the schema inferred from the small batch.
Finally, you can stream the data as follows (Step 3):
val inputDf = spark.readStream.format("kafka")
.option("kafka.bootstrap.servers", "node:9092")
.option("subscribe", "topicName")
.option("startingOffsets", "earliest")
.load()
val dataDf = inputDf.selectExpr("CAST(value AS STRING) as json")
.select( from_json($"json", schema=smallBatchSchema).as("data"))
.select("data.*")
Hope this helps!
It is possible using this construct:
myStream = spark.readStream.schema(spark.read.json("my_sample_json_file_as_schema.json").schema).json("my_json_file")..
How can this be? Well, as the spark.read.json("..").schema returns exactly a wanted inferred schema, you can use this returned schema as an argument for the mandatory schema parameter of spark.readStream
What I did was to specify a one-liner sample-json as input for inferring the schema stuff so it does not unnecessary take up memory. In case your data changes, simply update your sample-json.
Took me a while to figure out (constructing StructTypes and StructFields by hand was pain in the ..), therefore I'll be happy for all upvotes :-)
It is not possible. Spark Streaming supports limited schema inference in development with spark.sql.streaming.schemaInference set to true:
By default, Structured Streaming from file based sources requires you to specify the schema, rather than rely on Spark to infer it automatically. This restriction ensures a consistent schema will be used for the streaming query, even in the case of failures. For ad-hoc use cases, you can reenable schema inference by setting spark.sql.streaming.schemaInference to true.
but it cannot be used to extract JSON from Kafka messages and DataFrameReader.json doesn't support streaming Datasets as arguments.
You have to provide schema manually How to read records in JSON format from Kafka using Structured Streaming?
It is possible to convert JSON to a DataFrame without having to manually type the schema, if that is what you meant to ask.
Recently I ran into a situation where I was receiving massively long nested JSON packets via Kafka, and manually typing the schema would have been both cumbersome and error-prone.
With a small sample of the data and some trickery you can provide the schema to Spark2+ as follows:
val jsonstr = """ copy paste a representative sample of data here"""
val jsondf = spark.read.json(Seq(jsonstr).toDS) //jsondf.schema has the nested json structure we need
val event = spark.readStream.format..option...load() //configure your source
val eventWithSchema = event.select($"value" cast "string" as "json").select(from_json($"json", jsondf.schema) as "data").select("data.*")
Now you can do whatever you want with this val as you would with Direct Streaming. Create temp view, run SQL queries, whatever..
Taking Arnon's solution to the next step (since it's deprecated in spark's newer versions, and would require iterating the whole dataframe just for a type casting)
spark.read.json(df.as[String])
Anyways, as for now, it's still experimental.
The below code reads the messages from Kafka and the messages are in Avro so how do I parse the message and put it into a dataframe in Spark 2.2.0?
Dataset<Row> df = sparkSession.readStream()
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "topic1")
.load();
This https://github.com/databricks/spark-avro library had no example for streaming case.
how do I parse the message and put it into a dataframe in Spark 2.2.0?
That's your home exercise that is going to require some coding.
This https://github.com/databricks/spark-avro library had no example for streaming case.
I've been told (and seen a couple of questions here) that spark-avro does not support Spark Structured Streaming (aka Spark Streams). It works fine with non-streaming Datasets, but can't handle streaming ones.
That's why I wrote that this is something you have to code yourself.
That could look as follows (I use Scala for simplicity):
// Step 1. convert messages to be strings
val avroMessages = df.select($"value" cast "string")
// Step 2. Strip the avro layer off
val from_avro = udf { (s: String) => ...processing here... }
val cleanDataset = avroMessages.withColumn("no_avro_anymore", from_avro($"value"))
That would require developing a from_avro custom UDF that would do what you want (and would be similar to how Spark handles JSON format using from_json standard function!)
Alternatively (and in a slightly more advanced? / convoluted approach) write your own custom streaming Source for datasets in Avro format in Kafka and use it instead.
Dataset<Row> df = sparkSession.readStream()
.format("avro-kafka") // <-- HERE YOUR CUSTOM Source
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "topic1")
.load();
I'm yet to find out how doable avro-kafka format is. It is indeed doable, but does two things at once, i.e. reading from Kafka and doing Avro conversion, and am not convinced that's the way to do things in Spark Structured Streaming and in software engineering in general. I wished there were a way to apply one format after another, but that's not possible in Spark 2.2.1 (and is not planned for 2.3 either).
I think then that a UDF is the best solution for the time being.
Just a thought, you could also write a custom Kafka Deserializer that would do the deserialization while Spark loads messages.