How convert rdd row to a dataframe with json struct in pyspark? - apache-spark

Question

I am sending the following json to the path "/ home / host / test" so that the program can capture it using spark streaming and be able to make queries about it.
{"id": "1", description: "test"}
{"id": "1", description: "test"}
But when I perform the query it looks like the following structure
root
| --word: String (Nulleable = true)
and I get the following result:
+ ------------------- +
| word |
---------------------
| {"id": "1", "test"}
| {"id": "1", "test"}
I need the structure to look like this
root
| --id: String (Nulleable = true)
| --description string (Nulleable = true)
and I need to get a result like the following
----------------
| id | description
----------------
| "1" | "test" |
| "1" | "test" |
----------------
this is my pyspkark code
from __future__ import print_function
import os
import sys
from pyspark import SparkContext
from pyspark.sql.functions import col, explode
from pyspark.streaming import StreamingContext
from pyspark.sql import SQLContext, Row
from pyspark.sql import SQLContext
if __name__ == "__main__":
sc = SparkContext(appName="PythonSqlNetworkWordCount")
ssc = StreamingContext(sc, 3)
sqlcontextoriginal = SQLContext(sc)
# Create a socket stream on target ip:port and count the
# words in input stream of \n delimited text (eg. generated by 'nc')
lines = ssc.textFileStream("/home/host/test")
# Convert RDDs of the words DStream to DataFrame and run SQL query
def process(time, rdd):
print("========= %s =========" % str(time))
try:
# Get the singleton instance of SQLContext
sqlContext = SQLContext(rdd.context)
# Convert RDD[String] to RDD[Row] to DataFrame
rowRdd = rdd.map(lambda w: Row(word=w))
wordsDataFrame = sqlContext.createDataFrame(rowRdd).toJSON()
json = sqlContext.read.json(wordsDataFrame)
# Register as table
json.createOrReplaceTempView("words")
json.printSchema()
wordCountsDataFrame = sqlContext.sql("select * from words ")
wordCountsDataFrame.show()
except:
pass
lines.foreachRDD(process)
ssc.start()
ssc.awaitTermination()

Answer 1

Ok, i found the solution.
I had to use sql.read.json passing it as parameter the rdd directly.
json = sqlContext.read.json(rdd)