How to convert .CSV file to .Json file using Pyspark? - python-3.x

Question

I am having a problem in converting .csv file to multiline json file using pyspark.
I have a csv file read via spark rdd and I need to convert this to multiline json using pyspark.
Here is my code:
import json
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("jsonconversion").getOrCreate()
df = spark.read.format("csv").option("header","True").load(csv_file)
df.show()
df_json = df.toJSON()
for row in df_json.collect():
line = json.loads(row)
result =[]
for key,value in list(line.items()):
if key == 'FieldName':
FieldName =line['FieldName']
del line['FieldName']
result.append({FieldName:line})
res =result
with open("D:/tasklist/jsaonoutput.json",'a+')as f:
f.write(json.dumps(res, indent=4, separators=(',',':')))
I need the output in below format.
{
"Name":{
"DataType":"String",
"Length":4,
"Required":"Y",
"Output":"Y",
"Address": "N",
"Phone Number":"N",
"DoorNumber":"N/A"
"Street":"N",
"Locality":"N/A",
"State":"N/A"
}
}
My Input CSV file Looks like this:
I am new to Pyspark, Any leads to modify this code to a working code will be much appreciated.
Thank you in advance.

Answer 1

Try the following code. It first creates pandas dataframe from spark DF (unless you care doing some else with spark df, you can load csv file directly into pandas). From pandas df, it creates groups based on FieldName column and then writes to file where json.dumps takes care of formatting.
import json
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("jsonconversion").getOrCreate()
df = spark.read.format("csv").option("header","True").load(csv_file)
df.show()
df_pandas_grped = df.toPandas().groupby('FieldName')
final_dict = {}
for key, grp in df_pandas_grped:
final_dict[str(key)] = grp.to_dict('records')
with open("D:/tasklist/jsaonoutput.json",'w')as f:
f.write(json.dumps(final_dict,indent=4))