Pyspark - How can I convert parquet file to text file with delimiter - apache-spark

Question

I have a parquet file with the following schema:
|DATE|ID|
I would like to convert it into a text file with tab delimiters as follows:
20170403 15284503
How can I do this in pyspark?

Answer 1

In Spark 2.0+
spark.read.parquet(input_path)
to read the parquet file into a dataframe. DataFrameReader
spark.write.csv(output_path, sep='\t')
to write the dataframe out as tab delimited. DataFrameWriter

Answer 2

You can read your .parquet file in python using DataFrame and with the use of list data structure, save that in a text file. the sample code is here:
this code, reads word2vec (word to vector) that is output of spark mllib WordEmbeddings class in a .parquet file and convert it to tab delimiter .txt file.
import pandas as pd
import pyarrow.parquet as pq
import csv
data = pq.read_pandas('C://...//parquetFile.parquet', columns=['word', 'vector']).to_pandas()
df = pd.DataFrame(data)
vector = df['vector'].tolist()
word = df['word']
word = word.tolist()
k = [[]]
for i in range(0, word.__len__()):
l = []
l.append(word[i])
l.extend(vector[i])
k.append(l)
#you can not save data frame directly to .txt file.
#so, write df to .csv file at first
with open('C://...//csvFile.csv', "w", encoding="utf-8") as f:
writer = csv.writer(f)
for row in k:
writer.writerow(row)
outputTextFile = 'C://...//textFile.txt'
with open(outputTextFile, 'w') as f:
for record in k:
if (len(record) > 0):
for element in record:
#tab delimiter elements
f.write("%s\t" % element)
f.write("%s" % element)
#add enter after each records
f.write("\n")
I hope it helps :)