Write pyspark dataframe to file keeping nested quotes, but not "outer" ones? - apache-spark

Question

Is there a way to preserve nested quotes in pyspark dataframe value when writing to file (in my case, a TSV) while also getting rid of the "outer" ones (ie. those that denote a string value in a column)?
>>> dff = sparkSession.createDataFrame([(10,'this is "a test"'), (14,''), (16,'')], ["age", "comments"])
>>> dff.show()
+---+----------------+
|age| comments|
+---+----------------+
| 10|this is "a test"|
| 14| |
| 16| |
+---+----------------+
>>> dff.write\
.mode('overwrite')\
.option("sep", "\t")\
.option("quoteAll", "false")\
.option("emptyValue", "").option("nullValue", "")\
.csv('/tmp/test')
then
$ cat /tmp/test/part-000*
10 "this is \"a test\""
14
16
# what I'd want to see is
10 this is "a test"
14
16
# because I am later parsing based only on TAB characters, so the quote sequences are not a problem in that regard
Is there any way to write the dataframe in this desired format?
* as aside, more info about the args used can be found here

Answer 1

Set the escapeQuotes option to false:
>>> dff = spark.createDataFrame([(10,'this is "a test"'), (14,''), (16,'')], ["age", "comments"])
>>> dff.show()
+---+----------------+
|age| comments|
+---+----------------+
| 10|this is "a test"|
| 14| |
| 16| |
+---+----------------+
>>> dff.write\
... .mode('overwrite')\
... .option("sep", "\t")\
... .option("quoteAll", "false")\
... .option("emptyValue", "").option("nullValue", "")\
... .option("escapeQuotes", "false").csv('/tmp/test')
>>>
➜ ~ cd /tmp/test
➜ test ls
_SUCCESS part-00001-f702e661-15c2-4ab9-aef2-8dad5d923412-c000.csv part-00003-f702e661-15c2-4ab9-aef2-8dad5d923412-c000.csv
part-00000-f702e661-15c2-4ab9-aef2-8dad5d923412-c000.csv part-00002-f702e661-15c2-4ab9-aef2-8dad5d923412-c000.csv
➜ test cat part*
10 this is "a test"
14
16
➜ test