Spark does not recognize new lines, &amp, etc. from String - apache-spark

Question

I'm trying to process text data (Twitter tweets) with PySpark. Emojis and special characters are being red correctly but "\n", "&amp" appear to be escaped. Spark does not recognize them. Probably others too. One example tweet in my Spark DF would look like this:
"Hello everyone\n\nHow is it going? 😉 Take care & enjoy"
I would like Spark to read them correctly. The files are stored as parquet and I'm reading them like this:
tweets = spark.read.format('parquet')\
.option('header', 'True')\
.option('encoding', 'utf-8')\
.load(path)
Below are some sample input data, which I took from the original JSONL files (I stored the data as parquet later).
"full_text": "RT #OurWarOnCancer: Where is our FEDERAL vaccination
education campaign for HPV?! Where is our FEDERAL #lungcancer
screening program?! (and\u2026"
"full_text": "\u2b55\ufe0f#HPV is the most important cause of
#CervicalCancer But it doesn't just cause cervical cancer (see the figure\ud83d\udc47) \n\u2b55\ufe0fThat means they can be PREVENTED"
Reading directly from JSONL files results in the same recognizing problems.
tweets = spark.read.\
.option('encoding', 'utf-8')\
.json(path)
How can Spark recognize them correctly? Thank you in advance.

Answer 1

the below code might be helpful to solve your problem,
Input taken:
"Hello everyone\n\nHow is it going? 😉 Take care & enjoy"
"full_text": "RT #OurWarOnCancer: Where is our FEDERAL vaccination education campaign for HPV?! Where is our FEDERAL #lungcancer screening program?! (and\u2026 &"
"full_text": "\u2b55\ufe0f#HPV is the most important cause of #CervicalCancer But it doesn't just cause cervical cancer (see the figure\ud83d\udc47) \n\u2b55\ufe0fThat means they can be PREVENTED #theNCI #NCIprevention #AmericanCancer #cancereu #uicc #IARCWHO #EuropeanCancer #KanserSavascisi #AUTF_DEKANLIK #OncoAlert"
code to solve the problem:
from pyspark.sql.functions import *
df=spark.read.csv("file:///home/sathya/Desktop/stackoverflo/raw-data/input.tweet")
df1=df.withColumn("cleandata",regexp_replace('_c0', '&|\\\\n', ''))
df1.select("cleandata").show(truncate=False)
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|cleandata |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|Hello everyoneHow is it going? 😉 Take care & enjoy |
|"full_text": "RT #OurWarOnCancer: Where is our FEDERAL vaccination education campaign for HPV?! Where is our FEDERAL #lungcancer screening program?! (and\u2026 &" |
|"full_text": "\u2b55\ufe0f#HPV is the most important cause of #CervicalCancer But it doesn't just cause cervical cancer (see the figure\ud83d\udc47) \u2b55\ufe0fThat means they can be PREVENTED #theNCI #NCIprevention #AmericanCancer #cancereu #uicc #IARCWHO #EuropeanCancer #KanserSavascisi #AUTF_DEKANLIK #OncoAlert"|
+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+