Spark file cleansing and transformation - apache-spark

Question

I have a file with 5 columns tab separated value (tsv). I need to do data scrubbing and transformation.
case 1) Remove the special character (\001 and \x0D ) with ""
case 2) Filter rows which has less then 5 columns count as Bad_Row RDD
case 3) Iterate Bad_Row RDD and check if last character of row is "\n" then remove last character "\n" and append next row until we get column count 5
Sample File Formate
------------------------------
one two 12345 four five
aaa ppp 12345 ttt
bbb
ccc rrr 12355
yyy
ddd
eee iii 12845 rrr two
Good_Rows RDD
-------------------------------
one two 12345 four five
eee iii 12845 rrr two
BAD_Row RDD
-------------------------------
aaa ppp 12345 ttt
bbb
ccc rrr 12355
yyy
ddd
Remove "\n" from second line and append third line into second line and recalculate column count if column count is 5 then consider as good_row and into Good_Rows RDD.
Sample Code Snippet
def FilterData(rdd):
row=rdd.split("\t")
col_count=len(row)
if col_count!=5 :
return row
textFile1=sc.textFile("hdfs://localhost:9000/A/test.tsv")
Clean_RDD=textFile1.map(lambda x: x.replace("\\001|\\x0D","")) # case 1
Badrow_RDD=Clean_RDD.map(FilterData) # case 2
Please help to implement case 3
Thanks
Vishal

Answer 1

Below is the code for your 3rd problem in Scala.
val data=sc.textFile("file:/home/rieter/Test_Streaming/ab.txt").map(x=>x.split(" +").map(x=>x.trim)).filter(x=>(x.size<5)).flatMap(x=>x).collect
val data1=data.take(data.length-(data.length%5))
data1.splitAt(5)
Output
scala> data1.splitAt(5)
res59: (Array[String], Array[String]) = (Array(aaa, ppp, 12345, ttt, bbb),Array(ccc, rrr, 12355, yyy, ddd))