PySpark Schema structure to read nested data - apache-spark

Question

I am getting following error.
ValueError: field col4: Length of object (1) does not match with length of fields (2)
The data is in this format.
[
["N","S","3",null,null],
["N","P","4",[{"key1":"val1","key2":"val2"}],null],
["N","I","5",null,[{"key1":"val1","key2":"val2"}]],
["N","S","3",null,null]
]
The Schema I have defined is following:
schema = StructType(
StructField("col1", StringType(), True),
StructField("col2", StringType(), True),
StructField("col3", StringType(), True),
StructField("col4",
StructType(
StructField("key1", StringType(), True),
StructField("key2", StringType(), True)
)
),
StructField("col5",
StructType(
StructField("key1", StringType(), True),
StructField("key2", StringType(), True)
)
)
)
Please help me in identifying how I can read the data of this format.

Answer 1

Welcome to StackOverflow community.
Coming to your question, first you need to replace null with None, as null is not a keyword in either python or pyspark (unless you are using spark-sql).
Now regarding your schema - you need to define it as ArrayType wherever complex or list column structure is there. Inside that, you again need to specify StructType because within your list there is a dictionary with key and value pairs.
See below structure to visualize it better -
data = [["N","S","3",None,None], ["N","P","4",[{"key1":"val1","key2":"val2"}],None], ["N","I","5",None,[{"key1":"val1","key2":"val2"}]], ["N","S","3",None, None] ]
You need to convert this to a RDD as below -
data_rdd = sc.parallelize(data)
Once you're RDD is created then you need to create your dataframe using the schema I explained above -
from pyspark.sql.types import *
schema = schema = StructType(
[StructField("col1", StringType(), True),
StructField("col2", StringType(), True),
StructField("col3", StringType(), True),
StructField("col4",
ArrayType(
StructType([StructField("key1", StringType(), True),
StructField("key2", StringType(), True)])
)
),
StructField("col5",
ArrayType(
StructType([StructField("key1", StringType(), True),
StructField("key2", StringType(), True)])
)
)
]
)
df = spark.createDataFrame(data=data_rdd, schema=schema)
Output
df.show()
+----+----+----+--------------+--------------+
|col1|col2|col3| col4| col5|
+----+----+----+--------------+--------------+
| N| S| 3| null| null|
| N| P| 4|[{val1, val2}]| null|
| N| I| 5| null|[{val1, val2}]|
| N| S| 3| null| null|
+----+----+----+--------------+--------------+
Additionally, if you need the key and value as separate columns for both col4 and col5, in that case you need to create the schema as below -
schema = StructType(
[StructField("col1", StringType(), True),
StructField("col2", StringType(), True),
StructField("col3", StringType(), True),
StructField("col4",
ArrayType(
MapType(StringType(), StringType())
)
),
StructField("col5",
ArrayType(
MapType(StringType(), StringType())
)
)
]
)
from pyspark.sql.functions import *
df = spark.createDataFrame(data=sc.parallelize(data), schema=schema)
df.show(truncate=False)
#Input dataframe output -
+----+----+----+------------------------------+------------------------------+
|col1|col2|col3|col4 |col5 |
+----+----+----+------------------------------+------------------------------+
|N |S |3 |null |null |
|N |P |4 |[{key1 -> val1, key2 -> val2}]|null |
|N |I |5 |null |[{key1 -> val1, key2 -> val2}]|
|N |S |3 |null |null |
+----+----+----+------------------------------+------------------------------+
Finally, explode these columns col4 and col5 as below -
(df.withColumn('explode_col4', explode_outer( col('col4')))
.withColumn('explode_col5', explode_outer( col('col5')))
.select("col1", "col2", "col3", (explode_outer( col('explode_col4') ).alias('col4_key', 'col4_value')) , "explode_col5")
.select("col1", "col2", "col3", "col4_key", "col4_value", (explode_outer( col('explode_col5') ).alias('col5_key', 'col5_value')))
).show(truncate=False)
Output
+----+----+----+--------+----------+--------+----------+
|col1|col2|col3|col4_key|col4_value|col5_key|col5_value|
+----+----+----+--------+----------+--------+----------+
|N |S |3 |null |null |null |null |
|N |P |4 |key1 |val1 |null |null |
|N |P |4 |key2 |val2 |null |null |
|N |I |5 |null |null |key1 |val1 |
|N |I |5 |null |null |key2 |val2 |
|N |S |3 |null |null |null |null |
+----+----+----+--------+----------+--------+----------+