Modify a struct column in Spark dataframe - apache-spark

Question

I have a PySpark dataframe which contains a column "student" as follows:
"student" : {
"name" : "kaleem",
"rollno" : "12"
}
Schema for this in dataframe is:
structType(List(
name: String,
rollno: String))
I need to modify this column as
"student" : {
"student_details" : {
"name" : "kaleem",
"rollno" : "12"
}
}
Schema for this in dataframe must be:
structType(List(
student_details:
structType(List(
name: String,
rollno: String))
))
How to do this in Spark?

Answer 1

Use named_struct function to achieve this
1. Read the json as column
val data =
"""
| {
| "student": {
| "name": "kaleem",
| "rollno": "12"
| }
|}
""".stripMargin
val df = spark.read.json(Seq(data).toDS())
df.show(false)
println(df.schema("student"))
Output
+------------+
|student |
+------------+
|[kaleem, 12]|
+------------+
StructField(student,StructType(StructField(name,StringType,true), StructField(rollno,StringType,true)),true)
2. change the schema using named_struct
val processedDf = df.withColumn("student",
expr("named_struct('student_details', student)")
)
processedDf.show(false)
println(processedDf.schema("student"))
Output
+--------------+
|student |
+--------------+
|[[kaleem, 12]]|
+--------------+
StructField(student,StructType(StructField(student_details,StructType(StructField(name,StringType,true), StructField(rollno,StringType,true)),true)),false)
For python step#2 will work as is just remove val

Answer 2

With a library called spark-hats - This library extends Spark DataFrame API with helpers for transforming fields inside nested structures and arrays of arbitrary levels of nesting., you can do a lot of these transformations.
scala> import za.co.absa.spark.hats.Extensions._
scala> df.printSchema
root
|-- ID: string (nullable = true)
scala> val df2 = df.nestedMapColumn("ID", "ID", c => struct(c as alfa))
scala> df2.printSchema
root
|-- ID: struct (nullable = false)
| |-- alfa: string (nullable = true)
scala> val df3 = df2.nestedMapColumn("ID.alfa", "ID.alfa", c => struct(c as "beta"))
scala> df3.printSchema
root
|-- ID: struct (nullable = false)
| |-- alfa: struct (nullable = false)
| | |-- beta: string (nullable = true)
Your query would be
df.nestedMapColumn("student", "student", c => struct(c as "student_details"))

Answer 3

Spark 3.1+
To modify struct type columns, we can use withField and dropFields
F.col("Student").withField("student_details", F.col("student"))
F.col("Student").dropFields("name", "rollno")
Input:
from pyspark.sql import functions as F
df = spark.createDataFrame([(("kaleem", "12"),)], "student struct<name:string,rollno:string>")
df.printSchema()
# root
# |-- student: struct (nullable = true)
# | |-- name: string (nullable = true)
# | |-- rollno: string (nullable = true)
Script:
df = df.withColumn("student", F.col("Student")
.withField("student_details", F.col("student"))
.dropFields("name", "rollno")
)
Result:
df.printSchema()
# root
# |-- student: struct (nullable = true)
# | |-- student_details: struct (nullable = true)
# | | |-- name: string (nullable = true)
# | | |-- rollno: string (nullable = true)