Spark Querying on all keys of a map - apache-spark

Question

I have a parquet file with the following schema
|-- Name: string (nullable = true)
|-- Attendance: long (nullable = true)
|-- Efficiency: map (nullable = true)
| |-- key: string
| |-- value: double (valueContainsNull = true)
Where efficiency value ranges from -1 to +1 and the key is various categories such as Sports,Academics etc. I have up to 20 different keys.
I am trying to fetch the top 100 names ordered in descending by Attendance where efficiency[Key] is lesser than 0.
I am able to do this for one key. But i'm not able to figure out how I should be implementing this for all my keys simultaneously.
Code snippet for one key:
spark.sql("select Name,Attendance,Efficiency['Sports'] from data where Efficiency['Sports'] < 0 order by Attendance desc limit 100")
On doing some analysis I found that we would need to explode our map. But whenever I explode the number of rows in my table goes up and I am unable to fetch the top 100 names.
Sample Data for one key. The actual table has a map instead of the one column that is seen here
+--------------------+------------------+-------------+
|Name |Attendance |Efficiency[Sports]|
+--------------------+------------------+-------------+
|A |1000 |0.002 |
|B |365 |0.0 |
|C |1080 |0.193 |
|D |245 |-0.002 |
|E |1080 |-0.515 |
|F |905 |0.0 |
|G |900 |-0.001 |
Expected output : List of 100 names for each key
+-----------------------+--------------+
|Sports |Academics |
+-----------------------+--------------+
|A |A |
|B |C |
|C |D |
|D |E |
Any help on solving this would really helpful
Thanks

Answer 1

I hope this is what you are looking for
import org.apache.spark.sql.functions._
//dummy data
val d = Seq(
("a", 10, Map("Sports" -> -0.2, "Academics" -> 0.1)),
("b", 20, Map("Sports" -> -0.1, "Academics" -> -0.1)),
("c", 5, Map("Sports" -> -0.2, "Academics" -> 0.5)),
("d", 15, Map("Sports" -> -0.2, "Academics" -> 0.0))
).toDF("Name", "Attendence", "Efficiency")
//explode the map and get key value
val result = d.select($"Name", $"Attendence", explode($"Efficiency"))
//select value less than 0 and show 100
result.select("*").where($"value".lt(0))
.sort($"Attendence".desc)
.show(100)
Output:
+----+----------+---------+-----+
|Name|Attendence|key |value|
+----+----------+---------+-----+
|b |20 |Sports |-0.1 |
|b |20 |Academics|-0.1 |
|d |15 |Sports |-0.2 |
|a |10 |Sports |-0.2 |
|c |5 |Sports |-0.2 |
+----+----------+---------+-----+
Hope this helps!

Answer 2

Given the input dataframe as
+----+----------+-----------------------------------------+
|Name|Attendance|Efficiency |
+----+----------+-----------------------------------------+
|A |1000 |Map(Sports -> 0.002, Academics -> 0.002) |
|B |365 |Map(Sports -> 0.0, Academics -> 0.0) |
|C |1080 |Map(Sports -> 0.193, Academics -> 0.193) |
|D |245 |Map(Sports -> -0.002, Academics -> -0.46)|
|E |1080 |Map(Sports -> -0.515, Academics -> -0.5) |
|F |905 |Map(Sports -> 0.0, Academics -> 0.0) |
|G |900 |Map(Sports -> -0.001, Academics -> -0.0) |
+----+----------+-----------------------------------------+
Using udf function to iterate the Map for checking less than zero values. This can be done as below
import org.apache.spark.sql.functions._
val isLessThan0 = udf((maps: Map[String, Double]) => maps.map(x => x._2 < 0).toSeq.contains(true))
df.withColumn("lessThan0", isLessThan0('Efficiency))
.filter($"lessThan0" === true)
.orderBy($"Attendance".desc)
.drop("lessThan0")
.show(100, false)
you will have output as
+----+----------+-----------------------------------------+
|Name|Attendance|Efficiency |
+----+----------+-----------------------------------------+
|E |1080 |Map(Sports -> -0.515, Academics -> -0.5) |
|G |900 |Map(Sports -> -0.001, Academics -> -0.0) |
|D |245 |Map(Sports -> -0.002, Academics -> -0.46)|
+----+----------+-----------------------------------------+