How to count rows matching interrelated conditions? - apache-spark

Question

I have table that looks like this:
TripID | Name | State
1 | John | OH
2 | John | OH
3 | John | CA
4 | John | OH
1 | Mike | CA
2 | Mike | CA
3 | Mike | OH
I'd like to count the people who travelled to OH first followed by CA.
In the above case it'd be John only so the answer should be 1.
So I want to know how can we set a certain order in SQL filtering to filter the result?

Answer 1

I may have misunderstood your question perhaps, but if you're asking about:
how many people travelled to OH first and then to CA.
(The sketch of) the query could be as follows:
scala> trips.show
+------+----+-----+
|tripid|name|state|
+------+----+-----+
| 1|John| OH|
| 2|John| OH|
| 3|John| CA|
| 4|John| OH|
| 1|Mike| CA|
| 2|Mike| CA|
| 3|Mike| OH|
+------+----+-----+
scala> trips.orderBy("name", "tripid").groupBy("name").agg(collect_list("state")).show
+----+-------------------+
|name|collect_list(state)|
+----+-------------------+
|John| [OH, OH, CA, OH]|
|Mike| [CA, CA, OH]|
+----+-------------------+
As I see it now, you'd have two options:
(hard) Write a user-defined aggregate function (UDAF) that would do the aggregation (and would replace collect_list with a itinerary that'd contain distinct states).
(easier) Write a user-defined function (UDF) that would do a similar job as the UDAF above (but after collect_list have collected the values).
(easy) Use functions (like explode and/or window)
Let's do the easy solution (not necessarily the most effective!).
It turns out that the groupBy earlier is not really necessary (!) You can handle it using window aggregation alone (used twice).
import org.apache.spark.sql.expressions.Window
val byName = Window.partitionBy("name").orderBy("tripid")
val distinctStates = trips.withColumn("rank", rank over byName).dropDuplicates("name", "state").orderBy("name", "rank")
scala> distinctStates.show
+------+----+-----+----+
|tripid|name|state|rank|
+------+----+-----+----+
| 1|John| OH| 1|
| 3|John| CA| 3|
| 1|Mike| CA| 1|
| 3|Mike| OH| 3|
+------+----+-----+----+
// rank again but this time use the pre-calculated distinctStates dataset
val distinctStatesRanked = distinctStates.withColumn("rank", rank over byName).orderBy("name", "rank")
scala> distinctStatesRanked.show
+------+----+-----+----+
|tripid|name|state|rank|
+------+----+-----+----+
| 1|John| OH| 1|
| 3|John| CA| 2|
| 1|Mike| CA| 1|
| 3|Mike| OH| 2|
+------+----+-----+----+
val left = distinctStatesRanked.filter($"state" === "OH").filter($"rank" === 1)
val right = distinctStatesRanked.filter($"state" === "CA").filter($"rank" === 2)
scala> left.join(right, "name").show
+----+------+-----+----+------+-----+----+
|name|tripid|state|rank|tripid|state|rank|
+----+------+-----+----+------+-----+----+
|John| 1| OH| 1| 3| CA| 2|
+----+------+-----+----+------+-----+----+