Origin and last destination of a customer who has taken multiple flights - apache-spark

Question

For every customer id, I need to find out the first flight source and last flight destination using spark / pyspark
cid --> customerId
fid --> flight ID -- it is random
Input DataFrame:
cid
fid
origin
destination
1
f12
texas
newyork
1
f21
newyork
delhi
1
f08
delhi
japan
texas -> newyork -> japan
ouput dataframe:
cid
origin
destination
1
texas
japan
I tried doing it with joins but not sure how to merge the resultset after joins to filter only starting flight source and last flight destination

Answer 1

Update: In case the order is not certain:
df.groupBy("cid").agg(F.collect_list("origin").alias("origin"), F.collect_list("destination").alias("destination")).select(col("cid"), F.array_except(col("origin"), col("destination")).alias("origin"), F.array_except(col("destination"), col("origin")).alias("destination")).show(truncate=False)
Output:
Essentially, first, collect all origins and destinations as arrays.
As you can see, the stops (i.e. points except starting point and last destination) will common in both arrays.
If order is certain, this would also work:
df.groupBy("cid").agg(F.first("origin").alias("origin"), F.last("destination").alias("destination")).show()
Output:
Although this would not work if the order of rows is messed up.

Answer 2

Assuming that the passenger never takes more than one plane, and that therefore there is a unique path for them to travel
(assuming the start location start)
Run a query SELECT * FROM df where origin = start, that way you get all reachable destinations
store the destination in a list
run the above query with start = destinations for the next step
This will discover the above path.
In case start is not given, you can do something more complex, such as picking a random record, reaching alo destinations as explained above and then also walking back, reaching all sources, you then join these two paths and you have successfully reconstructed the walk.