Getting the latest value in a time range or null - apache-spark

Question

I have a huge data set e.g.
| Date | ID | Value |
+------------+----+-------+
| 10-10-2020 | 1 | 1 |
| 10-11-2020 | 1 | 2 |
| 10-12-2020 | 1 | 3 |
| 10-13-2020 | 1 | 4 |
| 10-10-2020 | 2 | 5 |
| 10-11-2020 | 2 | 6 |
| 10-12-2020 | 2 | 7 |
| 10-09-2020 | 3 | 8 |
| 10-08-2020 | 4 | 9 |
As you can see this example contains of 4 IDs within different date ranges.
I have a special logic, which calculates some derived values with RangeBetween function. Let's assume it is a simple sum over the defined time range.
What I need to do is to generate such a result (explained below):
| ID | Value sum (last 2 days) | Value sum (last 4 days) | Value sum (prev 2 days) | Value sum (prev 4 days) | Result (2 days) | Result (4 days) |
+----+-------------------------+-------------------------+-------------------------+-------------------------+-----------------+-----------------+
| 1 | 7 (3+4) | 10 (1+2+3+4) | 5 (3+2) | 6 (3+2+1) | 7 | 10 |
| 2 | 7 | 18 (5+6+7) | 11 (5+6) | 11 (5+6) | 7 | 18 |
| 3 | null | null | null | 8 | null | 0 |
//exclude | 4 | null | null | null | null | null | null |
This example assumes that today is 10-13-2020.
For each Id I need to get a sum of the value in 2 ranges: 2 and 4 days
1. the table contains 2 calculations for the same ranges starting from now and the day before (columns last and prev X days)
2. if all values exist in a range - simply result the sum of the range (example with ID = 1)
3. if some of values are not specified in a range assume it is zero (example with ID = 2)
4. if values do not exist in the defined range, but there is at least 1 value in the range with the day before - assume there was a sum yesterday, but no such today - set it to zero (example #3)
5. if no value values in the range and the day before - do not include in the result set (example #4)
Right now I have a code:
let last2Days =
Window
.PartitionBy('ID')
.OrderBy(Functions.Col('Date').Cast("timestamp").Cast("long"))
.RangeBetween(-1, 0)
let prev2Days =
Window
.PartitionBy('ID')
.OrderBy(Functions.Col('Date').Cast("timestamp").Cast("long"))
.RangeBetween(-2, -1)
df
.WithColumn('last2daysSum', Functions.Sum('value').Over(last2Days))
.WithColumn('prev2daysSum', Functions.Sum('value').Over(last4Days))
.WithColumn('result2Days', Functions.Col('last2daysSum'))
.Where(Functions.Col('Date').EqualTo(Functions.Lit('10-13-2020')))
This works for example #1 (when result is taken from last2daysSum)
1. is there a simple way to get a proper result for #2 (the latest record within defined time range)?
2. combine the previous question and condition `if last = null && prev != null then 0 else if last = null && prev = null then null else last` - example #3?
3. how to exclude records as per example #4?
Is that possible to solve it with no reshuffling?

Answer 1

For Question #1 If you only want to calculate for one specific date then a groupBy and agg is simpler and should execute faster. The trick is to use when inside aggregate functions like sum.
For Questions #2 and #3 you can coalesce to zero and filter out fully null rows before that. If you need to filter for a broader range than you want to display (so include rows that had values days before but do not now) you can add an extra calculation for the longer period a drop that after filtering. See below for code example.
import org.apache.spark.sql.functions._
val data = Seq(
("2020-10-10", 1, 1),
("2020-10-11", 1, 2),
("2020-10-12", 1, 3),
("2020-10-13", 1, 4),
("2020-10-10", 2, 5),
("2020-10-11", 2, 6),
("2020-10-12", 2, 7),
("2020-10-09", 3, 8),
("2020-10-08", 4, 9)
).toDF("Date", "ID", "Value").withColumn("Date", to_date($"Date"))
def sumLastNDays(now: java.sql.Timestamp, start: Int, end: Int = 0) =
sum(when($"Date".between(date_sub(lit(now), start-1), date_sub(lit(now), end)), $"Value"))
val now = java.sql.Timestamp.valueOf("2020-10-13 00:00:00")
data
.groupBy($"ID")
.agg(
sumLastNDays(now, 2).as("last2DaysSum"),
sumLastNDays(now, 4).as("last4DaysSum"),
sumLastNDays(now, 4, 2).as("prev2DaysSum"),
sumLastNDays(now, 5).as("last5DaysSum")
)
.filter($"last5DaysSum".isNotNull)
.drop($"last5DaysSum")
.withColumn("last4DaysSum", coalesce($"last4DaysSum", lit(0)))
.withColumn("last2DaysSum", coalesce($"last2DaysSum", lit(0)))
.withColumn("prev2DaysSum", coalesce($"prev2DaysSum", lit(0)))
.orderBy($"ID")
.show()
Result:
+---+------------+------------+------------+
| ID|last2DaysSum|last4DaysSum|prev2DaysSum|
+---+------------+------------+------------+
| 1| 7| 10| 3|
| 2| 7| 18| 11|
| 3| 0| 0| 0|
+---+------------+------------+------------+
Note: I'm not sure if you meant prev2Days to be the previous 2 day interval before the current 2 day interval or the yesterday's last 2 day interval, because in the expected results table ID 1 has Oct. 11-12 summed and ID 2 has Oct. 10-11 summed for prev2Days, but either way you can adjust the range params if you want something else. I assumed that prev2Days does not overlap with last2Days, just change it to sumLastNDays(now, 3, 1) if you want overlapping 2 day ranges.