Logical Conditioning on Pyspark - where versus aggregation with when - apache-spark

Question

Can anyone explain to me why these two conditions produce different outputs (even different count() )?
FIRST:
(df
.where(cond1)
.where((cond2) | (cond3))
.groupBy('id')
.agg(F.avg(F.column('col1')).alias('name'),
F.avg(F.column('col2')).alias('name'))
).count()
SECOND:
(df
.groupBy('id')
.agg(F.avg(F.when(((cond2) | (cond3))) & (cond1),
F.column('col1'))).alias('name'),
F.avg(F.when(((cond2) | (cond3)) & (cond1),
F.column('col2'))).alias('name'))
).count()

Answer 1

I just figured it out. when() returns None when it finds no match, but None is still a return, which means the aggregation takes into account all the values. When compared to a simple df grouped by the same column and just aggregated with no conditions, the result is the same.
On the other hand, where() filters the DataFrame, so the aggregation is only applied to the filtered version of the DataFrame, hence lower number of results

Answer 2

Without knowing what the conditions are, my understanding is that they are different processes: in the first case you first filter the rows you need to process, group by id and get the averages of the filtered data, that results to lets say x rows. In the second case, you group by id first, no rows filtering, and you tell spark to add a column named 'name' that holds the conditional average to the grouped df. You don't conditionally filter the rows, so you now have x+something more rows (depending on your conditions)
(df
.where(cond1) # remove rows by applying cond1
.where((cond2) | (cond3)) # remove rows by applying cond2, 3
.groupBy('id') # group *remaining* rows by id
.agg(F.avg(F.column('col1')).alias('name'), # then get the average
F.avg(F.column('col2')).alias('name'))
).count()
But:
(df
.groupBy('id') # group initial data by id
.agg(F.avg(F.when(((cond2) | (cond3))) & (cond1), # add a column to the grouped data that computes average conditionally
F.column('col1'))).alias('name'),
F.avg(F.when(((cond2) | (cond3)) & (cond1),
F.column('col2'))).alias('name'))
).count()
# the agg does not change the number of the rows.
Hope this helps (I think you've already figured it out though :) ). Good luck!