Does Spark SQL optimize queries with repeated expressions? - apache-spark

Question

Given the following
from pyspark.sql import functions, window
f = functions.rank()
w1 = window.Window.partitionBy("column")
w2 = window.Window.partitionBy("column")
col = functions.col("column * 42")
and a dataframe df, is there any difference in performance for
df.select(f.over(w1), f.over(w2))
vs
df.select(f.over(w1), f.over(w1))
?
What about
df.select(col + 1, col + 2)
vs
df.select(functions.expr("column * 42 + 1"), functions.expr("column * 42 + 2")
?
(Feel free to imagine arbitrarily complex expressions in place of column * 42)
I.e. is there any benefit in reusing Column- and Window-instances vs constructing these expressions on the fly?
I would expect for Spark SQL to properly optimize this but couldn't find a conclusive answer on that.
Also, should I be able to answer this question myself by inspecting the result of df.explain() and if so, what should I be looking for?

Answer 1

Feel free to imagine arbitrarily complex expressions in place of column * 42
...or even any non-deterministic expressions like generating random numbers or current timestamp.
Whenever you ask such a question use explain operator to see what Spark SQL deals with under the covers (that in fact should be irrelevant of the programming language and function or method in use, shouldn't it?)
So, what happens under the covers of the following non-deterministic query (or fully deterministic, but non-deterministic at the first glance):
val q = spark.range(1)
.select(
current_timestamp as "now", // <-- this should be the same as the following line?
current_timestamp as "now_2",
rand as "r1", // <-- what about this and the following lines?
rand as "r2",
rand as "r3")
scala> q.show(truncate = false)
+-----------------------+-----------------------+-------------------+------------------+------------------+
|now |now_2 |r1 |r2 |r3 |
+-----------------------+-----------------------+-------------------+------------------+------------------+
|2017-12-13 15:17:46.305|2017-12-13 15:17:46.305|0.33579358107333823|0.9478025260069644|0.5846726225651472|
+-----------------------+-----------------------+-------------------+------------------+------------------+
I'm actually a bit surprised to have noticed that rands all generated different results as I had assumed the results would be the same. The answer is in...the source code of rand where you can see that it uses different seeds if not defined explicitly (learnt it today! thanks).
def rand(): Column = rand(Utils.random.nextLong)
The answer is to use the version of rand with explicit seed as that will give you the same Rand logical operator with the same seed across the query.
val seed = 1
val q = spark.range(1)
.select(
current_timestamp as "now", // <-- this should be the same as the following line?
current_timestamp as "now_2",
rand(seed) as "r1", // <-- what about this and the following lines?
rand(seed) as "r2",
rand(seed) as "r3")
scala> q.show(false)
+-----------------------+-----------------------+-------------------+-------------------+-------------------+
|now |now_2 |r1 |r2 |r3 |
+-----------------------+-----------------------+-------------------+-------------------+-------------------+
|2017-12-13 15:43:59.019|2017-12-13 15:43:59.019|0.06498948189958098|0.06498948189958098|0.06498948189958098|
+-----------------------+-----------------------+-------------------+-------------------+-------------------+
Spark SQL knows what you used in a structured query since the high level API of Spark SQL called DataFrame or Dataset is just a wrapper around logical operators that are the same across languages (Python, Scala, Java, R, SQL).
Just look at the source code of any function and you will see a Catalyst expression (e.g. rand) or a Dataset operator (e.g. select) and you will see one or a tree of logical operators.
In the end, Spark SQL uses an rule-based optimizer that uses rules to optimize your query and find repetitions.
So, let's have a look at your case (which is more deterministic than rand).
(I'm using Scala but the differences are at language not optimization level)
import org.apache.spark.sql.expressions.Window
val w1 = Window.partitionBy("column").orderBy("column")
val w2 = Window.partitionBy("column").orderBy("column")
In your case you used rank that requires a dataset to be ordered so I add orderBy clause to make the window specification complete.
scala> w1 == w2
res1: Boolean = false
They are indeed different from Scala's point of view
val df = spark.range(5).withColumnRenamed("id", "column")
scala> df.show
+------+
|column|
+------+
| 0|
| 1|
| 2|
| 3|
| 4|
+------+
With the dataset (which is pretty much irrelevant to our discussion), let's create a structured query and explain it to see the physical plan which is what Spark SQL executes.
val q = df.select(rank over w1, rank over w2)
scala> q.explain
== Physical Plan ==
*Project [RANK() OVER (PARTITION BY column ORDER BY column ASC NULLS FIRST unspecifiedframe$())#193, RANK() OVER (PARTITION BY column ORDER BY column ASC NULLS FIRST unspecifiedframe$())#194]
+- Window [rank(column#156L) windowspecdefinition(column#156L, column#156L ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS RANK() OVER (PARTITION BY column ORDER BY column ASC NULLS FIRST unspecifiedframe$())#193, rank(column#156L) windowspecdefinition(column#156L, column#156L ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS RANK() OVER (PARTITION BY column ORDER BY column ASC NULLS FIRST unspecifiedframe$())#194], [column#156L], [column#156L ASC NULLS FIRST]
+- *Sort [column#156L ASC NULLS FIRST, column#156L ASC NULLS FIRST], false, 0
+- Exchange hashpartitioning(column#156L, 200)
+- *Project [id#153L AS column#156L]
+- *Range (0, 5, step=1, splits=8)
Let's use the numbered output so we can reference every line in the description.
val plan = q.queryExecution.executedPlan
scala> println(plan.numberedTreeString)
00 *Project [RANK() OVER (PARTITION BY column ORDER BY column ASC NULLS FIRST unspecifiedframe$())#193, RANK() OVER (PARTITION BY column ORDER BY column ASC NULLS FIRST unspecifiedframe$())#194]
01 +- Window [rank(column#156L) windowspecdefinition(column#156L, column#156L ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS RANK() OVER (PARTITION BY column ORDER BY column ASC NULLS FIRST unspecifiedframe$())#193, rank(column#156L) windowspecdefinition(column#156L, column#156L ASC NULLS FIRST, specifiedwindowframe(RowFrame, unboundedpreceding$(), currentrow$())) AS RANK() OVER (PARTITION BY column ORDER BY column ASC NULLS FIRST unspecifiedframe$())#194], [column#156L], [column#156L ASC NULLS FIRST]
02 +- *Sort [column#156L ASC NULLS FIRST, column#156L ASC NULLS FIRST], false, 0
03 +- Exchange hashpartitioning(column#156L, 200)
04 +- *Project [id#153L AS column#156L]
05 +- *Range (0, 5, step=1, splits=8)
With that you can see whether the query is similar to another and what are the differences if any. That's the most definitive answer you can get and...surprise...things may (and often will) change between Spark versions.
I.e. is there any benefit in reusing Column- and Window-instances vs constructing these expressions on the fly?
I would not think much about it as I'd expect Spark to handle it internally (and as you may have noticed I was surprised to have seen that rand works differently).
Just use explain to see the physical plan and you will be able to answer the question yourself.