Spark SQL - does renaming columns affect partitioning? - apache-spark

Question

I have written an explicitJoin API which renames the columns in a Dataset with either a l_ or r_ prefix to disambiguate and to solve problems with spark lineage, i.e columnName1#77 not found in columnName1#123, columnName2#55....
Part of the code is shown below:
def explicitJoin(other: Dataset[_], joinExpr: Column, joinType: String): ExplicitJoinExt = {
val left = dataset.toDF(dataset.columns.map("l_" + _): _*)
val right = other.toDF(other.columns.map("r_" + _): _*)
new ExplicitJoinExt(left.join(right, joinExpr, joinType))
}
Users may then pass a join expressions such as $"l_columnName1" === $"r_columnName1" && ... so that they are 100% explicit about what columns they are joining on.
I am experiencing a new issue where partitions are too large to load into memory (org.apache.spark.shuffle.FetchFailedException: Too large frame....) yet there was no problem reading the input (partitioned) Datasets.
Can renaming columns affect the underlying parititioning of the input Datasets/DataFrames?
EDIT
Example 1 - regular join
case class A(a: Int, b: String)
val l = (0 to 1000000).map(i => A(i, i.toString))
val r = (0 to 1000000).map(i => A(i, i.toString))
val ds1 = l.toDF.as[A].repartition(100, $"a")
val ds2 = r.toDF.as[A].repartition(100, $"a")
val joined = ds1.join(ds2, Seq("a"), "inner")
joined.explain
== Physical Plan ==
*Project [a#2, b#3, b#15]
+- *SortMergeJoin [a#2], [a#14], Inner
:- *Sort [a#2 ASC NULLS FIRST], false, 0
: +- Exchange hashpartitioning(a#2, 100)
: +- LocalTableScan [a#2, b#3]
+- *Sort [a#14 ASC NULLS FIRST], false, 0
+- ReusedExchange [a#14, b#15], Exchange hashpartitioning(a#2, 100)
Example 2 - Using my (possibly misguided) ExplicitJoinExt involving renames
val joined = ds1
.explicitJoin(ds2, $"l_a" === $"r_a", "inner") // Pimped on conversion to ExplicitJoin type, columns prefixed by l_ or r_. DS joined by expr and join type
.selectLeft // Select just left prefixed columns
.toDF // Convert back from ExplicitJoinExpr to DF
.as[A]
joined.explain
== Physical Plan ==
*Project [l_a#24 AS a#53, l_b#25 AS b#54]
+- *BroadcastHashJoin [l_a#24], [r_a#29], Inner, BuildRight
:- *Project [a#2 AS l_a#24, b#3 AS l_b#25]
: +- Exchange hashpartitioning(a#2, 100)
: +- LocalTableScan [a#2, b#3]
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)))
+- *Project [a#14 AS r_a#29]
+- Exchange hashpartitioning(a#14, 100)
+- LocalTableScan [a#14]
So, for the second join it would apear that we are repartitioning again - correct?

Answer 1

NO, I checked in SPARK 2.3.1. Renaming does not affect partitioning, at least not in this approach:
val ds11 = ds1.repartition(4)
NO, I checked this also. Renaming does not affect partitioning, at least not in this approach:
val ds11 = ds1.repartition(2, $"cityid")
EXPLAIN Output for:
val j = left.join(right, $"l_personid" === $"r_personid", "inner").explain
​reveals, in my case 2 and 4 as number of partitions:
== Physical Plan ==
*(2) BroadcastHashJoin [l_personid#641], [r_personid#647], Inner,
BuildRight, false
:- *(2) Project [personid#612 AS l_personid#641, personname#613 AS
l_personname#642, cityid#614 AS l_cityid#643]
: +- Exchange hashpartitioning(cityid#614, 2)
: +- LocalTableScan [personid#612, personname#613, cityid#614]
+- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)))
+- *(1) Project [personid#612 AS r_personid#647, personname#613 AS r_personname#648, cityid#614 AS r_cityid#649]
+- Exchange hashpartitioning(personid#612, 4)
+- LocalTableScan [personid#612, personname#613, cityid#614]
One can see that the renamed cols are mapped back to their original names.
In a test on a post elsewhere we were able to ascertain that new actions relying on AGGRegations or JOINs will default to 200 unless
sqlContext.setConf("spark.sql.shuffle.partitions", "some val")
is issued in the code setting this to the required value. If it is a small set of data being JOINed, etc. then the results may differ.

Answer 2

For those still encountering this issue: renaming columns does affect partitioning in Spark < 3.0.
Seq((1, 2))
.toDF("a", "b")
.repartition($"b")
.withColumnRenamed("b", "c")
.repartition($"c")
.explain()
Gives the following plan:
== Physical Plan ==
Exchange hashpartitioning(c#40, 10)
+- *(1) Project [a#36, b#37 AS c#40]
+- Exchange hashpartitioning(b#37, 10)
+- LocalTableScan [a#36, b#37]
This was fixed in this PR.