Does spark perform transformations row by row or together? - apache-spark

Question

I am trying to understand spark and have the following code where df,df2,df3,df4 are Dataset<Row>
df=df.join(df2,"ID");
df4=df4.join(df3,"ID");
df=df.union(df4);
long count=df.count();
My question is how do the transformation happen? In the above example, does the union waits for both joins to complete fully (i.e join for all rows is done) and then start the union? Or does it goes row by row in a pipelined way, where once join for one row is complete, the union transform starts on it (even though join for other rows is still going on)?
I tried searching on this but am not able to find any answer.

Answer 1

Use explain() to see what happen.
df.explain()
== Physical Plan ==
Union
:- *(5) Project [ID#3599L, VALUE#3600, VALUE#3604]
: +- *(5) SortMergeJoin [ID#3599L], [ID#3603L], Inner
: :- *(2) Sort [ID#3599L ASC NULLS FIRST], false, 0
: : +- Exchange hashpartitioning(ID#3599L, 200), true, [id=#1332]
: : +- *(1) Filter isnotnull(ID#3599L)
: : +- *(1) Scan ExistingRDD[ID#3599L,VALUE#3600]
: +- *(4) Sort [ID#3603L ASC NULLS FIRST], false, 0
: +- ReusedExchange [ID#3603L, VALUE#3604], Exchange hashpartitioning(ID#3599L, 200), true, [id=#1332]
+- *(10) Project [ID#3599L, VALUE#3600, VALUE#3609]
+- *(10) SortMergeJoin [ID#3599L], [ID#3608L], Inner
:- *(7) Sort [ID#3599L ASC NULLS FIRST], false, 0
: +- ReusedExchange [ID#3599L, VALUE#3600], Exchange hashpartitioning(ID#3599L, 200), true, [id=#1332]
+- *(9) Sort [ID#3608L ASC NULLS FIRST], false, 0
+- ReusedExchange [ID#3608L, VALUE#3609], Exchange hashpartitioning(ID#3599L, 200), true, [id=#1332]
Sort, SortMergeJoin, Project, and Union. This happens when you take an action, such as count() and before that, spark will just plan how to proceed.