union() method giving weird dataset values - apache-spark

Question

I want to pile up the rows of many datasets into one. The datasets are stored in array and i use union() to add the rows of the previous dataset, given the array of datasets i described :
ArrayList<Dataset<Row>> dropped_added_cols_dataset_list = ...
i do it like so :
AD = dropped_added_cols_dataset_list.get(0);
for (int i = 1; i < dropped_added_cols_dataset_list.size(); i++) {
AD = AD.union(dropped_added_cols_dataset_list.get(i));
}
But it seems as the final AD is showing weird values not related to those in each sub-dataset from the array described previously.
Anyone has an idea why ?

Answer 1

Let's say we have the following n (here n == 3) number of Datasets as input stored in your dropped_added_cols_dataset_list list.
df1:
+---+----+----+
| _1| _2| _3|
+---+----+----+
| 0|cat1|30.9|
| 2|cat2|22.1|
| 0|cat3|19.6|
| 1|cat4| 1.3|
+---+----+----+
df2:
+---+----+----+
| _1| _2| _3|
+---+----+----+
| 1|cat5|28.5|
| 2|cat6|26.8|
| 1|cat7|12.6|
| 1|cat8| 5.3|
+---+----+----+
df3:
+---+-----+----+
| _1| _2| _3|
+---+-----+----+
| 1| cat9|39.6|
| 2|cat10|29.7|
| 0|cat11|27.9|
| 1|cat12| 9.8|
+---+-----+----+
We can apply the unions for all of them in the list without any user-specific loop, by using reduce (Java docs for Dataset's specified reduce here, working for the ArrayList through the InterruptibleIterator class that inherits reduce from the scala.collection.TraversableOnce interface as seen here) to merge the Datasets together in couples at a time.
// Java (`dropped_added_cols_dataset_list` is of type ArrayList)
list.reduce((ReduceFunction<Dataset>) (v1, v2) -> v1.union(v2));
// Scala (`dropped_added_cols_dataset_list` is of type List)
dropped_added_cols_dataset_list.reduce(_ union _)
The resulting Dataset will look like this:
+---+-----+----+
| _1| _2| _3|
+---+-----+----+
| 0| cat1|30.9|
| 2| cat2|22.1|
| 0| cat3|19.6|
| 1| cat4| 1.3|
| 1| cat5|28.5|
| 2| cat6|26.8|
| 1| cat7|12.6|
| 1| cat8| 5.3|
| 1| cat9|39.6|
| 2|cat10|29.7|
| 0|cat11|27.9|
| 1|cat12| 9.8|
+---+-----+----+
Of course, for the unions to be successful, you need to be extra careful that all of the Datasets stored in your list have the exact same column schema.
It also needs to be said that this solution's performance is heavily dependent on the number of Datasets being merged, so a relatively large number of unions can result to non-linear time complexity.