How to update an entire Dataset Row? - apache-spark

Question

I want to update a row (having index numberInt) values of a given dataset dFIdx using values of another row from another dataset dFInitIdx (the row of the second dataset having different index j). I try in JAVA like the following :
for (String colName : dFInitIdx.columns())
dFIdx = dFIdx.where(col("id").equalTo(numberInt)).withColumn(colName,dFInitIdx.where(col("id").equalTo(j)).col(colName));
But i am getting this error :
Attribute(s) with the same name appear in the operation: id. Please
check if the right attribute(s) are used
How to achieve that update of one row in JAVA (preferably a one liner) ?
Thanks

Answer 1

Since both of your Datasets seem to have the same columns, you can use a join() method to merge them together based on your numberInt and j conditions, in order to select() (at least) the id column value of the first Dataset dFIdx and all the other columns from the second Dataset dFInitIdx.
dFIdx data sample:
+---+--------+---------+
| id|hundreds|thousands|
+---+--------+---------+
| 1| 100| 1000|
| 2| 200| 2000|
| 3| 300| 3000|
+---+--------+---------+
dFInitIdx data sample:
+---+--------+---------+
| id|hundreds|thousands|
+---+--------+---------+
| 1| 101| 1001|
| 2| 201| 2001|
| 3| 301| 3001|
+---+--------+---------+
Let's say that for the given data samples numberInt and j are (hardcoded) set as:
numberInt == 1
j == 2
The solution will look like this:
dFIdx.join(dFInitIdx, dFIdx("id").equalTo(numberInt) && dFInitIdx("id").equalTo(j))
.select(dFIdx("id"), dFInitIdx("hundreds"), dFInitIdx("thousands"))
And we can see the result of the query with show() as seen below:
+---+--------+---------+
| id|hundreds|thousands|
+---+--------+---------+
| 1| 201| 2001|
+---+--------+---------+