How does rdd.cache works? - apache-spark

Question

In the below code, I have created an rdd, cached it and created child rdds out of it. vertexRDD -> newVert1 -> newVert2 -> newVert3 -> newVert4 -> newVert5. I do a simple incremental operation in each RDD.
when cache is not enabled, the array values for newVert1, newVert2, newVert3 RDD are
0:ArrayBuffer(1, 0, 0, 0) 0:ArrayBuffer(2, 0, 0, 0) 0:ArrayBuffer(3, 0, 0, 0)
1:ArrayBuffer(0, 1, 0, 0) 1:ArrayBuffer(0, 2, 0, 0) 1:ArrayBuffer(0, 3, 0, 0)
2:ArrayBuffer(0, 0, 1, 0) 2:ArrayBuffer(0, 0, 2, 0) 2:ArrayBuffer(0, 0, 3, 0)
3:ArrayBuffer(0, 0, 0, 1) 3:ArrayBuffer(0, 0, 0, 2) 3:ArrayBuffer(0, 0, 0, 3)
But when the cache method is invoked on the parentRDD, the array values for newVert1, newVert2, newVert3 RDD are
0:ArrayBuffer(1, 0, 0, 0) 0:ArrayBuffer(3, 0, 0, 0) 0:ArrayBuffer(6, 0, 0, 0)
1:ArrayBuffer(0, 1, 0, 0) 1:ArrayBuffer(0, 3, 0, 0) 1:ArrayBuffer(0, 6, 0, 0)
2:ArrayBuffer(0, 0, 1, 0) 2:ArrayBuffer(0, 0, 3, 0) 2:ArrayBuffer(0, 0, 6, 0)
3:ArrayBuffer(0, 0, 0, 1) 3:ArrayBuffer(0, 0, 0, 3) 3:ArrayBuffer(0, 0, 0, 6)
It is something to do with foreach method on RDD. This method is incrementing the values of Array elements.
I have posted the results in the code as comments. I have posted the results in the code as comments below each print statement. Please help me in understanding the cache operation. Thanks
package com.examples
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
object ReCalculationModified {
def main(args: Array[String]) {
val conf = new SparkConf().setMaster("local[1]").setAppName("RecalculationModified")
val sc = new SparkContext(conf)
val vertex = Array(
(0, Array(0, 0, 0, 0)),
(1, Array(0, 0, 0, 0)),
(2, Array(0, 0, 0, 0)),
(3, Array(0, 0, 0, 0)))
val vertexRDD = sc.makeRDD(vertex).map(x => x).cache()
val newVert1 = vertexRDD.map {
case (vid, array) =>
array(vid) += 1
(vid, array)
}
println("--------------newVertex1-------------")
newVert1.foreach(x => println(x._1 + ":" + x._2.toBuffer))
println("--------------VertexRDD-------------")
vertexRDD.foreach(x => println(x._1 + ":" + x._2.toBuffer))
// Output of both vertexRDD and newVert1 is
// 0:ArrayBuffer(1, 0, 0, 0)
// 1:ArrayBuffer(0, 1, 0, 0)
// 2:ArrayBuffer(0, 0, 1, 0)
// 3:ArrayBuffer(0, 0, 0, 1)
val newVert2 = newVert1.map {
case (vid, array) =>
array(vid) += 1
(vid, array)
}
println("--------------newVertex2-------------")
newVert2.foreach(x => println(x._1 + ":" + x._2.toBuffer))
println("--------------VertexRDD-------------")
vertexRDD.foreach(x => println(x._1 + ":" + x._2.toBuffer))
// Output of both vertexRDD and newVert2 is
// 0:ArrayBuffer(3, 0, 0, 0)
// 1:ArrayBuffer(0, 3, 0, 0)
// 2:ArrayBuffer(0, 0, 3, 0)
// 3:ArrayBuffer(0, 0, 0, 3)
val newVert3 = newVert2.map {
case (vid, array) =>
array(vid) += 1
(vid, array)
}
println("--------------newVertex3-------------")
newVert3.foreach(x => println(x._1 + ":" + x._2.toBuffer))
println("--------------VertexRDD-------------")
vertexRDD.foreach(x => println(x._1 + ":" + x._2.toBuffer))
// Output of both vertexRDD and newVert3 is
// 0:ArrayBuffer(6, 0, 0, 0)
// 1:ArrayBuffer(0, 6, 0, 0)
// 2:ArrayBuffer(0, 0, 6, 0)
// 3:ArrayBuffer(0, 0, 0, 6)
val newVert4 = newVert3.map {
case (vid, array) =>
array(vid) += 1
(vid, array)
}
println("--------------newVertex4-------------")
newVert4.foreach(x => println(x._1 + ":" + x._2.toBuffer))
println("--------------VertexRDD-------------")
vertexRDD.foreach(x => println(x._1 + ":" + x._2.toBuffer))
// Output of both vertexRDD and newVert4 is
// 0:ArrayBuffer(10, 0, 0, 0)
// 1:ArrayBuffer(0, 10, 0, 0)
// 2:ArrayBuffer(0, 0, 10, 0)
// 3:ArrayBuffer(0, 0, 0, 10)
val newVert5 = newVert4.map {
case (vid, array) =>
array(vid) += 1
(vid, array)
}
println("--------------newVertex5-------------")
newVert5.foreach(x => println(x._1 + ":" + x._2.toBuffer))
println("--------------VertexRDD-------------")
vertexRDD.foreach(x => println(x._1 + ":" + x._2.toBuffer))
// Output of both vertexRDD and newVert5 is
// 0:ArrayBuffer(15, 0, 0, 0)
// 1:ArrayBuffer(0, 15, 0, 0)
// 2:ArrayBuffer(0, 0, 15, 0)
// 3:ArrayBuffer(0, 0, 0, 15)
}
}

Answer 1

The values are getting updated due to use of Array you had used. Arrays is mutable in Scala. See the below discussion.
Why no immutable arrays in scala standard library?
Also, cache is a transformation in Spark which caches an RDD in memory. Cache only avoids to run the DAG from starting. When cache is used, the processed data will be picked from the memory rather than going through the whole DAG and computing the same.
Caching is one of the features of Spark that makes it 10x to 100x faster than MapReduce