Extract Pairwise Correlation from Correlation Matrix in Spark - apache-spark

Question

I'm trying to extract the pairwise correlation (e.g. pearson) into a spark dataframe. I want to use the pairwise coreelation in table format in further queries and as machine learning input.
So here is a running example:
Data:
import org.apache.spark.sql.{SQLContext, Row, DataFrame}
import org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType, DoubleType}
import org.apache.spark.sql.functions._
// rdd
val rowsRdd: RDD[Row] = sc.parallelize(
Seq(
Row(2.0, 7.0, 1.0),
Row(3.5, 2.5, 0.0),
Row(7.0, 5.9, 0.0)
)
)
// Schema
val schema = new StructType()
.add(StructField("item_1", DoubleType, true))
.add(StructField("item_2", DoubleType, true))
.add(StructField("item_3", DoubleType, true))
// Data frame
val df = spark.createDataFrame(rowsRdd, schema)
Correlation Matrix
import org.apache.spark.ml.feature.VectorAssembler
import org.apache.spark.ml.linalg.Vectors
import org.apache.spark.sql.Row
import org.apache.spark.mllib.linalg._
import org.apache.spark.mllib.stat.Statistics
import org.apache.spark.rdd.RDD
val rows = new VectorAssembler().setInputCols(df.columns).setOutputCol("corr_features")
.transform(df)
.select("corr_features")
.rdd
val items_mllib_vector = rows.map(_.getAs[org.apache.spark.ml.linalg.Vector](0))
.map(org.apache.spark.mllib.linalg.Vectors.fromML)
val correlMatrix: Matrix = Statistics.corr(items_mllib_vector, "pearson")
The output is the correlation matrix of all elements. I would like to extract pairwise each element (i:j) with the correlation coefficent and the name of each element into a dataframe.
Needed output:
item_from | item_to | Correlation
item_1 | item_2 | -0.0096912
item_1 | item_3 | -0.7313071
item_2 | item_3 | 0.68910356

Answer 1

With some help I was able to find a solution:
Get the result into a local Array:
import scala.collection.mutable.ListBuffer
val pairwiseArr = new ListBuffer[Array[Double]]()
for( i <- 0 to correlMatrix.numRows-1){
for(j <- 0 to correlMatrix.numCols-1){
pairwiseArr += Array(i, j, correlMatrix.apply(i,j))
}
}
Transform the Array into a spark Dataframe:
case class pairRow(i: Double, j: Double, corr: Double)
val pairwiseDF = pairwiseArr.map(x => pairRow(x(0), x(1), x(2))).toDF()
display(pairwiseDF
)
Since the Array is a local one it's preferred to use ColumnSimilarities