Getting requirement failed exception on Applying K Means to PCA data - apache-spark

Question

I was trying K means from Spark ML, on some scaled features which are a column of a spark dataframe. It works as expected. However, as soon as I apply PCA on the same feature set, and try K means on that, I get the error
java.lang.IllegalArgumentException: requirement failed
My original data before PCA has 108 features, which have been Indexed and Assembled. K means runs fine on this and there are no nulls/NANs. After this, I'm setting the number of principal components as 6,running PCA. On the PCA data, K means is being run, and I am getting the above error.
A look at the stacktrace shows that the error is in fastSquaredDistance in MLUtils
.......at scala.Predef$.require(Predef.scala:212)
at org.apache.spark.mllib.util.MLUtils$.fastSquaredDistance(MLUtils.scala:486)
at org.apache.spark.mllib.clustering.KMeans$.fastSquaredDistance(KMeans.scala:589)
at org.apache.spark.mllib.clustering.KMeans$$anonfun$findClosest$1.apply(KMeans.scala:563)
at org.apache.spark.mllib.clustering.KMeans$$anonfun$findClosest$1.apply(KMeans.scala:557)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at org.apache.spark.mllib.clustering.KMeans$.findClosest(KMeans.scala:557)
at org.apache.spark.mllib.clustering.KMeans$.pointCost(KMeans.scala:580)
at org.apache.spark.mllib.clustering.KMeansModel$$anonfun$computeCost$1.apply(KMeansModel.scala:88)
at org.apache.spark.mllib.clustering.KMeansModel$$anonfun$computeCost$1.apply(KMeansModel.scala:88)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1336)
at scala.collection.TraversableOnce$class.fold(TraversableOnce.scala:212)
at scala.collection.AbstractIterator.fold(Iterator.scala:1336)
at org.apache.spark.rdd.RDD$$anonfun$fold$1$$anonfun$20.apply(RDD.scala:1087)
at org.apache.spark.rdd.RDD$$anonfun$fold$1$$anonfun$20.apply(RDD.scala:1087)
at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2120)
at org.apache.spark.SparkContext$$anonfun$33.apply(SparkContext.scala:2120)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
This throws an exception when the dimensions of the vectors are not the same, or when the vector norm is negative
I checked the size of all the vectors in the PCA column and all of the, have size 6( as expected) . As for non negative norm, I can't figure out how that's even possible.
Does anyone have leads into what could be happening?
If my question isn't formatted well, I apologize, it's my first time asking a question here
The code that's throwing the error is below
....import org.apache.spark.ml.clustering.KMeans
val kmeans = new KMeans().setK(6).setSeed(42L)
val model = kmeans.fit(scaledDataPcaNN).setFeaturesCol("pcaFeatures")
val predictions = model.transform(scaledDataPca)
val WSSSE = model.computeCost(scaledDataPca)