PCA in Spark 2.3.0 Examples with PySpark - apache-spark

Question

I have a Spark dataframe I would like to use to run a simple PCA example. I have looked at this example and notice this works because they transpose the features into vectors:
from pyspark.ml.linalg import Vectors
>>> data = [(Vectors.sparse(5, [(1, 1.0), (3, 7.0)]),),
... (Vectors.dense([2.0, 0.0, 3.0, 4.0, 5.0]),),
... (Vectors.dense([4.0, 0.0, 0.0, 6.0, 7.0]),)]
>>> df = spark.createDataFrame(data,["features"])
>>> pca = PCA(k=2, inputCol="features", outputCol="pca_features")
I am trying to reproduce the same kind of simple PCA by using a Spark Dataframe I have created my self. How would I transform my Spark DataFrame into a form similar to the above so I could run it with one input column and one output column?
I looked into using RowMatrix as shown here but I am not understanding if this is the way to go (see error below).
>>>from pyspark.mllib.linalg import Vectors
>>>from pyspark.mllib.linalg.distributed import RowMatrix
>>>from pyspark.ml.feature import PCA
>>>master = pd.read_parquet('master.parquet',engine='fastparquet')
>>>A = sc.parallelize(master)
>>>mat = RowMatrix(A)
>>>pc = mat.computePrincipalComponents(4)
Py4JJavaError: An error occurred while calling
o382.computePrincipalComponents. : org.apache.spark.SparkException:
Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times,
most recent failure: Lost task 0.0 in stage 1.0 (TID 1, localhost,
executor driver): org.apache.spark.api.python.PythonException:
Traceback (most recent call last)

Answer 1

In Pyspark for mllib library you need to convert all the features into a single feature vector.
You can do the same using a Vector Assembler:
https://spark.apache.org/docs/latest/ml-features.html#vectorindexer
from pyspark.ml.feature import VectorAssembler
assembler = VectorAssembler(inputCols=inputColumnsList,outputCol='features')
assembler.transform(df)
Where inputColsList contains a list of all the features u want to use