I'm using the corr from mllib with basic interface
like
val a:RDD[Double] = sc.makeRDD(Seq(1., 1., 0.))
val b:RDD[Double] = sc.makeRDD(Seq(1., -1., 0.))
val r = Statistics.corr(a, b)
println(r)
Is there a possibility to have casewise or pairwise removal of NAN and Infinity values?
By Default Mllib provides NAN as a result of corr in case of infinity or NAN values.
To my knowledge, there is no built-in function and you need to filter those values out by your own. One approach is to use java.Double (http://docs.oracle.com/javase/7/docs/api/java/lang/Double.html) functionality:
import java.lang.Double.isNaN
import java.lang.Double.isInfinite
val filtered1 = data1.filter((!isNaN(_))&&(!isInfinite(_)))
val filtered2 = data2.filter((!isNaN(_))&&(!isInfinite(_)))
val r = Statistics.corr(filtered1, filtered2)
println(r)
Related
I would like to ask what's the different between OneHotEncoder and mode_onehot_pipe
mode_onehot_pipe = Pipeline([
('encoder', SimpleImputer(strategy = 'most_frequent')),
('one hot encoder', OneHotEncoder(handle_unknown = 'ignore'))])
transformer = ColumnTransformer([
('one hot', OneHotEncoder(handle_unknown = 'ignore'), ['Gender', 'Age', 'Working_Status', 'Annual_Income', 'Visit_Duration', 'Spending_Time', 'Outlet_Location', 'Member_Card', 'Average_Spending']),
('mode_onehot_pipe', mode_onehot_pipe, ['Visit_Plan'])], remainder = 'passthrough')
Thanks a lot!
The main difference between the two is the way they handle nan values.
mode_onehot_pipe will replace nan by the most frequent value according to the SimpleImputer configuration while OneHotEncoder will create a category for nan values.
If you pass the same feature, you will end up with one extra feature for the OneHotEncoder which will represents the nan values.
I have a big pyspark data frame. I want to get its correlation matrix. I know how to get it with a pandas data frame.But my data is too big to convert to pandas. So I need to get the result with pyspark data frame.I searched other similar questions, the answers don't work for me.
Can any body help me? thanks!
Data example:
data example
Welcome to SO!
Example data
I prepared some dummy data for easier replication (perhaps next time you may supply some easy to copy data, too ;-)):
data = pd.DataFrame(np.random.random((10, 5)),
columns=["x{}".format(x) for x in range(5)])
df = spark.createDataFrame(data)
df.show()
And here is the data:
+-------------------+-------------------+-------------------+-------------------+--------------------+
| x0| x1| x2| x3| x4|
+-------------------+-------------------+-------------------+-------------------+--------------------+
| 0.9965335347601945|0.09311299224360992| 0.9273393764180728| 0.8523333283310564| 0.5040716744686445|
| 0.2341313103221958| 0.9356109544246494| 0.6377089480113576| 0.8129047787928055| 0.22215891357547046|
| 0.6310473705907303| 0.2040705293700683|0.17329601185489396| 0.9062007987480959| 0.44105687572209895|
|0.27711903958232764| 0.9434521502343274| 0.9300724702792151| 0.9916836130997986| 0.6869145183972896|
| 0.8247010263098201| 0.6029990758603708|0.07266306799434707| 0.6808038838294564| 0.27937146479120245|
| 0.7786370627473335|0.17583334607075107| 0.8467715537463528| 0.67702427694934| 0.8976402177586831|
|0.40620117097757724| 0.5080531043890719| 0.3722402520743703|0.14555317396545808| 0.7954133091360741|
|0.20876805543974553| 0.9755867281355178| 0.7570617946515066| 0.6974893162590945|0.054708580878511825|
|0.47979629269402546| 0.1851379589735923| 0.4786682088989791| 0.6809358266732168| 0.8829180507209633|
| 0.1122983875801804|0.45310988757198734| 0.4713203140134805|0.45333792855503807| 0.9189083355172629|
+-------------------+-------------------+-------------------+-------------------+--------------------+
Solution
There is a correlation function in the ml subpackage pyspark.ml.stat. However, it requires you to provide a column of type Vector. So you need to convert your columns into a vector column first using the VectorAssembler and then apply the correlation:
from pyspark.ml.stat import Correlation
from pyspark.ml.feature import VectorAssembler
# convert to vector column first
vector_col = "corr_features"
assembler = VectorAssembler(inputCols=df.columns, outputCol=vector_col)
df_vector = assembler.transform(df).select(vector_col)
# get correlation matrix
matrix = Correlation.corr(df_vector, vector_col)
If you want to get the result as a numpy array (on your driver), you can use the following:
matrix.collect()[0]["pearson({})".format(vector_col)].values
array([ 1. , -0.66882741, -0.06459055, 0.21802534, 0.00113399,
-0.66882741, 1. , 0.14854203, 0.09711389, -0.5408654 ,
-0.06459055, 0.14854203, 1. , 0.33513733, 0.09001684,
0.21802534, 0.09711389, 0.33513733, 1. , -0.37871581,
0.00113399, -0.5408654 , 0.09001684, -0.37871581, 1. ])
Building on the answer of #pansen, but to better visualize the result, you can also use...
1. easy visualization:
matrix = Correlation.corr(df_vector, 'corr_vector').collect()[0][0]
corr_matrix = matrix.toArray().tolist()
corr_matrix_df = pd.DataFrame(data=corr_matrix, columns = numeric_variables, index=numeric_variables)
corr_matrix_df .style.background_gradient(cmap='coolwarm').set_precision(2)
2. better visualization:
import seaborn as sns
import matplotlib.pyplot as plt
plt.figure(figsize=(16,5))
sns.heatmap(corr_matrix_df,
xticklabels=corr_matrix_df.columns.values,
yticklabels=corr_matrix_df.columns.values, cmap="Greens", annot=True)
Clearer:
from pyspark.ml.stat import Correlation
from pyspark.ml.feature import VectorAssembler
# convert to vector column first
vector_col = "corr_features"
assembler = VectorAssembler(inputCols=df.columns, outputCol=vector_col)
df_vector = assembler.transform(df).select(vector_col)
matrix = Correlation.corr(df_vector, vector_col)
cor_np = matrix.collect()[0][matrix.columns[0]].toArray()
And following on from #Artur, a plotly version:
import numpy as np
import plotly.graph_objects as go
matrix = Correlation.corr(df_vector, vector_col).collect()[0][0].toArray()
# to only show one triangle
m = matrix
m[np.triu_indices(m.shape[0], 0)] = None
corr_matrix = m.tolist()
corr_matrix_df = pd.DataFrame(data=corr_matrix, columns = numeric_columns, index=numeric_columns)
labels = corr_matrix_df.columns.values
fig = go.Figure(data=go.Heatmap(
z=corr_matrix_df,
x = labels,
y = labels,
text=corr_matrix_df.round(2),
texttemplate="%{text}",
textfont={"size":8},
colorscale='greens'
)
)
fig.update_xaxes(showticklabels=False)
fig.update_yaxes(autorange="reversed")
fig.show()
In general, all other things being equal, which would be expected to run more quickly?
val a = myRDD.sample(true, 0.01)
val b = myRDD.sample(false, 0.01)
Trained a RandomForest as this (Spark 1.6.0)
val numClasses = 4 // 0-2
val categoricalFeaturesInfo = Map[Int, Int]()
val numTrees = 9
val featureSubsetStrategy = "auto" // Let the algorithm choose.
val impurity = "gini"
val maxDepth = 6
val maxBins = 32
val model = RandomForest.trainClassifier(trainRDD, numClasses,
categoricalFeaturesInfo, numTrees,
featureSubsetStrategy, impurity,
maxDepth, maxBins)
input labels:
labels = labeledRDD.map(lambda lp: lp.label).distinct().collect()
for label in sorted(labels):
print label
0.0
1.0
2.0
But the output only contain only two classes:
metrics = MulticlassMetrics(labelsAndPredictions)
df_confusion = metrics.confusionMatrix()
display_cm(df_confusion)
Output:
83017.0 81.0 0.0
8703.0 2609.0 0.0
10232.0 255.0 0.0
Output from when I load the same model in pyspark and run it against the other data (parts of the above)
DenseMatrix([[ 1.75280000e+04, 3.26000000e+02],
[ 3.00000000e+00, 1.27400000e+03]])
It got better... I used pearson correlation to figure out which columns did not have any correlation. Deletes the ten lowest correlating columns and now I get ok results:
Test Error = 0.0401823
precision = 0.959818
Recall = 0.959818
ConfusionMatrix([[ 17323., 0., 359.],
[ 0., 1430., 92.],
[ 208., 170., 1049.]])
Using MLLib LinearRegressionWithSGD for the dummy data set (y, x1, x2) for y = (2*x1) + (3*x2) + 4 is producing wrong intercept and weights. Actual data used is,
x1 x2 y
1 0.1 6.3
2 0.2 8.6
3 0.3 10.9
4 0.6 13.8
5 0.8 16.4
6 1.2 19.6
7 1.6 22.8
8 1.9 25.7
9 2.1 28.3
10 2.4 31.2
11 2.7 34.1
I set the following input parameters and got the below model outputs
[numIterations, step, miniBatchFraction, regParam] [intercept, [weights]]
[5,9,0.6,5] = [2.36667135839938E13, weights:[1.708772545209758E14, 3.849548062850367E13] ]
[2,default,default,default] = [-2495.5635231554793, weights:[-19122.41357929275,-4308.224496146531]]
[5,default,default,default] = [2.875191315671051E8, weights: [2.2013802074495964E9,4.9593017130199933E8]]
[20,default,default,default] = [-8.896967235537095E29, weights: [-6.811932001659158E30,-1.5346020624812824E30]]
Need to know,
How do i get the correct intercept and weights [4, [2, 3]] for the above mentioned dummy data.
Will tuning the step size help in convergence? I need to run this in a automated manner for several hundred variables, so not keen to do that.
Should I scale the data? How will it help?
Below is the code used to generate these results.
object SciBenchTest {
def main(args: Array[String]): Unit = run
def run: Unit = {
val sparkConf = new SparkConf().setAppName("SparkBench")
val sc = new SparkContext(sparkConf)
// Load and parse the dummy data (y, x1, x2) for y = (2*x1) + (3*x2) + 4
// i.e. intercept should be 4, weights (2, 3)?
val data = sc.textFile("data/dummy.csv")
// LabeledPoint is (label, [features])
val parsedData = data.map { line =>
val parts = line.split(',')
val label = parts(2).toDouble
val features = Array(parts(0), parts(1)) map (_.toDouble)
LabeledPoint(label, Vectors.dense(features))
}
//parsedData.collect().foreach(x => println(x));
// Scale the features
/*val scaler = new StandardScaler(withMean = true, withStd = true)
.fit(parsedData.map(x => x.features))
val scaledData = parsedData
.map(x =>
LabeledPoint(x.label,
scaler.transform(Vectors.dense(x.features.toArray))))
scaledData.collect().foreach(x => println(x));*/
// Building the model: SGD = stochastic gradient descent
val numIterations = 20 //5
val step = 9.0 //9.0 //0.7
val miniBatchFraction = 0.6 //0.7 //0.65 //0.7
val regParam = 5.0 //3.0 //10.0
//val model = LinearRegressionWithSGD.train(parsedData, numIterations, step) //scaledData
val algorithm = new LinearRegressionWithSGD() //train(parsedData, numIterations)
algorithm.setIntercept(true)
algorithm.optimizer
//.setMiniBatchFraction(miniBatchFraction)
.setNumIterations(numIterations)
//.setStepSize(step)
//.setGradient(new LeastSquaresGradient())
//.setUpdater(new SquaredL2Updater()) //L1Updater //SimpleUpdater //SquaredL2Updater
//.setRegParam(regParam)
val model = algorithm.run(parsedData)
println(s">>>> Model intercept: ${model.intercept}, weights: ${model.weights}")
// Evaluate model on training examples
val valuesAndPreds = parsedData.map { point =>
val prediction = model.predict(point.features)
(point.label, point.features, prediction)
}
// Print out features, actual and predicted values...
valuesAndPreds.take(10).foreach({ case (v, f, p) =>
println(s"Features: ${f}, Predicted: ${p}, Actual: ${v}")
})
}
}
As described in the documentation
https://spark.apache.org/docs/1.0.2/mllib-optimization.html
selecting the best step-size for SGD methods can often be delicate.
I would try with lover values, for example
// Build linear regression model
var regression = new LinearRegressionWithSGD().setIntercept(true)
regression.optimizer.setStepSize(0.001)
val model = regression.run(parsedData)
Adding the stepsize did not help us much.
We used the following parameters to calculate the intercept/weights and loss and used the same to construct a linear regression model in order to predict our features. Thanks #selvinsource for pointing me in the correct direction.
val data = sc.textFile("data/dummy.csv")
// LabeledPoint is (label, [features])
val parsedData = data.map { line =>
val parts = line.split(',')
val label = parts(2).toDouble
val features = Array(parts(0), parts(1)) map (_.toDouble)
(label, MLUtils.appendBias(Vectors.dense(features)))
}.cache()
val numCorrections = 5 //10//5//3
val convergenceTol = 1e-4 //1e-4
val maxNumIterations = 20 //20//100
val regParam = 0.00001 //0.1//10.0
val (weightsWithIntercept, loss) = LBFGS.runLBFGS(
parsedData,
new LeastSquaresGradient(),//LeastSquaresGradient
new SquaredL2Updater(), //SquaredL2Updater(),SimpleUpdater(),L1Updater()
numCorrections,
convergenceTol,
maxNumIterations,
regParam,
Vectors.dense(0.0, 0.0, 0.0))//initialWeightsWithIntercept)
loss.foreach(println)
val model = new LinearRegressionModel(
Vectors.dense(weightsWithIntercept.toArray.slice(0, weightsWithIntercept.size - 1)),
weightsWithIntercept(weightsWithIntercept.size - 1))
println(s">>>> Model intercept: ${model.intercept}, weights: ${model.weights}")
// Evaluate model on training examples
val valuesAndPreds = parsedData.collect().map { point =>
var prediction = model.predict(Vectors.dense(point._2.apply(0), point._2.apply(1)))
(prediction, point._1)
}
// Print out features, actual and predicted values...
valuesAndPreds.take(10).foreach({ case (v, f) =>
println(s"Features: ${f}, Predicted: ${v}")//, Actual: ${v}")
})