Generating Difficult Classification Data Sets using scikit-learn - scikit-learn

Question

I am trying to generate a range of synthetic data sets using make_classification in scikit-learn, with varying sample sizes, prevalences (i.e., proportions of the positive class), and accuracies. Varying the sample size and prevalence is fairly straightforward, but I am having difficult generating any data sets that have less than 50% accuracy using logistic regression. Playing around with the number of informative columns, the number of clusters per class, and the flip_y parameter (which randomly flips the class of a given proportion of observations) seem to reduce the accuracy, but not as much as I would like. Is there a way to vary the parameters of make_classification in such a way to reduce this further (e.g., to 20%)?
Thanks!

Answer 1

Generally, the combination of a fairly low number of n_samples, a high probability of randomly flipping the label flip_y and a large number of n_classes should get you where you want.
You can try the following:
from sklearn.cross_validation import cross_val_score
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
lr = LogisticRegression()
# 2-class problem
X, y = make_classification(n_samples=100, n_informative=2, flip_y=0.8, random_state=42)
cross_val_score(estimator=lr, X=X, y=y, scoring='accuracy', cv=10)
# Output
array([ 0.54545455, 0.27272727, 0.45454545, 0.2 , 0.4 ,
0.5 , 0.7 , 0.55555556, 0.55555556, 0.44444444])
# 8-class problem
X, y = make_classification(n_samples=100, n_classes=8, n_informative=4, n_clusters_per_class=1, flip_y=0.5, random_state=42)
cross_val_score(estimator=lr, X=X, y=y, scoring='accuracy', cv=5)
# Output
array([ 0.16666667, 0.19047619, 0.15 , 0.16666667, 0.29411765])

Answer 2

In case you go with binary classification only, you should carefully choose flip_y. If, for example, you choose flip_y to be high, that means you flip almost every label, hence making the problem easier!. (the consistency is preserved)
Hence, in binary classification, flip_y is really min(flip_y,1-flip_y), and setting it as 0.5 will make the classification really hard.
Another thing you can do: after creating the data, do dimension reduction, using PCA:
from sklearn.cross_validation import cross_val_score
from sklearn.datasets import make_classification
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
clf = LogisticRegression()
X, y = make_classification(n_samples=10000, n_informative=18,n_features=20, flip_y=0.15, random_state=217)
print cross_val_score(estimator=clf, X=X, y=y, scoring='accuracy', cv=4)
#prints [ 0.80287885 0.7904 0.796 0.78751501]
pca = PCA(n_components=10)
X = pca.fit_transform(X)
print cross_val_score(estimator=clf, X=X, y=y, scoring='accuracy', cv=4)
#prints [ 0.76409436 0.7684 0.7628 0.75830332]
you can reduce n_components to get even poorer results, while having the original number of features:
pca = PCA(n_components=1)
X = pca.fit_transform(X)
X = np.concatenate((X, np.random.rand(X.shape[0],19)),axis=1) #concatenating random features
cross_val_score(estimator=clf, X=X, y=y, scoring='accuracy', cv=10)
print cross_val_score(estimator=clf, X=X, y=y, scoring='accuracy', cv=4)
#prints [ 0.5572 0.566 0.5552 0.5664]
Getting less than 50% accuracy is 'hard' - even when you take random vectors, the expectancy of accuracy is still 0.5:
X = np.random.rand(10000,20)
print np.average(cross_val_score(estimator=clf, X=X, y=y, scoring='accuracy', cv=100))
#prints 0.501489999
So 55% accuracy is considered very low.