Hyperparameters optimization gives worse results - scikit-learn

Question

I trained my random forest classifier as follows:
rf = RandomForestClassifier(n_jobs=-1, max_depth = None, max_features = "auto",
min_samples_leaf = 1, min_samples_split = 2,
n_estimators = 1000, oob_score=True, class_weight="balanced",
random_state=0)
​
rf.fit(X_train, y_train)
y_pred = rf.predict(X_test)
​
print("Confusion matrix")
print(metrics.confusion_matrix(y_test, y_pred))
print("F1-score")
print(metrics.f1_score(y_test, y_pred, average="weighted"))
print("Accuracy")
print(metrics.accuracy_score(y_test, y_pred))
print(metrics.classification_report(y_test, y_pred))
and got the following results:
Confusion matrix
[[558 42 2 0 1]
[ 67 399 84 3 2]
[ 30 135 325 48 7]
[ 5 69 81 361 54]
[ 8 17 7 48 457]]
F1-score
0.7459670332027826
Accuracy
0.7473309608540926
precision recall f1-score support
1 0.84 0.93 0.88 603
2 0.60 0.72 0.66 555
3 0.65 0.60 0.62 545
4 0.78 0.63 0.70 570
5 0.88 0.85 0.86 537
Then I decided to perform a hyperparameters optimization in order to improve this result.
clf = RandomForestClassifier(random_state = 0, n_jobs=-1)
param_grid = {
'n_estimators': [1000,2000],
'max_features': [0.2, 0.5, 0.7, 'auto'],
'max_depth' : [None, 10],
'min_samples_leaf': [1, 2, 3, 5],
'min_samples_split': [0.1, 0.2]
}
k_fold = StratifiedKFold(n_splits=10, shuffle=True, random_state=0)
clf = GridSearchCV(estimator=clf,
param_grid=param_grid,
cv=k_fold,
scoring='accuracy',
verbose=True)
clf.fit(X_train, y_train)
But it gave me worse results if I do y_pred = clf.best_estimator_.predict(X_test):
Confusion matrix
[[533 68 0 0 2]
[145 312 70 0 28]
[ 58 129 284 35 39]
[ 21 68 73 287 121]
[ 32 12 3 36 454]]
F1-score
0.6574507466273805
Accuracy
0.6654804270462633
precision recall f1-score support
1 0.68 0.88 0.77 603
2 0.53 0.56 0.55 555
3 0.66 0.52 0.58 545
4 0.80 0.50 0.62 570
5 0.70 0.85 0.77 537
I assume that it's happening because of scoring='accuracy'. Which score should I use to get the same or better result as my initial random forest?

Answer 1

Defining scoring='accuracy' in your gridsearch should not be responsible for this difference, because this would be the default anyway for a Random Forest Classifier.
The reason why you have an unexpected difference here is because you have specified class_weight="balanced" in your first random forest rf, but not in the second classifier clf. As a result, your classes are weighted differently when calculating the accuracy score, which eventually leads to different performance metrics.
To correct this, just define clf through:
clf = RandomForestClassifier(random_state = 0, n_jobs=-1, class_weight="balanced")