Multi-class ROC curve calculation for predicted and actual data - python-3.x

Question

I have a task to calculate ROC curve for this kind of data:
Initially the classes are strings.
Actual data
Predicted data
Confidence level
Class 1
Class 1
0.9997
Class 2
Class 3
0.9978
Class 4
Class 4
0.9988
Class 1
Class 1
0.9984
I encoded classes and made new columns for the labels:
Actual data
Predicted data
Confidence level
label_actual
label_predicted
Class 1
Class 1
0.9997
0
0
Class 2
Class 3
0.9978
1
2
Class 4
Class 4
0.9988
3
3
Class 1
Class 1
0.9984
0
0
Code for this transformations:
data = pd.read_csv('classification_data.csv')
data.actual_type = pd.Categorical(data.actual__type)
data.predicted_type = pd.Categorical(data.predicted_type)
data['label_actual'] = data.actual_type.cat.codes
data['label_predicted'] = data.predicted_type.cat.codes
I also have used these 3 function to compute ROC:
def calculate_tpr_fpr(y_real, y_pred):
'''
Calculates the True Positive Rate (tpr) and the True Negative Rate (fpr) based on real and predicted observations
Args:
y_real: The list or series with the real classes
y_pred: The list or series with the predicted classes
Returns:
tpr: The True Positive Rate of the classifier
fpr: The False Positive Rate of the classifier
'''
# Calculates the confusion matrix and recover each element
cm = confusion_matrix(y_real, y_pred)
TN = cm[0, 0]
FP = cm[0, 1]
FN = cm[1, 0]
TP = cm[1, 1]
# Calculates tpr and fpr
tpr = TP/(TP + FN) # sensitivity - true positive rate
fpr = 1 - TN/(TN+FP) # 1-specificity - false positive rate
return tpr, fpr
def get_all_roc_coordinates(y_real, y_proba):
'''
Calculates all the ROC Curve coordinates (tpr and fpr) by considering each point as a treshold for the predicion of the class.
Args:
y_real: The list or series with the real classes.
y_proba: The array with the probabilities for each class, obtained by using the `.predict_proba()` method.
Returns:
tpr_list: The list of TPRs representing each threshold.
fpr_list: The list of FPRs representing each threshold.
'''
tpr_list = [0]
fpr_list = [0]
for i in range(len(y_proba)):
threshold = y_proba[i]
y_pred = y_proba >= threshold
tpr, fpr = calculate_tpr_fpr(y_real, y_pred)
tpr_list.append(tpr)
fpr_list.append(fpr)
return tpr_list, fpr_list
def plot_roc_curve(tpr, fpr, scatter = True, ax = None):
'''
Plots the ROC Curve by using the list of coordinates (tpr and fpr).
Args:
tpr: The list of TPRs representing each coordinate.
fpr: The list of FPRs representing each coordinate.
scatter: When True, the points used on the calculation will be plotted with the line (default = True).
'''
if ax == None:
plt.figure(figsize = (5, 5))
ax = plt.axes()
if scatter:
sns.scatterplot(x = fpr, y = tpr, ax = ax)
sns.lineplot(x = fpr, y = tpr, ax = ax)
sns.lineplot(x = [0, 1], y = [0, 1], color = 'green', ax = ax)
plt.xlim(-0.05, 1.05)
plt.ylim(-0.05, 1.05)
plt.xlabel("False Positive Rate")
plt.ylabel("True Positive Rate")
The code I used to calculate and plot ROC curve:
classes_labeled = data['label_actual'].unique()
plt.figure(figsize = (12, 8))
bins = [i/20 for i in range(20)] + [1]
roc_auc_ovr = {}
for i in range(len(classes)):
# Gets the class
c = classes_labeled[i]
# Prepares an auxiliar dataframe to help with the plots
df_aux = pd.DataFrame()
df_aux['class'] = [1 if y == c else 0 for y in data['label_actual']]
df_aux['prob'] = data['label_predicted']
df_aux = df_aux.reset_index(drop = True)
# Plots the probability distribution for the class and the rest
ax = plt.subplot(2, 5, i+1)
sns.histplot(x = "prob", data = df_aux, hue = 'class', color = 'b', ax = ax, bins = bins)
ax.set_title(c)
ax.legend([f"Class: {c}", "Rest"])
ax.set_xlabel(f"P(x = {c})")
# Calculates the ROC Coordinates and plots the ROC Curves
ax_bottom = plt.subplot(2, 5, i+6)
tpr, fpr = get_all_roc_coordinates(df_aux['class'], df_aux['prob'])
plot_roc_curve(tpr, fpr, scatter = False, ax = ax_bottom)
ax_bottom.set_title("ROC Curve OvR")
# Calculates the ROC AUC OvR
roc_auc_ovr[c] = roc_auc_score(df_aux['class'], df_aux['prob'])
plt.tight_layout()
# Displays the ROC AUC for each class
avg_roc_auc = 0
i = 0
for k in roc_auc_ovr:
avg_roc_auc += roc_auc_ovr[k]
i += 1
print(f"{k} ROC AUC OvR: {roc_auc_ovr[k]:.4f}")
print(f"average ROC AUC OvR: {avg_roc_auc/i:.4f}")
The results I've got:
0 ROC AUC OvR: 0.0069
3 ROC AUC OvR: 0.9878
1 ROC AUC OvR: 0.7296
4 ROC AUC OvR: 0.9993
2 ROC AUC OvR: 0.7675
average ROC AUC OvR: 0.6982
The problem is that the 0 Class is calculated wrong. You can see that on the first line of the results. How can I fix that?
Thank you.