Explanation for coverage_error metric in scikit learn - scikit-learn

Question

I am not understanding how the coverage_error is calculated in scikit learn, available in sklearn.metrics module. Explanation in the docs is as below:
The coverage_error function computes the average number of labels that have to be included in the final prediction such that all true labels are predicted.
For eg:
import numpy as np
from sklearn.metrics import coverage_error
y_true = np.array([[1, 0, 0], [0, 1, 1]])
y_score = np.array([[1, 0, 0], [0, 1, 1]])
print coverage_error(y_true, y_score)
1.5
As per my understanding, here we need to include 3 labels from the prediction to get all labels in y_true. So coverage error = 3/2, ie, 1.5. But I am not able to understand what happens in the below cases:
>>> y_score = np.array([[1, 0, 0], [0, 0, 1]])
>>> print coverage_error(y_true, y_score)
2.0
>>> y_score = np.array([[1, 0, 1], [0, 1, 1]])
>>> print coverage_error(y_true, y_score)
2.0
How come the error is same in both the cases?

Answer 1

You can have a look at User Guide 3.3.3. Multilabel ranking metrics
with
One thing you need to take care is how to compute ranks and break ties in ranking y_score.
To be specific, the first case:
In [4]: y_true
Out[4]:
array([[1, 0, 0],
[0, 1, 1]])
In [5]: y_score
Out[5]:
array([[1, 0, 0],
[0, 0, 1]])
For the 1st sample, the 1st true label is true, and the rank of 1st score is 1.
For the 2ed sample, the 2ed and 3rd true label are true, and the ranks of score are 3 and 1 respectively, so the max rank is 3.
The average is (3+1)/2=2.
the second case:
In [7]: y_score
Out[7]:
array([[1, 0, 1],
[0, 1, 1]])
For the 1st sample, the 1st true label is true, and the rank of 1st score is 2.
For the 2ed sample, the 2ed and 3rd true label are true, and the ranks of score are 2 and 2 respectively, so the max rank is 2.
The average is (2+2)/2=2.
Edit:
The rank is within one sample of y_score. The formula says the rank of a label is the number of labels (including itself) whose score is greater than or equal to its score.
It is just like sort the labels by y_score, and the label with largest score is ranked 1, the second largest is ranked 2, the third largest is ranked 3, etc. But if the second and third largest labels have the same score, they are both ranked 3.
Notice that y_score is
Target scores, can either be probability estimates of the positive class, confidence values, or binary decisions.
The goal is to have all true labels predicted, so we need to include all the labels with higher or equal scores than the true label.