Coverage error measures how far we need to go through the ranked list of labels to cover all true labels. It calculates the average number of labels that must be included to ensure all true labels are covered for each sample.
In scikit-learn, the coverage_error() function calculates this metric by taking the true labels and predicted probabilities as inputs. The result is the average number of labels that must be included to cover all the true labels. Lower values indicate better performance, with a perfect score of 1 meaning the true labels are always the top-ranked labels.
Coverage error is used for multilabel classification problems where each instance can have multiple labels. It is not suitable for single-label classification problems or scenarios where label ranking is not important.
from sklearn.datasets import make_multilabel_classification
from sklearn.model_selection import train_test_split
from sklearn.multioutput import MultiOutputClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import MultiLabelBinarizer
from sklearn.metrics import coverage_error
# Generate synthetic multilabel dataset
X, y = make_multilabel_classification(n_samples=1000, n_classes=5, n_labels=3, random_state=42)
# Binarize the labels
mlb = MultiLabelBinarizer()
y = mlb.fit_transform(y)
# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Train a logistic regression classifier
clf = MultiOutputClassifier(LogisticRegression(max_iter=1000, random_state=42))
clf.fit(X_train, y_train)
# Predict on test set
y_pred = clf.predict(X_test)
# Calculate coverage error
cov_err = coverage_error(y_test, y_pred)
print(f"Coverage Error: {cov_err:.2f}")
Running the example gives an output like:
Coverage Error: 1.99
The steps are as follows:
- Generate a synthetic multilabel classification dataset using
make_multilabel_classification(). - Binarize the labels using
MultiLabelBinarizer(). - Split the dataset into training and test sets using
train_test_split(). - Train a logistic regression classifier using
LogisticRegression. - Predict probabilities for the test set using
predict_proba(). - Calculate coverage error using
coverage_error()by comparing true labels to predicted probabilities.
First, we generate a synthetic multilabel classification dataset using the make_multilabel_classification() function from scikit-learn. This function creates a dataset with 1000 samples and 5 classes, where each sample can have up to 3 labels.
Next, we binarize the labels using the MultiLabelBinarizer() function. This converts the list of labels for each sample into a binary format suitable for multilabel classification.
We then split the dataset into training and test sets using the train_test_split() function. We use 80% of the data for training and reserve 20% for testing.
With our data prepared, we train a logistic regression classifier using the LogisticRegression class wrapped in a MultiOutputClassifier. We set the maximum number of iterations to 1000 to ensure convergence. The fit() method is called on the classifier object, passing in the training features (X_train) and labels (y_train) to learn the underlying patterns in the data.
After training, we use the trained classifier to predict labels for the test set by calling the predict() method with X_test. This generates predicted labels for each class for each sample in the test set.
Finally, we evaluate the coverage error of our classifier using the coverage_error() function. This function takes the true labels (y_test) and the predicted labels (y_pred) as input and calculates the average number of labels that must be included to cover all the true labels. The resulting coverage error score is printed, giving us a quantitative measure of our classifier’s performance.
This example demonstrates how to use the coverage_error() function from scikit-learn to evaluate the performance of a multilabel classification model.