Scikit-Learn Get RandomizedSearchCV "multimetric_" Attribute

RandomizedSearchCV is a versatile tool for hyperparameter optimization that allows you to evaluate models using multiple metrics simultaneously.

The multimetric_ attribute of a fitted RandomizedSearchCV object indicates whether multiple evaluation metrics were used during the search.

Knowing if multiple metrics were used is important because it affects how you interpret the search results and select the best model.

When multimetric_ is True, the best_index_, best_score_, and best_params_ attributes correspond to the hyperparameter configuration that performed best according to the refit metric, which is the first metric specified in the scoring parameter.

from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

# Generate a random multiclass classification dataset
X, y = make_classification(n_samples=100, n_classes=3, n_informative=5, n_redundant=5, random_state=42)

# Set up a RandomForestClassifier
rf = RandomForestClassifier(random_state=42)

# Define hyperparameter distributions to sample from
param_dist = {
    'n_estimators': randint(5, 10),
    'max_depth': [3, 5, 10, None],
    'min_samples_split': randint(2, 10),
}

# Run random search with multiple evaluation metrics
random_search = RandomizedSearchCV(rf, param_distributions=param_dist, n_iter=10, cv=5,
                                   scoring=['accuracy', 'f1_weighted'], refit='accuracy', random_state=42)
random_search.fit(X, y)

# Access the multimetric_ attribute
is_multimetric = random_search.multimetric_

# Print the value of multimetric_
print("Multiple metrics used?", is_multimetric)

Running the example gives an output like:

Multiple metrics used? True

The steps are as follows:

1. Prepare a synthetic multiclass classification dataset using make_classification.
2. Configure a RandomForestClassifier and define distributions to sample hyperparameters from.
3. Run RandomizedSearchCV with the classifier, hyperparameter distributions, 10 iterations, 5-fold cross-validation, and multiple evaluation metrics (‘accuracy’ and ‘f1_weighted’).
4. After fitting, access the multimetric_ attribute from the random_search object.
5. Print the value of multimetric_ to check if multiple metrics were used.

By checking the multimetric_ attribute, you can determine if the random search used multiple evaluation metrics, which is crucial for interpreting the search results correctly and selecting the best model based on the appropriate metric.