RandomizedSearchCV is a versatile tool for hyperparameter optimization that allows you to evaluate models using multiple metrics simultaneously.
The multimetric_
attribute of a fitted RandomizedSearchCV object indicates whether multiple evaluation metrics were used during the search.
Knowing if multiple metrics were used is important because it affects how you interpret the search results and select the best model.
When multimetric_
is True, the best_index_
, best_score_
, and best_params_
attributes correspond to the hyperparameter configuration that performed best according to the refit metric, which is the first metric specified in the scoring
parameter.
from sklearn.datasets import make_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint
# Generate a random multiclass classification dataset
X, y = make_classification(n_samples=100, n_classes=3, n_informative=5, n_redundant=5, random_state=42)
# Set up a RandomForestClassifier
rf = RandomForestClassifier(random_state=42)
# Define hyperparameter distributions to sample from
param_dist = {
'n_estimators': randint(5, 10),
'max_depth': [3, 5, 10, None],
'min_samples_split': randint(2, 10),
}
# Run random search with multiple evaluation metrics
random_search = RandomizedSearchCV(rf, param_distributions=param_dist, n_iter=10, cv=5,
scoring=['accuracy', 'f1_weighted'], refit='accuracy', random_state=42)
random_search.fit(X, y)
# Access the multimetric_ attribute
is_multimetric = random_search.multimetric_
# Print the value of multimetric_
print("Multiple metrics used?", is_multimetric)
Running the example gives an output like:
Multiple metrics used? True
The steps are as follows:
- Prepare a synthetic multiclass classification dataset using
make_classification
. - Configure a
RandomForestClassifier
and define distributions to sample hyperparameters from. - Run
RandomizedSearchCV
with the classifier, hyperparameter distributions, 10 iterations, 5-fold cross-validation, and multiple evaluation metrics (‘accuracy’ and ‘f1_weighted’). - After fitting, access the
multimetric_
attribute from therandom_search
object. - Print the value of
multimetric_
to check if multiple metrics were used.
By checking the multimetric_
attribute, you can determine if the random search used multiple evaluation metrics, which is crucial for interpreting the search results correctly and selecting the best model based on the appropriate metric.