Scikit-Learn HalvingRandomSearchCV Hyperparameter Optimization

HalvingRandomSearchCV is an efficient approach for hyperparameter tuning that progressively eliminates low-performing candidates, making it a faster alternative to RandomizedSearchCV. It uses a “halving” strategy, where at each iteration, it retains the best performing half of the candidates and allocates more resources to them.

The key hyperparameters of HalvingRandomSearchCV are factor (the proportion of candidates to retain at each iteration), min_resources (the minimum resources each candidate should be allocated), and max_resources (the maximum resources to allocate per candidate).

The HalvingRandomSearchCV class is experimental, using it requires that we import enable_halving_search_cv.

This example demonstrates hyperparameter tuning for a classification problem, but the approach can be similarly used for regression tasks.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.experimental import enable_halving_search_cv
from sklearn.model_selection import HalvingRandomSearchCV
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score
import numpy as np

# generate multi-class classification dataset
X, y = make_classification(n_samples=1000, n_features=5, n_classes=2, random_state=42)

# split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# define the model
model = DecisionTreeClassifier()

# define the hyperparameter configuration space
params = {
    'max_depth': [3, 5, 7, 9],
    'min_samples_split': [2, 5, 10, 20],
    'min_samples_leaf': [1, 2, 4, 8]
}

# configure the search
search = HalvingRandomSearchCV(model, params, n_candidates=10, factor=2, min_resources=100, max_resources=500, n_jobs=-1, scoring='accuracy', cv=5, random_state=42)

# execute search
result = search.fit(X_train, y_train)

# summarize result
print('Best Score: %.3f' % result.best_score_)
print('Best Params: %s' % result.best_params_)

# evaluate the best model on the test set
best_model = result.best_estimator_
yhat = best_model.predict(X_test)
acc = accuracy_score(y_test, yhat)
print("Test Accuracy: %.3f" % acc)

# make a prediction on new data
row = [[-2.02220122,  0.31563495, -0.82797464, -1.41477081, -1.10395094]]
yhat = best_model.predict(row)
print('Predicted Class: %d' % yhat[0])

Running the example gives an output like:

Best Score: 0.865
Best Params: {'min_samples_split': 5, 'min_samples_leaf': 8, 'max_depth': 9}
Test Accuracy: 0.860
Predicted Class: 0

The key steps in this example are:

1. A synthetic multi-class classification dataset is generated using make_classification() and split into training and test sets.

2. A DecisionTreeClassifier is defined as the model to be tuned, with its hyperparameters left unspecified.

3. The hyperparameter search space is defined as a dictionary (params), specifying the values to try for max_depth, min_samples_split, and min_samples_leaf.

4. HalvingRandomSearchCV is configured with the model, hyperparameter space, number of candidates, halving factor, resource limits, and 5-fold cross-validation.

5. The search is executed with fit(), and the best hyperparameters and score are reported.

6. The best model is evaluated on the test set to get an unbiased estimate of its performance.

7. Finally, the tuned model is used to make a prediction on a new data sample.

This example showcases how HalvingRandomSearchCV can efficiently find good hyperparameter settings for a DecisionTreeClassifier, demonstrating its utility for hyperparameter tuning in scikit-learn.