Scikit-Learn HalvingGridSearchCV Hyperparameter Optimization

HalvingGridSearchCV is an efficient method for hyperparameter optimization that uses successive halving to iteratively eliminate underperforming configurations. It is useful for tuning any supervised learning algorithm in scikit-learn.

The key hyperparameters include the estimator (base model), param_grid (hyperparameter values to search), scoring (performance metric), and n_jobs (number of parallel jobs).

The HalvingGridSearchCV class is experimental, using it requires that we import enable_halving_search_cv.

It is suitable for classification and regression tasks.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.experimental import enable_halving_search_cv
from sklearn.model_selection import HalvingGridSearchCV

# generate binary classification dataset
X, y = make_classification(n_samples=1000, n_features=5, n_classes=2, random_state=42)

# split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# define the base model
model = LogisticRegression()

# define the hyperparameter grid
param_grid = {
    'C': [0.1, 1, 10],
    'solver': ['liblinear', 'saga']
}

# create the HalvingGridSearchCV object
search = HalvingGridSearchCV(model, param_grid, cv=5, scoring='accuracy', n_jobs=-1)

# fit the grid search
search.fit(X_train, y_train)

# report the best score and hyperparameters
print(f'Best score: {search.best_score_:.3f}')
print(f'Best params: {search.best_params_}')

# evaluate the best model on the test set
best_model = search.best_estimator_
accuracy = best_model.score(X_test, y_test)
print(f'Test accuracy: {accuracy:.3f}')

# make a prediction with the best model
new_data = [[2.1, 1.8, 3.5, 1.2, 0.8]]
prediction = best_model.predict(new_data)
print(f'Prediction: {prediction[0]}')

Running the example gives an output like:

Best score: 0.859
Best params: {'C': 0.1, 'solver': 'saga'}
Test accuracy: 0.885
Prediction: 1

The steps are as follows:

1. Prepare the dataset by generating a synthetic classification problem and splitting it into train and test sets.

2. Define the base model (LogisticRegression) and the hyperparameter grid to search over.

3. Create the HalvingGridSearchCV object, specifying the model, parameter grid, cross-validation strategy, scoring metric, and parallel jobs.

4. Fit the grid search on the training data. HalvingGridSearchCV efficiently explores the search space.

5. Retrieve the best model and its hyperparameters. Evaluate the tuned model on the test set.

6. Use the optimized model to make predictions on new, unseen data.

This example demonstrates how to efficiently tune hyperparameters using HalvingGridSearchCV. By leveraging successive halving, it can find well-performing configurations faster than traditional grid search, making it a valuable tool for model optimization in scikit-learn.