Configure MLPRegressor "validation_fraction" Parameter

The validation_fraction parameter in scikit-learn’s MLPRegressor controls the proportion of training data to set aside as a validation set for early stopping.

Multi-layer Perceptron (MLP) is a type of artificial neural network used for regression tasks. Early stopping is a regularization technique that helps prevent overfitting by monitoring the model’s performance on a validation set during training.

The validation_fraction parameter determines the size of the validation set used for early stopping. A larger value provides a more reliable estimate of generalization performance but reduces the amount of data available for training.

The default value for validation_fraction is 0.1, meaning 10% of the training data is used for validation.

In practice, values between 0.1 and 0.3 are commonly used, depending on the size of the dataset and the complexity of the problem.

from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPRegressor
from sklearn.metrics import mean_squared_error
import numpy as np

# Generate synthetic dataset
X, y = make_regression(n_samples=1000, n_features=10, noise=0.1, random_state=42)

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train with different validation_fraction values
val_fractions = [0.1, 0.2, 0.3]
mse_scores = []

for val_frac in val_fractions:
    mlp = MLPRegressor(hidden_layer_sizes=(100, 50), max_iter=1000,
                       validation_fraction=val_frac, random_state=42)
    mlp.fit(X_train, y_train)
    y_pred = mlp.predict(X_test)
    mse = mean_squared_error(y_test, y_pred)
    mse_scores.append(mse)
    print(f"validation_fraction={val_frac}, MSE: {mse:.3f}")

# Find best validation_fraction
best_val_frac = val_fractions[np.argmin(mse_scores)]
print(f"Best validation_fraction: {best_val_frac}")

Running the example gives an output like:

validation_fraction=0.1, MSE: 6.708
validation_fraction=0.2, MSE: 6.708
validation_fraction=0.3, MSE: 6.708
Best validation_fraction: 0.1

The key steps in this example are:

Generate a synthetic regression dataset
Split the data into train and test sets
Train MLPRegressor models with different validation_fraction values
Evaluate the mean squared error of each model on the test set
Identify the best validation_fraction based on the lowest MSE

Some tips and heuristics for setting validation_fraction:

Start with the default value of 0.1 and adjust based on dataset size and model performance
Use a larger fraction for smaller datasets to ensure a representative validation set
Consider using cross-validation for more robust performance estimation, especially with smaller datasets

Issues to consider:

A too small validation set may not provide reliable estimates of generalization performance
A too large validation set reduces the amount of data available for training, potentially impacting model performance
The optimal validation_fraction may vary depending on the complexity of the problem and the size of the dataset

See Also