Configure SGDRegressor "max_iter" Parameter

The max_iter parameter in scikit-learn’s SGDRegressor controls the maximum number of iterations over the training data (epochs).

Stochastic Gradient Descent (SGD) is an optimization algorithm used for training various linear models. It updates the model parameters iteratively based on batches of training data. The max_iter parameter sets an upper limit on these iterations.

Increasing max_iter allows the model more opportunities to converge to an optimal solution. However, setting it too high can lead to overfitting and increased computational cost.

The default value for max_iter is 1000. In practice, values between 100 and 10000 are commonly used, depending on the dataset size and complexity.

from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.linear_model import SGDRegressor
from sklearn.metrics import mean_squared_error

# Generate synthetic dataset
X, y = make_regression(n_samples=1000, n_features=10, noise=0.1, random_state=42)

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train with different max_iter values
max_iter_values = [100, 1000, 5000, 10000]
mse_scores = []

for max_iter in max_iter_values:
    sgd = SGDRegressor(max_iter=max_iter, random_state=42)
    sgd.fit(X_train, y_train)
    y_pred = sgd.predict(X_test)
    mse = mean_squared_error(y_test, y_pred)
    mse_scores.append(mse)
    print(f"max_iter={max_iter}, MSE: {mse:.3f}")

Running the example gives an output like:

max_iter=100, MSE: 0.010
max_iter=1000, MSE: 0.010
max_iter=5000, MSE: 0.010
max_iter=10000, MSE: 0.010

The key steps in this example are:

Generate a synthetic regression dataset
Split the data into train and test sets
Train SGDRegressor models with different max_iter values
Evaluate the mean squared error of each model on the test set

Some tips and heuristics for setting max_iter:

Start with the default value of 1000 and adjust based on model convergence
Monitor the model’s performance on a validation set to avoid overfitting
Consider using early stopping if the model converges before reaching max_iter

Issues to consider:

Larger datasets may require more iterations for convergence
Very high max_iter values can lead to overfitting and increased training time
The optimal number of iterations can vary depending on learning rate and regularization

See Also