The n_iter_no_change
parameter in scikit-learn’s SGDClassifier
controls the early stopping criterion based on the number of consecutive iterations without improvement.
Stochastic Gradient Descent (SGD) is an iterative optimization algorithm used for training various linear models. The SGDClassifier
implements SGD for classification tasks, updating model parameters based on one sample at a time.
The n_iter_no_change
parameter determines how many iterations to continue training without improvement before stopping. This helps prevent overfitting and reduces unnecessary computation when the model has converged.
The default value for n_iter_no_change
is 5.
In practice, values between 2 and 10 are commonly used, depending on the dataset size and complexity.
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import SGDClassifier
from sklearn.metrics import accuracy_score
import numpy as np
# Generate synthetic dataset
X, y = make_classification(n_samples=10000, n_features=20, n_informative=10,
n_redundant=5, n_classes=3, random_state=42)
# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Train with different n_iter_no_change values
n_iter_no_change_values = [2, 5, 10, 20]
accuracies = []
for n in n_iter_no_change_values:
sgd = SGDClassifier(loss='log_loss', n_iter_no_change=n, random_state=42)
sgd.fit(X_train, y_train)
y_pred = sgd.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
accuracies.append(accuracy)
print(f"n_iter_no_change={n}, Accuracy: {accuracy:.3f}, n_iter_: {sgd.n_iter_}")
Running the example gives an output like:
n_iter_no_change=2, Accuracy: 0.674, n_iter_: 72
n_iter_no_change=5, Accuracy: 0.669, n_iter_: 106
n_iter_no_change=10, Accuracy: 0.679, n_iter_: 165
n_iter_no_change=20, Accuracy: 0.674, n_iter_: 248
The key steps in this example are:
- Generate a synthetic multi-class classification dataset
- Split the data into train and test sets
- Train
SGDClassifier
models with differentn_iter_no_change
values - Evaluate the accuracy of each model on the test set
Some tips and heuristics for setting n_iter_no_change
:
- Start with the default value of 5 and adjust based on model performance
- Smaller values may lead to early stopping, potentially underfitting
- Larger values allow more iterations, potentially improving convergence but increasing computation time
- Monitor the actual number of iterations (
n_iter_
attribute) to understand the impact
Issues to consider:
- The optimal value depends on the dataset size, complexity, and learning rate
- Too small values may stop training prematurely, while too large values may cause overfitting
- Consider using this parameter in conjunction with
tol
for more precise control over convergence - The effect of
n_iter_no_change
may vary with different loss functions and regularization settings