Configure LogisticRegression "solver" Parameter

The solver parameter in scikit-learn’s LogisticRegression controls the algorithm used for optimization.

Logistic Regression is a linear model for binary classification that uses a logistic function to model the probability of a binary outcome.

The solver parameter specifies which algorithm to use for solving the optimization problem. Different solvers can have varying performance and computational efficiency.

The default solver is lbfgs. Other common values include liblinear, sag, saga, and newton-cg.

In practice, the choice of solver can depend on the size and nature of the dataset. For example, liblinear is suitable for small datasets and sparse data, while sag and saga are more efficient for large datasets.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Generate synthetic dataset
X, y = make_classification(n_samples=1000, n_features=20, n_informative=15,
                           n_redundant=5, random_state=42)

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train with different solver values
solver_values = ['lbfgs', 'liblinear', 'sag', 'saga', 'newton-cg']
accuracies = []

for solver in solver_values:
    lr = LogisticRegression(solver=solver, max_iter=1000, random_state=42)
    lr.fit(X_train, y_train)
    y_pred = lr.predict(X_test)
    accuracy = accuracy_score(y_test, y_pred)
    accuracies.append(accuracy)
    print(f"solver={solver}, Accuracy: {accuracy:.3f}")

Running the example gives an output like:

solver=lbfgs, Accuracy: 0.825
solver=liblinear, Accuracy: 0.825
solver=sag, Accuracy: 0.825
solver=saga, Accuracy: 0.825
solver=newton-cg, Accuracy: 0.825

The key steps in this example are:

Generate a synthetic binary classification dataset with informative and noise features.
Split the data into train and test sets.
Train LogisticRegression models with different solver values.
Evaluate the accuracy of each model on the test set.

Some tips and heuristics for setting solver:

Choose lbfgs for small datasets and dense data.
Use liblinear for small datasets and sparse data.
Opt for sag and saga for large datasets.
Prefer newton-cg for multi-class problems.

Issues to consider:

Different solvers have varying computational efficiencies and memory requirements.
Some solvers are better suited for certain types of data and problem sizes.
The choice of solver can impact convergence and accuracy.

See Also