Scikit-Learn "StackingClassifier" versus "VotingClassifier"

Combining multiple machine learning models can often improve predictive performance. In scikit-learn, StackingClassifier and VotingClassifier are two ensemble methods that facilitate this process.

StackingClassifier combines multiple models (base learners) by training a meta-model on their outputs. Key hyperparameters include estimators (list of base models), final_estimator (meta-model), and cv (cross-validation strategy).

VotingClassifier combines multiple models by averaging their predictions, either through hard voting (majority rule) or soft voting (weighted average of probabilities). Key hyperparameters include estimators (list of base models) and voting (type of voting: ‘hard’ or ‘soft’).

The main difference between these classifiers is that StackingClassifier uses a meta-model to learn from the base models’ predictions, potentially yielding better performance but requiring more complexity and computational resources. VotingClassifier is simpler and faster but may not capture complex relationships between base models’ predictions.

StackingClassifier is beneficial when you want to leverage a meta-model for improved performance. VotingClassifier is ideal for quick ensemble methods that are easy to implement and interpret.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import StackingClassifier, VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score, f1_score

# Generate synthetic binary classification dataset
X, y = make_classification(n_samples=1000, n_classes=2, random_state=42)

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Define base learners
base_learners = [
    ('lr', LogisticRegression()),
    ('dt', DecisionTreeClassifier()),
    ('svm', SVC(probability=True))
]

# Fit and evaluate VotingClassifier with hard voting
voting_clf = VotingClassifier(estimators=base_learners, voting='hard')
voting_clf.fit(X_train, y_train)
y_pred_voting = voting_clf.predict(X_test)
print(f"VotingClassifier accuracy: {accuracy_score(y_test, y_pred_voting):.3f}")
print(f"VotingClassifier F1 score: {f1_score(y_test, y_pred_voting):.3f}")

# Fit and evaluate StackingClassifier
stacking_clf = StackingClassifier(estimators=base_learners, final_estimator=LogisticRegression())
stacking_clf.fit(X_train, y_train)
y_pred_stacking = stacking_clf.predict(X_test)
print(f"\nStackingClassifier accuracy: {accuracy_score(y_test, y_pred_stacking):.3f}")
print(f"StackingClassifier F1 score: {f1_score(y_test, y_pred_stacking):.3f}")

Running the example gives an output like:

VotingClassifier accuracy: 0.860
VotingClassifier F1 score: 0.863

StackingClassifier accuracy: 0.870
StackingClassifier F1 score: 0.875

1. Generate a synthetic binary classification dataset using make_classification.
2. Split the data into training and test sets using train_test_split.
3. Define base learners: LogisticRegression, DecisionTreeClassifier, and SVC.
4. Instantiate VotingClassifier with hard voting, fit it on the training data, and evaluate its performance on the test set.
5. Instantiate StackingClassifier with a LogisticRegression meta-model, fit it on the training data, and evaluate its performance on the test set.
6. Compare the test set performance (accuracy and F1 score) of both models.