Configure GradientBoostingRegressor "loss" Parameter

The loss parameter in scikit-learn’s GradientBoostingRegressor specifies the loss function to be minimized during training.

GradientBoostingRegressor is a powerful ensemble technique that builds regression trees sequentially to minimize a specified loss function. The loss parameter can significantly impact the model’s performance and robustness.

The loss parameter controls the error measurement used to fit the model. Different loss functions are suited for different types of regression problems and can affect model accuracy and robustness. The default value for loss is “squared_error”. Common values include “huber” and “absolute_error”, which can be useful for dealing with outliers.

from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_squared_error

# Generate synthetic dataset
X, y = make_regression(n_samples=1000, n_features=10, noise=0.1, random_state=42)

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train with different loss values
loss_values = ["squared_error", "huber", "absolute_error"]
mse_scores = []

for loss in loss_values:
    gbr = GradientBoostingRegressor(loss=loss, random_state=42)
    gbr.fit(X_train, y_train)
    y_pred = gbr.predict(X_test)
    mse = mean_squared_error(y_test, y_pred)
    mse_scores.append(mse)
    print(f"loss={loss}, MSE: {mse:.3f}")

Running the example gives an output like:

loss=squared_error, MSE: 1234.753
loss=huber, MSE: 1373.886
loss=absolute_error, MSE: 1876.078

The key steps in this example are:

Generate a synthetic regression dataset with noise features
Split the data into train and test sets
Train GradientBoostingRegressor models with different loss values
Evaluate the MSE of each model on the test set

Some tips and heuristics for setting loss:

Use “squared_error” for standard regression tasks
“Huber” is robust to outliers, making it useful for datasets with significant noise
“Absolute_error” minimizes absolute deviations, useful when outliers are present

Issues to consider:

The choice of loss can affect training time and model robustness
Evaluate different loss functions based on specific problem characteristics
Properly tuning the loss parameter can lead to better generalization and handling of outliers

See Also