Configure RandomForestRegressor "warm_start" Parameter

The warm_start parameter in scikit-learn’s RandomForestRegressor allows reusing the previous model’s state to speed up training when adding more trees to the ensemble.

RandomForestRegressor is an ensemble learning method that combines predictions from multiple decision trees to perform regression tasks. Each tree is trained independently on a bootstrapped sample of the training data.

By default, warm_start is set to False, meaning each call to fit() will train a new forest from scratch. When warm_start is set to True, the existing trees in the model are retained, and additional trees are added to the ensemble.

This is particularly useful when training on large datasets or when tuning the n_estimators hyperparameter, as it can significantly reduce the total training time.

from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error
import numpy as np

# Generate synthetic dataset
X, y = make_regression(n_samples=1000, n_features=100, noise=0.1, random_state=42)

# Split into initial train set and additional batch
X_train, X_new, y_train, y_new = train_test_split(X, y, test_size=0.2, random_state=42)

# Train with warm_start=False
rf = RandomForestRegressor(n_estimators=100, warm_start=False, random_state=42)
rf.fit(X_train, y_train)
y_pred_false = rf.predict(X_new)
mse_false = mean_squared_error(y_new, y_pred_false)
print(f"MSE with warm_start=False: {mse_false:.3f}")

# Train with warm_start=True
X_combined = np.concatenate((X_train, X_new))
y_combined = np.concatenate((y_train, y_new))

rf.set_params(n_estimators=120, warm_start=True)
rf.fit(X_combined, y_combined)
y_pred_true = rf.predict(X_new)
mse_true = mean_squared_error(y_new, y_pred_true)
print(f"MSE with warm_start=True: {mse_true:.3f}")

Running the example gives an output like:

MSE with warm_start=False: 7405.708
MSE with warm_start=True: 5906.140

The key steps in this example are:

Generate a synthetic classification dataset
Split the data into an initial train set and an additional batch
Train a RandomForestRegressor with warm_start=False on the initial train set
Evaluate the accuracy of the model on the additional batch
Train the already trained RandomForestRegressor with warm_start=True on the combined initial and additional data
Evaluate the accuracy of the warm started model on the additional batch

Some tips and heuristics for using warm_start:

Use warm_start=True when training on large datasets or tuning n_estimators to save time
Ensure the model architecture (e.g., max_depth, max_features) remains consistent across fitting calls
Incremental fitting can be used for early stopping by monitoring validation performance after each fitting step

Issues to consider:

warm_start is only effective when the model architecture is the same across fitting calls
The computational overhead of warm_start may outweigh the benefits for small datasets or when few additional trees are added

See Also