SKLearner Home | About | Contact | Examples

Gradient Boosting

Helpful examples of using Gradient Boosting machine learning algorithms in scikit-learn.

The Gradient Boosting algorithm is an ensemble learning technique used for both classification and regression tasks.

It builds a predictive model in a stage-wise manner by combining multiple weak learners, typically decision trees, to form a strong learner.

The key idea is to sequentially add trees that correct the errors of the existing ensemble. Each new tree is fitted to the residual errors of the previous trees, with the goal of minimizing the overall prediction error. This is achieved through gradient descent optimization, where the algorithm seeks to minimize a specified loss function.

Gradient Boosting is highly effective at producing accurate and robust models, particularly when dealing with complex datasets. It includes variants like Gradient Boosting Machines (GBM), XGBoost, and LightGBM, which offer enhancements in speed and performance.

However, it can be computationally intensive and prone to overfitting if not properly regularized with techniques such as learning rate adjustment, tree pruning, or early stopping.

ExamplesTags
Configure GradientBoostingClassifier "ccp_alpha" Parameter
Configure GradientBoostingClassifier "criterion" Parameter
Configure GradientBoostingClassifier "init" Parameter
Configure GradientBoostingClassifier "learning_rate" Parameter
Configure GradientBoostingClassifier "loss" Parameter
Configure GradientBoostingClassifier "max_depth" Parameter
Configure GradientBoostingClassifier "max_features" Parameter
Configure GradientBoostingClassifier "max_leaf_nodes" Parameter
Configure GradientBoostingClassifier "min_impurity_decrease" Parameter
Configure GradientBoostingClassifier "min_samples_leaf" Parameter
Configure GradientBoostingClassifier "min_samples_split" Parameter
Configure GradientBoostingClassifier "min_weight_fraction_leaf" Parameter
Configure GradientBoostingClassifier "n_estimators" Parameter
Configure GradientBoostingClassifier "n_iter_no_change" Parameter
Configure GradientBoostingClassifier "random_state" Parameter
Configure GradientBoostingClassifier "subsample" Parameter
Configure GradientBoostingClassifier "tol" Parameter
Configure GradientBoostingClassifier "validation_fraction" Parameter
Configure GradientBoostingClassifier "verbose" Parameter
Configure GradientBoostingClassifier "warm_start" Parameter
Configure GradientBoostingRegressor "alpha" Parameter
Configure GradientBoostingRegressor "ccp_alpha" Parameter
Configure GradientBoostingRegressor "criterion" Parameter
Configure GradientBoostingRegressor "init" Parameter
Configure GradientBoostingRegressor "learning_rate" Parameter
Configure GradientBoostingRegressor "loss" Parameter
Configure GradientBoostingRegressor "max_depth" Parameter
Configure GradientBoostingRegressor "max_features" Parameter
Configure GradientBoostingRegressor "max_leaf_nodes" Parameter
Configure GradientBoostingRegressor "min_impurity_decrease" Parameter
Configure GradientBoostingRegressor "min_samples_leaf" Parameter
Configure GradientBoostingRegressor "min_samples_split" Parameter
Configure GradientBoostingRegressor "min_weight_fraction_leaf" Parameter
Configure GradientBoostingRegressor "n_estimators" Parameter
Configure GradientBoostingRegressor "n_iter_no_change" Parameter
Configure GradientBoostingRegressor "random_state" Parameter
Configure GradientBoostingRegressor "subsample" Parameter
Configure GradientBoostingRegressor "tol" Parameter
Configure GradientBoostingRegressor "validation_fraction" Parameter
Configure GradientBoostingRegressor "verbose" Parameter
Configure GradientBoostingRegressor "warm_start" Parameter
Configure HistGradientBoostingClassifier "categorical_features" Parameter
Configure HistGradientBoostingClassifier "class_weight" Parameter
Configure HistGradientBoostingClassifier "early_stopping" Parameter
Configure HistGradientBoostingClassifier "interaction_cst" Parameter
Configure HistGradientBoostingClassifier "l2_regularization" Parameter
Configure HistGradientBoostingClassifier "learning_rate" Parameter
Configure HistGradientBoostingClassifier "loss" Parameter
Configure HistGradientBoostingClassifier "max_bins" Parameter
Configure HistGradientBoostingClassifier "max_depth" Parameter
Configure HistGradientBoostingClassifier "max_features" Parameter
Configure HistGradientBoostingClassifier "max_iter" Parameter
Configure HistGradientBoostingClassifier "max_leaf_nodes" Parameter
Configure HistGradientBoostingClassifier "min_samples_leaf" Parameter
Configure HistGradientBoostingClassifier "monotonic_cst" Parameter
Configure HistGradientBoostingClassifier "n_iter_no_change" Parameter
Configure HistGradientBoostingClassifier "random_state" Parameter
Configure HistGradientBoostingClassifier "scoring" Parameter
Configure HistGradientBoostingClassifier "tol" Parameter
Configure HistGradientBoostingClassifier "validation_fraction" Parameter
Configure HistGradientBoostingClassifier "verbose" Parameter
Configure HistGradientBoostingClassifier "warm_start" Parameter
Configure HistGradientBoostingRegressor "categorical_features" Parameter
Configure HistGradientBoostingRegressor "early_stopping" Parameter
Configure HistGradientBoostingRegressor "interaction_cst" Parameter
Configure HistGradientBoostingRegressor "l2_regularization" Parameter
Configure HistGradientBoostingRegressor "learning_rate" Parameter
Configure HistGradientBoostingRegressor "loss" Parameter
Configure HistGradientBoostingRegressor "max_bins" Parameter
Configure HistGradientBoostingRegressor "max_depth" Parameter
Configure HistGradientBoostingRegressor "max_features" Parameter
Configure HistGradientBoostingRegressor "max_iter" Parameter
Configure HistGradientBoostingRegressor "max_leaf_nodes" Parameter
Configure HistGradientBoostingRegressor "min_samples_leaf" Parameter
Configure HistGradientBoostingRegressor "monotonic_cst" Parameter
Configure HistGradientBoostingRegressor "n_iter_no_change" Parameter
Configure HistGradientBoostingRegressor "quantile" Parameter
Configure HistGradientBoostingRegressor "random_state" Parameter
Configure HistGradientBoostingRegressor "scoring" Parameter
Configure HistGradientBoostingRegressor "tol" Parameter
Configure HistGradientBoostingRegressor "validation_fraction" Parameter
Configure HistGradientBoostingRegressor "verbose" Parameter
Configure HistGradientBoostingRegressor "warm_start" Parameter
Scikit-Learn "GradientBoostingClassifier" versus "HistGradientBoostingClassifier"
Scikit-Learn "GradientBoostingRegressor" versus "HistGradientBoostingRegressor"
Scikit-Learn GradientBoostingClassifier Model
Scikit-Learn GradientBoostingRegressor Model
Scikit-Learn GridSearchCV GradientBoostingClassifier
Scikit-Learn GridSearchCV GradientBoostingRegressor
Scikit-Learn GridSearchCV HistGradientBoostingClassifier
Scikit-Learn GridSearchCV HistGradientBoostingRegressor
Scikit-Learn HistGradientBoostingClassifier Model
Scikit-Learn HistGradientBoostingRegressor Model
Scikit-Learn RandomizedSearchCV GradientBoostingClassifier
Scikit-Learn RandomizedSearchCV GradientBoostingRegressor
Scikit-Learn RandomizedSearchCV HistGradientBoostingClassifier
Scikit-Learn RandomizedSearchCV HistGradientBoostingRegressor