Gradient Boosting

Gradient Boosting builds an ensemble of weak learners (usually shallow trees) in sequence, where each new model corrects the errors of the previous ones.

Intuition

Start with a simple base prediction (e.g., mean of targets).
Fit a new tree to the residuals (errors) of the current model.
Add this new tree to the ensemble with a learning rate.
Repeat for many iterations to gradually minimize the loss function.

GradientBoostingClassifier with scikit-learn

Basic Gradient Boosting example

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

gb = GradientBoostingClassifier(
    n_estimators=200,
    learning_rate=0.05,
    max_depth=3,
    random_state=42
)
gb.fit(X_train, y_train)

y_pred = gb.predict(X_test)
print(classification_report(y_test, y_pred))

Advanced Gradient Boosting (XGBoost, LightGBM)

Modern gradient boosting libraries add powerful optimizations:

XGBoost: regularization, tree pruning, parallelization.
LightGBM: histogram‑based splits, leaf‑wise growth, very fast on large datasets.
CatBoost: strong support for categorical features.

Key Hyperparameters

n_estimators: number of boosting stages (too high → overfitting, too low → underfitting).
learning_rate: how much each tree contributes; lower values often need more trees.
max_depth / max_leaf_nodes: control tree complexity.
subsample: using < 1.0 adds randomness and can improve generalization.

Previous: Ensemble Learning Next: Dimensionality Reduction