Lab 7 — Boosting: rounds, learning rate, three libraries

Author

Hicham Janati — FSML

The lecture built gradient boosting: small trees added one after the other, each one fitted to the pseudo-residuals of the sum so far and scaled by a learning rate. This lab uses it on the file of the last two labs, the 40 000 borrowers, raw. The random forest of the previous lab scored 0.888 on the test borrowers. You meet HistGradientBoostingClassifier and its two main dials, the number of rounds and the learning rate, then the same model in XGBoost and LightGBM.

Boxes as always: 🔮 predict before running (no AI) · 🧠 explain in your own words · ✍️ short task, boilerplate given.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score, log_loss

BLUE, RED, GREEN, ORANGE, GRAY = "#1f6fb2", "#c92a1a", "#1c7a43", "#e8871a", "#8a8a8a"
plt.rcParams["figure.figsize"] = (6.5, 3.6)
plt.rcParams["axes.spines.top"] = plt.rcParams["axes.spines.right"] = False

Part 1 — The data

The file and the split of the previous labs: the test borrowers are the same, so the scores compare directly with those of the forest.

df = pd.read_csv("https://hichamjanati.github.io/media/teaching/SMV/borrowers.csv")
y = df["FinancialDistressNextTwoYears"]
X = df.drop(columns="FinancialDistressNextTwoYears")

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25,
                                                    stratify=y, random_state=0)
print(f"train {len(X_train)} borrowers · test {len(X_test)} borrowers ({y_test.sum()} in distress)")

def train_test_auc(model):
    """ROC-AUC of a fitted model on the training rows and on the test rows."""
    return (roc_auc_score(y_train, model.predict_proba(X_train)[:, 1]),
            roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]))

Part 2 — A first boosting model

HistGradientBoostingClassifier is the boosting of the lecture, with the log-loss. Each round adds a small tree (at most 31 leaves, max_leaf_nodes), fitted to the pseudo-residuals and scaled by the learning rate (learning_rate, 0.1), for up to max_iter rounds (100). Like the trees of the previous lab, it accepts missing values and needs no scaler.

🔮 Predict. Will boosting with its default settings beat the test ROC-AUC of the forest (0.888)? Why?

✍️ Your answer:

…

✍️ Your turn. Fit hgb, a HistGradientBoostingClassifier(random_state=0), on the training rows.

# TODO: hgb
auc_train, auc_test = train_test_auc(hgb)
print(f"boosting: train ROC-AUC {auc_train:.3f}   test ROC-AUC {auc_test:.3f}   rounds fitted {hgb.n_iter_}")

Part 3 — Rounds and learning rate

To follow the model round by round, we cut a validation quarter out of the training rows. The test rows stay untouched.

X_fit, X_val, y_fit, y_val = train_test_split(X_train, y_train, test_size=0.25,
                                              stratify=y_train, random_state=0)
print(f"fit {len(X_fit)} borrowers · validation {len(X_val)} borrowers")

3.1 How many rounds?

staged_predict_proba replays a fitted model round by round: it gives the predicted probabilities after 1 round, after 2 rounds, and so on. log_loss computes the loss that boosting minimizes, the lecture’s log-loss averaged over the borrowers.

def loss_curve(model, X, y):
    """Log-loss of a fitted boosting model on the borrowers (X, y), after each round."""
    return np.array([log_loss(y, p[:, 1]) for p in model.staged_predict_proba(X)])

🔮 Predict. Adding trees never hurt the forest of the previous lab. Will adding rounds hurt boosting? Why?

✍️ Your answer:

…

✍️ Your turn. Fit hgb_300 on the fit rows: learning rate 0.1, 300 rounds, early_stopping=False and random_state=0.

# TODO: hgb_300

train_loss = loss_curve(hgb_300, X_fit, y_fit)
val_loss = loss_curve(hgb_300, X_val, y_val)
best_round = int(np.argmin(val_loss)) + 1
rounds = np.arange(1, 301)

plt.plot(rounds, train_loss, color=BLUE, label="fit rows")
plt.plot(rounds, val_loss, color=RED, label="validation rows")
plt.axvline(best_round, color=GRAY, ls=":", label=f"minimum at round {best_round}")
plt.xlabel("rounds"); plt.ylabel("log-loss"); plt.legend()
plt.show()
print(f"validation log-loss: round {best_round} {val_loss.min():.3f}, round 300 {val_loss[-1]:.3f}")
print(f"log-loss on the fit rows at round 300: {train_loss[-1]:.3f}")

3.2 The learning rate

🔮 Predict. With a learning rate of 0.3, then of 0.02, will the validation minimum come before or after round 32? Why?

✍️ Your answer:

…

curves = {0.1: val_loss}
for lr in [0.3, 0.02]:
    model = HistGradientBoostingClassifier(learning_rate=lr, max_iter=300, early_stopping=False,
                                           random_state=0).fit(X_fit, y_fit)
    curves[lr] = loss_curve(model, X_val, y_val)

for (lr, v), col in zip(sorted(curves.items()), [GREEN, BLUE, RED]):
    plt.plot(rounds, v, color=col, label=f"learning rate {lr}")
    print(f"learning rate {lr}: minimum {v.min():.3f} at round {np.argmin(v) + 1}, round 300 {v[-1]:.3f}")
plt.ylim(0.155, 0.25)
plt.xlabel("rounds"); plt.ylabel("validation log-loss"); plt.legend()
plt.show()

🧠 Explain. What does a small learning rate buy, and what does it cost?

✍️ Your answer:

…

3.3 Early stopping

In Part 2, hgb fitted fewer rounds than its max_iter=100. Above 10 000 training rows, HistGradientBoostingClassifier uses early stopping by default: it sets aside 10 % of the training rows (validation_fraction=0.1), computes their loss after every round, and stops when that loss has not improved for 10 rounds (n_iter_no_change=10). n_iter_ is the number of rounds it fitted.

✍️ Your turn. Fit hgb_es on all the training rows: learning rate 0.02, up to 2 000 rounds, early_stopping=True and random_state=0.

# TODO: hgb_es
auc_train, auc_test = train_test_auc(hgb_es)
print(f"rounds fitted {hgb_es.n_iter_}: train ROC-AUC {auc_train:.3f}   test ROC-AUC {auc_test:.3f}")

🧠 Explain. On which rows was the stopping round chosen? Is the test ROC-AUC of hgb_es a fair score?

✍️ Your answer:

…

Part 4 — Same model, three libraries

HistGradientBoostingClassifier is scikit-learn’s version of the algorithm of LightGBM. XGBoost and LightGBM are separate libraries; both come with scikit-learn wrappers, XGBClassifier and LGBMClassifier, with the same .fit and .predict_proba, and both accept missing values. The parameters have different names (defaults in parentheses):

HistGradientBoosting XGBoost LightGBM
learning rate learning_rate (0.1) learning_rate (0.3) learning_rate (0.1)
number of rounds max_iter (100) n_estimators (100) n_estimators (100)
size of a tree max_leaf_nodes (31), grown leaf by leaf max_depth (6), grown level by level: at most 2depth leaves num_leaves (31), grown leaf by leaf
l2 penalty λ on the leaf values l2_regularization (0) reg_lambda (1) reg_lambda (0)
early stopping on above 10 000 rows off off

🔮 Predict. With the same settings, how far apart do you expect the test ROC-AUCs of the three libraries? Why?

✍️ Your answer:

…

✍️ Your turn. Translate hgb_ref into xgb and lgbm with the table: same learning rate, number of rounds, tree size and λ. XGBoost limits the depth, not the leaves: max_depth=5 allows at most 25 = 32 leaves, the closest to 31. Give LightGBM verbose=-1 to silence it.

from xgboost import XGBClassifier
from lightgbm import LGBMClassifier

hgb_ref = HistGradientBoostingClassifier(learning_rate=0.05, max_iter=200, max_leaf_nodes=31,
                                         l2_regularization=1.0, early_stopping=False, random_state=0)
# TODO: xgb
# TODO: lgbm

same_settings = {}
for name, model in [("HistGradientBoosting", hgb_ref), ("XGBoost", xgb), ("LightGBM", lgbm)]:
    same_settings[name] = train_test_auc(model.fit(X_train, y_train))
pd.DataFrame(same_settings, index=["train ROC-AUC", "test ROC-AUC"]).T.round(3)

Each library also has its own defaults, those of the table: the same call with no settings gives three different models.

🔮 Predict. Which library’s defaults will overfit most? Why?

✍️ Your answer:

…

defaults = {"HistGradientBoosting": HistGradientBoostingClassifier(random_state=0),
            "XGBoost": XGBClassifier(), "LightGBM": LGBMClassifier(verbose=-1)}
pd.DataFrame({name: train_test_auc(model.fit(X_train, y_train)) for name, model in defaults.items()},
             index=["train ROC-AUC", "test ROC-AUC"]).T.round(3)