import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import roc_auc_score, log_loss
BLUE, RED, GREEN, ORANGE, GRAY = "#1f6fb2", "#c92a1a", "#1c7a43", "#e8871a", "#8a8a8a"
plt.rcParams["figure.figsize"] = (6.5, 3.6)
plt.rcParams["axes.spines.top"] = plt.rcParams["axes.spines.right"] = FalseLab 7 — Boosting: rounds, learning rate, three libraries
The lecture built gradient boosting: small trees added one after the other, each one fitted to the pseudo-residuals of the sum so far and scaled by a learning rate. This lab uses it on the file of the last two labs, the 40 000 borrowers, raw. The random forest of the previous lab scored 0.888 on the test borrowers. You meet HistGradientBoostingClassifier and its two main dials, the number of rounds and the learning rate, then the same model in XGBoost and LightGBM.
Boxes as always: 🔮 predict before running (no AI) · 🧠 explain in your own words · ✍️ short task, boilerplate given.
Part 1 — The data
The file and the split of the previous labs: the test borrowers are the same, so the scores compare directly with those of the forest.
df = pd.read_csv("https://hichamjanati.github.io/media/teaching/SMV/borrowers.csv")
y = df["FinancialDistressNextTwoYears"]
X = df.drop(columns="FinancialDistressNextTwoYears")
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=0)
print(f"train {len(X_train)} borrowers · test {len(X_test)} borrowers ({y_test.sum()} in distress)")
def train_test_auc(model):
"""ROC-AUC of a fitted model on the training rows and on the test rows."""
return (roc_auc_score(y_train, model.predict_proba(X_train)[:, 1]),
roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]))Part 2 — A first boosting model
HistGradientBoostingClassifier is the boosting of the lecture, with the log-loss. Each round adds a small tree (at most 31 leaves, max_leaf_nodes), fitted to the pseudo-residuals and scaled by the learning rate (learning_rate, 0.1), for up to max_iter rounds (100). Like the trees of the previous lab, it accepts missing values and needs no scaler.
🔮 Predict. Will boosting with its default settings beat the test ROC-AUC of the forest (0.888)? Why?
✍️ Your answer:
…
✍️ Your turn. Fit hgb, a HistGradientBoostingClassifier(random_state=0), on the training rows.
# TODO: hgb
auc_train, auc_test = train_test_auc(hgb)
print(f"boosting: train ROC-AUC {auc_train:.3f} test ROC-AUC {auc_test:.3f} rounds fitted {hgb.n_iter_}")Part 3 — Rounds and learning rate
To follow the model round by round, we cut a validation quarter out of the training rows. The test rows stay untouched.
X_fit, X_val, y_fit, y_val = train_test_split(X_train, y_train, test_size=0.25,
stratify=y_train, random_state=0)
print(f"fit {len(X_fit)} borrowers · validation {len(X_val)} borrowers")3.1 How many rounds?
staged_predict_proba replays a fitted model round by round: it gives the predicted probabilities after 1 round, after 2 rounds, and so on. log_loss computes the loss that boosting minimizes, the lecture’s log-loss averaged over the borrowers.
def loss_curve(model, X, y):
"""Log-loss of a fitted boosting model on the borrowers (X, y), after each round."""
return np.array([log_loss(y, p[:, 1]) for p in model.staged_predict_proba(X)])🔮 Predict. Adding trees never hurt the forest of the previous lab. Will adding rounds hurt boosting? Why?
✍️ Your answer:
…
✍️ Your turn. Fit hgb_300 on the fit rows: learning rate 0.1, 300 rounds, early_stopping=False and random_state=0.
# TODO: hgb_300
train_loss = loss_curve(hgb_300, X_fit, y_fit)
val_loss = loss_curve(hgb_300, X_val, y_val)
best_round = int(np.argmin(val_loss)) + 1
rounds = np.arange(1, 301)
plt.plot(rounds, train_loss, color=BLUE, label="fit rows")
plt.plot(rounds, val_loss, color=RED, label="validation rows")
plt.axvline(best_round, color=GRAY, ls=":", label=f"minimum at round {best_round}")
plt.xlabel("rounds"); plt.ylabel("log-loss"); plt.legend()
plt.show()
print(f"validation log-loss: round {best_round} {val_loss.min():.3f}, round 300 {val_loss[-1]:.3f}")
print(f"log-loss on the fit rows at round 300: {train_loss[-1]:.3f}")3.2 The learning rate
🔮 Predict. With a learning rate of 0.3, then of 0.02, will the validation minimum come before or after round 32? Why?
✍️ Your answer:
…
curves = {0.1: val_loss}
for lr in [0.3, 0.02]:
model = HistGradientBoostingClassifier(learning_rate=lr, max_iter=300, early_stopping=False,
random_state=0).fit(X_fit, y_fit)
curves[lr] = loss_curve(model, X_val, y_val)
for (lr, v), col in zip(sorted(curves.items()), [GREEN, BLUE, RED]):
plt.plot(rounds, v, color=col, label=f"learning rate {lr}")
print(f"learning rate {lr}: minimum {v.min():.3f} at round {np.argmin(v) + 1}, round 300 {v[-1]:.3f}")
plt.ylim(0.155, 0.25)
plt.xlabel("rounds"); plt.ylabel("validation log-loss"); plt.legend()
plt.show()🧠 Explain. What does a small learning rate buy, and what does it cost?
✍️ Your answer:
…
3.3 Early stopping
In Part 2, hgb fitted fewer rounds than its max_iter=100. Above 10 000 training rows, HistGradientBoostingClassifier uses early stopping by default: it sets aside 10 % of the training rows (validation_fraction=0.1), computes their loss after every round, and stops when that loss has not improved for 10 rounds (n_iter_no_change=10). n_iter_ is the number of rounds it fitted.
✍️ Your turn. Fit hgb_es on all the training rows: learning rate 0.02, up to 2 000 rounds, early_stopping=True and random_state=0.
# TODO: hgb_es
auc_train, auc_test = train_test_auc(hgb_es)
print(f"rounds fitted {hgb_es.n_iter_}: train ROC-AUC {auc_train:.3f} test ROC-AUC {auc_test:.3f}")🧠 Explain. On which rows was the stopping round chosen? Is the test ROC-AUC of hgb_es a fair score?
✍️ Your answer:
…
Part 4 — Same model, three libraries
HistGradientBoostingClassifier is scikit-learn’s version of the algorithm of LightGBM. XGBoost and LightGBM are separate libraries; both come with scikit-learn wrappers, XGBClassifier and LGBMClassifier, with the same .fit and .predict_proba, and both accept missing values. The parameters have different names (defaults in parentheses):
| HistGradientBoosting | XGBoost | LightGBM | |
|---|---|---|---|
| learning rate |
learning_rate (0.1)
|
learning_rate (0.3)
|
learning_rate (0.1)
|
| number of rounds |
max_iter (100)
|
n_estimators (100)
|
n_estimators (100)
|
| size of a tree |
max_leaf_nodes (31), grown leaf by leaf
|
max_depth (6), grown level by level: at most 2depth leaves
|
num_leaves (31), grown leaf by leaf
|
| l2 penalty λ on the leaf values |
l2_regularization (0)
|
reg_lambda (1)
|
reg_lambda (0)
|
| early stopping | on above 10 000 rows | off | off |
🔮 Predict. With the same settings, how far apart do you expect the test ROC-AUCs of the three libraries? Why?
✍️ Your answer:
…
✍️ Your turn. Translate hgb_ref into xgb and lgbm with the table: same learning rate, number of rounds, tree size and λ. XGBoost limits the depth, not the leaves: max_depth=5 allows at most 25 = 32 leaves, the closest to 31. Give LightGBM verbose=-1 to silence it.
from xgboost import XGBClassifier
from lightgbm import LGBMClassifier
hgb_ref = HistGradientBoostingClassifier(learning_rate=0.05, max_iter=200, max_leaf_nodes=31,
l2_regularization=1.0, early_stopping=False, random_state=0)
# TODO: xgb
# TODO: lgbm
same_settings = {}
for name, model in [("HistGradientBoosting", hgb_ref), ("XGBoost", xgb), ("LightGBM", lgbm)]:
same_settings[name] = train_test_auc(model.fit(X_train, y_train))
pd.DataFrame(same_settings, index=["train ROC-AUC", "test ROC-AUC"]).T.round(3)Each library also has its own defaults, those of the table: the same call with no settings gives three different models.
🔮 Predict. Which library’s defaults will overfit most? Why?
✍️ Your answer:
…
defaults = {"HistGradientBoosting": HistGradientBoostingClassifier(random_state=0),
"XGBoost": XGBClassifier(), "LightGBM": LGBMClassifier(verbose=-1)}
pd.DataFrame({name: train_test_auc(model.fit(X_train, y_train)) for name, model in defaults.items()},
index=["train ROC-AUC", "test ROC-AUC"]).T.round(3)