import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_curve, roc_auc_score
BLUE, RED, GREEN, ORANGE, GRAY = "#1f6fb2", "#c92a1a", "#1c7a43", "#e8871a", "#8a8a8a"
plt.rcParams["figure.figsize"] = (6.5, 3.6)
plt.rcParams["axes.spines.top"] = plt.rcParams["axes.spines.right"] = FalseLab 6 — Trees and random forests: who will be in financial distress?
A bank wants to rank its borrowers by risk: who will be in financial distress in the next two years. We come back to the file of the previous lab: 40 000 borrowers, a third of them with a missing income, and a few impossible ratios. The logistic regression needed two repairs of the file to score well. Here we fit a tree and a random forest on the raw columns, and score them with the ROC-AUC.
Boxes as always: 🔮 predict before running (no AI) · 🧠 explain in your own words · ✍️ short task, boilerplate given.
Part 1 — The data
df = pd.read_csv("https://hichamjanati.github.io/media/teaching/SMV/borrowers.csv")
print(df.shape)
df.head()Each row is a borrower. What the columns mean:
| column | meaning |
|---|---|
FinancialDistressNextTwoYears
|
the target: 1 if the borrower was 90 days or more late on a payment in the next two years, 0 otherwise |
BorrowedToLimitRatio
|
total amount borrowed (still owed) on credit cards and personal lines of credit, divided by the sum of their credit limits |
Age
|
age of the borrower, in years |
NumberOfTimes90DaysLate
|
number of times the borrower was 90 days or more late on a payment |
DebtToIncomeRatio
|
monthly debt payments, alimony and living costs, divided by the monthly income; missing when the income is missing |
MonthlyIncome
|
monthly income |
NumberOfOpenCreditLinesAndLoans
|
number of open loans (car loan, mortgage) and lines of credit (credit cards) |
NumberOfRealEstateLoansOrLines
|
number of mortgage and real estate loans, home equity lines of credit included |
y = df["FinancialDistressNextTwoYears"]
X = df.drop(columns="FinancialDistressNextTwoYears")We split the data just like in the previous lab:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25,
stratify=y, random_state=0)
print(f"train {len(X_train)} borrowers · test {len(X_test)} borrowers ({y_test.sum()} in distress)")
def train_test_auc(model):
"""ROC-AUC of a fitted model on the training rows and on the test rows."""
return (roc_auc_score(y_train, model.predict_proba(X_train)[:, 1]),
roc_auc_score(y_test, model.predict_proba(X_test)[:, 1]))Part 2 — A tree
DecisionTreeClassifier grows a tree with the lecture’s CART algorithm; its .predict_proba returns the share of distress among the training borrowers of the leaf. It accepts missing values: at each split it learns to which side the rows with a missing value go, the lecture’s default direction. So the tree gets the raw columns: no imputer, no scaler.
2.1 A tree you can read
We start with a tree of depth 2. plot_tree draws a fitted tree: each box shows its number (node #), its question, the share of the training borrowers that reach it (samples), and the shares of no distress and of distress among them (value).
tree2 = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X_train, y_train)
plt.figure(figsize=(11, 4.6))
plot_tree(tree2, feature_names=list(X_train.columns), class_names=["no distress", "distress"],
proportion=True, impurity=False, filled=True, rounded=True, fontsize=9, precision=3,
node_ids=True)
plt.show()
auc_train, auc_test = train_test_auc(tree2)
print(f"depth-2 tree: train ROC-AUC {auc_train:.3f} test ROC-AUC {auc_test:.3f}")🧠 Explain. Read one path in words: among the borrowers who were never 90 days late and have a debt-to-income ratio of at most 0.988, what share is in distress, and how large is that group?
✍️ Your answer:
…
2.2 How deep?
We choose the depth with GridSearchCV as in Lab 4, scored by the ROC-AUC (scoring="roc_auc"). With return_train_score=True it also stores the score on the training folds. The model is a tree alone, not a pipeline, so the key of the grid is the plain "max_depth".
🔮 Predict. The depth goes from 2 to 15. Sketch the shape of two curves against the depth: the training ROC-AUC and the cross-validated ROC-AUC.
✍️ Your answer:
…
✍️ Your turn. Tune the max_depth of a DecisionTreeClassifier(random_state=0) over depths by 5-fold cross-validation, scored by the ROC-AUC and keeping the training scores.
depths = [2, 3, 4, 5, 6, 8, 10, 12, 15]
# TODO: tree_cv
print(f"best depth {tree_cv.best_params_['max_depth']} CV ROC-AUC {tree_cv.best_score_:.3f}")res = pd.DataFrame(tree_cv.cv_results_)
plt.plot(depths, res["mean_train_score"], "o-", color=BLUE, label="training folds")
plt.plot(depths, res["mean_test_score"], "o-", color=RED, label="validation folds (CV)")
plt.xlabel("max_depth"); plt.ylabel("ROC-AUC"); plt.legend()
plt.show()
res[["param_max_depth", "mean_train_score", "mean_test_score"]].round(3)🧠 Explain. From which depth does the tree overfit, and which depth do you pick?
✍️ Your answer:
…
tree = tree_cv.best_estimator_
auc_train, auc_test = train_test_auc(tree)
print(f"depth-{tree_cv.best_params_['max_depth']} tree: train ROC-AUC {auc_train:.3f} test ROC-AUC {auc_test:.3f}")2.3 A messy file, a good score
The logistic regression of the previous lab scored 0.745 on this file with the missing values filled by the median, and 0.882 only after two repairs: dropping the training borrowers with an impossible ratio, and adding a missing indicator. The tree got the raw columns, and scores 0.867 in cross-validation.
🔮 Predict. Will the tree do better if we fill the missing values with the median first? Why?
✍️ Your answer:
…
✍️ Your turn. Tune the depth of tree_imputed, a pipeline of SimpleImputer(strategy=“median”) and a tree, with the same grid and cross-validation as tree_cv.
# TODO: tree_imputed
print(f"median imputation: CV ROC-AUC {tree_imputed.best_score_:.3f} {tree_imputed.best_params_}")In the depth-2 tree of 2.1, the number of training borrowers with a missing income in each leaf:
leaf = pd.Series(tree2.apply(X_train), index=X_train.index, name="node")
pd.crosstab(leaf, X_train["MonthlyIncome"].isna().rename("income missing"))🧠 Explain. Why does the imputer change nothing for the tree?
✍️ Your answer:
…
🔮 Predict. The logistic regression needed its columns standardized. Would a scaler change the tree? Why?
✍️ Your answer:
…
✍️ Your turn. Check it with tree_scaled, a pipeline of StandardScaler() and a tree, tuned like tree_imputed.
# TODO: tree_scaled
print(f"standardized columns: CV ROC-AUC {tree_scaled.best_score_:.3f} {tree_scaled.best_params_}")We did not even filter out the 53 impossible ratios. For a tree, a huge ratio is only a large one, and a split can set it apart: the depth-5 tree splits the ratio at 10.76, which puts 20 of the 53 in a leaf of their own. Dropping them, with the cutoff of 10 of the previous lab, gives 0.868 in cross-validation, against 0.867.
Part 3 — A random forest
A RandomForestClassifier grows n_estimators trees. Each tree is fitted on a bootstrap sample of the training rows, each split may only use a random subset of the columns (by default √7, rounded down to 2, of our 7), and the forest averages the probabilities of its trees. Like a single tree, it accepts missing values. We tune one dial: min_samples_leaf=1 gives the fully grown trees of the lecture, 50 keeps at least 50 borrowers in every leaf.
3.1 The size of the leaves
🔮 Predict. Will the forest’s test ROC-AUC be above or below the 0.869 of the tree? Why?
✍️ Your answer:
…
✍️ Your turn. Tune a RandomForestClassifier(n_estimators=100, n_jobs=-1, random_state=0) over min_samples_leaf in [1, 50] by 5-fold cross-validation, scored by the ROC-AUC and keeping the training scores.
# TODO: forest_cv
forest = forest_cv.best_estimator_
print(pd.DataFrame(forest_cv.cv_results_)[["param_min_samples_leaf", "mean_train_score", "mean_test_score"]].round(3))
auc_train, auc_test = train_test_auc(forest)
print(f"forest: train ROC-AUC {auc_train:.3f} test ROC-AUC {auc_test:.3f}")The models of the lab, side by side:
final = {"tree, depth 2": tree2, "tree, depth 5": tree, "random forest": forest}
pd.DataFrame([train_test_auc(m) for m in final.values()], index=list(final),
columns=["train ROC-AUC", "test ROC-AUC"]).round(3)for name, color in [("tree, depth 5", ORANGE), ("random forest", GREEN)]:
scores = final[name].predict_proba(X_test)[:, 1]
fpr_m, tpr_m, _ = roc_curve(y_test, scores)
plt.plot(fpr_m, tpr_m, color=color, label=f"{name} ({roc_auc_score(y_test, scores):.3f})")
plt.plot([0, 1], [0, 1], color=GRAY, ls="--")
plt.xlabel("false positive rate"); plt.ylabel("true positive rate (recall)")
plt.legend(title="test ROC-AUC")
plt.show()3.2 How many trees?
🔮 Predict. Sketch the test ROC-AUC of the forest as the number of trees grows from 1 to 300. Can too many trees overfit? Why?
✍️ Your answer:
…
✍️ Your turn. Complete the loop: rf is a forest of n trees with leaves of at least leaf borrowers, fitted on the training rows.
n_trees = [1, 5, 20, 50, 100, 300]
rows = []
for leaf in [1, 50]:
for n in n_trees:
# TODO: rf
rows.append([leaf, n, *train_test_auc(rf)])
by_n = pd.DataFrame(rows, columns=["min_samples_leaf", "n_estimators", "train ROC-AUC", "test ROC-AUC"])
for leaf, color in [(1, ORANGE), (50, GREEN)]:
part = by_n[by_n["min_samples_leaf"] == leaf]
plt.plot(part["n_estimators"], part["test ROC-AUC"], "o-", color=color, label=f"min_samples_leaf={leaf}")
plt.xscale("log"); plt.xlabel("number of trees"); plt.ylabel("test ROC-AUC"); plt.legend()
plt.show()
by_n.round(3)3.3 Which columns does the forest use?
forest.feature_importances_ gives one number per column: the decrease of impurity brought by all the splits on that column, weighted by the share of the training borrowers that reach them, averaged over the trees. The numbers add up to 1.
🔮 Predict. Which two columns will the forest rely on most? Why?
✍️ Your answer:
…
✍️ Your turn. Build importances, a Series of the importances of forest indexed by the column names.
# TODO: importances
importances.sort_values().plot.barh(color=GREEN)
plt.xlabel("importance (impurity decrease)")
plt.show()
importances.sort_values(ascending=False).round(3)