Random Forests
A random forest trains many decision trees, each on a bootstrap sample of the training rows and choosing among a random few features at each split, then averages their predicted probabilities, so it changes less with the rows it was given than one tree does; RandomForestClassifier(n_estimators=200, random_state=0) builds one, it's harder to read than a single tree, and gradient boosting is the next model to learn.
Journeyman · AI Basics · Python · Updated
After this lesson you can
- Train a RandomForestClassifier with n_estimators and random_state, and compare it with a single tree by cross-validation
- Explain how bootstrap samples and random feature choices make a forest's trees differ, and why averaging them is steadier than one tree
Key takeaways
- One deep tree changes a lot when its training rows change. Many different trees, averaged, change less.
- Each tree in a random forest learns from a bootstrap sample, rows drawn at random with repeats, and looks at a random few features at each split.
- scikit-learn's forest averages its trees' predicted probabilities, and random_state fixes both kinds of randomness so every run builds the same forest.
- A forest's settings are tuned like a tree's, by cross-validation, and a forest is harder to read than one tree.
Decision Trees showed one tree's weak spot: grow it deep and it memorizes the rows it learned from. There's a second weak spot that's harder to see. Give the same tree slightly different rows, and it can come out asking quite different questions. The random forest is the model most people reach for next, in competitions, in projects and at work, because it turns that weakness into a strength.
The example is Harbor View Inn's bookings.csv, from Exercise: Fair Model Showdown. Each code block is one notebook cell, run in order: the first one loads the data, and the others use what it made.
One tree is jumpy
Here are two trees with no depth limit, each trained on a different half of the 2,400 training bookings, then asked about the same 2,400 bookings:
import numpy as np
import pandas as pd
from sklearn.compose import make_column_transformer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder
from sklearn.tree import DecisionTreeClassifier
bookings = pd.read_csv("bookings.csv")
features = ["lead_time_days", "nights", "guests", "deposit", "channel", "repeat_guest", "special_requests"]
X_train, X_test, y_train, y_test = train_test_split(
bookings[features], bookings["canceled"], test_size=0.2, stratify=bookings["canceled"], random_state=42
)
prep = make_column_transformer((OneHotEncoder(handle_unknown="ignore"), ["channel"]), remainder="passthrough")
folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
half_a = X_train.sample(frac=0.5, random_state=1)
half_b = X_train.drop(half_a.index)
tree_a = Pipeline([("prep", prep), ("model", DecisionTreeClassifier(random_state=0))]).fit(half_a, y_train[half_a.index])
tree_b = Pipeline([("prep", prep), ("model", DecisionTreeClassifier(random_state=0))]).fit(half_b, y_train[half_b.index])
disagree = (tree_a.predict(X_train) != tree_b.predict(X_train)).mean()
print("The two trees disagree on", round(disagree * 100, 1), "% of the bookings")
The two trees disagree on 23.1 % of the bookings
Same kind of model, same settings, and they give different answers for almost one booking in four. Statistics calls this jumpiness variance: how much a model changes with the particular rows it learned from.
Ask a crowd
Ask one person to guess how many jelly beans are in a jar, and the guess may be far off. Ask two hundred people and average their guesses, and the misses in both directions tend to cancel out, as long as the people didn't all copy each other. A random forest is that crowd, made of trees. A model built from many models is an ensemble (said "on-SAHM-bul").
For the crowd to help, the trees must differ, and the forest makes them differ in two ways.
First, each tree learns from a bootstrap sample: as many rows as the training set has, drawn at random with replacement, so some rows come up twice and some not at all. Here's one draw of 10 row numbers:
rng = np.random.default_rng(0)
rows = rng.choice(10, size=10, replace=True).tolist()
print("Drawn:", sorted(rows))
print("Left out:", sorted(set(range(10)) - set(rows)))
Drawn: [0, 0, 0, 1, 2, 3, 5, 6, 8, 8]
Left out: [4, 7, 9]
Second, at each split, a tree may choose only among a random few of the columns, not all of them, so one strong column can't make every tree ask the same first question.

RandomForestClassifier
RandomForestClassifier builds the crowd. n_estimators is how many trees, and random_state fixes both kinds of randomness, the samples and the columns, so every run builds the same forest. Scored by cross-validation beside one tree:
for name, model in [
("one tree, no limit", DecisionTreeClassifier(random_state=0)),
("forest, 200 trees", RandomForestClassifier(n_estimators=200, random_state=0)),
("forest, 200 trees, depth 6", RandomForestClassifier(n_estimators=200, max_depth=6, random_state=0)),
]:
scores = cross_val_score(Pipeline([("prep", prep), ("model", model)]), X_train, y_train, cv=folds)
print(name, "| mean", round(scores.mean(), 3), "| spread", round(scores.std(), 3))
one tree, no limit | mean 0.765 | spread 0.016
forest, 200 trees | mean 0.82 | spread 0.01
forest, 200 trees, depth 6 | mean 0.846 | spread 0.012
Two hundred unlimited trees, averaged, do far better than one: 0.82 against 0.765. A forest is still a model with settings, though. By default its trees grow without a depth limit, and on these noisy bookings, limiting them to depth 6 helps more. That's a setting to tune with a grid search, model__max_depth, exactly as Searching for Good Settings tunes a single tree.
The two-halves test from the first section, with forests instead of trees:
forest = Pipeline([("prep", prep), ("model", RandomForestClassifier(n_estimators=200, random_state=0))])
forest_a = forest.fit(half_a, y_train[half_a.index]).predict(X_train)
forest_b = forest.fit(half_b, y_train[half_b.index]).predict(X_train)
print("The two forests disagree on", round((forest_a != forest_b).mean() * 100, 1), "% of the bookings")
The two forests disagree on 17.8 % of the bookings
Steadier than the trees' 23.1%. That's what the randomness is for: scikit-learn's guide says it lowers the variance, sometimes at the cost of a slight rise in bias.
How the forest decides, and what it costs
forest = Pipeline([("prep", prep), ("model", RandomForestClassifier(n_estimators=200, max_depth=6, random_state=0))])
forest.fit(X_train, y_train)
trees = forest.named_steps["model"].estimators_
print("Trees in the forest:", len(trees))
print("Columns each split chooses from:", trees[0].max_features_, "of", forest.named_steps["model"].n_features_in_)
print("Forest's chance of canceling, 3 bookings:", forest.predict_proba(X_train.head(3))[:, 1].round(3))
Trees in the forest: 200
Columns each split chooses from: 3 of 9
Forest's chance of canceling, 3 bookings: [0.186 0.027 0.09 ]
The one-hot encoder made 9 columns, and each split chooses among 3 of them: the square root of 9, the default for a classifier (max_features="sqrt"). And the forest's chance is the average of its trees' chances:
prepared = forest.named_steps["prep"].transform(X_train.head(3))
average = sum(tree.predict_proba(prepared)[:, 1] for tree in trees) / len(trees)
print("Average of the 200 trees' chances:", average.round(3))
Average of the 200 trees' chances: [0.186 0.027 0.09 ]
You'll often hear that a forest's trees "vote". In the method's original paper each tree votes for one answer; scikit-learn's trees vote with their probabilities instead, which are averaged, and predict picks the answer with the higher average chance.
The costs are real. Two hundred trees take longer to train than one. And you can draw one tree and read its questions; nobody can read two hundred. Permutation Importance, in Module 5, shows a way to see what a model leans on without reading it.
The next model to learn after this one is gradient boosting: trees built one after another, each fixing the mistakes of the ones before. HistGradientBoostingClassifier is scikit-learn's fast version, and its guide calls it a competitive alternative to random forests, worth trying side by side.
Check yourself
2 questions to try on the lesson page.
So what
One tree is jumpy; a random forest averages many trees made different on purpose, which is steadier, at the cost of time and of a model you can no longer read.
Sources
- scikit-learn, Ensembles: Gradient boosting, random forests, bagging, voting, stacking, User Guide, Random forests: each tree is built from a bootstrap sample (drawn with replacement) of the training set; a random subset of the features, sized by
max_features, is considered at each split; the two sources of randomness decrease the forest's variance, since single trees have high variance and tend to overfit, sometimes at the cost of a slight increase in bias; scikit-learn combines classifiers by averaging their probabilistic predictions instead of letting each vote for a single class;n_estimatorsis the number of trees, more is better but slower, and results stop getting much better past a critical number;max_features="sqrt"is a good default for classification. Histogram-based gradient boosting: a competitive alternative to random forests, worth comparing on your own problem. - scikit-learn 1.9.1, RandomForestClassifier, API reference: a meta estimator that fits many decision tree classifiers on sub-samples of the dataset and uses averaging to improve accuracy and control over-fitting;
n_estimators=100,max_features="sqrt",bootstrap=Trueandmax_depth=Noneby default, with fully grown, unpruned trees;random_statecontrols both the bootstrapping of the samples and the sampling of the features at each split;estimators_holds the fitted trees;predictgives the class with the highest mean probability across the trees, andpredict_probais the mean of the trees' class probabilities. - scikit-learn 1.9.1, HistGradientBoostingClassifier, API reference: histogram-based gradient boosting classification tree, much faster than
GradientBoostingClassifierfor big datasets (10,000 samples or more).