Hyperparameters: Settings You Choose

A hyperparameter is a setting you choose before fit, such as a tree's max_depth, while fit learns the rest, such as the tree's questions; get_params and set_params read and change the settings, the defaults are only a starting point, and settings are chosen with cross-validation on the training rows, never with the test set.

Journeyman · AI Basics · Python · Updated

By

After this lesson you can

  • Tell a model's hyperparameters from what it learns in fit, and read and change them with get_params and set_params
  • Explain why defaults are only a starting point, and why choosing settings by their test score makes the test score dishonest

Key takeaways

  • A hyperparameter is a setting you choose before training, such as max_depth, n_neighbors or C. What fit learns, such as a tree's questions or coef_, is the model's own.
  • get_params() lists a model's settings and their values, and set_params() changes them before the next fit.
  • Defaults are a starting point. A tree with no max_depth grows until it memorizes its training rows.
  • Choose settings by their cross-validated score on the training rows. Choosing them by their test score turns the test set into training data, and the test score stops being honest.

The Harbor View Inn loses money on every room that's booked and then canceled, so its manager wants a model that flags the bookings likely to cancel, to send those guests a reminder. You can train a decision tree in one line. But how deep should it grow? Two levels of questions, or ten, or as many as it likes? The tree can't learn that answer from the bookings; you have to give it. Every model you train, in a class project or at work, comes with settings like this, and choosing them well is a large part of the job.

The example is Harbor View Inn's bookings.csv, 3,000 made-up bookings, about a quarter of them canceled, from Exercise: Fair Model Showdown. Each code block is one notebook cell, run in order: the first one loads the data, and the others use what it made.

Settings before, learning during

Think of baking bread. You set the oven's temperature and the timer before the loaf goes in; how the loaf turns out is what happens inside. A model works the same way. Some things you set before it sees any data, and some it works out from the data during fit.

The first cell holds back a test set, a fifth of the bookings, with the same share of cancellations on both sides (stratify, as Stratified Splits explains). The number columns are the features for now; the booking channel, a category, joins them in the next lesson.

import pandas as pd
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.tree import DecisionTreeClassifier

bookings = pd.read_csv("bookings.csv")
features = ["lead_time_days", "nights", "guests", "deposit", "repeat_guest", "special_requests"]
X_train, X_test, y_train, y_test = train_test_split(
    bookings[features], bookings["canceled"], test_size=0.2, stratify=bookings["canceled"], random_state=42
)
print("Training bookings:", len(X_train))
print("Share canceled:", round(y_train.mean(), 3))
Training bookings: 2400
Share canceled: 0.242

Now a tree, with its depth set before training:

tree = DecisionTreeClassifier(max_depth=3, random_state=0)
print("Before fit, max_depth:", tree.max_depth)
tree.fit(X_train, y_train)
print("After fit, depth:", tree.get_depth())
print("After fit, leaves:", tree.get_n_leaves())
print("After fit, first question:", features[tree.tree_.feature[0]], "<=", tree.tree_.threshold[0])
Before fit, max_depth: 3
After fit, depth: 3
After fit, leaves: 8
After fit, first question: lead_time_days <= 93.5

max_depth=3 was your choice. The questions the tree asks, starting with "was it booked 93.5 days ahead or less?", it learned from the bookings. A setting you choose before training is a hyperparameter (said "hyper-parameter"). n_neighbors for nearest neighbors and C for logistic regression are hyperparameters too; a logistic regression's coef_ (read "coef underscore") is learned. The trailing underscore is scikit-learn's mark for something fit worked out. One word of warning for reading its docs: scikit-learn calls the settings parameters, the name it gives everything passed when a model is made.

Reading and changing settings

Every scikit-learn model lists its settings with get_params(), as a dictionary of names and values:

settings = DecisionTreeClassifier(random_state=0).get_params()
for name in ["criterion", "max_depth", "min_samples_leaf", "random_state"]:
    print(name, "=", settings[name])
criterion = gini
max_depth = None
min_samples_leaf = 1
random_state = 0

The dictionary holds every setting; these are four of them. set_params() changes a setting on a model you already made, and hands the model back:

tree = DecisionTreeClassifier(random_state=0)
tree.set_params(max_depth=4)
print(tree.get_params()["max_depth"])
print(tree)
4
DecisionTreeClassifier(max_depth=4, random_state=0)

A changed setting counts from the next fit. Printing a model shows the settings that differ from the defaults, which is a quick way to check what you set.

Defaults are a starting point

The listing above says max_depth = None: by default, a tree keeps asking questions until its leaves are as pure as the data allows. Here's what that does with the inn's bookings:

folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
deep = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
print("Depth it grew to:", deep.get_depth())
print("Training accuracy:", round(deep.score(X_train, y_train), 3))
scores = cross_val_score(DecisionTreeClassifier(random_state=0), X_train, y_train, cv=folds)
print("Cross-validated accuracy:", round(scores.mean(), 3))
Depth it grew to: 26
Training accuracy: 0.978
Cross-validated accuracy: 0.767

StratifiedKFold cuts folds the way Cross-Validation describes, shuffled and seeded, while keeping each fold's share of canceled bookings the same. Right about 98% of the bookings it learned from, and about 77% of the ones it was tested on: the overfitting Overfitting and Underfitting warns about. A default is a safe guess made by people who have never seen your data, not an answer.

So try a few depths, each scored by cross-validation on the training rows only:

for depth in [2, 4, 8, None]:
    tree.set_params(max_depth=depth)
    scores = cross_val_score(tree, X_train, y_train, cv=folds)
    print("max_depth", depth, "mean accuracy", round(scores.mean(), 3))
max_depth 2 mean accuracy 0.834
max_depth 4 mean accuracy 0.84
max_depth 8 mean accuracy 0.818
max_depth None mean accuracy 0.767

Depth 4 does best here, and the default does worst. Tuning means exactly this loop: try settings, score each one honestly, keep the best.

Why the test set stays out of it

Why not score each depth on the 600 test bookings instead? Because then the test set is choosing the model. Each look teaches you which depth happens to suit those 600 bookings, the best of several test scores is partly luck, and once you report it, it's no longer a score on bookings the model never saw. scikit-learn's guide says the same: tweak settings until the test score is best, and knowledge of the test set "leaks" into the model, so the score stops telling you how it does on new data. It's the leak Validation Sets: Keep the Test Set for Last warns about.

Settings are chosen on the training rows, with cross-validation; the test set is used once, at the end, for the model you chose.

Check yourself

2 questions to try on the lesson page.

So what

Every model has settings you choose, its defaults are a starting point, and the settings are chosen by cross-validation on the training rows, so the test set can still give you an honest score at the end.

Sources

  • scikit-learn, Tuning the hyper-parameters of an estimator, User Guide: hyper-parameters are parameters that are not directly learnt within estimators, passed as arguments to the estimator's constructor; estimator.get_params() finds their names and current values.
  • scikit-learn, Glossary of Common Terms and API Elements, parameter, get_params, set_params and attributes: scikit-learn calls the constructor's settings parameters, which statisticians may call hyperparameters, such as max_depth and random_state; get_params lists them and their values; set_params changes them between making a model and fitting it; attributes set in fit end in a single underscore.
  • scikit-learn 1.9.1, DecisionTreeClassifier, API reference: max_depth=None by default, which expands nodes until all leaves are pure or hold fewer than min_samples_split samples; get_depth and get_n_leaves.
  • scikit-learn, Cross-validation: evaluating estimator performance, User Guide, its opening: when settings are tweaked until the estimator performs best, knowledge about the test set can leak into the model and the evaluation metrics no longer report on generalization; a test set should still be held out for final evaluation.
  • scikit-learn 1.9.1, StratifiedKFold, API reference: a variation of KFold whose folds keep the percentage of samples for each class; shuffle and random_state as in KFold.

Key terms

Related