Validation Sets: Keep the Test Set for Last
Every time you look at the test score while choosing a model or a setting, the choice learns a little about the test rows, so choose with a validation set or with cross-validation on the training part, and score the test set once, at the end, for the model you chose.
Journeyman · AI Basics · Python · Updated
After this lesson you can
- Split rows three ways, into training, validation and test parts, and say what each part is for
- Choose a setting with a validation set or cross-validation on the training part, and score the test set only once, at the end
Key takeaways
- Choosing a setting by its test score makes the test rows part of the choosing, so the test score is no longer an honest guess for new data.
- Three parts keep it honest. Learn from the training rows, choose with the validation rows, and score the test rows once, at the end.
- Cross-validation on the training part can do the validation set's job, so only two parts are needed, training and test.
- The winner's own validation score tends to flatter it, which is why the test set gets the last word.
A student who sees the final exam's questions while studying will ace it, and the grade will say nothing about what they know. Models get the same unfair look in a quieter way: you try twelve depths, check each one's test score, and keep the best. You never trained on the test showings, so it feels honest. It isn't quite. You'll face this choice in every project that compares settings or models, and the habit that avoids it costs one extra line of code.
The example is Lakeside Cinema's showings.csv, from Exercise: Honest Scores. Each code block is one notebook cell, run in order: the first one loads the data, and the others use what it made.
The test set, used for choosing
Here's the tempting way: twelve depths, each scored on the 180 test showings, and keep the best.
import pandas as pd
from sklearn.model_selection import KFold, cross_val_score, train_test_split
from sklearn.tree import DecisionTreeRegressor
showings = pd.read_csv("showings.csv")
features = ["day_number", "weekday", "start_hour", "is_new_release", "screen_seats"]
X_train, X_test, y_train, y_test = train_test_split(
showings[features], showings["tickets_sold"], test_size=0.2, random_state=42
)
test_scores = {}
for depth in range(1, 13):
tree = DecisionTreeRegressor(max_depth=depth, random_state=0).fit(X_train, y_train)
test_scores[depth] = tree.score(X_test, y_test)
best = max(test_scores, key=test_scores.get)
print("Best depth by the test set:", best, "| test R²", round(test_scores[best], 2))
Best depth by the test set: 5 | test R² 0.69
What's wrong with 0.69? The depth was picked because it did best on these 180 showings. The test showings helped choose, so they're no longer showings the choice never saw. Of twelve tries, the winner is partly the one that got lucky on these particular rows, and its score carries that luck. scikit-learn's guide puts it in one rule: test data should never be used to make choices about the model.
Three parts: training, validation, test
The fix is a third part. The validation set is held back from training to choose with; the test set waits until the choosing is over. Here the 720 training showings are split again, a quarter for validation:
X_fit, X_valid, y_fit, y_valid = train_test_split(X_train, y_train, test_size=0.25, random_state=42)
print("Learn from:", len(X_fit), "| validation:", len(X_valid), "| test:", len(X_test))
Learn from: 540 | validation: 180 | test: 180

Each depth learns from the 540, is scored on the validation 180, and the best one is refit on all 720 training showings. Then, once, the test set:
valid_scores = {}
for depth in range(1, 13):
tree = DecisionTreeRegressor(max_depth=depth, random_state=0).fit(X_fit, y_fit)
valid_scores[depth] = tree.score(X_valid, y_valid)
best = max(valid_scores, key=valid_scores.get)
print("Best depth by validation:", best, "| validation R²", round(valid_scores[best], 2))
final = DecisionTreeRegressor(max_depth=best, random_state=0).fit(X_train, y_train)
print("Test R², looked at once:", round(final.score(X_test, y_test), 2))
Best depth by validation: 5 | validation R² 0.71
Test R², looked at once: 0.69
The same depth won here, 5, and that's common. The difference is what the final number means: 0.69 now comes from showings that played no part in any choice. Notice the winner's validation score, 0.71, is a little higher than its test score: the score that crowned a winner tends to flatter it, which is exactly why a fresh set gets the last word.
Cross-validation instead of a validation set
A validation set of 180 showings is one roll of the dice, as Cross-Validation showed. Cross-validation on the training part does the same job with every training row taking a turn, so you need only two parts:
folds = KFold(n_splits=5, shuffle=True, random_state=0)
mean_scores = {}
for depth in range(1, 13):
tree = DecisionTreeRegressor(max_depth=depth, random_state=0)
mean_scores[depth] = cross_val_score(tree, X_train, y_train, cv=folds).mean()
best = max(mean_scores, key=mean_scores.get)
print("Best depth by cross-validation:", best, "| mean R²", round(mean_scores[best], 2))
final = DecisionTreeRegressor(max_depth=best, random_state=0).fit(X_train, y_train)
print("Test R², looked at once:", round(final.score(X_test, y_test), 2))
Best depth by cross-validation: 5 | mean R² 0.7
Test R², looked at once: 0.69
That's the everyday pattern, and the one Exercise: Honest Scores uses: split off the test set first, choose with cross-validation on the rest, then score the test set once and report that number as it is, even if it's lower than you hoped. If you go back and change the model because of it, the test set has been used for choosing, and the next honest number needs new data.
Do it: split showings three ways
Lakeside Cinema's analyst has shuffled the showings' row numbers and wants them dealt into three parts.
The inputs:
rows: the showings' row numbers, already shuffled, in the order to deal them out.validation_percent: the share of the rows to hold back for validation, as a whole number of percent.test_percent: the share of the rows to hold back for testing, as a whole number of percent.
For example, ten rows with 20% for validation and 20% for testing:
rows = [7, 2, 9, 4, 0, 5, 8, 1, 6, 3]
| Part | How many | Rows |
|---|---|---|
| Training | 10 − 2 − 2 = 6 | 7, 2, 9, 4, 0, 5 |
| Validation | 10 × 20 // 100 = 2 | 8, 1 |
| Test | 10 × 20 // 100 = 2 | 6, 3 |
So three_way_split(rows, 20, 20) returns ([7, 2, 9, 4, 0, 5], [8, 1], [6, 3]).
Your job: write three_way_split(rows, validation_percent, test_percent).
- The test part has
len(rows) * test_percent // 100rows, and the validation partlen(rows) * validation_percent // 100. - The training part is the first rows, the validation part the rows after them, and the test part the last rows, each in the order
rowsgives them. - It returns the three lists in that order:
(training, validation, test). - With a percent of 0, that part is an empty list.
Practiced live on the lesson page: write the Python right there and check it.
Check yourself
2 questions to try on the lesson page.
So what
Choose with validation data or cross-validation on the training part, keep the test set out of every choice, and score it once at the end, so the number you report is a fair guess for showings nobody has seen yet.
Sources
- scikit-learn, Cross-validation: evaluating estimator performance, User Guide: a validation set is held out to evaluate while training proceeds, with final evaluation on the test set; when cross-validation is used, a test set should still be held out for final evaluation, but the validation set is no longer needed.
- scikit-learn, Common pitfalls and recommended practices, Data leakage: test data should never be used to make choices about the model, and
fitis never called on the test data. - scikit-learn example, Nested versus non-nested cross-validation: using the same data to choose a model's settings and to evaluate it lets information leak into the choice, and the settings that maximize that score give an overly optimistic score.
- scikit-learn 1.9.1, train_test_split, API reference:
test_sizeas a share of the rows it's given, andrandom_statefor the same split on every call.