Splitting Data with train_test_split

train_test_split shuffles the examples and holds a share back as test data (test_size=0.2 holds back a fifth, and 25% when no size is given), random_state fixes the shuffle so every run splits the same way, and the split comes before anything learns, so the test homes stay unseen.

Journeyman · AI Basics · Python · Updated

By

After this lesson you can

  • Split features and labels into training and test parts with train_test_split, test_size and random_state, and check the parts' sizes
  • Explain what random_state fixes and what it doesn't, and why the split comes before anything learns from the data

Key takeaways

  • train_test_split(X, y, test_size=0.2, random_state=42) gives back four parts in this order, X_train, X_test, y_train and y_test.
  • test_size is the share held back for testing. With no test_size or train_size, scikit-learn holds back 25%.
  • random_state fixes the shuffle, so the same number gives the same split on every run. It doesn't make the split a good one, and another number gives other test rows.
  • Split before anything learns from the data, and fit only on the training part, so the test examples stay ones the model never saw.

Your First scikit-learn Model: fit and predict trained a model on all 400 Maple Grove sales and guessed a price for a home it had never seen. How good are those guesses? The model can't be checked on the homes it learned from: it has already seen their prices. Every model you build, in a class, a project or a job, needs some examples kept out of its sight so it can be tested honestly, and scikit-learn has one function for keeping them out.

The example is Maple Grove Homes' home_sales.csv, from Exercise: Home Price Guesser. Each code block is one notebook cell, run in order: the first one loads the data, and the others use what it made.

Holding homes back

A student who memorized last year's answer key scores 100% on last year's test and might know nothing about this year's, as Training and Testing: Never Grade Your Own Homework explains. A model is the same. So before it learns anything, some homes go aside. The ones it learns from are the training data; the ones it's tested on, homes it never saw, are the test data.

train_test_split shuffles the rows and cuts them in two:

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

homes = pd.read_csv("home_sales.csv")
X = homes[["size_sqft"]]
y = homes["sale_price"]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
print("Training homes:", len(X_train))
print("Test homes:", len(X_test))
Training homes: 320
Test homes: 80

test_size=0.2 holds back a fifth of the homes: 80 of the 400. The four parts come back in this order: the training features, the test features, the training labels and the test labels. X and y are split the same way, so y_test holds the prices of exactly the homes in X_test.

The 400 Maple Grove sales shuffled like a deck of cards and cut into two piles: 320 training homes, which the model learns from with fit, and 80 test homes, set apart behind a fence and used only to check the guesses predict makes for them

The shuffle and random_state

The rows are shuffled first, so the test homes come from all over the file, not from its end:

X_test.head(3)
     size_sqft
209       2050
280       1830
33        1350

The numbers on the left are the rows' labels in the original table: the first test home is row 209. (VS Code draws a table like this as a grid.)

random_state=42 is the shuffle's seed, the number it starts from. The same seed gives the same shuffle on every run, on every computer, so your split matches this lesson's:

for seed in [42, 42, 7]:
    parts = train_test_split(X, y, test_size=0.2, random_state=seed)
    print("random_state", seed, "holds back rows", parts[1].index[:5].tolist())
random_state 42 holds back rows [209, 280, 33, 210, 93]
random_state 42 holds back rows [209, 280, 33, 210, 93]
random_state 7 holds back rows [265, 65, 120, 132, 378]

parts[1] is the second part, X_test. Leave random_state out, and each run shuffles differently, so your numbers wouldn't match anyone's, even your own from yesterday.

What the seed doesn't do: it doesn't make the split a good one. Any whole number works (42 is only a habit), and another seed holds back other homes, so a model's score can move a little with it. scikit-learn's own advice is to check that a result holds up across a few different seeds.

When no size is given

Leave out test_size, and scikit-learn holds back a quarter of the rows:

parts = train_test_split(X, y, random_state=42)
print("Training homes:", len(parts[0]))
print("Test homes:", len(parts[1]))
Training homes: 300
Test homes: 100

That's the default: 25% when neither test_size nor train_size is given. Write the size you want anyway, so anyone reading your code sees it.

Split first, then learn

The two parts share no home, and together they hold every one:

print("Homes in both parts:", len(set(X_train.index) & set(X_test.index)))
print("Homes in either part:", len(set(X_train.index) | set(X_test.index)))
Homes in both parts: 0
Homes in either part: 400

Then the model learns from the training part only, and guesses for the test homes:

model = LinearRegression()
model.fit(X_train, y_train)
guesses = model.predict(X_test)
for i in range(3):
    print(X_test["size_sqft"].iloc[i], "sq ft: guessed", round(guesses[i]), "and it sold for", y_test.iloc[i])
2050 sq ft: guessed 313464 and it sold for 335000
1830 sq ft: guessed 283153 and it sold for 317500
1350 sq ft: guessed 217019 and it sold for 228000

.iloc[i] reads the row at position i. These guesses are honest ones: the model never saw these three prices. Errors and the Mean Absolute Error turns misses like these into one number.

The order matters for everything that learns from data, not only the model. Later in the course, steps that fill in gaps or rescale columns learn averages from the data too; if they see the test rows, facts about the test homes leak into training and the test score looks better than the model will really do. Split first, and nothing that learns ever touches the test part. These homes aren't in time order, so a shuffled split fits; for data in time order, the most recent rows are held back instead.

Check yourself

2 questions to try on the lesson page.

So what

A score means something only when it comes from examples the model never saw, so the split comes first, its size is written down, and its seed is fixed so anyone can get the same split again.

Sources

  • scikit-learn 1.9.1, train_test_split, API reference: test_size as a share of the rows, 0.25 when neither test_size nor train_size is given, shuffle on by default, random_state for reproducible output across calls, and the train-test split of each input it's given.
  • scikit-learn, Glossary of Common Terms and API Elements, random_state: an integer gives the same results across calls, it's worth checking results across a few distinct seeds, and None gives different results each call; and data leakage: knowledge of the test data included in training by mistake.
  • scikit-learn, Cross-validation: evaluating estimator performance, User Guide, its opening: testing a model on the data it learned from is a mistake, and a test set is held out to avoid it.
  • scikit-learn, Common pitfalls and recommended practices: data leakage, and why the test data must never be used to fit any step, the model or the preparation steps.

Key terms

Related