Scaling Features

Features measured in different units (pounds, years, a 0 or 1) pull unevenly on any model that measures distance or penalizes large weights, so standardizing rescales each one with the training data's mean and standard deviation, which StandardScaler learns with fit on the training part and applies with transform to both parts.

Journeyman · AI Basics · Python · Updated

By

After this lesson you can

  • Standardize a number by subtracting the training mean and dividing by the training standard deviation, by hand and with StandardScaler
  • Explain which models care about the scale of features, and why a scaler is fit on the training data only

Key takeaways

  • When features have different units, the one with the biggest numbers wins any distance. Twelve pounds outweigh ten years.
  • Standardizing turns a value into how many standard deviations it sits from the mean, the value minus the training mean, divided by the training standard deviation.
  • StandardScaler learns the mean and standard deviation with fit on the training part only, then transform applies those same numbers to the training and the test part.
  • Nearest neighbors and logistic regression care about scale. A decision tree doesn't, because it asks one feature at a time.

A pet adoption center wants to know which new arrivals may wait a long time for a home, so it can feature them early. Its records mix pounds, years and yes-or-no answers, and many models can't tell that 12 pounds and 10 years aren't the same size of difference. You'll rescale features like these whenever a model compares examples by distance or keeps its weights small, which covers some of the most common models you'll use.

The example for this chapter is Happy Tails Pet Adoption: 800 pets in pets.csv, our own made-up data, with each pet's species, age, weight, size, color, whether its page has a photo, and whether it was adopted within 30 days. The file comes with Exercise: Adoption Pipeline, beside the seed script that made it. Each code block is one notebook cell, run in order.

When pounds drown out years

Distance Between Examples measured how alike two examples are with a straight-line distance. Here's a new arrival, a 1-year-old dog of 40 pounds with a photo on its page, and two pets the center already knows: a 1.2-year-old of 28 pounds with a photo, and an 11-year-old of 41 pounds without one. Which one is it most like?

import math

new_pet = (1.0, 40, 1)
pets = {
    "a 1.2-year-old, 28 lbs, with a photo": (1.2, 28, 1),
    "an 11-year-old, 41 lbs, no photo": (11.0, 41, 0),
}

for name, pet in pets.items():
    print(name, "is", round(math.dist(new_pet, pet), 2), "away")
a 1.2-year-old, 28 lbs, with a photo is 12.0 away
an 11-year-old, 41 lbs, no photo is 10.1 away

The distance says the old dog is the closer match, because 12 pounds of difference count for more than 10 years and a missing photo together. That's an accident of units: weigh the pets in ounces and weight would win by even more. Nearest neighbors would ask the wrong pets for their vote.

Standardizing: distance from the usual, in spreads

The fix puts every feature on the same footing. For each feature, take the value, subtract the feature's mean, and divide by its standard deviation (how far values typically sit from the mean). The result says how many standard deviations a value sits above or below the usual: 0 is average, 1 is one spread above it, −1 one below. This is standardizing, and your textbook may call the result a z-score: z = (x − mean) ÷ standard deviation, read "z equals x minus the mean, over the standard deviation".

scikit-learn's StandardScaler does it for every column. For now this lesson keeps the 760 pets whose weight was recorded; Missing Values in a Model deals with the 40 blanks.

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

pets = pd.read_csv("pets.csv")
known = pets.dropna(subset=["weight_lbs"])
features = ["age_years", "weight_lbs", "has_photo"]
X = known[features]
y = known["adopted_in_30_days"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, stratify=y, random_state=42
)

scaler = StandardScaler()
scaler.fit(X_train)
for name, mean, spread in zip(features, scaler.mean_, scaler.scale_):
    print(name, "mean:", round(mean, 2), "standard deviation:", round(spread, 2))
age_years mean: 3.85 standard deviation: 2.83
weight_lbs mean: 23.26 standard deviation: 24.1
has_photo mean: 0.7 standard deviation: 0.46

fit learned each column's mean (mean_, "mean underscore") and standard deviation (scale_) from the 570 training pets. Its standard deviation divides by the count, like statistics.pstdev, not by one less, like statistics.stdev; with hundreds of rows the two barely differ. Now the three pets, measured in spreads:

import math

three = pd.DataFrame([[1.0, 40, 1], [1.2, 28, 1], [11.0, 41, 0]], columns=features)
scaled = scaler.transform(three)
print("To the 1.2-year-old:", round(math.dist(scaled[0], scaled[1]), 2))
print("To the 11-year-old:", round(math.dist(scaled[0], scaled[2]), 2))
To the 1.2-year-old: 0.5
To the 11-year-old: 4.15

Ten years is more than three standard deviations of age, and 12 pounds is half of one standard deviation of weight. In spreads, the young dog is clearly the closer match.

The same three pets measured twice. In pounds and years, the new 1-year-old dog sits 12.0 from the 1.2-year-old and 10.1 from the 11-year-old, so the old dog looks closer; after standardizing, the 1.2-year-old is 0.5 away and the 11-year-old 4.15 away

Fit on training, transform both

The scaler learned its numbers from the training pets only, and transform measures every pet, training or test, with those same numbers. The test pets come out close to a mean of 0 and a standard deviation of 1, but not exactly, which is how it should be: they're new pets measured with the training ruler.

train_scaled = scaler.transform(X_train)
test_scaled = scaler.transform(X_test)
print("Test means:", test_scaled.mean(axis=0).round(2))
print("Test standard deviations:", test_scaled.std(axis=0).round(2))
Test means: [ 0.09  0.08 -0.16]
Test standard deviations: [1.07 1.02 1.06]

Fitting the scaler on all 760 pets would let the test pets' ages and weights shape the ruler, so facts about the test part would leak into training. fit_transform(X_train) fits and transforms the training part in one call; the test part only ever gets transform.

Which models care

Nearest neighbors, scored on the same test pets with and without scaling:

from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier

raw = KNeighborsClassifier(n_neighbors=15).fit(X_train, y_train)
scaled_knn = KNeighborsClassifier(n_neighbors=15).fit(train_scaled, y_train)
print("Neighbors, raw units:", round(raw.score(X_test, y_test), 3))
print("Neighbors, standardized:", round(scaled_knn.score(test_scaled, y_test), 3))

tree_raw = DecisionTreeClassifier(max_depth=3, random_state=0).fit(X_train, y_train)
tree_scaled = DecisionTreeClassifier(max_depth=3, random_state=0).fit(train_scaled, y_train)
same = (tree_raw.predict(X_test) == tree_scaled.predict(test_scaled)).all()
print("Tree, same guess for every test pet:", same)
Neighbors, raw units: 0.679
Neighbors, standardized: 0.758
Tree, same guess for every test pet: True

Neighbors improved because age and the photo finally count. The tree gave the same guess for every test pet: each of its questions looks at one feature ("is age at most 4.5?"), and stretching a column moves the threshold but not which pets fall on each side. Logistic regression cares too, in a different way: C keeps all its weights small by the same rule, which is only fair when features share a scale, and its search for the weights finishes in fewer steps on scaled data.

MinMaxScaler is the other common choice: it squeezes each feature into 0 to 1 using the training minimum and maximum. Either way, the scaler learns from the training data and nothing else.

Do it: standardize a new pet's weight

A rabbit has arrived at Happy Tails, and its weight has to be measured with the same ruler as the training rabbits' before a model can use it.

The inputs:

  • train_weights: the training rabbits' weights in pounds.
  • value: one weight to standardize.
  • new_weights: several new rabbits' weights.

For example, with eight training rabbits:

train_weights = [2, 4, 4, 4, 5, 5, 7, 9]
Step Work Result
Mean (2 + 4 + 4 + 4 + 5 + 5 + 7 + 9) ÷ 8 = 40 ÷ 8 5.0
Squared distances from the mean 9, 1, 1, 1, 0, 0, 4, 16 total 32
Standard deviation, dividing by the count √(32 ÷ 8) = √4 2.0
A new rabbit of 8 pounds (8 − 5.0) ÷ 2.0 1.5

So training_numbers(train_weights) returns (5.0, 2.0), standardize(8, 5.0, 2.0) returns 1.5, and standardize_all([8, 3], train_weights) returns [1.5, -1.0].

Your job: write three functions.

  • training_numbers(train_weights) returns a pair: the mean, then the standard deviation that divides by the count (statistics.pstdev, as StandardScaler does).
  • standardize(value, mean, spread) returns the value minus the mean, divided by the spread.
  • standardize_all(new_weights, train_weights) returns a list with each new weight standardized by the training rabbits' mean and standard deviation, never the new rabbits' own.

Practiced live on the lesson page: write the Python right there and check it.

Check yourself

2 questions to try on the lesson page.

So what

A model that measures distance or keeps its weights small only treats features fairly when they share a scale, and the scale has to come from the training data alone, so new data is measured with the same ruler.

Sources

  • scikit-learn 1.9.1, StandardScaler, API reference: the standard score z = (x − u) / s, where u and s are the mean and standard deviation of the training samples; fit computes them and transform uses them on later data; mean_ and scale_; a biased estimator for the standard deviation, equivalent to numpy.std(x, ddof=0); and that the L1 and L2 regularizers of linear models assume features centered around 0 with variance in the same order, so a feature with a much larger variance can dominate.
  • scikit-learn, Preprocessing data: Standardization, User Guide: standardization is a common requirement for many estimators; the scaler computes the mean and standard deviation on a training set to re-apply the same transformation to the testing set; MinMaxScaler scales features to lie between a minimum and maximum, often 0 and 1.
  • scikit-learn 1.9.1, MinMaxScaler, API reference: X_std = (X − X.min) / (X.max − X.min), feature_range (0, 1) by default, and fit computes the minimum and maximum used for later scaling.
  • scikit-learn, Decision Trees, User Guide: trees require little data preparation, while other techniques often require data normalization.
  • scikit-learn, Cross-validation: evaluating estimator performance, User Guide: preprocessing such as standardization should be learned from a training set and applied to held-out data.
  • Python documentation, statistics: fmean gives the arithmetic mean and pstdev the population standard deviation.

Key terms

Related