Nearest Neighbors
A nearest neighbors classifier guesses a new example's class by finding the k training examples closest to it and letting them vote, with KNeighborsClassifier(n_neighbors=5) storing the training rows instead of learning a rule, a small k following every quirk, a large k blurring the classes together, and a tied vote, which in the lesson's run went to the class listed first.
Journeyman · AI Basics · Python · Updated
After this lesson you can
- Train a KNeighborsClassifier with n_neighbors, and read a guess's neighbors, distances and votes with kneighbors and predict_proba
- Explain how a small or large k changes the guesses, how ties are settled, and why nearest neighbors stores examples instead of learning a rule
Key takeaways
- k nearest neighbors finds the k training examples closest to a new one and gives it the class most of them have.
- KNeighborsClassifier's fit stores the training rows instead of learning a rule, so every guess searches them all again.
- A small k follows every odd example, and k=1 gets every training row right. A large k blurs the classes, and k is a setting you choose.
- With two classes, an odd k can't tie. With more classes it still can, and in this lesson's run a tie went to the class listed first.
- The distances use the features' raw units, so grams decide which fruits count as neighbors until the features are scaled.
Distance Between Examples found the one known fruit nearest a new one and copied its label. Trusting a single fruit is risky: what if that one apple was an odd, pale apple sitting among pears? Ask a few neighbors instead and go with the majority, the way you'd ask several friends before choosing a restaurant. That's k nearest neighbors (k-NN for short; k is how many neighbors you ask). You'll meet it in recommendations ("shoppers like you also bought…"), in finding similar records, and as a quick, honest first model to compare others against.
The example is fruit.csv from Exercise: Fruit Sorter, split as in Decision Trees. Each code block is one notebook cell, run in order.
Ask five neighbors
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
fruit = pd.read_csv("fruit.csv")
features = ["weight_g", "diameter_cm", "sugar_brix", "skin_redness"]
X = fruit[features]
y = fruit["fruit"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42, stratify=y)
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
new_fruit = pd.DataFrame({"weight_g": [185], "diameter_cm": [7.2],
"sugar_brix": [13.4], "skin_redness": [7.5]})
print("Guess:", knn.predict(new_fruit)[0])
print("Classes:", knn.classes_)
print("Votes:", knn.predict_proba(new_fruit))
Guess: apple
Classes: ['apple' 'pear' 'plum']
Votes: [[1. 0. 0.]]
n_neighbors=5 is k (5 is also the default). predict_proba is the share of the votes for each class, in classes_ order: all five neighbors are apples. Each neighbor has one equal vote by default (weights="uniform"), and the class with the most votes wins: a majority vote.
kneighbors shows who voted, as each neighbor's distance and its position in the training rows:
distances, positions = knn.kneighbors(new_fruit)
neighbors = X_train.iloc[positions[0]].assign(
fruit=y_train.iloc[positions[0]], distance=distances[0].round(2))
print(neighbors)
weight_g diameter_cm sugar_brix skin_redness fruit distance
261 186 8.1 13.5 6.4 apple 1.74
165 186 7.7 14.5 8.4 apple 1.81
95 186 7.1 15.2 7.9 apple 2.10
94 183 8.0 13.1 7.1 apple 2.21
396 185 6.5 13.2 5.4 apple 2.22
The first is the apple from Distance Between Examples, at the same 1.74. Look at the weights: every neighbor weighs 183 to 186 grams, while their redness runs from 5.4 to 8.4. The distances are in raw units, so grams decide who counts as a neighbor, the problem Scaling Features in Module 4 solves. Here all five happen to be apples, but they were picked for their weight far more than for their color.
Small k, large k
for k in [1, 5, 15, 51, 151]:
model = KNeighborsClassifier(n_neighbors=k)
model.fit(X_train, y_train)
train_right = (model.predict(X_train) == y_train).sum()
test_right = (model.predict(X_test) == y_test).sum()
print("k =", k, "| training", train_right, "of 360 | test", test_right, "of 90")
k = 1 | training 360 of 360 | test 78 of 90
k = 5 | training 333 of 360 | test 78 of 90
k = 15 | training 321 of 360 | test 75 of 90
k = 51 | training 282 of 360 | test 61 of 90
k = 151 | training 265 of 360 | test 61 of 90
With k = 1, every training fruit's nearest neighbor is itself, so the training score is perfect and means nothing: the same memorizing a tree with no depth limit showed. A small k is jumpy, following every odd fruit. A large k is blurry: 151 neighbors reach far past the fruits that are really alike, and the guesses slide toward whatever is common nearby. scikit-learn's guide puts it this way: a larger k quiets the noise but makes the boundaries between classes less distinct, and the best k depends on the data. k is a setting you choose, not something fit learns; Module 4 shows how to choose it fairly.
Ties
With an even k, the vote can split. Four fruits with only their redness, two pears and two apples, and k = 2:
redness = pd.DataFrame({"skin_redness": [2.0, 7.0, 3.0, 8.0]})
kinds = ["pear", "apple", "pear", "apple"]
two = KNeighborsClassifier(n_neighbors=2)
two.fit(redness, kinds)
one_fruit = pd.DataFrame({"skin_redness": [4.9]})
print("Neighbors' distances:", two.kneighbors(one_fruit)[0].round(2))
print("Votes:", two.predict_proba(one_fruit))
print("Guess:", two.predict(one_fruit)[0])
Neighbors' distances: [[1.9 2.1]]
Votes: [[0.5 0.5]]
Guess: apple
One pear at 1.9, one apple at 2.1: a tie, and in this run the guess is apple, the class listed first in classes_, even though the pear is nearer. That's what the run showed, not a promise: scikit-learn's documentation doesn't say how a tied vote is settled. With two classes, an odd k (3, 5, 7) can't tie. With three classes, five votes can still split 2, 2 and 1, so an odd k only makes ties rarer. (weights="distance" gives nearer neighbors bigger votes, another way to settle them.)
It remembers instead of learning
A tree's fit boils 360 fruits down to a few questions. k-NN's fit keeps all 360 fruits, and every guess measures the distance to them again. Your textbook calls this instance-based learning: the model is the examples themselves. The honest trade:
- Strengths: nothing to learn up front, any number of classes, and boundaries of any shape, because they follow the examples.
- Weaknesses: guessing slows down and takes more memory as the examples grow; the units of each feature decide the distances; and every feature counts, even one that tells you nothing, because all of them go into the distance.
Do it: let the nearest fruits vote
Three Rivers has the distances from a new fruit to each known fruit, and wants the class the k nearest vote for.
The inputs:
neighbors: a list of pairs(distance, label), one for each known fruit, in any order.k: how many of the nearest fruits get a vote.
For example, with k = 3:
neighbors = [(1.9, "apple"), (0.8, "pear"), (2.5, "pear"), (1.2, "apple"), (1.5, "apple"), (3.1, "plum")]
| Nearest | Distance | Label |
|---|---|---|
| 1st | 0.8 | pear |
| 2nd | 1.2 | apple |
| 3rd | 1.5 | apple |
Apple has 2 votes and pear has 1, so majority_vote(neighbors, 3) returns "apple".
Your job: write majority_vote(neighbors, k).
- It keeps the
kpairs with the smallest distances. - It counts how many of them have each label.
- It returns the label with the most votes.
- If two or more labels tie for the most votes, it returns the one that comes first in alphabetical order (as
KNeighborsClassifierdid in the lesson).
Practiced live on the lesson page: write the Python right there and check it.
Check yourself
2 questions to try on the lesson page.
So what
Nearest neighbors is the most human of models, "go with what the most similar cases did", and its honesty is its weakness: it knows nothing beyond its examples, its distances and the k you choose.
Sources
- scikit-learn, Nearest Neighbors, User Guide: neighbors-based classification is instance-based, non-generalizing learning that stores the training data instead of building a general model; a query point gets the class with the most representatives among its nearest neighbors (a simple majority vote); the best k is highly data-dependent, and a larger k suppresses noise but makes the classification boundaries less distinct;
weights="uniform"(the default) or"distance". - scikit-learn 1.9.1, KNeighborsClassifier, API reference:
n_neighborsdefaults to 5; the default metric is the Euclidean distance;kneighborsreturns the distances to and indices of the neighbors;predict_probagives the class probabilities in lexicographic order; when neighbors k and k+1 are at identical distances with different labels, the result depends on the order of the training data. - scikit-learn, Importance of Feature Scaling, example: features on very different ranges give k-nearest neighbors a completely different model from scaled data, because the feature with the bigger values mostly decides the distances.