Permutation Importance
noun
say it: “pur-myoo-TAY-shun im-POR-tunss” /ˌpɝmjuˈteɪʃən ɪmˈpɔrtəns/
A measure of how much a model depends on a feature, found by shuffling that feature's values and seeing how much worse the model scores.
What is permutation importance?
Permutation importance measures how much a model leans on each feature: shuffle that one column so it no longer
matches its rows, score the model again, and see how much worse it does. A big drop means the model relies on it. It
works for any model; scikit-learn's is permutation_importance(), best run on test data.
from sklearn.ensemble import RandomForestRegressor
from sklearn.inspection import permutation_importance
homes = [[1200, 3], [1800, 7], [1500, 2], [2400, 9], [900, 4], [2000, 1]] # sqft, house number
bills = [80, 110, 95, 150, 60, 125] # monthly energy bill, dollars
model = RandomForestRegressor(random_state=0).fit(homes, bills)
result = permutation_importance(model, homes, bills, n_repeats=5, random_state=0)
print(result.importances_mean.round(2)) # [1.52 0.11]: sqft matters, the house number barely
In a sentence
- “Shuffling the house numbers barely changed the score, so their permutation importance was near 0.”
- “Permutation importance showed the energy model leaned mostly on square footage.”
Level: Freelancer · More AI Basics terms at this level
Updated