When Machine Learning Is the Wrong Tool

Machine learning is the wrong tool when a rule is already known, when a query on the data already holds the answer, when there are too few examples or the data doesn't hold the answer, when every decision must be explained exactly, or when a wrong guess costs too much, and a six-question checklist catches each case.

Apprentice · AI Basics · Python · Updated

By

After this lesson you can

  • Decide whether a problem needs machine learning, a written rule or a query, and say why
  • Spot the warning signs that machine learning won't work for a problem, such as too few examples or features that don't hold the answer

Key takeaways

  • When the rule is known, such as a sales tax, write the rule. A model would only guess at an answer you can work out exactly.
  • When the answer is already in the data, such as which members haven't visited this month, ask the data with a query instead of guessing.
  • A model needs many examples with answers, and features that really hold the answer. Twelve past floods aren't enough to learn from.
  • When every decision must be explained exactly, or a wrong guess costs too much, a written rule or a person is the safer choice.

After Sunrise Bakery's owner sees the weekday averages work, a nephew who's keen on technology has a suggestion: "Use machine learning for everything!" Work out the sales tax with it, find the loyalty members who've stopped coming, even predict the next time the river floods the bakery's street. Some of that is a good idea, and some of it is a bad one. Knowing which is one of the most useful skills in this course: you'll be the person in the meeting, or the group project, who asks "do we even need a model for this?"

When the rule is already known

The town's sales tax is 7%. That's a rule someone wrote down, and it's exact:

TAX_RATE = 0.07


def with_tax(price):
    return round(price * (1 + TAX_RATE), 2)


print(with_tax(6.00))
print(with_tax(25.00))
6.42
26.75

A model could learn to guess the tax from a thousand past receipts, and it would get close. But close is worse than exact, and the day the tax rate changes, the rule changes in one line while the model would keep guessing the old answer until it was trained again. When the rule is known and can be written down, write the rule.

In your textbook, a hand-written rule of thumb like the owner's "Saturdays are busy" is called a heuristic. A short heuristic is often the right first step. Machine learning starts to earn its place when the rules get so many and so tangled that nobody can keep them up to date: a guess that depends on the weekday, the weather, the holidays and the season all at once.

When a query already holds the answer

"Which loyalty members haven't visited in over a month?" sounds like a prediction, but it isn't one: the answer is already in the records. It needs a question asked of the data, called a query, not a guess:

days_since_last_visit = {"card 101": 3, "card 102": 27, "card 103": 41, "card 104": 12, "card 105": 35}

not_this_month = [card for card, days in days_since_last_visit.items() if days > 30]
print(not_this_month)
['card 103', 'card 105']

A query gives the exact answer every time. A model would give a guess about something that's already known. The difference to listen for: a model guesses something you don't know yet (tomorrow's loaves, whether a member will come back), and a query looks up something you already know (who hasn't come).

When the examples can't teach it

A model learns only what its examples hold, and three things stop it:

  • Too few examples. The bakery's street has flooded twelve times in fifty years. Twelve examples can't show a model what leads to a flood; it would mostly learn the wobble of those twelve.
  • No answers to learn from. A supervised model needs examples that come with their answers. A brand-new cinnamon roll has no sales history at all.
  • Features that don't hold the answer. Guessing tomorrow's loaves from the color of the delivery van can't work, however clever the model, because the van has nothing to do with how much bread people buy. A model can always find some pattern in a small table by luck; that's why it's scored on test data and compared with a baseline.

When a wrong guess costs too much

Every model is wrong sometimes: the weekday averages missed day 113 by about 23 loaves. That's fine for bread. It isn't fine when:

  • Every decision must be explained exactly. Each employee's pay follows the written rules for hours and overtime, and anyone must be able to check the arithmetic.
  • A wrong guess could hurt someone. Whether a loaf contains nuts must come from the recipe, never from a model that's right "most of the time".

A checklist

Before building a model, ask these six questions:

  1. Can someone write the rule down? Then write the rule.
  2. Is the answer already in the data? Then ask the data with a query.
  3. Are there many past examples, with their answers?
  4. Would the features be known at the moment of the guess, and could they plausibly hold the answer?
  5. Can the people who use it live with some wrong guesses?
  6. Does each decision have to be explained exactly?

A model fits when the answers to 1 and 2 are no, 3, 4 and 5 are yes, and 6 is no. And once it's built, it still has to beat its baseline on test data, as The Simplest Guess: Baselines showed.

Check yourself

2 questions to try on the lesson page.

So what

Machine learning is one tool among several: write the rule when it's known, query the data when the answer is already there, and build a model only when it has the examples to learn from and its mistakes can be lived with.

Sources

  • Martin Zinkevich, Rules of Machine Learning: Best Practices for ML Engineering, Google for Developers: Rule #1, don't be afraid to launch a product without machine learning, because machine learning requires data and a simple heuristic can get you far; Rule #3, choose machine learning over a complex heuristic, because a complex heuristic is unmaintainable.
  • scikit-learn, Glossary of Common Terms and API Elements: supervised learning needs the expected prediction (the label) for each sample when the model is fitted.
  • scikit-learn, Cross-validation: evaluating estimator performance, User Guide, its opening: a model scored on the data it learned from can look perfect and still predict nothing useful on new data, which is why a test set is held out.

Key terms

Related