Exercise: Market Day Warm-Up

Write five functions that load a farmers' market's 260 weekly sales from a CSV file, count the missing values in each column, fill the blank sales counts with the median on a copy and flag them, pick out one stall's rows, and work out each stall's share of the rows, and let pytest check what each one returns.

Apprentice · Python · Updated

By

The Hometown Farmers' Market kept a year of weekly sales for its five stalls, and like most real records, a few weeks' counts never made it into the file. Every model in this course starts the way this exercise does: load the table, find what's missing, fill the gaps without hiding them, and look at the rows you care about. You'll write the five functions that do it, tests call each one and check what it hands back, and a notebook comes with the exercise for looking at the data while you work.

Get the exercise code

Open this lesson's GitHub repository, github.com/ask-codey/exercise-ml-market-day-warm-up, also linked from the Exercise box beside the lesson. Clone it into your python-course folder. In VS Code's terminal:

cd ~/python-course
git clone https://github.com/ask-codey/exercise-ml-market-day-warm-up.git

Git makes a folder, exercise-ml-market-day-warm-up, with the exercise code inside. Get the Exercise Code with Git walks through cloning step by step, if you'd like a reminder. Open that folder in VS Code (File > Open Folder), open its terminal with Terminal > New Terminal, and make the exercise's virtual environment. On Windows (PowerShell):

python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txt

On a Mac (zsh):

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

The prompt starts with (.venv) once the virtual environment is on. The exercise's README says what each command does, and what to do if PowerShell won't run the second one.

The folder holds:

  • market_sales.csv: 260 rows, our own made-up data: one row for each of the five stalls in each of 52 weeks, with the columns week, stall, product, price, units_sold and weather. Twelve units_sold values are blank.
  • seed_data.py: the script that made the CSV, with the pattern it planted and which stalls' counts went missing. You don't need to run it: the committed CSV is what keeps everyone's numbers the same.
  • market_day.py: the five functions, for you to finish.
  • explore.ipynb: a notebook for looking at the sales and trying your code. Choose the .venv kernel with Select Kernel (top right) before you run it. The tests don't read it.
  • report.py: runs your five functions on the whole file and prints what each one gave. Ready to use.
  • tests/: the tests, which call each function and check what comes back, and conftest.py, which gives each test a fresh copy of the sales. You don't need to change them.
  • requirements.txt and pytest.ini: the packages the exercise needs, at exact versions, and where pytest finds the tests and your code.

What to do

Every function in market_day.py is already declared with its name and parameters, and for now it stops with a NotImplementedError that says Not built yet. The docstring under each one says what it should do, with examples:

def fill_missing_units(sales):
    """Return a copy of sales with every blank units_sold filled in, and a flag column.

    Fill each blank units_sold with the median of the units_sold values that are
    there. Add a True/False column named units_filled that is True on each row
    that was blank (and so was filled) and False on every other row, so nobody
    mistakes a filled-in guess for a real count.

    Work on a copy: sales itself must come out unchanged, still with its blanks
    and without a units_filled column.

    With units_sold 10, blank, 30, 20: the median of 10, 30 and 20 is 20, so
    units_sold becomes 10, 20, 30, 20 and units_filled is False, True, False, False.
    """
    raise NotImplementedError("Not built yet.")

In each function, replace the raise NotImplementedError("Not built yet.") line with your own code:

  1. load_sales(path) reads the CSV file at path and returns it as a DataFrame. With market_sales.csv: 260 rows and 6 columns.
  2. count_missing(sales) returns a dictionary of every column's name and how many of its values are missing. With market_sales.csv: {'week': 0, 'stall': 0, 'product': 0, 'price': 0, 'units_sold': 12, 'weather': 0}.
  3. fill_missing_units(sales) returns a copy of sales with each blank units_sold filled with the median of the counts that are there, plus a units_filled column that is True on the rows it filled. With market_sales.csv: the median is 28, so each of the 12 blanks becomes 28, and 12 rows are flagged. The table it was given still has its 12 blanks afterward.
  4. rows_for_stall(sales, stall) returns the rows whose stall is stall. rows_for_stall(sales, "Honey Hollow") gives Honey Hollow's 52 weeks; a stall that isn't in the table, such as "Moonlight Pies", gives 0 rows.
  5. stall_shares(sales) returns a dictionary of each stall and its share of the rows, from 0 to 1, most common first. With market_sales.csv: 0.2 for each of the five stalls. With a table of three Honey Hollow rows and one Sunny Patch row: {'Honey Hollow': 0.75, 'Sunny Patch': 0.25}.

Check your work

Run the tests from the exercise folder, with the virtual environment on:

python -m pytest

At first all 16 tests fail. That's expected: each function you finish turns its tests green. A failing test's message says what it called, what it expected and what came back, for example len(rows_for_stall(sales, 'Honey Hollow')) should return 52, but it returned 260. Besides the market's file, the tests use small tables of a few rows written inside the test file, so every answer can be worked out by hand.

To see your functions at work on the whole file, run python report.py.

Hints

  • Look before you write. Run the notebook's first cells: sales.info() shows which column has fewer values than rows, and sales.isna().sum() counts the blanks. A First Look: head, info and describe and Finding Missing Values: NaN and isna show both.
  • A Series is one step from a dictionary. isna().sum() and value_counts(normalize=True) each give back a Series, with the column names or the stalls as its index. to_dict() turns a Series into a dictionary whose keys are that index.
  • Flag first, then fill. Once a blank is filled, nothing tells you it was ever blank, so make the units_filled column before you call fillna. Dropping or Filling Missing Values shows the flag column and the median.
  • A copy keeps the original safe. sales.copy() gives you a table of your own to change. Adding a column to sales itself would change the caller's table.
  • A mask picks the rows. sales["stall"] == stall answers True or False for every row; Filtering Rows with a Condition shows how to keep the True ones.
  • Shares are counts divided by the total. Counting What's There: value_counts and unique shows normalize=True.

When you're done

Every test is green: pytest's last line says 16 passed. And python report.py prints:

1) rows and columns: (260, 6)
2) missing values: {'week': 0, 'stall': 0, 'product': 0, 'price': 0, 'units_sold': 12, 'weather': 0}
3) median of units_sold: 28.0
4) rows filled in: 12
5) blanks left after filling: 0
6) blanks still in the original table: 12
7) Honey Hollow's weeks: 52
8) Honey Hollow's median week: 16.0
9) each stall's share of the rows: {'Maple Lane Farm': 0.2, 'Honey Hollow': 0.2, 'Riverbend Greens': 0.2, 'Stone Oven Bakery': 0.2, 'Sunny Patch': 0.2}
10) stalls among the blank rows: {'Honey Hollow': 0.5, 'Sunny Patch': 0.25, 'Maple Lane Farm': 0.17, 'Stone Oven Bakery': 0.08}

Now read lines 3, 8 and 10 together. Every blank got the whole market's median, 28 units, but half of the blanks are Honey Hollow's, and Honey Hollow's usual week is 16. Was 28 a fair guess for a honey week? A model trained on this table would learn from those 28s as if someone had counted them, which is why the units_filled column matters: it tells whoever uses the table next which numbers are guesses. In the notebook, look at the filled rows (filled[filled["units_filled"]]), then work out what a fairer guess for each one would be.

Submit your exercise

The exercise is how you finish this lesson. When your work is done, put it on GitHub in a repo of your own, then submit that repo's address here. Hand In Your Work with Git walks through each step.

  1. On GitHub, click +, then New repository. Name it (for example exercise-ml-market-day-warm-up), choose Public, and click Create repository. Leave out the README: your folder already has one.

  2. In VS Code's terminal, in your exercise folder, send your work to your new repo. Point your copy at your own repo, then commit and push:

    git remote set-url origin https://github.com/your-name/exercise-ml-market-day-warm-up.git
    git add .
    git commit -m "My solution"
    git push -u origin main
    
  3. Paste your repo's address, such as https://github.com/your-name/exercise-ml-market-day-warm-up, into Submit your exercise on this page. The site checks that the repo is public and holds your commits, and then the lesson is complete.

Changed something after you submitted? Push it, and submit again: your latest submission is the one that counts.

So what

Loading a table, counting its gaps, filling them with a flag that says so, and checking which rows the gaps fall in: that's the first look a dataset deserves before any model learns from it.

Key terms

Related