Exercise: Weather Station Cleanup

Write a cleaning routine for a school weather station's messy year of readings, turning -999 into missing values, flagging and blanking readings that can't be true, checking the result with assert, cleaning in one function that leaves the raw table alone, and saving a clean file without touching the raw one, and let pytest check each step.

Journeyman · Python · Updated

By

The science department at Ridgeview School wants a clean year of readings from the weather station on its roof, one it can trust and make again when June's readings arrive. The station's file isn't ready for that: it writes -999 when its sensor drops out, reads near 150 °F when the sun shines on the sensor, reports humidity above 100% on foggy mornings, and sent some readings twice after its network came back. You'll write the five functions of a cleaning routine, and tests call each one and check what it does. A notebook comes with the exercise for looking at the data while you work.

Get the exercise code

Open this lesson's GitHub repository, github.com/ask-codey/exercise-data-weather-station-cleanup, also linked from the Exercise box beside the lesson. Clone it into your python-course folder. In VS Code's terminal:

cd ~/python-course
git clone https://github.com/ask-codey/exercise-data-weather-station-cleanup.git

Git makes a folder, exercise-data-weather-station-cleanup, with the exercise code inside. Open that folder in VS Code (File > Open Folder), open its terminal with Terminal > New Terminal, and make the exercise's virtual environment. On Windows (PowerShell):

python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -r requirements.txt

On a Mac (zsh):

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

The prompt starts with (.venv) once the virtual environment is on; the exercise's README says what each command does. Then make the data, once:

python seed.py
Wrote data/raw/readings.csv: 13344 rows.

The folder holds:

  • seed.py: makes data/raw/readings.csv, a school year of readings every 30 minutes, our own made-up data. Its fixed random seed means everyone gets the same rows.
  • station.py: the five functions, for you to finish.
  • explore.ipynb: a notebook for looking at the readings and trying your code. Choose the .venv kernel with Select Kernel (top right) before you run it. The tests don't read it.
  • report.py: cleans the readings with your functions, saves the clean file in data/clean and prints what changed. Ready to use.
  • tests/: the tests, which call each function and check what comes back, and conftest.py, which gives every test its own fresh copy of the readings. You don't need to change them.
  • requirements.txt and pytest.ini: the packages the exercise needs, at exact versions, and where pytest finds the tests and your code.

What to do

Every function in station.py is already declared with its name and parameters, and for now it stops with a NotImplementedError that says Not built yet. The docstring under each one says what it should do, with examples:

def replace_sentinels(df):
    """Return a new DataFrame with every -999 replaced by NaN.

    The station writes -999 in temp_f and humidity_pct when its sensor drops out: it means
    "no reading". Every -999, in any column, becomes NaN, and every other value stays as it was.

    temp_f 61.2, -999, 58.4 -> 61.2, NaN, 58.4

    On the readings seed.py makes: 61 temperatures and 61 humidities become NaN.
    """
    raise NotImplementedError("Not built yet.")

In each function, replace the raise NotImplementedError("Not built yet.") line with your own code:

  1. replace_sentinels(df) returns a new table with every -999 turned into NaN: temperatures 61.2, -999, 58.4 become 61.2, NaN, 58.4.
  2. flag_suspect(df) returns a new table with a suspect column, True for a row whose temp_f is below -30 or above 120, or whose humidity_pct is below 0 or above 100, and False for every other row, a missing value included. Then it makes each of those impossible values NaN. Temperatures 63.6, 152.1, 65.2 become 63.6, NaN, 65.2, with suspect False, True, False. A humidity of exactly 100 is fine.
  3. check_readings(df) checks a cleaned table with four asserts, each with a message that names what failed: every temp_f that's there is between -30 and 120 (the message names temp_f); every humidity_pct that's there is between 0 and 100 (it names humidity_pct); no timestamp appears twice (it names timestamp); and no more than 5% of temp_f is missing (it says missing). One missing temperature out of 20 passes; two out of 10 doesn't. A table that passes gives back None and prints nothing.
  4. clean_readings(raw) runs every step in order: replace_sentinels, then flag_suspect, then drops each row whose timestamp is already in a row above it (keeping the first), then check_readings, and returns the result. On the seeded readings, 13,344 rows go in and 13,296 come out, 71 of them suspect, and raw is exactly as it was.
  5. save_clean(df, folder) writes df to readings_clean.csv in folder, without the index, making the folder if it isn't there, and returns the file's Path. If the folder's name is raw, it raises a ValueError and writes nothing.

replace_sentinels, flag_suspect and clean_readings return a new table and leave the one they were given unchanged; the tests check that too. A Cleaning Function, Step by Step shows why a step that changes a column starts with result = df.copy().

Check your work

Run the tests from the exercise folder, with the virtual environment on:

python -m pytest

At first all 21 tests fail. That's expected: each function you finish turns its tests green. A failing test's message says what it called, what it expected and what came back, for example flag_suspect(df)['temp_f'] for 63.6, 152.1, 65.2 should give [63.6, None, 65.2], but it gave [63.6, 152.1, 65.2], where None stands for a missing value.

To see your functions at work on the whole school year, run python report.py.

Hints

  • Look before you write. Run the notebook's cells: describe() shows the -999s and the impossible highs, as Values That Can't Be Right: Sentinels and Outliers does.
  • One method does replace_sentinels' whole job, and it hands back a new table by itself.
  • Flag first, then blank. Once a value is NaN, nothing shows it was ever impossible. mask(condition) blanks the values where the condition is True, and a comparison with a missing value is False, so missing values are never flagged.
  • between says False for a missing value, so check_readings runs its range checks after dropna(), as Checking Your Data with assert does. .all() turns a column of answers into one.
  • Order matters in clean_readings. Flag before the sentinels are gone and every -999 looks "below -30". pipe lets you write the steps as a chain, one per line.
  • Path(folder).name is the last part of a folder's path, such as raw in data/raw, as Saving the Clean Data: to_csv and to_excel shows.

When you're done

Every test is green: pytest's last line says 21 passed. And python report.py prints:

1) raw rows: 13344
2) raw temperatures of -999: 61
3) clean rows: 13296
4) suspect rows: 71
5) temperatures missing: 67
6) hottest temperature: 97.6
7) average temperature: 43.9
8) raw rows after cleaning: 13344
9) saved: data/clean/readings_clean.csv

Now look at lines 5 and 6: 67 temperatures are missing, 61 of them from sentinels and 6 from the sun on the sensor, and the hottest reading left is a real heat wave. If next September's file arrives with a new kind of mistake, which of your five functions would you change, and which check would tell you it's there?

Submit your exercise

The exercise is how you finish this lesson. When your work is done, put it on GitHub in a repo of your own, then submit that repo's address here. Hand In Your Work with Git walks through each step.

  1. On GitHub, click +, then New repository. Name it (for example exercise-data-weather-station-cleanup), choose Public, and click Create repository. Leave out the README: your folder already has one.

  2. In VS Code's terminal, in your exercise folder, send your work to your new repo. Point your copy at your own repo, then commit and push:

    git remote set-url origin https://github.com/your-name/exercise-data-weather-station-cleanup.git
    git add .
    git commit -m "My solution"
    git push -u origin main
    
  3. Paste your repo's address, such as https://github.com/your-name/exercise-data-weather-station-cleanup, into Submit your exercise on this page. The site checks that the repo is public and holds your commits, and then the lesson is complete.

Changed something after you submitted? Push it, and submit again: your latest submission is the one that counts.

So what

A cleaning routine is a recipe you can trust: stand-ins become missing values, impossible readings are flagged before they're blanked, the checks run every time, and the raw file stays as the evidence. When next year's readings arrive, one function call cleans them the same careful way.

Key terms

Related