Impossible Values and a Cleaning Routine
Some values can't be right: a temperature of -999, humidity above 100%. Using a school weather station's readings, turn stand-in values into missing ones, flag what looks wrong instead of deleting it, check your data with assert, and put every cleaning step into one function that turns raw data into a clean file the same way every time.
Updated
Values That Can't Be Right
- Values That Can't Be Right: Sentinels and Outliers (Apprentice): A stand-in such as -999 for "no reading" and a sensor that reads 152 °F in the sun both throw off an average, so replace turns stand-ins into NaN, describe and between find the values that can't be true, and a suspect column flags them before they're blanked, while real but unusual readings stay.
- Checking Your Data with assert (Journeyman): assert stops the code with an AssertionError and your message whenever a condition is False, so checks on ranges with between and all, on repeated timestamps and on missing values, gathered in one check_readings function, catch a cleaning step that went wrong every time the cleaning runs.
A Cleaning Routine
- A Cleaning Function, Step by Step (Journeyman): Cleaning steps scattered across notebook cells are hard to run again the same way, so each step becomes a function that starts with result = df.copy() and returns a new table, one clean(raw) function runs them all in order, method chaining with pipe, assign and rename writes it as one readable recipe, and in pandas 3 a single value is changed with .loc.
- Saving the Clean Data: to_csv and to_excel (Apprentice): A clean table lives only as long as the notebook's kernel, so to_csv with index=False saves it as a file the next notebook can load, to_excel saves a workbook for people who use Excel, and keeping data/raw and data/clean folders, with a save function that refuses the raw folder, means the raw file is never written over.
Exercise: Weather Station Cleanup
- Exercise: Weather Station Cleanup (Journeyman): Write a cleaning routine for a school weather station's messy year of readings, turning -999 into missing values, flagging and blanking readings that can't be true, checking the result with assert, cleaning in one function that leaves the raw table alone, and saving a clean file without touching the raw one, and let pytest check each step.
Up Next: Answering Questions
- Up Next: Answering Questions (Apprentice): Look back at what Module 2 taught, finding and handling missing values, fixing types, removing duplicates, tidying text and dates, catching impossible values and putting it all in a cleaning function, try a cleaning log of your own, and see what Module 3 adds, filtering, sorting, grouping, pivot tables, joins and reshaping.