Up Next: Cleaning Data
Look back at what Module 1 taught you, put it to work on a first look at a table of your own, and see what Module 2's cleaning tools will let you do with messy data.
Apprentice · Python · Updated
After this lesson you can
- Describe what Module 2's cleaning tools will let you do with messy data, such as missing values, numbers stored as text and duplicate rows
Key takeaways
- After Module 1 you can work in a notebook without being fooled by run order, load a table from CSV, Excel or JSON, and say what's in it.
- A first look at any table is the same few lines, shape, info, describe and value_counts, with a sentence about what each one shows.
- Real data arrives messy, with gaps, numbers typed as text, rows entered twice and values that can't be right, and pandas won't always warn you.
- Module 2 teaches you to find and fix each of those problems, and to put every fix into one cleaning function you can run again.
At the start of this module, a table of data was something you scrolled through in a spreadsheet. Now you can load one into Python from three kinds of file and find out what's in it with a handful of lines. That's the first hour of almost any data job, and you can do it.
This lesson looks back at what you can do, gives you a first look to try on data of your own, and shows you what Module 2 adds: cleaning the messy data that real files are full of.
What you can do now
After Module 1, you can:
- work in a notebook in VS Code, with a virtual environment that has pandas, and use Restart and Run All so a notebook's answers are ones the file can reproduce;
- get exercise code with Git and hand in your work;
- use the Python data work leans on: functions, lists, dictionaries, and rows of data as a list of dictionaries;
- make tables with pandas: a Series for one column, a DataFrame for a whole table, with math on whole columns at once;
- load data from a CSV file, an Excel workbook (the right sheet, title rows skipped) and JSON, nested lists included;
- take a first look:
head,info,describeandvalue_counts.
Here's a first look at Sunnyside Farmers Market's season of sales, the chapter's example, in a few lines:
import pandas as pd
sales = pd.read_csv("sales.csv")
print("rows:", len(sales))
print("vendors:", sales["vendor_id"].nunique())
print("units missing:", len(sales) - sales["units"].count())
print("average price:", sales["price"].mean().round(2))
rows: 1900
vendors: 15
units missing: 12
average price: 10.63
At the start of the module, those five lines would have meant nothing. Now you can read every one.
Put it together: a first look of your own
Here's a way to prove it to yourself. Pick a table you care about: a budget you keep in a spreadsheet, a sports team's results, a dataset from a class, a reading log. Save it as a CSV file, or use the Excel workbook as it is, and make a notebook beside it that answers:
- How big is it?
shape, and what one row stands for. - What's in each column?
info(): the types, and which columns have gaps. - What's normal?
describe(), with one sentence in plain words about a number column's middle and spread. - How many of each?
value_counts()on a column you'd group by.
Put a Markdown cell after each result that says what it means, then run Restart and Run All. For a stretch, many apps let you download your own data as JSON: load it, and flatten a nested list with json_normalize. Keep personal details out of anything you share.
Where it gets messy
The market's sales were kind to you: twelve blank units, and everything else where it should be. Real files are rarely that tidy. Volunteers type unknown for an age nobody knew, or 59 lbs with the unit in the weight. A form gets submitted twice. A sensor reports 152 °F on a mild afternoon. And pandas doesn't always complain. Here are three weights as a volunteer typed them:
weights = pd.Series(["12 lbs", "5.5", "about 15"])
print(weights.dtype)
print(weights.sum())
str
12 lbs5.5about 15
The weights are text, so sum() glued them together instead of adding them. No error, and a total that's nonsense.
What Module 2 unlocks
Module 2 is about cleaning, the part of the job where most of an analyst's time goes:
- Missing values: find every gap, including words like
unknownthat mean "missing", then decide honestly whether to drop or fill each one. The first chapter's example is an animal shelter's intake log. - Types and duplicates: turn text such as
$75and12 lbsinto numbers you can add up, and remove the rows a volunteer entered twice without losing real return visits. - Text and dates: tidy names with stray spaces and capital letters, make categories match, and turn text into dates and times you can do math with.
- Values that can't be right: catch impossible readings, check your data with
assert, and put every step into one cleaning function that saves a clean file.
By the end of it, a messy file goes in and a clean one comes out, with every change written down where you can check it.
Check yourself
2 questions to try on the lesson page.
So what
Module 1 taught you to open a table and say what's in it; Module 2 teaches you to make it trustworthy, because every chart and answer after that depends on clean data.