Up Next: Charts
Look back at what Module 3 taught (picking, filtering and sorting rows, new columns, grouping, pivot tables, totals by period, stacking, joining and reshaping tables), try five questions about a school's records with all of it, and see what Module 4 adds, charts that show an answer at a glance and keep it honest.
Journeyman · Python · Updated
After this lesson you can
- Describe what Module 4's charts will let you show about the answers you can now find with pandas
Key takeaways
- After Module 3 you can answer a specific question from one table or several, by filtering, grouping, joining and reshaping, and check each step with a row count.
- Five questions about Oakridge's records, each answered by a function that returns a small table, put every one of those tools to work.
- A table of numbers is exact but slow to read; most people who ask a question won't read one.
- Module 4 draws the answers as line, bar, scatter and histogram charts with matplotlib, and keeps their titles, axes and colors honest.
When this module started, you could load a table and say what was in it. Now you can answer the questions people actually ask: which homeroom did best, which club's members improved most, who isn't in a club yet. That's the everyday work of a data analyst, in a lab, an office or a class project.
This lesson looks back at what you can do, gives you five questions to answer with all of it, and shows what Module 4 adds: charts that carry an answer to people who won't read a table.
What you can do now
After Module 3, you can:
- pick the rows and columns you need with
locandiloc, and filter with conditions joined by&,|and~; - sort and make top lists with
sort_values,nlargestandnsmallest; - add new columns from math,
np.whereandpd.cut; - group and summarize with
groupbyandagg, as shares of a total, in pivot tables and crosstabs, and by week or month withresample; - stack files with
pd.concat, join tables withmergeand the join that fits, and catch a join that goes wrong withindicator=True,validate=and a row count; - reshape between wide and long with
meltandpivot.
Here's one question that uses most of it, on Oakridge Middle School's records: which club's members improved most from fall to spring?
import pandas as pd
fall = pd.read_csv("fall_scores.csv")
spring = pd.read_csv("spring_scores.csv")
clubs = pd.read_csv("clubs.csv")
memberships = pd.read_csv("memberships.csv")
both = pd.concat([fall.assign(term="fall"), spring.assign(term="spring")], ignore_index=True)
year = both.melt(id_vars=["student_id", "term"], var_name="quiz", value_name="score")
change = year.pivot_table(index="student_id", columns="term", values="score")
change["gain"] = change["spring"] - change["fall"]
members = memberships.merge(clubs, on="club_id", validate="many_to_one")
gains = members.merge(change, on="student_id").groupby("club_name")["gain"].mean()
gains.round(1).sort_values(ascending=False)
club_name
Robotics 4.6
Chess Club 4.2
Coding Club 4.1
Art Club 3.9
Drama 3.8
Choir 3.6
Garden Club 2.9
Name: gain, dtype: float64
Stacked, melted, averaged per student with pivot_table (here averaging is the point), joined twice with a check, grouped and sorted: you can read every line. A student in two clubs counts in both, which is right for a question about clubs.
Try it: five questions from the principal
The best way to find out how much you can do is to answer real questions with no step-by-step instructions. Use the five files from Exercise: Report Card Merge, and answer these, one function each that returns a small table:
- Which homeroom had the highest spring average? All nine homerooms, highest first.
- Which quiz was hardest in each term? The lowest average quiz, fall and spring.
- Which club's members scored highest in the spring? Every club with members, and how many members each has.
- Which ten students improved most from fall to spring? With their first names and homerooms.
- What share of each grade is in at least one club? As a percentage, one decimal place.
Write one function at a time, and check each answer on a few rows by hand before you trust it. Count the rows before and after every join: a membership that points at a club that doesn't exist, and a student in two clubs, are both waiting in the files. There's no test suite for this one: when your answers match what you work out by hand on a few students, they work.
Where tables hit a wall
The club table above is exact, and it still takes a moment to read. Two things limit every answer you can give so far:
- A table is slow to read. Seven numbers are fine; twelve months of sales for six stores aren't. The pattern is in there, and the reader has to dig it out.
- Most people who ask won't read one. A principal, a manager or a client wants to see the answer, not search for it in a grid.
What Module 4 unlocks
Module 4 draws your answers with matplotlib, the package most Python charts are drawn with:
- Four charts: a line chart for change over time, a bar chart to compare groups (like the club table above), a scatter plot for two things that may move together, and a histogram for how values are spread.
- Choosing the chart that fits the question, with a title that states the finding and labels that give the units.
- Honest charts: bars that start at zero, side-by-side charts on the same scale, and colors that everyone can tell apart.
- Several charts on one page, saved as an image for a report.
The tables you've learned to build are what every chart is drawn from, so nothing you did in this module is wasted: Module 4 makes it visible.
Check yourself
2 questions to try on the lesson page.
So what
Module 3 gave you answers; Module 4 makes them seen, by drawing each table as a chart someone can read in a few seconds.