Up Next: Charts

Look back at what Module 3 taught (picking, filtering and sorting rows, new columns, grouping, pivot tables, totals by period, stacking, joining and reshaping tables), try five questions about a school's records with all of it, and see what Module 4 adds, charts that show an answer at a glance and keep it honest.

Journeyman · Python · Updated

By

After this lesson you can

  • Describe what Module 4's charts will let you show about the answers you can now find with pandas

Key takeaways

  • After Module 3 you can answer a specific question from one table or several, by filtering, grouping, joining and reshaping, and check each step with a row count.
  • Five questions about Oakridge's records, each answered by a function that returns a small table, put every one of those tools to work.
  • A table of numbers is exact but slow to read; most people who ask a question won't read one.
  • Module 4 draws the answers as line, bar, scatter and histogram charts with matplotlib, and keeps their titles, axes and colors honest.

When this module started, you could load a table and say what was in it. Now you can answer the questions people actually ask: which homeroom did best, which club's members improved most, who isn't in a club yet. That's the everyday work of a data analyst, in a lab, an office or a class project.

This lesson looks back at what you can do, gives you five questions to answer with all of it, and shows what Module 4 adds: charts that carry an answer to people who won't read a table.

What you can do now

After Module 3, you can:

  • pick the rows and columns you need with loc and iloc, and filter with conditions joined by &, | and ~;
  • sort and make top lists with sort_values, nlargest and nsmallest;
  • add new columns from math, np.where and pd.cut;
  • group and summarize with groupby and agg, as shares of a total, in pivot tables and crosstabs, and by week or month with resample;
  • stack files with pd.concat, join tables with merge and the join that fits, and catch a join that goes wrong with indicator=True, validate= and a row count;
  • reshape between wide and long with melt and pivot.

Here's one question that uses most of it, on Oakridge Middle School's records: which club's members improved most from fall to spring?

import pandas as pd

fall = pd.read_csv("fall_scores.csv")
spring = pd.read_csv("spring_scores.csv")
clubs = pd.read_csv("clubs.csv")
memberships = pd.read_csv("memberships.csv")

both = pd.concat([fall.assign(term="fall"), spring.assign(term="spring")], ignore_index=True)
year = both.melt(id_vars=["student_id", "term"], var_name="quiz", value_name="score")
change = year.pivot_table(index="student_id", columns="term", values="score")
change["gain"] = change["spring"] - change["fall"]

members = memberships.merge(clubs, on="club_id", validate="many_to_one")
gains = members.merge(change, on="student_id").groupby("club_name")["gain"].mean()
gains.round(1).sort_values(ascending=False)
club_name
Robotics       4.6
Chess Club     4.2
Coding Club    4.1
Art Club       3.9
Drama          3.8
Choir          3.6
Garden Club    2.9
Name: gain, dtype: float64

Stacked, melted, averaged per student with pivot_table (here averaging is the point), joined twice with a check, grouped and sorted: you can read every line. A student in two clubs counts in both, which is right for a question about clubs.

Try it: five questions from the principal

The best way to find out how much you can do is to answer real questions with no step-by-step instructions. Use the five files from Exercise: Report Card Merge, and answer these, one function each that returns a small table:

  • Which homeroom had the highest spring average? All nine homerooms, highest first.
  • Which quiz was hardest in each term? The lowest average quiz, fall and spring.
  • Which club's members scored highest in the spring? Every club with members, and how many members each has.
  • Which ten students improved most from fall to spring? With their first names and homerooms.
  • What share of each grade is in at least one club? As a percentage, one decimal place.

Write one function at a time, and check each answer on a few rows by hand before you trust it. Count the rows before and after every join: a membership that points at a club that doesn't exist, and a student in two clubs, are both waiting in the files. There's no test suite for this one: when your answers match what you work out by hand on a few students, they work.

Where tables hit a wall

The club table above is exact, and it still takes a moment to read. Two things limit every answer you can give so far:

  • A table is slow to read. Seven numbers are fine; twelve months of sales for six stores aren't. The pattern is in there, and the reader has to dig it out.
  • Most people who ask won't read one. A principal, a manager or a client wants to see the answer, not search for it in a grid.

What Module 4 unlocks

Module 4 draws your answers with matplotlib, the package most Python charts are drawn with:

  • Four charts: a line chart for change over time, a bar chart to compare groups (like the club table above), a scatter plot for two things that may move together, and a histogram for how values are spread.
  • Choosing the chart that fits the question, with a title that states the finding and labels that give the units.
  • Honest charts: bars that start at zero, side-by-side charts on the same scale, and colors that everyone can tell apart.
  • Several charts on one page, saved as an image for a report.

The tables you've learned to build are what every chart is drawn from, so nothing you did in this module is wasted: Module 4 makes it visible.

Check yourself

2 questions to try on the lesson page.

So what

Module 3 gave you answers; Module 4 makes them seen, by drawing each table as a chart someone can read in a few seconds.

Key terms

Related