Meet pandas: A Table You Can Program
pandas is the Python library for tables, imported as pd, and pd.DataFrame turns a list of dictionaries or a dictionary of lists into a DataFrame, so a question such as sales by store takes one line instead of a loop.
Apprentice · Python · Updated
After this lesson you can
- Build a DataFrame with pd.DataFrame from a list of dictionaries or a dictionary of lists, after import pandas as pd
Key takeaways
- pandas is a library for working with tables in Python, like a spreadsheet you control with code. It's imported as pd, read "p d", the short name almost everyone uses.
- pd.DataFrame turns a list of dictionaries (one for each row) or a dictionary of lists (one for each column) into a DataFrame, pandas' table.
- pandas works on a whole column in one step, so a total for each store takes one line instead of a loop and a dictionary.
- A missing price stops a hand-written loop with an error, but pandas leaves it out quietly, so you need to know where the gaps are.
Whenever you have rows of data and a question about them, which store sold the most or which product sells out first, you need a table that can do the math for you. Writing a loop for every question works for one question and gets tiring by the third. Analysts, scientists and students in data classes use pandas for this, and every lesson from here on does too.
The loop you'd write by hand
Here is a morning of Bean Street Coffee's orders, from the chain's three stores, as a list of dictionaries: one dictionary for each order. Totaling the sales by store by hand takes a dictionary of running totals and a loop:
orders = [
{"store": "Downtown", "drink": "latte", "quantity": 2, "unit_price": 4.45},
{"store": "Campus", "drink": "drip coffee", "quantity": 1, "unit_price": 2.95},
{"store": "Harbor", "drink": "cold brew", "quantity": 1, "unit_price": 4.15},
{"store": "Downtown", "drink": "mocha", "quantity": 1, "unit_price": 4.75},
{"store": "Campus", "drink": "mocha", "quantity": 3, "unit_price": 4.75},
{"store": "Harbor", "drink": "chai", "quantity": 2, "unit_price": 4.35},
]
totals = {}
for order in orders:
sale = order["quantity"] * order["unit_price"]
totals[order["store"]] = totals.get(order["store"], 0) + sale
print(totals)
{'Downtown': 13.65, 'Campus': 17.2, 'Harbor': 12.85}
It works, and it's one question. Sales by drink is another loop, the average order another, and one order with no price stops the loop with an error. Answering a Question by Hand, and Where It Breaks walked through each of those.
pandas, imported as pd
pandas is a library: code other people wrote and share, which your program imports and uses. It's a third-party library, so it doesn't come with Python; it's installed into your virtual environment with pip, as Setup: pandas and Notebook Support showed. Its job is tables.
You'll see it imported the same way everywhere, import pandas as pd, read "import pandas as p d". The as pd gives the library a short name, so you write pd. in front of what it offers. A list of dictionaries becomes a table with pd.DataFrame:
import pandas as pd
table = pd.DataFrame(orders)
table
store drink quantity unit_price
0 Downtown latte 2 4.45
1 Campus drip coffee 1 2.95
2 Harbor cold brew 1 4.15
3 Downtown mocha 1 4.75
4 Campus mocha 3 4.75
5 Harbor chai 2 4.35
A DataFrame is pandas' table: each dictionary became a row, each key a named column, and pandas numbered the rows from 0 down the left. VS Code draws the table as a grid; this lesson shows it as pandas prints it as text.
The same question in one line
A DataFrame works on a whole column at a time. Multiply two columns and every row is multiplied, with no loop. Then one line answers the store question:
table["total"] = table["quantity"] * table["unit_price"]
table.groupby("store")["total"].sum()
store
Campus 17.20
Downtown 13.65
Harbor 12.85
Name: total, dtype: float64
The same three totals as the loop. Read the second line as "group the rows by store, then add up each group's total". Grouping with groupby teaches it in full; for now, notice how little you wrote. Other questions are one line each too:
print("orders:", len(table))
print("cups:", table["quantity"].sum())
print("biggest order: quot;, table["total"].max())
orders: 6
cups: 10
biggest order: $ 14.25
A table from columns
Data doesn't always arrive as rows. When you have one list for each column, pass a dictionary of lists: each key is a column's name, and its list holds that column's values, top to bottom.
menu = pd.DataFrame({
"drink": ["drip coffee", "latte", "cold brew", "mocha"],
"price": [2.95, 4.45, 4.15, 4.75],
})
menu
drink price
0 drip coffee 2.95
1 latte 4.45
2 cold brew 4.15
3 mocha 4.75
Every list must be the same length, because each row needs a value in every column. Give the menu three drinks and two prices and pandas stops with ValueError: All arrays must be of the same length.
When a price is missing
Here's the order that broke the loop: a tea typed in with no price, None. In plain Python, 1 * None stops with a TypeError. pandas carries on:
orders.append({"store": "Harbor", "drink": "tea", "quantity": 1, "unit_price": None})
table = pd.DataFrame(orders)
table["total"] = table["quantity"] * table["unit_price"]
table.groupby("store")["total"].sum()
store
Campus 17.20
Downtown 13.65
Harbor 12.85
Name: total, dtype: float64
Harbor's total didn't change: pandas left the tea out without a word. That's handy, and it's also why you'll always look for gaps before you trust a number (Finding Missing Values: NaN and isna shows how).
pandas isn't the only table library. Polars is a newer one built on the same idea, and fast on very large files. pandas is the one most classes, jobs and answers online use, so it's the one this course teaches.
Check yourself
2 questions to try on the lesson page.
So what
A DataFrame turns rows of data into a table that answers questions in a line, so your time goes into the questions instead of the loops.