Testing pandas Functions with Small Tables

A real week of data is too big to check by hand, so a test runs the function on a few rows made by hand, where the right answer is clear, and checks one number, a column or a whole table with assert_frame_equal, then the empty table, the missing value and the table it was given.

Journeyman · Python · Updated

By

After this lesson you can

  • Test a pandas function on a small DataFrame made by hand, checking a number with assert, a column with tolist() or a table with assert_frame_equal
  • Test an analysis function on an empty table and a missing value, and check that it leaves the table it was given unchanged
  • Test a function that writes a file by giving it a path in pytest's tmp_path folder and reading the file back

Key takeaways

  • A real file is too big to check by hand, so a test builds a table of three or four rows whose answer you can work out on paper first.
  • Check a number with assert, a column with tolist(), and a whole table with pd.testing.assert_frame_equal, which names the column that differs.
  • assert_frame_equal compares each column's type too, so 18 and 18.0 don't match; check_dtype=False compares the values only.
  • Test what real weeks bring, an empty table and a missing value, and check that the function leaves the table it was given unchanged.
  • A function that writes a file gets a path in pytest's tmp_path, a fresh empty folder for each test, and the test reads the file back.

Every Monday, the Greenway Food Truck's owner reads a report of last week's sales and decides how much to order and where to park. A number in that report is something a person acts on, so the functions behind it need proof that they're right. You'll write tests like these for any analysis someone relies on: a class project's results, a lab's figures, a weekly report at work.

A table small enough to check by hand

sales_report.py holds the report's functions, and every week's file has the same columns: sold_at, location, item, quantity, price and payment. Here is the total for the week of October 26, in a notebook:

import pandas as pd

from sales_report import total_revenue

sales = pd.read_csv("data/incoming/week-2026-10-26.csv")
print("rows:", len(sales))
print("revenue:", total_revenue(sales))
rows: 390
revenue: 4455.0

Is $4,455.00 right? Checking 390 rows with a calculator takes an afternoon, and next week you'd have to do it again. Comparing with last week's report proves nothing, because the same function made it. Running the function on the real file only shows that it gives an answer.

The way out is an example so small that you know the answer before the code runs. Here is the function:

def total_revenue(sales):
    """The week's revenue in dollars: quantity times price for each sale, added up."""
    return round(float((sales["quantity"] * sales["price"]).sum()), 2)

And a test, in tests/test_sales_report.py, with three sales made by hand: 2 Street Tacos at $9.00 is $18.00, 2 Horchatas at $3.50 is $7.00, and 3 Elotes at $4.50 is $13.50, so the total is $38.50.

import pandas as pd

from sales_report import total_revenue


def three_sales():
    return pd.DataFrame({
        "location": ["City Park", "City Park", "Stadium"],
        "item": ["Street Tacos", "Horchata", "Elote"],
        "quantity": [2, 2, 3],
        "price": [9.00, 3.50, 4.50],
    })


def test_total_revenue_of_three_sales_is_38_50():
    assert total_revenue(three_sales()) == 38.5

three_sales() builds a fresh table each time a test calls it, with only the columns the functions read. Run python -m pytest, as Your First Tests with pytest shows:

tests/test_sales_report.py .                                             [100%]

One dot, and the last line says 1 passed. The answer was worked out on paper first; a test that copies whatever the code returns proves nothing.

Checking a column, and a whole table

revenue_by_location(sales) returns a table: each location and its revenue, the most first. To check one column, turn it into a plain list with tolist():

def test_revenue_by_location_puts_city_park_first():
    result = revenue_by_location(three_sales())
    assert result["location"].tolist() == ["City Park", "Stadium"]

To check the whole table, build the table you expect and compare the two with pd.testing.assert_frame_equal, read "assert frame equal". pd.testing is pandas' toolbox for tests. It checks the column names, their order, every value, the row labels and each column's type:

def test_revenue_by_location_of_three_sales():
    expected = pd.DataFrame({
        "location": ["City Park", "Stadium"],
        "revenue": [25.0, 13.5],
    })
    pd.testing.assert_frame_equal(revenue_by_location(three_sales()), expected)

When the tables differ, it says where. On a quiet day, the Stadium takes $18 and City Park $9, and a test that writes the expected revenues as [18, 9] fails:

E       AssertionError: Attributes of DataFrame.iloc[:, 1] (column name="revenue") are different
E
E       Attribute "dtype" are different
E       [left]:  float64
E       [right]: int64

The values match, and the types don't: 18 makes a column of whole numbers, int64, while the function's revenue is float64. Write [18.0, 9.0], or add check_dtype=False when the type doesn't matter to the report and only the values should be compared.

The empty week and the missing price

The three sales show an ordinary week. Real weeks also bring the cases a sample hides, called edge cases:

def test_total_revenue_of_an_empty_week_is_0():
    empty = three_sales().iloc[0:0]
    assert total_revenue(empty) == 0.0


def test_total_revenue_leaves_out_a_sale_with_no_price():
    sales = three_sales()
    sales.loc[1, "price"] = None
    assert total_revenue(sales) == 31.5

iloc[0:0] keeps the columns and no rows: a week the truck was in the shop. The second test removes the Horchatas' price, and writing the test settles what the report does with it: that sale is left out, so $18.00 plus $13.50 is $31.50. A missing value can't be checked with ==, because NaN (read "not a number") isn't equal to anything, itself included. Use pd.isna(value) for one value; assert_frame_equal counts NaN in the same place in both tables as a match.

Leave the table you were given alone

Someone tidies revenue_by_location to save a line, and adds the column straight to the table it was given:

def revenue_by_location(sales):
    sales["revenue"] = sales["quantity"] * sales["price"]
    table = sales.groupby("location", as_index=False)["revenue"].sum()
    return table.sort_values("revenue", ascending=False, ignore_index=True)

Its answer is still right, but the caller's sales now has an extra column, and the next function that reads it sees a surprise. A test keeps a copy, calls the function, and compares:

def test_revenue_by_location_leaves_its_table_unchanged():
    sales = three_sales()
    before = sales.copy()
    revenue_by_location(sales)
    pd.testing.assert_frame_equal(sales, before)
E       AssertionError: DataFrame are different
E
E       DataFrame shape mismatch
E       [left]:  (3, 5)
E       [right]: (3, 4)

Three rows, and five columns where there were four. The fix is in the function: start with result = sales.copy(), or add the column with assign, which hands back a new table and leaves sales as it was.

A function that writes a file: tmp_path

save_revenue_table(sales, path) writes the location table to a CSV file. A test shouldn't write into your project folder, where it could overwrite a real report. pytest has tmp_path: name it as the test's parameter, and pytest hands the test a new, empty folder of its own:

def test_save_revenue_table_writes_two_locations(tmp_path):
    path = tmp_path / "revenue.csv"
    save_revenue_table(three_sales(), path)
    assert path.read_text(encoding="utf-8") == "location,revenue\nCity Park,25.0\nStadium,13.5\n"

tmp_path / "revenue.csv" (read "tmp path slash revenue dot c s v") is a file's path in that folder. read_text reads the whole file back as one string, with \n at the end of each line, so one assert checks every character the report will contain.

Check yourself

2 questions to try on the lesson page.

So what

A table of three rows you worked out on paper turns "the report looks fine" into proof, and the empty week, the missing price and the untouched input are where reports quietly go wrong.

Key terms

Related