pandas: Series and DataFrames
Meet pandas, the library analysts use to work with tables in Python: a Series is one column, a DataFrame is a whole table. Load a farmers market's sales from CSV, its vendor list from an Excel workbook and its reviews from nested JSON, then take a first look at each: its size, its columns and types, its summary numbers and what it counts.
Updated
Meet pandas
- Meet pandas: A Table You Can Program (Apprentice): pandas is the Python library for tables, imported as pd, and pd.DataFrame turns a list of dictionaries or a dictionary of lists into a DataFrame, so a question such as sales by store takes one line instead of a loop.
- Series: One Column of Values (Apprentice): A Series is pandas' column, one list of values with a label for each value (its index) and one dtype, and math such as price * 1.07 or sum, mean, max and round works on every value at once, with no loop.
- DataFrames: Rows, Columns and the Index (Apprentice): A DataFrame is pandas' table, and shape, columns and dtypes describe it, df['col'] picks one column as a Series, df[['a', 'b']] picks several as a smaller table, a new column can be made from two others in one line, and the index labels every row.
Loading Data
- Loading a CSV File (Apprentice): pd.read_csv loads a CSV file into a DataFrame, usecols and nrows load only the columns and rows you need, sep reads a file whose values are split by semicolons, and a FileNotFoundError means the path doesn't lead from the folder the notebook runs in to the file.
- Loading an Excel Workbook (Apprentice): pd.read_excel loads a sheet of an Excel workbook into a DataFrame with openpyxl doing the reading, sheet_name picks the sheet by its name, skiprows steps over title rows above the table, and sheet_name=None loads every sheet at once as a dictionary of DataFrames.
- Loading JSON, Flat and Nested (Apprentice): pd.read_json loads a JSON list of records into a DataFrame, lines=True reads JSON Lines with one record on each line, and pd.json_normalize with record_path and meta turns a nested list, such as each vendor's reviews, into one row for each item.
A First Look
- A First Look: head, info and describe (Apprentice): head, tail and sample show a few of a DataFrame's rows, info lists every column's type and how many values are there (the first hint of missing values), and describe sums up each number column in eight numbers, from count and mean to the quartiles, with include='all' adding the text columns.
- Counting What's There: value_counts and unique (Apprentice): value_counts counts how many rows hold each value of a column, most common first, normalize=True turns the counts into shares of the whole, unique lists the different values in a column and nunique counts them.
Exercise: Farmers Market First Look
- Exercise: Farmers Market First Look (Apprentice): Write five functions that load a farmers market's sales from a CSV file, its vendor list from the right sheet of an Excel workbook and its nested JSON reviews as one row each, then sum up a table and count its top categories, and let pytest check what each one returns.
Up Next: Cleaning Data
- Up Next: Cleaning Data (Apprentice): Look back at what Module 1 taught you, put it to work on a first look at a table of your own, and see what Module 2's cleaning tools will let you do with messy data.