Finding Descriptive Statistics for Columns in a DataFrame

When we're presented with a new DataFrame, it can be a lot to deal with. A great way to familiarize ourselves with all the new information is to look at descriptive statistics (sometimes known as summary statistics) for all applicable variables.

The Movie Dataset

To demonstrate these functions, we'll use a DataFrame of five different movies, including information about their release date, how much money they made in US dollars, and a personal rating out of 10.

import pandas as pd

#Creates a DataFrame of "movie", "release date", "domestic gross", "worldwide gross", "personal rating", and "international box office" columns
df = pd.DataFrame([
  {"movie": "The Truman Show", "release date": "1996-06-05", "domestic box office": 125618201, "worldwide box office": 264118201, "personal rating": 10, "international box office": 138500000}, 
  {"movie": "Rogue One: A Star Wars Story", "release date": "2016-12-16", "domestic box office": 532177324, "worldwide box office": 1055135598, "personal rating": 9, "international box office": 522958274}, 
  {"movie": "Iron Man", "release date": "2008-05-02", "domestic box office": 318604126, "worldwide box office": 585171547, "personal rating": 7, "international box office": 266567421}, 
  {"movie": "Blade Runner", "release date": "1982-06-25", "domestic box office": 32656328, "worldwide box office": 39535837, "personal rating": 8, "international box office": 6879509}, 
  {"movie": "Breakfast at Tiffany's", "release date": "1961-10-05", "domestic box office": 9551904, "worldwide box office": 9794721, "personal rating": 7, "international box office": 242817}
movierelease datedomestic box officeworldwide box officepersonal ratinginternational box office
0The Truman Show1996-06-0512561820126411820110138500000
1Rogue One: A Star Wars Story2016-12-1653217732410551355989522958274
2Iron Man2008-05-023186041265851715477266567421
3Blade Runner1982-06-25326563283953583786879509
4Breakfast at Tiffany's1961-10-05955190497947217242817
Creating the movie DataFrame

List of Functions

Pandas has a great selection of functions for calculating descriptive statistics. In most cases, we only want to use these on columns with float and int dtypes, not strings. For example, we can't calculate the average movie title!

We'll go into detail about how to use these later. But for now, here are the most common and useful functions.

  • .count()
    • Returns how many non-null values are in a column
    • In other words, how many rows actually have a value for this column?
  • .sum()
    • Returns the sum of all values in a column
  • .mean()
    • Returns the mean (average) of the values in a column
  • .median()
    • Returns the median of the values in a column
  • .var()
    • Returns the variance of the values in a column
  • .std()
    • Returns the standard deviation of the values in the column
    • aka the square root of the variance
    • NOTE: Pandas automatically calculates the sample standard deviation, not the population standard deviation. To calculate the population standard deviation, switch the degrees of freedom to 0 by typing the parameter ddof = 0 in the parenthesis.
  • .min() and .max()
  • .quantile()

Now, let's see these functions in action.

Finding a Descriptive Statistic for a Single Column

The most practical use of descriptive statistics is to apply the functions to a single column. This allows us to store the result in a variable and save it for future analysis.

We do this by specifying the column in brackets before applying the function. Let's say we wanted to find the average personal rating of these 5 movies.

df["personal rating"].mean()
Calculating average (mean) personal rating

Finding a Descriptive Statistic for All Columns

If we don't specify the column first, the function will return a list of that statistic for each column. But be careful: this could produce an error, since not every column in the DataFrame contains floats and ints!

domestic box office         125618201.0
worldwide box office        264118201.0
personal rating                     8.0
international box office    138500000.0
dtype: float64
Calculating median for all columns

The Holy Grail: Finding All of the Basic Descriptive Statistic

All of the aforementioned functions find one descriptive statistic at a time. But if we want a simple way to see all this information at once, there's also a function for that: .describe(). There are a few different ways to use this function, which are detailed below.

Entire DataFrame

If we apply .describe() to an entire DataFrame, it returns a brand new DataFrame with rows that correspond to all essential descriptive statistics. By default, it will only include the columns with integer and float dtypes.

domestic box officeworldwide box officepersonal ratinginternational box office
Using .describe() on an entire DataFrame

That one line of code returns something pretty powerful.

One Column

If you want to find all descriptive statistics for a single column at once, .describe() can do that, too. With only one column, the results are returned as a list.

df["worldwide box office"].describe()
count    5.000000e+00
mean     3.907512e+08
std      4.369559e+08
min      9.794721e+06
25%      3.953584e+07
50%      2.641182e+08
75%      5.851715e+08
max      1.055136e+09
Name: worldwide box office, dtype: float64
Applying .describe() to a numerical column

However, when we apply .describe() to a column of strings, we don't get an error. Instead, .describe() gives us a list of statistics that are more applicable to the string dtype.

count                   5
unique                  5
top       The Truman Show
freq                    1
Name: movie, dtype: object
Applying .describe() to a string column

Subsets of Columns

We can describe smaller subsets of columns, too. Just use double brackets to insert a list of the column names, with each name separated by a comma. The result will be a DataFrame.

df[["domestic box office", "worldwide box office"]].describe()
domestic box officeworldwide box office
.describe() with a subset of columns

However, this is only effective when both columns contain numbers (floats and/or ints) or when both columns contain strings. If you select columns with contrasting dtypes, it will only show the numerical descriptive statistics by default.

df[["movie", "personal rating"]].describe()
personal rating
.describe() with contrasting dtypes