Finding Descriptive Statistics for Columns in a DataFrame

When we're presented with a new DataFrame, it can be a lot to deal with. A great way to familiarize ourselves with all the new information is to look at descriptive statistics (sometimes known as summary statistics) for all applicable variables.

The Movie Dataset

To demonstrate these functions, we'll use a DataFrame of five different movies, including information about their release date, how much money they made in US dollars, and a personal rating out of 10.

import pandas as pd

#Creates a DataFrame of "movie", "release date", "domestic gross", "worldwide gross", "personal rating", and "international box office" columns
df = pd.DataFrame([
  {"movie": "The Truman Show", "release date": "1996-06-05", "domestic box office": 125618201, "worldwide box office": 264118201, "personal rating": 10, "international box office": 138500000}, 
  {"movie": "Rogue One: A Star Wars Story", "release date": "2016-12-16", "domestic box office": 532177324, "worldwide box office": 1055135598, "personal rating": 9, "international box office": 522958274}, 
  {"movie": "Iron Man", "release date": "2008-05-02", "domestic box office": 318604126, "worldwide box office": 585171547, "personal rating": 7, "international box office": 266567421}, 
  {"movie": "Blade Runner", "release date": "1982-06-25", "domestic box office": 32656328, "worldwide box office": 39535837, "personal rating": 8, "international box office": 6879509}, 
  {"movie": "Breakfast at Tiffany's", "release date": "1961-10-05", "domestic box office": 9551904, "worldwide box office": 9794721, "personal rating": 7, "international box office": 242817}
])
df

List of Functions

Pandas has a great selection of functions for calculating descriptive statistics. In most cases, we only want to use these on columns with float and int dtypes, not strings. For example, we can't calculate the average movie title!

We'll go into detail about how to use these later. But for now, here are the most common and useful functions.

• .count()
  • Returns how many non-null values are in a column
  • In other words, how many rows actually have a value for this column?
• .sum()
  • Returns the sum of all values in a column
• .mean()
  • Returns the mean (average) of the values in a column
• .median()
  • Returns the median of the values in a column
• .var()
  • Returns the variance of the values in a column
• .std()
  • Returns the standard deviation of the values in the column
  • aka the square root of the variance
  • NOTE: Pandas automatically calculates the sample standard deviation, not the population standard deviation. To calculate the population standard deviation, switch the degrees of freedom to 0 by typing the parameter ddof = 0 in the parenthesis.
• .min() and .max()
  • Returns the minimum value in a column
  • See the guide for Finding Specific Values in a Column
• .quantile()
  • Returns the quantiles of the values in a column
  • Must input a parameter to specify the quantile
  • See the guide for Finding Quantiles of a Column in a DataFrame

Now, let's see these functions in action.

Finding a Descriptive Statistic for a Single Column

The most practical use of descriptive statistics is to apply the functions to a single column. This allows us to store the result in a variable and save it for future analysis.

We do this by specifying the column in brackets before applying the function. Let's say we wanted to find the average personal rating of these 5 movies.

df["personal rating"].mean()

8.2

Finding a Descriptive Statistic for All Columns

If we don't specify the column first, the function will return a list of that statistic for each column. But be careful: this could produce an error, since not every column in the DataFrame contains floats and ints!

df.median()

domestic box office         125618201.0
worldwide box office        264118201.0
personal rating                     8.0
international box office    138500000.0
dtype: float64

The Holy Grail: Finding All of the Basic Descriptive Statistic

All of the aforementioned functions find one descriptive statistic at a time. But if we want a simple way to see all this information at once, there's also a function for that: .describe(). There are a few different ways to use this function, which are detailed below.

Entire DataFrame

If we apply .describe() to an entire DataFrame, it returns a brand new DataFrame with rows that correspond to all essential descriptive statistics. By default, it will only include the columns with integer and float dtypes.

df.describe()

That one line of code returns something pretty powerful.

One Column

If you want to find all descriptive statistics for a single column at once, .describe() can do that, too. With only one column, the results are returned as a list.

df["worldwide box office"].describe()

count    5.000000e+00
mean     3.907512e+08
std      4.369559e+08
min      9.794721e+06
25%      3.953584e+07
50%      2.641182e+08
75%      5.851715e+08
max      1.055136e+09
Name: worldwide box office, dtype: float64

However, when we apply .describe() to a column of strings, we don't get an error. Instead, .describe() gives us a list of statistics that are more applicable to the string dtype.

df["movie"].describe()

count                   5
unique                  5
top       The Truman Show
freq                    1
Name: movie, dtype: object

Subsets of Columns

We can describe smaller subsets of columns, too. Just use double brackets to insert a list of the column names, with each name separated by a comma. The result will be a DataFrame.

df[["domestic box office", "worldwide box office"]].describe()

However, this is only effective when both columns contain numbers (floats and/or ints) or when both columns contain strings. If you select columns with contrasting dtypes, it will only show the numerical descriptive statistics by default.

df[["movie", "personal rating"]].describe()