Finding Descriptive Statistics for Columns in a DataFrame
When we're presented with a new DataFrame, it can be a lot to deal with. A great way to familiarize ourselves with all the new information is to look at descriptive statistics (sometimes known as summary statistics) for all applicable variables.
The Movie Dataset
To demonstrate these functions, we'll use a DataFrame of five different movies, including information about their release date, how much money they made in US dollars, and a personal rating out of 10.
Pandas has a great selection of functions for calculating descriptive statistics. In most cases, we only want to use these on columns with float and int dtypes, not strings. For example, we can't calculate the average movie title!
We'll go into detail about how to use these later. But for now, here are the most common and useful functions.
.count()
Returns how many non-null values are in a column
In other words, how many rows actually have a value for this column?
.sum()
Returns the sum of all values in a column
.mean()
Returns the mean (average) of the values in a column
.median()
Returns the median of the values in a column
.var()
Returns the variance of the values in a column
.std()
Returns the standard deviation of the values in the column
aka the square root of the variance
NOTE: Pandas automatically calculates the sample standard deviation, not the population standard deviation. To calculate the population standard deviation, switch the degrees of freedom to 0 by typing the parameter ddof = 0 in the parenthesis.
Finding a Descriptive Statistic for a Single Column
The most practical use of descriptive statistics is to apply the functions to a single column. This allows us to store the result in a variable and save it for future analysis.
We do this by specifying the column in brackets before applying the function. Let's say we wanted to find the average personal rating of these 5 movies.
If we don't specify the column first, the function will return a list of that statistic for each column. But be careful: this could produce an error, since not every column in the DataFrame contains floats and ints!
```
domestic box office 125618201.0
worldwide box office 264118201.0
personal rating 8.0
international box office 138500000.0
dtype: float64
```
The Holy Grail: Finding All of the Basic Descriptive Statistic
All of the aforementioned functions find one descriptive statistic at a time. But if we want a simple way to see all this information at once, there's also a function for that: .describe(). There are a few different ways to use this function, which are detailed below.
Entire DataFrame
If we apply .describe() to an entire DataFrame, it returns a brand new DataFrame with rows that correspond to all essential descriptive statistics. By default, it will only include the columns with integer and float dtypes.
That one line of code returns something pretty powerful.
One Column
If you want to find all descriptive statistics for a single column at once, .describe() can do that, too. With only one column, the results are returned as a list.
```
count 5.000000e+00
mean 3.907512e+08
std 4.369559e+08
min 9.794721e+06
25% 3.953584e+07
50% 2.641182e+08
75% 5.851715e+08
max 1.055136e+09
Name: worldwide box office, dtype: float64
```
However, when we apply .describe() to a column of strings, we don't get an error. Instead, .describe() gives us a list of statistics that are more applicable to the string dtype.
```
count 5
unique 5
top The Truman Show
freq 1
Name: movie, dtype: object
```
Subsets of Columns
We can describe smaller subsets of columns, too. Just use double brackets to insert a list of the column names, with each name separated by a comma. The result will be a DataFrame.
However, this is only effective when both columns contain numbers (floats and/or ints) or when both columns contain strings. If you select columns with contrasting dtypes, it will only show the numerical descriptive statistics by default.