Bechdel Test

In this DISCOVERY Data Science Project, you will do real data science in less than an hour and you will earn this project's card to your collection when you fully complete this project! 🎉

Data Source: Bechdel Test

The Bechdel Test is a simple way of measuring the representation of women in a film or other work of fiction. A work earns points for each of three criteria:

  1. The work must have at least two women in it,

  2. who talk to each other,

  3. about something other than a man.

The test was popularized by Alison Bechdel's 1985 comic strip called "The Rule" and has grown in popularity as a simple way to measure the representation of women in works of fiction. The website BechdelTest.com provides both a searchable database and a publicly-available API that includes over 10,000 films and their Bechdel Test ratings, allowing users to explore and analyze patterns in gender representation in cinema.

In this MicroProject, you will explore using a JSON-based API, grouping data using a pivot table (df.pivot_table(...)), and creating a stacked area chart to create a data visualization that shows the change in Bechdel Test ratings over time. Let's nerd out! 🎉

Background Knowledge

To finish this MicroProject, we assume you already know:

Let's get started! :)

Part 1: Importing the Bechdel Test Dataset

BechdelTest.com had previously provided an API as an easy-to-access data source that contains the Bechdel Test ratings for thousands of movies. The API provided a list of over 10,000 movies in a commonly used format called "JSON" that can be read in using pd.read_json(...).

In 2025, their API was removed. However, since their content is licensed as as Attribution-NonCommercial 3.0 Unported, we are able to provide a local copy of the data they had previously made available. A replica of their API is provided here for you here: https://waf-server-01.cs.illinois.edu/static/getAllMovies.json.

Using pd.read_json, create a DataFrame df_bechdel to store all the movies and their Bechdel Test ratings:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Cleaning the Dataset

The DataFrame you collected comes from BechdelTest.com, containing their collection of data. As with many real data sources, there may be typos, errors, or incorrect values in the data.

For example, the 2000 film "Grandma Got Run Over by a Reindeer" was incorrectly listed as being released in the year 200 instead of 2000 -- that's a big difference for a simple typo!! (We reported this error and it has been fixed, but there may be others.)

Additionally, many of the early films in the dataset, including 1874's "Passage de Venus", are technical films at the very beginning of "motion pictures". Passage de Venus, for example, is a six-second film showing Venus pass between the Earth and the Sun and not a work of fiction that would be traditionally analyzed using the Bechdel Test.

To limit our data to modern film and remove any erroneous data, create a new DataFrame df_clean that contains the data from df_bechdel where:

  1. All movies made before 1900 are removed. (The data in df_clean should only be movies made in 1900 or later.)
  2. All movies made in the future are removed. (It's impossible to have a movie from 2035, since that's the future.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 1: Importing the Bechdel Test Dataset

Part 2: Creating a Pivot Table for Analysis

With over 10,000 movies that were released over 100 years, creating a summary of our data may be helpful for analysis! Specifically, it would be very useful to get a breakdown of the number of movies for each Bechdel Test rating from every year.

In the simplest terms, we would create a dataset with the following overall structure:

  • In 1900, there were 11 movies all with a 0 rating (their score on the Beckdel test).
  • In 1910, there were 2 movies with a 0 rating and 1 movie with a 3 ratiing.
  • In 1989, there were 81 movies; 12 with a 0, 16 with a 1, 11 with a 2, and 41 with a 3 rating.

Translating this design idea to a table, a simple design for the table would have:

  • Each row of our table to be a year,
  • Each column of our table to be a rating (0, 1, 2, and 3), AND
  • The values inside of our table to count the number of times a movie was made in that year with that specific rating.

That means the table would look something like:

Year0123
190011000
...
19102001
...
198912161141
...

Part 2.1: Starting a Pivot Table with index and aggfunc:

The groupby and pivot table operations are two different ways to summarize a DataFrame using pandas. In general, the groupby operation resembles how databases work and the pivot_table operation resembles how spreadsheets works, but both methods can create almost any summary of data!

To make a pivot table, you specify the structure of the table you want pandas to generate along with how you want multiple rows to be combined (or aggregated) together.

The simplest of pivot tables require us to specify only two things:

  1. What column name in our original DataFrame do we want to use for the row labels for our pivot table? (This is called the index in a pivot table.)
  2. How do we want to combine multiple rows together? (This is called the aggfunc in a pivot table.)

For example, if we want to count the number of movies with unique values for rating, our initial pivot table can be created using the following code:

df_clean.pivot_table(index="rating", aggfunc="count")
#                    ^^^^^^^^^^^^^^ 
#          Each *row* (index) in the pivot table will aggregate rows with the same `rating` together.
#                                   ^^^^^^^^^^^^^^^
#                      The method of aggregation (aggfunc) will be to count the number of rows aggregated together.

Create your first pivot table, and store it in a Python variable df, that summarizes your cleaned DataFrame, df_clean, by counting the number of films receiving each of the four ratings:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 2.1: Starting a Pivot Table

Part 2.2: Creating a Pivot Table for Years

Using what you know, create another pivot table. However, in this new pivot table summarize the count of how many films are in the dataset for each year instead each row being a rating. Just as before, store the pivot table you created in the Python variable df:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 2.2: Creating a Pivot Table for Years

Part 2.3: Columns in a Pivot Table

The pivot table you just created counts how many rows have data for each year and rating combination.

For example, looking at your pivot table above, if the year 1900 reports 11 for the column rating, our pivot table informs us that there were 11 movies in the dataset that contains a data in the column rating when the year was 1900. However, we want to know the breakdown of Bechdel Test scores for those 11 movies.

The pivot table you created has two function parameters so far: index and aggfunc. The third function parameter we will include is columns. The value set for the columns parameter will specify what column from the original DataFrame should be presented in each column. This columns value is combined together with the rows value (index) we already have specified.

Now, create a pivot table with three function parameters:

  • Each row (index parameter) in our pivot table is one year from the original DataFrame,
  • Each column (columns parameter) in our pivot table is one rating from the original DataFrame, and
  • The values are aggregated together by the "count" function (aggfunc parameter).

Call this pivot table df and we'll check to make sure it looks good:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 2.3: Columns in a Pivot Table

Part 2.4: Adding the values Parameter to your Pivot Table

In your latest pivot table, you have a summary for the number of movies rated 0, 1, 2, and 3 for every year of data in the dataset, but it is repeated for every id record, every imdbid, and every title. It's a lot of extra columns! 😢

The values parameter allows us to specify the column from the original DataFrame whose values we want to aggregate. Currently, we have three columns of data we aren't doing anything specific with except aggregating them together: id, imdbid, and title. We do not need all three.

Since all three columns are identical, it's completely up to you to choose any one of the three columns to keep as the values that we're going to analyze.

  • Do you want to keep the count of how many movies have data for the id column in your original dataset? If so, use values="id".
  • Do you want to keep the count of how many movies have data for the imdbid column in your original dataset? If so, use values="imdbid".
  • Do you want to keep the count of how many movies have data for the title column in your original dataset? If so, use values="title".

Extend your pivot table to now include all four parameters (index, aggfunc, columns and values) and store your improved pivot table in the variable df:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 2.4: Adding the values Parameter to your Pivot Table

Part 2.5: Specify the fill_value

Finally, you may notice that there are no movies with a 3 in any of the early 1900s. Since there is no data for that row/column combination, pandas leaves the value blank and reports a NaN or "Not a Number".

The fill_value parameter allows us to give a default value when there is no data. Since we know no data indicates that there were zero movies that that rating in our dataset, setting fill_value=0 fills all the missing data with zeroes.

Extend your pivot table to have five parameters, ensuring that empty values get replaced with a 0:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Step 2.5: Specify the fill_value

Part 3: Visual Analysis of the Data

You have an extremely detailed summary of the entire Bechdel Test dataset -- over 10,000 movies across more than 100 years! A simple exploratory data visualization would help us understand all of this data and a line chart is a great way to get started!

Create a simple line chart by using df.plot.line(), no parameters needed:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 3.1: Transforming Counts into Proportions

In the graph above, you can see that the total number of movies in the dataset has increased dramatically from the 1900s until today. However, except for seeing that the number of movies increased, it's impossible to tell if movies are scoring, on average, higher or lower today than they did in the early 1900s.

A stacked area chart is a useful visualization for showing how the proportion of different categories changes over time. A stacked area chart is similar to a stacked bar chart where each data value will be stacked on top of the previous data. The most common application of a stacked area chart is graphing different proportions of data.

To create a graph of the proportion of movies at each ranking, add the following four additional columns to your DataFrame df:

  • "%0", that contains the percentage of movies with a 0 ranking in a given year,
  • "%1", that contains the percentage of movies with a 1 ranking in a given year,
  • "%2", that contains the percentage of movies with a 2 ranking in a given year,
  • "%3", that contains the percentage of movies with a 3 ranking in a given year,
  • Hint: You may find adding a "Total" column helpful to make your calculations easier.

Note that the 0,1,2,3 column names are integers, not strings.

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 3.1: Transforming Counts into Proportions

Part 3.2: Finding the Percentage of Movies Per Year with Each Rating:

To visualize only the percentage columns, create a new df_pct that contains only the %0, %1, %2, and %3 columns:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 3.2: Finding the Percentage of Movies Per Year with Each Rating

Part 3.3: Visualizing the Percentage of Movies / Year with Each Rating:

Since df_pct has rows that always add up to 1, this data is PERFECT for a stacked area graph. In the cell below, df.plot.area() is used to create a stacked area visualization to view the growth of the percentage of each movie's rating. Run the cell to view the graph!

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Area Chart Analysis

The stacked area chart shows the proportion of movies with a score of 0, 1, 2, and 3. Here are some questions to think about:

  • Has the proportion of movies rated as 3 generally increased over the past century?
  • Were there periods of time where the general trend reversed?

And feel free to nerd out with this visualization:

  • You can change the colors used in this visualization, and any other visualization, by changing the colormap parameter. The page Choosing Colormaps in Matplotlib lists all of the available colormaps included with pandas!
  • Feel free to add grid=True, which will help with data analysis as it is well known that it's tricky for humans to imagine a straight line when the background is slanted. (It's really crazy how different the data looks with grid lines turned on!)
  • Finally, you're free to change this graph an any other way you want! It's your graph to transform! :)

Validate and Complete This Project!

Congratulations on finishing this DISCOVERY Data Science Project! 🎉🎉

To validate your entire project, your entire code will run from top-to-bottom on this page and each test case will be validated one final time. If everything looks good, you'll earn the card for completing this project: