Project #13: Creating Choropleth Maps from DataFrames with folium

In this DISCOVERY Data Science Project, you will do real data science in less than an hour and you will earn this project's card to your collection when you fully complete this project! 🎉

Data Visualization: Choropleth Maps

Geographical data visualizations are some of the most impactful forms of data visualization, as these visualizations easily allow the user to locate places familiar to themselves. One popular geographical visualization is a choropleth map -- a map that shades a geographical region to visually encode data about the region. As an example, population density maps and per-capita income maps are common choropleth maps.

Understanding how to use an external library, and read the documentation provided by a library developer, is a critical skill to always continuing to learn and expand your Data Science skills! In this project, you will learn about the folium Python library -- https://python-visualization.github.io/folium/ -- to create choropleth maps from a DataFrame!

Let's nerd out! :)

Part 1: Exploring the folium Python library

All widely-used Python libraries will have extensive examples and it is often easy to get started by viewing an example of the library's code by the authors of the library. The folium project provides a "Getting Started" guide that includes a section on choropleth maps: https://python-visualization.github.io/folium/latest/getting_started.html#Choropleth-maps

When I take a look at the code, which we provide below, I see that the provided code has four distinct sections:

  1. Data Import: The first six lines import data about the United States geography (state_geo) and then the unemployment into a DataFrame (state_data),
  2. Map Creation: The next line of code creates a blank map, and sets the initial latitude/longitude and zoom level to provide a view of the entire United States.
  3. Data Visualization: The next several lines of code is one giant call to folium.Choropleth, which configures the data visualization on the map.
  4. Rendering: The final two lines are used to display the map inside of your notebook.

Try it out, and see your first choropleth map! 🗺️

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 2: The Dataset: University of Illinois Demographics by State

The Division of Management Information (DMI) at The University of Illinois is a service unit that provides current and historical student enrollment information statistics. One of the many datasets that DMI provides is the "Final Statistical Abstract" that provides "a summary of student information on the tenth day of the term".

Only students taking at least one on-campus, credit-bearing class are included in these reports. The following categories of students are excluded: auditors (students taking only non-credit classes); students taking only extramural or off-campus classes; Medical Scholars taking no on-campus, non-MSP classes. (Note: Illini Center MBA students are included in these statistics.)

To help focus on building the choropleth maps, we have extracted the data shown in the teal subtable titled "By Permanent Home Address" and provided it for you as uiuc-dmi-students-by-permanent-home-address.csv.

Part 2.1: Create a DataFrame of Illinois Students by State

In the Python variable df, create a DataFrame containing the University of Illinois student demographic by state by loading the provided CSV file uiuc-dmi-students-by-permanent-home-address.csv:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

🔬 Checkpoint Tests 🔬

⚙️ Test Case: Part 2.1: Create a DataFrame of Illinois Students by State

Part 3: Making Our Own Choropleth Map

One of the best ways to begin to use a new library is to modify existing code to create your own visualization!

This project will walk you through modifying folium's "Getting started" guide that you saw provided in Part 1.

Part 3.1: Using the Illinois Dataset

In the "Getting Started" example, you used the dataset they provided about unemployment by state. In what we labeled as "Section 1: Data Import", this was imported into the variable state_data.

  • Modification #1: Instead of loading in the dataset they've provided, change state_data to be equal to df (the DataFrame containing Illinois data).

Incorporate Modification #1 below and run the code (you should get an error!, continue to Part 3.2 to fix it):

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 3.2: Fix the KeyError: 'Unemployment'

After making Modification #1, you will find yourself with a KeyError: 'Unemployment'. A KeyError indicates that Python attempted to look for data in a column that does not exist.

This hint leads us to look into the "Section 3: Data Visualization" for any mentions of the columns that are being used. Specifically, you will find:

  ...
  data=state_data,
  columns=["State", "Unemployment"],    # <-- Line about `columns`
  key_on="feature.id",
  ...

To fix this:

  • Look back at your DataFrame in Part 2. What column name contains the total number of students from Illinois in each state?
  • Modification #2: Replace the "Unemployment" column name with the column name that contains the total number of students at Illinois for each state.

Return to Part 3.1 and incorporate Modification #2 to your visualization.

Part 3.3: Fix the Mapping to Each States

After making Modification #2, the entire map is gray! :( When using folium, a gray location indicates there is no data available for that location -- but our DataFrame has the data!

The mapping between data in your DataFrame and the visual encoding of the state on the data visualization is done through the key_on field. The key_on field specifically tells folium how to map your data to the visualization.

In the "Quick Start" guide, the key_on field is properties.id (key_on="feature.id").

  • This indicates that the key that maps the data to the visual encoding is in the "features" dictionary, and then the "id" field of each entry.

Examining an entry in the "features" dictionary by running the code below, you can see that 'id': 'AL' represents the state of Alabama by the two-letter postal code:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 3.3 (Continued)

Unfortunately for us, the state column in our dataset contains the full state name (ex: "Alabama") instead of the two letter abbreviation.

There are two ways to solve this problem:

  1. We can modify the state_geo variable so that the id field contains the full state name, OR
  2. We can modify the key_on attribute to reference the field with the full state name (instead of id)

Looking at the JSON object above, you can find the full state name ("Alabama") NOT in 'id', but in 'properties' and then in 'names'. This makes the second way to solve this problem way easier than the first.

To do this,

  • Modification #3: Replace the key_on= attribute to be equal to feature.properties.name (the location of the full state name).

Return to Part 3.1 and incorporate Modification #3 to your visualization.

Part 3.4: Fix the Legend

After making Modification #3, we see the majority of students at the University of Illinois comes from Illinois -- but what about the other states? They all look the same!

Before we worry about that in Part 4, let's fix one other thing that looks odd before we continue: Why does our legend say "Unemployment Rate?"

  • Modification #4: Replace the legend_name= attribute with a descriptive title of the data you're visualizing.

Return to Part 3.1 and incorporate Modification #4 to your visualization.

⚙️ Test Case: Part 3: Making Our Own Choropleth Map

Part 4: Scaling Data with Data Spanning Multiple Orders of Magnitude

The visualization implies that everyone at Illinois comes from Illinois, and no where else? Let's explore the raw data and ensure there's not an error in the data.

Part 4.1: Sorting the Data by Enrollment

To be certain that there is not an error in the data, first sort the data based on the total number of students for each state:

(Not sure how to do this? The DISCOVERY guide "Sorting a DataFrame Using Pandas" can help refresh your knowledge on sorting a DataFrame)

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 4.2: Finding the State with the Largest Enrollment besides Illinois

Now, find the single row for the US state that has the second largest representation of students at The University of Illinois and store it in df_secondLargestState:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 4.2: Finding the State with the Largest Enrollment besides Illinois

Part 5: Scaling Data

The data in our dataset spans many orders of magnitude -- from just a single-digit number of students from Alaska to thousands of students from California and Illinois, it is incredibly hard to create a simple scale to visually represent numbers from as small as 2 all the way to over 20,000.

A common way to visually represent data with many orders of magnitude is to use a log scale. Here's some information about "base10 logarithm" or $log_{10}$:

  • The logarithm function is the inverse of the exponent function. (A base10 logarithm is the inverse of $10^{x}$.)

  • The base10 logarithm is easy to understand since it roughly counts the number of possible zeros in a number:

    • 1 has one digit (no zeros), and $log_{10}(1) = 0$
    • 10 has two digits (one possible zero), and $log_{10}(10) = 1$
    • 100 has three digits (two possible zeros), and $log_{10}(100) = 2$
    • 1000 has four digits (three possible zeros), and $log_{10}(1000) = 3$
    • 10,000 has four digits (four possible zeros), and $log_{10}(10000) = 4$
    • ...some people prefer to think of a number's base10 logarithm as the number of digits minus one, instead of the number of zeros.

The $log_{10}$ will approximate partial values:

  • 33 is between 10 and 100, so we expect the base10 logarithm to be between 1 and 2. It's 1.52.
  • 72 is also between 10 and 100, so we expect the base10 logarithm to be between 1 and 2, and closer to 2. It's 1.86.
  • The number of students at Illinois from Illinois, 30872, has five digits so its base10 logarithm should be between 4 and 5. It's 4.49.

Using the np.log10 function

The numpy library, commonly imported as np, provides a function that will transform a column of data into the log10 values. Create a new column, Total_log10, that uses the np.log10(...) function with the Total column:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 5: Scaling Data

Part 6: Creating a log10-scaled Choropleth Map

Finally, let's try remaking our choropleth map from before using our new column Total_log10! 🎉

  • First, return to Part 3.1 and copy and paste all your code into the cell below.
  • Then, complete the final two modifications:
    • Modify the code to use the column Total_log10 instead of Total.
    • Add "(log scale)" to the end of the legend_name.
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Analysis

The choropleth map you have just created shows an easy-to-read distribution of our entire dataset into a single data visualization. A reader can now quickly answer contextually relevant questions very quickly, including:

  • What states that border Illinois sends the fewest students to the University of Illinois?
  • Other than California, what other Western states do many University of Illinois students call home?
  • ...and many other similar questions!
⚙️ Test Case: Part 6: Creating a log10-scaled Choropleth Map

Validate and Complete This Project!

Congratulations on finishing this DISCOVERY Data Science Project! 🎉🎉

To validate your entire project, your entire code will run from top-to-bottom on this page and each test case will be validated one final time. If everything looks good, you'll earn the card for completing this project: