Project #4: United States Congress

In this DISCOVERY Data Science Project, you will do real data science in less than an hour and you will earn this project's card to your collection when you fully complete this project! 🎉

Data Source: @unitedstates' Current Members of Congress

The @unitedstates GitHub organization described itself as a "shared commons of data and tools for the United States. Made by the public, used by the public." Their most popular work is the congress-legislators collection of data, which contains every member "of the United States Congress (1789-Present), congressional committees (1973-Present), committee membership (current only), and presidents and vice presidents of the United States in YAML, JSON, and CSV format."

The dataset of currently serving Members of Congress, known as the legislators-current dataset, is in CSV format at https://unitedstates.github.io/congress-legislators/legislators-current.csv.

In this MicroProject, you'll both explore working with the date/time "data type" or "dtype" in pandas and create several exploratory data visualizations! Let's nerd out! 🎉

Background Knowledge

To finish this MicroProject, we assume you already know:

Let's get started! :)

Part 1: Importing the legislators-current dataset

Create a DataFrame in the Python variable named df that contains the legislators-current data located at the following URL:

https://unitedstates.github.io/congress-legislators/legislators-current.csv
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Understanding the Truncation of Rows and Columns in a DataFrame's Output

To save space, pandas will default to showing a truncated list of rows and columns by using the ... to skip over columns when there are too many columns.

The pandas library provides over a hundred different configurable options for DataFrames that you, a data scientist, can control using the pd.get_option("option.name") and pd.set_option("option.name", value) functions.

The option that controls the maximum number of columns that are visible before Python begins to truncate your output is named display.max_columns.

Run the code below to see what the current, default value for display.max_columns:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Updating the display.max_columns Value

After finding the default option, change the number of max_columns displayed in this notebook to 50 columns (instead of the default 20 columns) by using the set_option command explained in the previous section:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

In addition to setting the maximum columns to 50, you can feel free to change:

Display the DataFrame

With the updated display.max_columns value, display the DataFrame again and note that all columns are visible by scrolling right to view the entire DataFrame:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 1: Importing the legislators-current dataset

Part 2: Working with Dates in pandas via Congressional Birthdays

One of the trickiest things to work with in data science are dates. One of the first challenges is that dates can be represented in many, many ways:

  • In the United States, December 13, 1989 is commonly written as 12/13/1989, 12/13/89, or 12-13-89. Each of these contain the month, then day, and then year.
  • In Europe, the same December 13, 1989 is commonly written as 13/12/1989, 13/12/89 or 13-12-1989. Instead of listing the month first, the day is listed first, then month, and then year.
  • In many other places in the world, the same December 13, 1989 is written as 1989/12/13 or 1989-12-13. This format lists the year first, then month, and then day.

In 1988, a group of computer scientists met together and published an international standard for the "worldwide exchange and communication of date and time-related data". This standard is referred to as ISO 8601 and is the universal format to easily work with dates as data.

The ISO 8601 standard says that dates must be encoded in only one of two formats:

  • Option #1: YYYY-MM-DD
  • Option #2: YYYYMMDD (same order, but without the -)

Let's look at the birthday column in the dataset and see if this standard is followed in our dataset:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 2.1: Converting an ISO formatted date into a DateTime data type

At the bottom of the output above, pandas will report the datatype -- or dtype -- of the data is an object:

Example:

...
Name: birthday, Length: 539, dtype: object

In general, there are three major data types commonly used in a DataFrame:

  • Numbers, commonly with a dtype of int64 (whole numbers) or float64 (numbers with decimals),
  • Strings, commonly with a dtype of object, AND
  • Date/Time, commonly with a dtype of datetime64[ns]

When the dtype of a column is a number, we can perform numerical operations on the data (ex: sum, mean, etc). When the dtype of the data is a string, we can perform string operations on the data (ex: contains, etc). Finally, when the dtype is a Date/Time, we can perform datetime operations on it (ex: month, day, etc).

To convert data from a string to a date, the pandas library provides the function:

pd.to_datetime( df["column_with_iso_date_strings"] )

Create a new column in your DataFrame df called birthday_dt that stores the birthday date as a DateTime:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

The data should look the same, but the dtype will be updated to datetime64[ns] instead of object.

⚙️ Test Case: Part 2.1: Converting an ISO formatted date into a DateTime data type

Part 2.2: Using DateTime Operations

When your data in a DataFrame is the dtype of datetime64[ns], we are able to access date-specific functions for the column by using the .dt functions. Several common functions we can now use on the birthday_dt column include:

df["birthday_dt"].dt.month   # Returns the numeric month; ex: 1 for January through 12 for December
df["birthday_dt"].dt.day     # Returns the numeric day of the month; ex: 13 for Dec. 13, 1989 (`1989-12-13`)
df["birthday_dt"].dt.year    # Returns the numeric year; ex: 1989

df["birthday_dt"].dt.dayofweek   # Returns a numeric day of the week; 0 for Monday through 6 for Sunday

df["birthday_dt"].dt.hour    # Returns the numeric hour based on a 24-hour clock
df["birthday_dt"].dt.minute  # Returns the numeric minute
df["birthday_dt"].dt.second  # Returns the numeric second

# ..and several others...

Finding Birthdays

When every seat in Congress is filled, there are a total of 541 members (100 senators, 435 voting representatives, and 6 non-voting representatives from non-voting districts and territories). Since there are at most 366 days in a leap year, is it always someone's birthday in Congress?

To nerd out with some exploratory data analysis to answer this question, we will graph every day of the year as a scatter plot with x-axis containing the day of birth of each member of congress and the y-axis containing the month of birth of each member of Congress.

To create this scatter plot, we need both pieces of data in individual columns in our DataFrame:

  1. Add a new column to df called dayOfBirth that contains the day of birth for the member of Congress
  2. Add a new column to df called monthOfBirth that contains the month of birth for the member of Congress
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 2.2: Using DateTime Operations

Part 2.3: Create a Scatter Plot

Using df.plot.scatter, create a scatter plot with the x-axis data being the day of birth of each member of Congress and the y-axis data being the month of birth of each member of Congress:

(If you need a refresher, the DISCOVERY page "Basic Data Visualization in Python" steps you through the code to build a scatter plot!)

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Analysis: A Day Without a Birthday Celebration?

The scatter plot of days/months of birth is an example of an exploratory data visualization. This visualization alone does not provide much context about the data itself and may be difficult for others to immediately understand, but it helps a data scientist understand the data to ask additional questions.

Using your exploratory data visualization, write a date in ISO format that no member of Congress will have a birthday (using the current year as the year) and store that date in the Python variable noCongressBirthdaysCelebrated:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 2.3: A Day Without a Birthday Celebration

In addition to providing the x and y values to the df.plot.scatter function, you can provide an alpha value to your scatter plot to make the data points partially transparent to see where there are multiple data points at a single location.

Create another scatter plot of the same data, adding to the function the option alpha = 0.2 to make each data point 80% transparent. (Feel free to play with the alpha value to make it exactly how you like it!)

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Most days of the year have a single member of Congress with a birthday on that day. In your exploratory data visualization with alpha transparency, days with only a single birthday will visually show as a very pale dot.

When two members of Congress have a birthday on the same day, the two partially transparent dots are drawn on top of each other making the data point darker.

Write a date in ISO format for a date that at least four members of congress will celebrate their birthday together (using the current year as the year) and store it in the variable popularCongressBirthday:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 2.4: A Popular Day for Birthdays

Part 3: The House of Representatives and The Senate

There's one final question about birthdays -- are senators or representatives older?

Part 3.1: Calculating the Age of Each Member of Congress

One unique features of the datetime dtype is that, just like numeric data, you can add and subtract dates! Unlike numeric data, however, the difference between two dates is a span of time called a "time delta". In the cell below, we calculate the "time delta" from the current moment in time (datetime.datetime.now()) to the time that each member of Congress was born:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

By default, the largest unit of time that Python natively understands is "1 day" (exactly 86,400 seconds). A more meaningful measure for us, as humans, is to represent someone's age in terms of years instead of days.

In Python, when you have a "time delta", we can divide by another time delta to get a numeric value.

In the following cell, we divide the previous calculation by 365.25 days (an very close approximation for the number of days in a year, accounting for leap years):

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Assign the calculation we did above to a new column in the DataFrame named age to store each member's age:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 3.1: Calculating the Age of Each Member of Congress

Part 3.2: Senators vs. Representatives

In this dataset, each row is one person who is a member of the United States Congress. The United States Congress is bicameral, being made up of both the House of Representatives and the Senate. In the dataset:

  • Representatives in the House of Representatives represent a district, which is encoded in the district column. Their type column contains the string "rep".
  • Senators are elected for six years, with one-third of the senators up for election every two years. The senate_class denotes which election class a senator is in and when they are up for election. A senator's type column contains the string "sen".

In the following two cells, create two new DataFrames, df_house and df_senate, that contain members from only the House of Representatives (df_house) and the Senate (df_senate):

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

🔬 MicroProject Checkpoint Tests 🔬

⚙️ Test Case: Part 3.2: Senators vs. Representatives

Part 3.3: Box Plots

Finally, let's visually answer the question: are senators or representatives older?

To create a box plot to view the distribution of ages, we can use df.plot.box(...). The three options we absolutely need to use to create a box plot include:

  • by="type", to inform pandas to create a different box for each "type" of member of Congress (senators and representatives)
  • column="age", to inform pandas to use the age column when computing the Q1, Q2, Q3, and outliers
  • title="Current Age of Members of Congress, by Chamber", to provide a descriptive title

Additionally, you may want to optionally add any, all, or none of the following options:

  • vert=False, if you prefer a horizontal box plot
  • grid=True, if you prefer gridlines on your box plot
  • figsize=(6, 4), to resize the shape of your plot (adjusting 6 and 4 as necessary)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Box Plot Analysis

The box plot shows the distribution of every member of Congress is a single visualization. From this, you can answer a lot of questions:

  • Is the average median age of a representatives or senators older?
  • Is the youngest representative older than the youngest senator? How old is that person?
  • Is the oldest representative younger than the oldest senator? How old is that person?
  • What's the oldest age you can be and still be younger than the half of all senators in the Senate?
  • ... and many other questions, all from one visualization! :)

Validate and Complete This Project!

Congratulations on finishing this DISCOVERY Data Science Project! 🎉🎉

To validate your entire project, your entire code will run from top-to-bottom on this page and each test case will be validated one final time. If everything looks good, you'll earn the card for completing this project: