Project #11: Learning Handwritten Digits with AI

In this DISCOVERY Data Science Project, you will do real data science in less than an hour and you will earn this project's card to your collection when you fully complete this project! 🎉

Data Source: MNIST Database of Handwritten Digits

The National Institute of Standards and Technology (NIST), an institute run by the government of the United States, provides a collection of "Special Databases" that "contain digital data objects such as images, software, and videos" and can be found at https://www.nist.gov/srd/related-data-products-and-links/special-databases-and-special-software. Two specific early "Special Databases" have become some of the most famous datasets in machine learning and artificial intelligence:

  • SD-3 contains handwriting samples from 2,100 United States census workers during the 1990 census.
  • SD-7 contains handwriting samples from 500 high school students in Maryland.

As part of the 1992 paper "Comparison of classifier methods: a case study in handwritten digit recognition" by Léon Bottou et al., the authors describe the creation of the "Modified NIST" (MNIST) dataset that:

  • Included samples from 500 unique writers (250 from each NIST dataset), and
  • Normalized all images to 28×28 pixel gray scale images

This MNIST dataset is now one of the most famous datasets in machine learning and artificial intelligence. This dataset contains a collection of 70,000 images of hand-written digits and each are labeled with the number (ex: a picture of a "0" and labeled as 0). In this project, you'll explore the MNIST dataset and learn about various clustering algorithms to help a computer learn to recognize handwritten digits. Let's nerd out! 🎉

Background Knowledge

To complete this project, we assume you already know:

  • All topics covered in DISCOVERY Module 1: Basics of Data Science with Python (review the module here)
  • Machine Learning Models in Python with sk-learn and Clustering sections of DISCOVERY Module 5: Towards Machine Learning (review the module here)

Let's get started! :)

Part 1: Loading the MNIST Dataset

As a classic dataset, the MNIST dataset is available online in many places and many formats. To focus on machine learning and building on the skills you already have, we have already transformed the ARFF file found on OpenML into a CSV file and provided this CSV file as part of this project.

Since this is a large dataset (over 120 MiB uncompressed), the file is provided as a compressed file -- that won't cause any problems as pandas can read compressed files that contain a CSV inside using pd.read_csv.

Every value in this dataset is a whole number between 0 and 255, so we can tell pandas to store the data as uint8 (an 8-bit unsigned integer) instead of the 64-bit integers it would use by default. That keeps the DataFrame eight times smaller, which matters a lot for a dataset this size.

(Want to nerd out on why eight bits is exactly enough to count from 0 to 255 -- and what happens when a value doesn't fit? The NumPy guide to data types covers the fixed-size numeric types and integer overflow, and the DISCOVERY guide "Python Data Types" reviews the types you already know.)

Using pd.read_csv(..., dtype="uint8"), load the mnist_784.csv.xz file into a DataFrame in the Python variable df:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 1.1: Viewing a Subset of Pixel Columns

The first column in our DataFrame -- label -- is the digit represented by the following 784 pixels (pixel1 - pixel784). Each pixel column has a value from 0 to 255 representing 256 different grayscale values. You can see a lot of variety in the pixel grayscale values near the center of the image, ex: viewing pixel300, pixel301, pixel302:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 1.2: Creating the List of All Pixel Column Names

In order to build our model using only pixel data, we need to capture a list containing only the pixel columns (and not including the index, label, or any other columns we might add to our dataset). To list all of the columns in the dataset, we can use df.columns:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

When we have a list of elements, we can exclude elements by taking the subset range of a set of elements. The syntax to get a subset of a list requires a start index of the subset range and the index to stop at (and NOT include):

As an example, if we consider this list of the five vowels:

vowels = ["A", "E", "I", "O", "U"]
# index:  [0]  [1]  [2]  [3]  [4]

The format of the subset range for the list of vowels will be vowels[startIndex:stopIndex]. For example:

vowels[1:3]   # Starts at index [1], stops BEFORE reaching index [3]
# Output: ["E", "I"]           #  (...includes only [1] and [2])
vowels[0:3]   # ...start at index [0], still stopping BEFORE reaching index [3]
# Output: ["A", "E", "I"]
vowels[1:4]
# Output: ["E", "I", "O"]

Either (or both) of the starting index or the stopping index can be excluded. If a starting index is excluded, it is understood by Python to mean "starting from the beginning of the list"; if the ending index is excluded, it is understood by Python to mean "stopping only after the end of the list". For example:

vowels[1:]   # Starts at index [1], going until the end of the list
# Output: ["E", "I", "O", "U"]
vowels[:3]   # Starting at the beginning of the list, stopping BEFORE reaching index [3]
# Output: ["A", "E", "I"]
vowels[:]    # Starting at the beginning of the list and going all the way to the end -- [:] is the whole list
# Output: ["A", "E", "I", "O", "U"]

Using this idea, create a list of all column names that contain data about pixels (i.e. every column except the label column) -- you will want to use df.columns from the previous code cell and select only a subset of all the columns.

Store this list of all the pixels in a new Python variable named PIXEL_COLUMNS:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 1.2: Creating a List of All Pixel Column Names

Part 1.3: Rendering the Data

Using the DataFrame df you loaded the MNIST dataset into and the PIXEL_COLUMNS list of pixel columns, we have provided an already completed function that renders any index (or list of indexes) in the DataFrame. Run the following code to define the renderDigit function we provided:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Now that we have the renderDigit function, let's render and explore some data by running the following cells:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

There are also some really challenging digits:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 2: Splitting the Data into the Training and Test Datasets

When working with any AI algorithm, it's critical to reserve a portion of the data for testing the model that we're going to train. In this way we both avoid "over-fitting" our algorithm to specific data and have a reliable way to test if our model works with data that has never been seen before.

In the MNIST dataset, the first 60,000 rows of data are used for the training dataset and the final 10,000 rows are used for the testing dataset. In the following cell, create two DataFrames (named train and test) that contain the training and testing portion of the entire MNIST dataset:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Selecting a Subset of the Testing Data

The full MNIST dataset is very large, 784 features across 70,000 data points, which is over 50,000,000 data points!

  • You CAN skip this cell and complete the project using the full dataset.
  • We have designed and tested all the test cases to run with both the full dataset and the subset below.
  • If you use the full dataset, the kNN algorithm will be take quite some time time to run (taking over five minutes to run on most devices).

If you want to make sure that your project cells run in a reasonable amount of time, uncomment BOTH of the lines of code below that will reduce the size of the MNIST dataset to 50% of the training data and 10% of the test dataset:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 2: Testing/Training Split

Part 3: Using K-Means Clustering

We know from this project that the MNIST training dataset contains 60,000 labeled images, each containing the numbers 0 through 9 (ten total possible digits).

Part 3.1: Initialize the K-Means Clustering Model

One of the simplest machine learning algorithms is k-means clustering.

In our DISCOVERY lecture on clustering, we explore the k-means algorithm. With 784 features for each of the 60,000 data points, clustering all 60,000 data points requires significant processing time and memory.

The sklearn library provides the MiniBatchKMeans version of k-means that will batch similar data points together to reduce the processing time and memory. In the cell below, initialize the MiniBatchKMeans with a total of 10 clusters (one cluster for each of the ten possible digits).

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 3.2: Training your MiniBatchKMeans model with Training Data

Before training our model, we first need to set up our independent variables (ind) to be equal to the 784 columns you identified as the PIXEL_COLUMNS earlier in this project as your independent variable:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Now, train (fit) your MiniBatchKMeans model on your entire train DataFrame:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 3.3: Labeling Training Data with Clusters

Since k-means is an unsupervised algorithm, we have found ten distinct clusters but we don't know what's in each cluster. To find the best label for each cluster, we have to find the label that is most frequent in each cluster.

To do this, we first need to identify which cluster each training data row belongs to. To do this, add a new column to your DataFrame called cluster that contains the cluster that each row of your train DataFrame is in. (Hint: You can do this by assigning the result of model.predict( train[PIXEL_COLUMNS] ) to the new column.)

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 3.3: Training and Labeling your KMeans model with Training Data

Part 3.4: Identifying Each Cluster's Label

Now we can consider any example cluster, and we'll start with 4. Create a DataFrame named train_cluster4 that contains all of the training data that is labeled to be in cluster 4:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

The following code randomly chooses eight of the rows in train_cluster4 and renders them, resulting in eight sample images from within that cluster. If KMeans is a good clustering algorithm, all of the images should be the same digit almost every time since they're all in the same cluster. Run the code below to see how the clustering did:

(You can run this code multiple times to view different samples.)

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Even though it's not perfect, the majority of rows in a particular cluster should have the same label (the true numerical digit for that row) -- the most popular label in each cluster can be chosen as the label for the entire cluster. For example, here's the distribution of the labels within train_cluster4:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

The most common label for all the training images in train_cluster4 appears first and is known as the "mode" of the label category. It will be the "label" of the entire cluster. We can generalize this approach to find the mode for every cluster by grouping by the "cluster" and finding the mode of the "label" column:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Finally, we can use this set of cluster labels to label each cluster with the predicted label.

The final code we provide is a function that uses the cluster value in each row and finds the corresponding label from the cluster_label DataFrame that we just found:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Use train.apply(labelCluster, axis=1) to run the labelCluster function for each row in the DataFrame. Add the result of this function to the train DataFrame as a new column called predicted:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 3.4: Identifying Each Cluster's Label

Part 3.5: Label the Testing Data with a Prediction

In the previous code, we labeled the training data with the predicted label. However, the testing data is data that our model has never seen before and should be used to find the accuracy of our model.

First, using model, add a new column "cluster" to the test DataFrame. (This will be very similar to adding it to the train DataFrame, but now we're using a different DataFrame but the same trained model.)

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Next, apply the labelCluster function to every row in your test DataFrame and add a new column called predicted to your test DataFrame:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 3.5: Label the Testing Data with a Prediction

Part 3.6: Calculate the Prediction Error

The correct prediction for each row in our DataFrame is when the label matches the predicted label. A reliable algorithm should have a very low error rate, and any error rate above 5% will be quickly noticed by humans.

Find the number of rows where the label does not match the predicted label in the test DataFrame:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Convert the number of errors into a percentage (decimal form) of the 10,000 test images:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 3.6: Calculate the Prediction Error

Part 4: Using k-Nearest Neighbors (kNN) Classifier

In Part 3, you found that clustering the features using KMeans failed to accurately predict data. Many times in AI and machine learning, the first approach you try doesn't yield great results. Finding the right algorithm for a specific dataset is very hard and knowing hundreds of different algorithms and when to apply them makes you a great data scientist.

Let's try another simple classification algorithm -- but this time, let's see if a supervised algorithm can give us a better result!

Part 4.1: Finding Neighbors

The k-Nearest Neighbor algorithm is simple: find the k closest points from your training data and use the most commonly occurring label. That's it!

Let's start by taking the very first row in our test dataset and call it our sample:

(Hint: what is the index of the first row in the test dataset? How do you get a row given its index?)

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Finding the Correct Sample

Once we have our sample, we can find the difference between the values in each pixel column by subtracting the entire training DataFrame from the sample to get a DataFrame of the differences:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Using the distance column that was just calculated, find the 5 rows in your train dataset that have the smallest distance and store those five rows in the DataFrame neighbors:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

In the above DataFrame, you should find that all of the neighbors of the first sample are labeled with 7. This means we predict that the label of our sample is 7. Is that what it was labeled when we found our sample at the beginning of Part 4.1?

⚙️ Test Case: Part 4.1: Finding Neighbors

Part 4.2: Training the KNeighborsClassifier

Similar to KMeans in Part 3, the KNeighborsClassifier in the sklearn library is a machine learning algorithm. Unlike KMeans, KNeighborsClassifier is a supervised algorithm where we'll supply both the independent and dependent variables in our training step.

First, initialize an instance of the KNeighborsClassifier using all the default parameters (you don't need to specify anything in the initialization) as model2:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Second, identify the independent and dependent variables:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Finally, fit the train data to our KNeighborsClassifier that we called model2:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 4.3: Understanding the KNeighborsClassifier Neighbors

First, let's refresh what neighbors we found for the first row in our test DataFrame:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Before we make a prediction, let's check if the neighbors we found ourselves matches the ones found by the KNeighborsClassifier algorithm below:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

If everything is correct so far, the two previous cells should be identical in distance (676.584.. vs 676.584.., etc) -- our manual process of finding the nearest neighbor matches exactly what our kNN classifier did! 🎉

Part 4.4: Labeling Predictions Using the KNeighborsClassifier

Let's first make a prediction using our sample, and see if our prediction matches:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Now, add a new predicted column to the entire test dataset by predicting the value of the full test dataset by using your model2 classifier:

⚠️ This may take some time! ⚠️

  • Using the recommended subset of the dataset, kNN needs to check the distance between each of the 784 features of 1,000 test images to the 784 features of all 30,000 test images. That is over 23 billion (23,520,000,000) possible calculations!
  • If you are using the full dataset, this number grows by a factor of 20x -- to over 470,400,000,000 possible calculations!
  • Even a fast computer will take a significant amount of time!!
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 4.4: Labeling Predictions Using the KNeighborsClassifier

Part 4.5: Viewing the Predictions

Run the cell below to view a random select of eight test images that KNN identified to be the same number:

  • (Each time you run the cell, you will get a new set of eight images that KNN predicts to be that digit.)
  • (The number below the image is the label provided in the MNIST dataset.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 4.6: Calculate Prediction Error

Just like with KMeans clustering, the correct prediction for each row in our DataFrame is when the label matches the predicted label. A reliable algorithm should have a very low error rate, and any error rate above 5% will be quickly noticed by humans.

Now that we labeled our test DataFrame using kNN, find the number or rows where the label does not match the predicted label in the test DataFrame:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Just like before, convert this to a percentage (in decimal form):

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 4.6: Calculate Prediction Error

Part 5: Comparing Error Between Models

Running the following cell will provide a summary of the two error rates you calculated on the MNIST dataset:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Exploring how different models perform with different datasets, and understanding why a model works and doesn't work, is one of the most foundational things in Data Science:

  • Which model performed better on the MNIST test data?
  • Was there a significant difference in the error rates?

You can now use your trained model to predict writing you have never seen -- and recognize any handwritten digit! 🎉

Note on Project Validation

⚠️ The project validation will take some time! ⚠️

  • Validating the project will run all of your code again, which will take the same amount of time your code originally took to run.
  • This will take a significant amount of time using the full dataset.

Validate and Complete This Project!

Congratulations on finishing this DISCOVERY Data Science Project! 🎉🎉

To validate your entire project, your entire code will run from top-to-bottom on this page and each test case will be validated one final time. If everything looks good, you'll earn the card for completing this project: