Project #7: Creating an Image Mosaic

Project Overview

A Data Science DISCOVERY alum had their puppy, Rosie, featured on the official University of Illinois Instagram, and we wondered: What if we could make an image of Rosie entirely from images posted to the Illinois Instagram?

By dividing the image into a square grid and replacing each region with a matching Illinois image, we created an image mosaic of Rosie:

→

The following images zoom into the mosaic to show how the individual tiles form the finished picture:

The "Rosie Mosaic", made entirely of images from @illinois1867	5x Zoom (the top-left of the zoomed region is Rosie's nose)	25x Zoom	50x Zoom

Mosaic Terminology

1. The base image is the image that your mosaic will recreate. You will choose one from the provided Illinois image collection.

2. The tile images are the images used to build the mosaic. This project uses the same collection of 300 provided Illinois images as tiles.

   • Before you begin, make sure to preview the 300 tile images so you can know what we're working with! :)

Let's get started and create your own mosaic! 🎉

Section 1: Loading the DISCOVERY Library

To access an image as a DataFrame, we have provided the DISCOVERY library to help process the image into a DataFrame. The library is available to be imported using import DISCOVERY.

Specifically, the function we provided is DISCOVERY.loadImageAsDataFrame("sample.png"), which loads an image and returns the image's pixel data as a DataFrame.

To use this function:

• First, import DISCOVERY by using import DISCOVERY (all caps, because DISCOVERY!)
• Then, image = DISCOVERY.loadImageAsDataFrame("sample.png") to load the sample.png image into a DataFrame named image
• Finally, type image as the last line in the cell to display the contents of the DataFrame you just created

Try that out in the cell below:

...

The Sample Image

You just loaded a sample image -- it is tiny, just 3 pixels by 3 pixels -- and almost invisible. Below is the a giant version of the nine pixels you just loaded by loading sample.png:

Compare the DataFrame you created in Python with the image above. There are a few key observations:

• Each row of the DataFrame contains a position (x and y) and a color (composed of a red r, green g, and blue b component).
• The first row of the DataFrame (index [0]) contains information about the pixel located at x=0, y=0. The color of the pixel is r=0 (no red), g=0 (no green), and b=255 (the maximum value of blue). This tells us that the top-left pixel is a bright blue color.
• The next row of the DataFrame (index [1]) has the y value of 1, giving us information about the pixel located at (x=0, y=1). Visually it is a bright green pixel, and the DataFrame shows a color value of rgb(0, 255, 0).
• ...this goes on for all nine pixels...

Finding a Single Pixel

The code you used to load the image stored the image as a DataFrame. Using the DataFrame image, write the code to select the single row from the DataFrame that has information about the GREEN 🟩 pixel (second row, first column) and store that in the variable green_pixel:

green_pixel = ...\ngreen_pixel

Section 2: Accessing Color Data

Every color visible on a computer screen is made up of the three primary colors of light -- red, green, and blue. Your monitor displays color by varying the intensity of the red light, green light, and blue light emitted for every pixel on your screen. Since images are primary displayed on computer screens, the default "color space" is to represent colors as red, green, and blue.

When you have a pixel, you will always see the three components. For example, the contents of green_pixel is [0, 255, 0]:

• The first 0 means we have 0 / 255 (0%) red light
• The second 255 means we have 255 / 255 (100%) green light
• The final 0 means we have 0 / 255 (0%) blue light

Section 2.1: Accessing Illini Orange

Using the image you loaded in Section 1, create a DataFrame that contains ONLY the three pixels on the top row of the DataFrame. Store that subset of the DataFrame in the variable image_topRowSubset:

image_topRowSubset = ...\nimage_topRowSubset

The top row should contain only three pixels. Check your DataFrame to make sure that you have only:

• A blue pixel, where b = 255 and both r = 0 and g = 0.
• A red pixel, where r = 255 and both g = 0 and b = 0.
• A white pixel, where r, g, and b are all 255.

Once we have a subset, we can use our DataFrame operations on our subset. For example, let's look at the median value for each color.

• The red components of the three pixels is [0, 255, 255]. This means the median red value should be 255.
• However, the green components of the three pixels is [0, 0, 255]. This means the median red value should be 0.

Calculate the median r, g, and b values of the image_topRowSubset image below:

(Hint: You can find the median of a DataFrame column with the .median() function on the DataFrame's column data.)

r_median = ...\nr_median

g_median = ...\ng_median

b_median = ...\nb_median

Section 3: Finding the Average Color of the Sample Image

Now, find the average color (not the median anymore!) of the entire image by finding r_avg, g_avg, and b_avg:

r_avg = ...\nr_avg

g_avg = ...\ng_avg

b_avg = ...\nb_avg

Section 4: Find the Average Color of ANY Image

Building off Section 3, we now want to find the average color of ANY image. To do that, we need to create a function that is given a DataFrame image and returns the average color of that image.

Write this in the function findAverageColor below. You must return the average color as a dictionary with the values:

• r_avg, for the average red color,
• g_avg, for the average green color,
• b_avg, for the average blue color

A dictionary is a data structure that stores multiple values. You have used a dictionary in "Step 4" of your simulation code, where you use d to accumulate all real-world values. Accumulate the values r_avg, g_avg, and b_avg just like you would in the simulation:

def findAverageImageColor(image):
  ...

  # Return a dictionary of average color:
  d = {"r_avg" : r_avg, "g_avg": g_avg, "b_avg": b_avg}
  return d

Write the entire findAverageImageColor function to find the average color of the image passed into the function:

# Define the function:\ndef findAverageImageColor(image):\n  ...\n \n \n# ...and we'll test it by calling the function on the sample image we've been working with:\nfindAverageImageColor(image)

Loading and Testing with the Rosie Image

So far, you have been working with a tiny (9 pixel) image. Let's start working with a much larger image!

Using the DISCOVERY.loadImageAsDataFrame(...) function again, load the file named rosie.jpg into a Python variable named rosie. This file contains the image of Rosie you saw at the beginning of the lab (and displayed again here).

# Load the "rosie.jpg" image:\nrosie = ...\nrosie

Find the Average Color of the Rosie Image

Run the following cell to find the average color or the Rosie Image you just loaded:

# Find the average color of the "rosie.jpg":\nfindAverageImageColor(rosie)

Section 5: Dividing Your Base Image Into Small Grid Tiles

To create a mosaic from an image, we must split the base image into small regions to be replaced with the tile images. To accomplish this, we need a function that will find the subset of pixels found in a region of an image.

• Thinking about Section 2, you found the subset of the image that included all pixels in the top row.
• Other subsets of the sample sample.png 3x3 pixel image might be a 2x2 square of pixels, or a 1x3 rectangle, or any number of other subsets.

Since we might need any rectangular region of an image, lets create a function that returns a rectangular region on an input image that starts at (x0, y0) and spans width pixels wide and height pixels tall.

Your findImageSubset function

Specifically, create a function findImageSubset that finds the subset of the image starting at (x0, y0), spanning width pixels wide and height pixels tall. Your function must return the subset of all the pixels in that region of the image.

• Example: findImageSubset(rosie, x0=0, y0=0, width=3, height=3) -- returns subset of all the pixels in the square defined by: x=0...2 and y=0...2 (9 total pixels)

• Example: findImageSubset(rosie, x0=5, y0=5, width=5, height=5) -- returns subset of all the pixels in the square defined by: x=5...9 and y=5...9 (25 total pixels)

• Example: findImageSubset(rosie, x0=5, y0=0, width=5, height=5) -- returns subset of all the pixels in the square defined by: x=5...9 and y=0...4 (25 total pixels)

• Example: findImageSubset(rosie, x0=15, y0=0, width=20, height=10) -- returns subset of all the pixels in the square defined by: x=15...34 and y=0...9 (200 total pixels)

# Define the function:\ndef findImageSubset(image, x0, y0, width, height):\n  ...\n \n# ...and we'll test it by finding a subset of the "rosie.jpg" image:\nfindImageSubset(rosie, x0=15, y0=0, width=20, height=10)

Section 6: Finding the Average Color of a Subset

You have created two functions:

1. A function that finds the average color of a DataFrame of pixels (findAverageImageColor), AND
2. A function that finds a subset of pixels of an image (findImageSubset)

Create a new function, findAverageImageSubsetColor that combines both of them and returns the average color of a subset of the image:

# Complete the function:\ndef findAverageImageSubsetColor(image, x0, y0, width, height):\n  # 1. Find the subset:\n  ...\n \n  # 2. Find and return the average color of the subset:\n  ...\n \n \n# ...and we'll test it by finding a subset of the "rosie.jpg" image:\nfindAverageImageSubsetColor(rosie, x0=15, y0=0, width=20, height=10)

Section 7: Finding the Average Color of Your Tile Images

To create an image mosaic, we need to find the average pixel color of every one of our tile images so that you can find the best tile image to use when you're creating your mosaic. The code below is already complete and does the following:

• Goes through each image file in your path directory,
• Finds the average pixel color of each image using your findAverageColor function from Section 4,
• Finally, creates a new DataFrame df_tiles with the average color of each image and returns that DataFrame.

Make sure to run this code -- we'll use it in the very next section!

import pandas as pd\n \ndef createTilesDataFrame(pathOrList):\n  data = []\n \n  # If the data is a list, use the list of files; otherwise use the path:\n  if type(pathOrList) is list:\n    fileNames = pathOrList\n  else:\n    fileNames = DISCOVERY.listTileImagesInPath(pathOrList)\n \n  # Loop through all images in the `path` directory:\n  for tileImageFileName in fileNames:\n    # Load the image as a DataFrame and find the average color:\n    image = DISCOVERY.loadImageAsDataFrame(tileImageFileName)\n    averageColor = findAverageImageColor(image)\n \n    # Store the fileName and average colors in a dictionary:\n    d = { "fileName": tileImageFileName, "r": averageColor["r_avg"], "g": averageColor["g_avg"], "b": averageColor["b_avg"] }\n    data.append(d)\n \n  # Create the `df_tiles` DataFrame:\n  df_tiles = pd.DataFrame(data)\n  return df_tiles

Understanding the Output

Here's the DataFrame output that the function creates for some sample images. The DataFrame has four columns:

• fileName, containing the name of the tile image
• r, g, and b, containing the average color for the image

We will use this on your tile images directory when we put this all together to make your mosaic, but since that will contain hundreds of images we're going to start with just two sample images:

createTilesDataFrame(["sample.png", "sample2.png", "rosie.jpg"])

Section 8: Finding the Best Match

There's just one last function to create before we can start making mosaics! This function needs to find the best tile for a given average color.

To do this, your function will use two pieces of data:

1. Your function will use the DataFrame that contains all of the tile images. This will be the DataFrame from Section 7, df_tiles, and contains the tile image filename and each tile's r, g, and b average pixel colors.
2. Your function will be called with the average color of the current region of your base image that you're searching for a tile image to replace. The color of the base image is passed into your function as r_avg, g_avg, and b_avg.

Using this DataFrame of all tile images and the (r_avg, g_avg, and b_avg) form the base image, create the findBestTile that must find the best tile image from df_tiles for a given average color. You should do this by finding the single row in df_tiles that has the smallest distance from the average color.

Example

Imagine you just have three tiles, so your df_tiles DataFrame is:

If your image subset has r_avg = 10, g_avg = 200, and b_avg = 20, we can use the distance formula (Pythagorean's Theorem) to find how "far away" the average color is from each of the tile images:

1. For the red tile (255, 0, 0), the distance away is $d = \sqrt{(255 - 10)^2 + (0 - 200)^2 + (0 - 20)^2} = 316.8990375$
2. For the green tile (0, 255, 0), the distance away is $d = \sqrt{(0 - 10)^2 + (255 - 200)^2 + (0 - 20)^2} = 59.37171044$
3. For the blue tile (0, 0, 255), the distance away is $d = \sqrt{(0 - 10)^2 + (0 - 200)^2 + (255 - 20)^2} = 308.7474696$

We find that the green tile is the closest since it has the minimum distance. The green tile should be returned.

Hints

• It will be VERY helpful to add an extra column to your DataFrame -- like df["distance"] = ....
• Once you have the distance calculated, how do you return just the smallest row?

# Complete the function to find the best tile:\ndef findBestTile(df_tiles, r_avg, g_avg, b_avg):\n  ...\n \n \n# ...and test it to make sure the best match for a red pixel among our samples is our dog Rosie:\nfindBestTile(\n  createTilesDataFrame(["sample.png", "sample2.png", "rosie.jpg"]),\n  r_avg=255, g_avg=0, b_avg=0\n)

Section 9: Selecting Your Base Image

The base image is the picture your mosaic will recreate. In your first mosaic, you should use "rosie.jpg" but you will be able to use your image later!

# Replace this with any filename from the Illinois Image Explorer:\nbaseImageFileName = "rosie.jpg"\n# baseImageFileName = "https://waf-server-01.cs.illinois.edu/static/getAllMovies.png"\nbaseImageFileName

Section 10: Loading the Tile Images

This project includes 300 University of Illinois images in the tiles folder. They are already available to Python in your browser, so there is nothing to upload.

Run the cell below to select that folder and verify the collection:

tileImageFolder = "tiles"\ntile_files = DISCOVERY.listTileImagesInPath(tileImageFolder)\nlen(tile_files)

Section 11: Create Your Mosaic!

Two settings control the finished mosaic:

• maximumTilesX controls how many tiles can appear across the image. More tiles add detail but take longer to process.
• tileHeight controls the size of each displayed tile. Larger tiles make a larger final image.

We have provided reasonable defaults for the Rosie image:

• maximumTilesX = 40 will use 40 tiles across (translating 5×5 regions of the base image into mosaic tiles)
• tileHeight = 10 will make each tile 10 pixels tall (translating each of those 5×5 regions into a 10×10 tile)

We recommend running the mosaic first with these default values and then changing them later.

maximumTilesX = 40\ntileHeight = 10

Section 11.1: Find the Average Color of Every Tile

This code uses your function from Section 7 to create df_tiles, a DataFrame of every tile image. We have supplied you with 300 tile images so we can get straight to making the mosaic.

⚠️⚠️ This takes longer than the earlier cells because Python must inspect all 300 images -- up to one minute or more to run! ⚠️⚠️

df_tiles = createTilesDataFrame(tileImageFolder)\ndf_tiles

Section 11.2: Load Your Base Image

baseImage = DISCOVERY.loadImageAsDataFrame(baseImageFileName)\nwidth = int(baseImage.x.max()) + 1\nheight = int(baseImage.y.max()) + 1\nbaseImage

Section 11.3: Build and Display the Mosaic

Run the code below to divide your base image into regions, find the closest matching tile for each region, and display the completed mosaic.

⚠️⚠️ This takes longer than the earlier cells because it generates the mosaic -- up to one minute or more to run! ⚠️⚠️

import io\nimport math\nfrom PIL import Image\n \npixelsPerTile = max(1, math.ceil(width / maximumTilesX))\nmosaicWidth = (width // pixelsPerTile) * pixelsPerTile\nmosaicHeight = (height // pixelsPerTile) * pixelsPerTile\ntilesX = mosaicWidth // pixelsPerTile\ntilesY = mosaicHeight // pixelsPerTile\n \nmosaic = Image.new("RGB", (tilesX * tileHeight, tilesY * tileHeight))\nfor x in range(0, mosaicWidth, pixelsPerTile):\n  for y in range(0, mosaicHeight, pixelsPerTile):\n    averageColor = findAverageImageSubsetColor(baseImage, x, y, pixelsPerTile, pixelsPerTile)\n    replacement = findBestTile(\n      df_tiles,\n      averageColor["r_avg"],\n      averageColor["g_avg"],\n      averageColor["b_avg"],\n    )\n    tile = DISCOVERY.getTileImage(replacement["fileName"].values[0], tileHeight)\n    mosaic.paste(tile, ((x // pixelsPerTile) * tileHeight, (y // pixelsPerTile) * tileHeight))\n \nmosaicOutput = io.BytesIO()\nmosaic.save(mosaicOutput, format="PNG")\nmosaicOutput

Section 12: Experiment and Save

Try a different base image from the images provided or adjust maximumTilesX and tileHeight, then rerun Sections 11.2 and 11.3.

• To save your mosaic, right-click the rendered image and choose Save Image As.
• On a touch screen, press and hold the image.

Validation Will Be Slow!

⚠️⚠️ The validation runs you entire project, so it will take the total time of all your cells to run! This may take several minutes to complete! ⚠️⚠️