AI versus Human: Response Analysis

MicroProject Overview

Detecting if something has been generated by a human or AI can be difficult. This MicroProject explores this topic by looking at 100 questions identical questions answered by both ChatGPT and a human.

Hugging Face is a company that has developed a platform for natural language processing (NLP) applications. They have created and shared a large collection of pre-trained models, datasets, and learning resources which are open-source and available for the public to use. Using Hello-SampleAI from their HC3 dataset dataset, this MicroProject explores the length of responses, sentiment score, and subjectivity scores of the responses.

Data Science Skills

In this microproject, you will explore the Bechdel Test and strengthen the following Data Science skills:

  • Using string accessor methods of DataFrame data,
  • Using the TextBlob library for sentiment, polarity, and subjectivity analysis,
  • Using a t-test to test for significance, and
  • Working with parallel data (human vs. AI) in a single dataset

Let's nerd out!

First Time Doing a MicroProject?

Each MicroProject starts with a notebook that we provide to you to get started! You will need to configure a git repository to connect to our `microprojects` remote where we release the starter notebook.

Fetch the Initial Files

In your terminal, navigate to your GitHub repository and merge the initial files by running the following commands:

git fetch microprojects
git merge microprojects/microproject-ai-vs-human-response-analysis --allow-unrelated-histories -m "Merging initial files"

Complete the Notebook

If the commands above were successful, you have merged in the initial files to start on the MicroProject.

  • Find the new microproject-ai-vs-human-response-analysis folder.
  • Open microproject-ai-vs-human-response-analysis.ipynb and complete the MicroProject!

Commit and Grade Your Notebook

Once you have finished your notebook, you must use the built-in GitHub Action to preform automated grading of your MicroProject notebook! You will need to commit your work and then manually run the GitHub Action.

Commit Your Work

To commit your notebook, run the standard git commands in your terminal:

git add -u
git commit -m "microproject completed"
git push

Grade Your Notebook

To grade your notebook, you will need to visit your GitHub repository in your browser.

  • Visit your GitHub repository in your browser
  • Click on the "Actions" tab
  • Under "Workflows", find the workflow for this microproject
  • Click the "Run Workflow" in the blue box, and then the green "Run Workflow"
  • After about 10 seconds, you should see a new job that has started running
    • You can click on the job to watch it run in real-time
    • It will take ~1 minute to run and grade
  • Once the running is complete, the autograding summary will be available!