Project #14: AI vs Human: Response Analysis

In this DISCOVERY Data Science Project, you will do real data science in less than an hour and you will earn this project's card to your collection when you fully complete this project! 🎉

Data Source: Hugging Face HC3 Dataset

Hugging Face is a company that has developed a platform for natural language processing (NLP) applications. They have created and shared a large collection of pre-trained models, datasets, and learning resources which are open-source and available for the public to use.

Hello-SimpleAI is a small group of researchers (PhD students and engineers) that released the HC3 dataset. This dataset compares human responses (from Reddit, Wikipedia, and other sources) and AI responses to the same question, with details available on their GitHub repository (@Hello-SimpleAI/chatgpt-comparison-detection) or via the Hugging Face API.

Are responses from AI different from humans? Let's nerd out with this data and find out! 🎉

Background Knowledge

To finish this project, we assume you already know:

Let's get started! :)

Part 1: Loading the Dataset

For this project, we have provided the first 100 samples of the Hello-SimpleAI dataset that was gathered from the following URL: https://datasets-server.huggingface.co/rows?dataset=Hello-SimpleAI%2FHC3&config=all&split=train&offset=0&length=100

This data has been cleaned up and provided to you as huggingface-hello-simpleAI-100.csv. This dataset has 100 questions that were asked and answered on the "Explain Like I'm Five" (ELI5) subreddit, with the human_answer from reddit and a chatgpt_answer from ChatGPT.

Load the cleaned up CSV file dataset as a DataFrame df:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

View Sample Outputs

Run the following cell to view a random question and the human and AI answers to get a feeling for this data (run it again to view a different random question).

  • Note that this text has already been tokenized, an early step in almost all text analysis that breaks a sentence into the small tokens for further processing, and may have extra spaces not naturally seen outside of NLP.
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 1: Loading the Dataset

Part 2: Response Length Analysis

This project attempts to identify features that may be distinctive between a human response and an AI response. One common critique of current AI systems is that the answers are often excessively long. Our first hypothesis to investigate is that an AI response may be longer than a human response.

To explore this, we will explore the average length of an AI response and see if there's any statistical significance to the difference from the human response.

Part 2.1: Finding the Response Length

To perform the analysis of response length, we first need to add two new columns in our DataFrame for each of our 100 sample questions. When working with columns that contain string data, we work with the individual strings as data using the following generic syntax:

df["column"].str.STRING_FUNCTION()

...where STRING_FUNCTION() is any of the string accessor methods.

Specifically, one string accessor method is:

[...].str.len(): Compute the length of each element in the Series/Index.

To begin our analysis of the length of each response, add two new columns to the DataFrame:

  • human_answer_len, containing the length (in characters) of the human response
  • chatgpt_answer_len, containing the length (in characters) of the ChatGPT response
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 2.2: Compute the Average Answer Lengths

Now that we have the lengths of both responses, save the average length of the responses in the variables human_avg and ai_avg:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 2.2: Compute the Average Answer Lengths

Part 2.3: Testing if the Difference in Length is Statistically Significant

To determine if the difference between the means is statistically significant, we're going to conduct a t-test. Let's first state our null and alternative hypotheses.

Null hypothesis: There is no significant difference between the means of the two response lengths.

Alternative hypothesis: There is a significant difference between the means of the two response lengths.

Use the scipy.stats.ttest_ind() function to get the t-stat (as t_stat) and p-value (as p_value).

You can find the function documentation here: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 2.4: Choosing Your Conclusion

Using the calculated p-value and an alpha level of 0.05, we can determine if there is a significant difference between the two mean values.

For each set of comments, un-comment exactly one line in each set corresponding to the true statement:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 2.4: Response Length Analysis

Part 3: Sentiment Analysis

A second common critique of current AI systems is the answers are often more subjective and optimistic than human responses.

There is a field of Data and Computer Science called "Natural Language Processing (NLP)" that involves algorithms to analyze the "polarity" and "subjectivity" score of a piece of text. In the next two parts of this project, we will explore the polarity and subjectivity of human and AI responses.

Part 3.1: Using TextBlob to Find the Polarity Scores

TextBlob is a Python library for processing textual data. It provides a simple API for diving into common natural language processing (NLP) tasks such as part-of-speech tagging, noun phrase extraction, sentiment analysis, classification, translation, and more.

Among the many features, TextBlob can calculate a polarity score for any given piece of text. A "polarity score" ranges from -1 to 1, where:

  • A score of -1 is very negative,
  • A score of 0 is neutral, and
  • A score of 1 is very positive.

For example, the text "You will fail DISCOVERY if you do not take the final exam" is a negative statement and has a polarity score of -0.25. See this for yourself by running the following below:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 3.2: Creating a polarity Function

To allow us to easily find the "polarity score" of hundreds of pieces of text using pandas, we need a function that will take a string s as input and return the polarity score.

Complete the polarity function below, by:

  • Reading the TextBlob Tutorial: Quickstart, specifically focusing on the "Sentiment Analysis" portion of the tutorial.
  • Finish the polarity so that the function returns the polarity score of the input string s.
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Now, let's test your function to make sure it's working:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 3.3: Finding the Polarity of Human Answers

Once you have defined a custom function, you can use it on every row of data by using:

df["column"].apply( customFunctionName )

Note that customFunctionName is ONLY the function name and does NOT contain the function call (there are no parentheses after the function name; it's just the name of the function). You are providing ONLY the name of the function and the apply function will internally call the function.

Using the syntax above, create a new column named human_polarity that finds the polarity of the human_answer data using the .apply operation on a column of the DataFrame:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 3.4: Finding the Polarity of ChatGPT Answers

Now, find the polarity of ChatGPT answers and save each polarity in the column chatgpt_polarity:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 3: Sentiment Analysis - Polarity

Analysis: Overall Results So Far

Run the following cell to see a summary of the results you found so far:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 4: Sentiment Analysis - Subjectivity

TextBlob can also calculate a subjectivity score for any given piece of text. A "subjectivity score" ranges from 0 to 1, where:

  • A score of 0 is not a subjective statement, and
  • A score of 1 is a very subjective statement

Using the same library and design as Part 2, create a function called subjectivity that returns the subjectivity score of a given string s:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Let's try it out:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 4.1: Finding the Subjectivity of Human and ChatGPT Answers

Add two new columns to the DataFrame, human_subjectivity and chatgpt_subjectivity, that stores the subjectivity of each answer for all of the questions.

You may need to look back to the previous section to review how to use the .apply function.

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 4: Sentiment Analysis - Subjectivity

Analysis: Overall Results So Far

Run the following cell to see a summary of the results you found so far:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 5: Testing if the Difference in Subjectivity is Statistically Significant

To determine if the difference between the subjectivity is statistically significant, we're going to conduct another t-test. Let's first state our null and alternative hypotheses.

Null hypothesis: There is no significant difference between the means of the human and ChatGPT subjectivity.

Alternative hypothesis: There is a significant difference between the means of the human and ChatGPT subjectivity.

Part 5.1: Performing the t-test

Use the scipy.stats.ttest_ind function again to get the t-stat and p-value, storing it as t_stat_subjectivity and p_value_subjectivity

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Part 5.2: Choosing Your Conclusion

Using the calculated p-value and an alpha level of 0.05, we can determine if there is a significant difference between the two mean values.

For each set of comments, un-comment exactly one line in each set corresponding to the true statement:

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)
⚙️ Test Case: Part 5: Testing if the Difference in Subjectivity is Statistically Significant

Analysis: Results

Run the following cell to see a summary of the results of all of the metrics (we finished the t-test for polarity for you as well):

Reset Code Run All to Here Python Output:
(Run your code to see your code result's here.)

Validate and Complete This Project!

Congratulations on finishing this DISCOVERY Data Science Project! 🎉🎉

To validate your entire project, your entire code will run from top-to-bottom on this page and each test case will be validated one final time. If everything looks good, you'll earn the card for completing this project: