Starting Your Own Data Science Project

Starting a data science project of your own can be an intimidating task. In this guide, we will help give you a head start on building your own project. We'll walk through: how to find a dataset, ways to start analyzing it, and some steps to spark ideas for you to explore!

Collecting Your Data

First, we need to find a dataset to perform our analysis on. It's best to find a dataset about a topic you are passionate about. As well, having prior knowledge about the data you're analyzing will help you understand the context behind certain columns which inform your analysis. Kaggle.com is a great resource to find a dataset on just about anything. If you are interested in sports for example, you can find more specific datasets at websites such as sports-reference.com.

For this example, we'll use a dataset of songs from our favorite artist at DISCOVERY; Taylor Swift!

The dataset we'll be using can be found on Kaggle, and contains information about every Taylor Swift song. Kaggle datasets also contain information about what the columns in the dataset represent.

Our columns in this dataset are Name (song name), Album, release_date, length, popularity, danceability, acousticness, energy, instrumentalness, liveness, loudness, speechiness, valence, and tempo. This may seem like a lot of information, so we'll take it slow and analyze a few of the variables at a time. Our first step will be to load the DataFrame, and become familiar with it so we can perform our analysis.

df = pd.read_csv('spotify_taylorswift.csv')
df.head(5)

Exploring the Dataset

This dataset has 171 rows, and by using the pandas function df.dtypes(), we can see the data types of each column. Variables such as album name, artist name etc. are coded as objects, while audio information is numerical and is represented by integer or float values.

When working with datasets from the internet, the data might not always be perfectly formatted. There could be missing values, or data may have been entered incorrectly due to human error. Due to the imperfect nature of some datasets, it is important to understand what each column is supposed to be representing. This dataset seems to have the data types we'd expect for each column, so we can proceed.

If your dataset has null or missing values, or an incorrect data type (such as an object where an int or float value should be), you can look at our guide on handling missing data here to see how to correct it.

print('Dataframe Length: ' + str(len(df)))
print(df.dtypes)

Dataframe length: 171
name                 object
album                object
artist               object
release_date         object
length                int64
popularity            int64
danceability        float64
acousticness        float64
energy              float64
instrumentalness    float64
liveness            float64
loudness            float64
speechiness         float64
valence             float64
tempo               float64
dtype: object

Descriptive Analysis

Our next step will be to explore the dataset in order to find relationships that interest us. It can be helpful to plot out variables of interest, and as you can see below, we plotted histograms for the popularity, danceability, and tempo variables.

Popularity is given as an integer ranging from 0-100, danceability is a metric ranging from 0-1 measuring how suitable a song is for dancing, and tempo measures beats per minute. Note that danceability appears to be normally distributed.

We can use functions such as df.describe() or df.corr() to see if there are any strong numerical relationships or variables with especially high/low values that we may want to inspect further.

df.describe()

df.corr()

df.popularity.plot.hist()
plt.xlabel('Popularity')
df.danceability.plot.hist()
plt.xlabel('Danceability')
df.tempo.plot.hist()
plt.xlabel('Tempo')

Generating/ Exploring Hypothesis

After looking over the dataset as a whole, we can start to manipulate it to look into our areas of interest.

By this point, we should have a solid understanding of the structure of our dataset. Now, you will think of 2-3 hypotheses or questions. These hypotheses or questions will be the basis of your project.

For example, how does X affect Y in this dataset? Can we predict Z using A? After formulating our hypothesis, creating DataFrames that take subsets of our main df, or grouping the data, is an excellent way to test our theories.

In this example, we'll examine differences in various categories of audio information based on album. First, let's group the data by album, and create different DataFrames for each albums. This will allow us to examine differences in Taylor Swift songs depending on the album a song was from.

One question we might like to answer is: "Is the median danceability in Taylor Swift songs different in each Album?"

As we can see in the box plot below, the median danceability of her songs in Lover and Folklore are quite different. Creating plots and visualizations like this can help us to see patterns in our data that we might otherwise miss.

df.groupby('album').agg('mean').reset_index()

evermore = df[ df.album == 'evermore (deluxe version)']
lover = df[ df.album == 'Lover']
print(lover.danceability.mean())
print(evermore.danceability.mean())

sns.set(rc={'figure.figsize':(5,5)})
sns.boxplot(x = 'album', y = 'danceability', data = df)
sns.set(rc={'figure.figsize':(5,5)})
sns.boxplot(x = 'album', y = 'acousticness', data = df)

Statistical and Predictive Analysis

After diving further into your hypothesis, the time has come to perform our final analysis.

Now, we can take an observation we had (or something we want to predict) and create hypothesis tests, confidence intervals, linear regressions, or clustering models on our data. The steps above should help you identify variables that may be related or may affect another variable. Use these variables to conduct the above mentioned tests and prove, or disprove, your hypotheses.

This guide is only a suggestion on ways to get started. Your datasets will likely look quite different from the one we used here, but the principles we discussed can help you get started if you aren't sure how to begin! Good luck future data scientist.