DataFrames with Conditionals

The use of conditionals allows us to select a subset of rows based on the value in each row. Writing a conditional to select rows based on the data in a single column is straightforward and was used when we selected all of the courses taught by the Statistics department with the following code:

df[ df.Subject == "STAT" ]     # dot notation, using df.Subject

df[ df["Subject"] == "STAT" ]  # index notation, using df["Subject"]

Complex Conditionals with Multiple Parts

As we want to answer more complex questions, we need increasingly complex conditionals. To help understand how a computer works, you may be familiar with the idea that computers ultimately only think in zeros and ones:

• When a computer stores a zero, we consider that to be False.
• When a computer stores a one, we consider that to be True.

When we use conditionals, we are assigning a truth value to every single row in the DataFrame.

• With our conditional df[df.Subject == "STAT"], all rows where the Subject data was "STAT" was assigned a truth value of True and kept in the final result; all other rows were labeled False and discarded.

All programming languages allows us to combine conditionals together in two key ways: with an AND (&) or with an OR (|).

Multiple Conditionals Joined with AND (&)

When we combine two conditionals, we can ask Python to keep only the result where both the first conditional AND the second conditional are both True.

Writing a conditional with multiple parts requires the use of parenthesis around each individual conditional and an operation joining the two conditionals together. For example, using the Course Catalog dataset, we want all of the courses that are taught by Computer Science (CS) with a course number less than 300:

df[ (df.Subject == "CS") & (df.Number < 300) ]
# [ (First Conditional) AND (Second Conditional) ]

Both the first (Subject is exactly equal to "CS") and second (Number is less than 300) conditionals are checked independently. Since an AND (&) is used to join these two conditionals, the final truth value if True only when both conditionals are True:

df[ (df.Subject == "CS") & (df.Number < 300) ]

Python allows us to continue to apply conditionals together infinitely long -- so it's no problem to have three conditionals:

df[ (df.Subject == "CS") & (df.Number < 300) & (df["Credit Hours"] == "3 hours.") ]

Multiple Conditionals Joined with OR (|)

Alternatively, Python can combine two conditionals together and keep the result when either the first conditional OR the second conditional is True (this includes when they're both True as well!). There are two major applications when this is useful:

1. Selecting multiple values of data from the same column (ex: all courses in "ARTD" OR "ARTE" OR "ARTF").
2. Selecting multiple values from different columns and keeping all matches (ex: all courses in "PSYC" OR courses that are only 1 credit hour).

Selecting Multiple Values of Data from the Same Column

Looking at the first example above, the University of Illinois has a lot of courses in art across many different sub-areas of art including: Art Design ("ARTD"), Art Education ("ARTE"), Art Foundation ("ARTF"), Art History ("ARTH"), and Art Studio ("ARTS").

To include ALL courses from all five sub-areas of art listed above, we must join them together with an OR (|). Notice that it is necessary to specify each conditional completely each time even though we are always comparing the subject since Python has to evaluate each conditional independently and then combine the results together:

df[ (df.Subject == "ARTD") | (df.Subject == "ARTE") | (df.Subject == "ARTF") | (df.Subject == "ARTH") | (df.Subject == "ARTS") ]

Selecting Multiple Values from Different Columns and Keeping All Matches

To be considered a "full-time student" at most universities, you must be enrolled in at least 12 credit hours. If you are only enrolled in 11 credit hours, you may be interested in any course that will bump you up to exactly 12 credit hours (ex: a course worth exactly one credit hour) or a course you may be interested in (ex: something from the psychology ("PSYC") department).

To include ALL of the results of all courses that are either one credit hour OR in the psychology department, we need an OR:

df[ (df["Credit Hours"] == "1 hours.") | (df["Subject"] == "PSYC") ]

Combining ANDs and ORs

The most complex conditionals will require a combination of both AND and OR statements. These can get incredibly tricky, but we can remember that Python will always process conditionals by only combining two conditionals together at a time.

Since Python combines only two conditionals together at any given time, it is critical we use parenthesis to ensure we specify the order that we want these conditionals combined. For example, let's explore only junior level (300-399) courses in Chemistry or Physics. To do so:

• The subject of the course must be CHEM or PHYS.
• The course number must be greater than or equal to 300.
• The course number must also be less than 400.

Naively writing this conditional results in the following code:

df[ (df.Subject == "CHEM") | (df.Subject == "PHYS") & (df.Number >= 300) & (df.Number < 400) ]

Default Order of Evaluation: AND before OR

If we do not use additional parenthesis, Python will always combine the ANDs first and then the ORs and will do so in left-to-right order. This means that:

1. The first set of two conditionals combined will be the first AND conditional: (df.Subject == "PHYS") & (df.Number >= 300). The result contains all courses in PHYS with a number larger than 300.

2. The second set of two conditionals will be the result from #1 with the second AND: (Result of Step #1) & (df.Number < 400). The result contains all courses in PHYS with a number from 300-399.

3. The final set of conditionals will be combined using OR: (df.Subject == "CHEM") | (Result of Step #2). Since this is an OR, the result is ALL CHEM courses and then only the PHYS courses in the number 300-399.

We can verify our result by running the code:

df[ (df.Subject == "CHEM") | (df.Subject == "PHYS") & (df.Number >= 300) & (df.Number < 400) ]

Notice that the code appears correct until we scroll down! The courses in Chemistry start at 300, but the last five rows show us that the courses in Physics include 500-level courses -- yikes!

Order of Evaluation: Using Parenthesis to Specify Order

Python uses parenthesis in a similar way to basic mathematics where the inner-most operations are done first. In our example, we want to make sure that all Chemistry and Physics courses are combined first, and only then can we limit the range of course numbers to the junior level.

By grouping both of these logical operations together, our new conditional can be thought of as a combination of two complex conditionals:

1. (df.Subject == "CHEM") | (df.Subject == "PHYS"), selecting only that are Chemistry OR Physics

2. (df.Number >= 300) & (df.Number < 400), selecting only courses between 300 AND 399.

Joining these two conditionals together with an AND results in the exact output we expect:

df[ ((df.Subject == "CHEM") | (df.Subject == "PHYS")) & ((df.Number >= 300) & (df.Number < 400)) ]

Example Walk-Throughs with Worksheets

Video 1: DataFrame Conditionals using The Party Dataset

Video 2: DataFrame Conditionals using The Berkeley Dataset

Video 3: DataFrame Conditionals using The Course Catalog Dataset

Practice Questions