7 Adv Data wrangling P2

🚩 Pre-Class Learnings

To prepare for this lesson, do the followings:

  • Read Chapter 14, 15, and 19 in D4DS
  • Do checkpoint in Moodle

🔥 Data Story Critique

Go to https://geonarrative.usgs.gov/muledeer255/ then answer the following questions:

  • What is the data story?
  • What is effective?
  • What could be improved?
ImportantICA Instructions

Before starting, review the ICA Instructions ⭐ for details on pair programming and activity procedures.

🧩 Learning Goals

By the end of this lesson, you should be able to:

  • Manipulate and explore strings using the stringr package
  • Construct regular expressions to find patterns in strings

Helpful Cheatsheets

The stringr cheatsheet (HTML, PDF) will be useful to have open and reference.

Motivation: 30 Years of American Anxieties

In 2018 the data journalism organization The Pudding featured a story called 30 Years of American Anxieties about themes in 30 years of posts to the Dear Abby column (an American advice column).

One way to understand themes in text data is to conduct a qualitative analysis, a methodology in which multiple readers read through instances of text several times to reach a consensus about themes.

Another way to understand themes in text data is computational text analysis.

  • This is what we will explore today.

Both qualitative analysis and computational tools can be used in tandem. Often, using computational tools can help focus a close reading of select texts, which parallels the spirit of a qualitative analysis.

To prepare ourselves for a computational analysis, let’s learn about strings.

Strings

Strings are objects of the character class (abbreviated as <chr> in tibbles).

When you print out strings, they display with double quotes:

some_string <- "banana"
some_string

Working with strings generally will involve the use of regular expressions, a tool for finding patterns in strings.

Regular expressions (regex, for short) look like the following:

"^the" (Strings that start with "the")
"end$" (Strings that end with "end")

Before getting to regular expressions, let’s go over some fundamentals about working with strings. The stringr package (available within tidyverse) is great for working with strings.

Creating strings

Creating strings by hand is useful for testing out regular expressions.

To create a string, type any text in either double quotes (") or single quotes '. Using double or single quotes doesn’t matter unless your string itself has single or double quotes.

string1 <- "This is a string"
string2 <- 'If I want to include a "quote" inside a string, I use single quotes'
string3 <- c(string1, string2) # string / character vector (of greater than length 1)


class(string1)
class(string2)
class(string3)

length(string1)
length(string2)
length(string3)

We can view these strings “naturally” (without the opening and closing quotes) with str_view():

str_view(string1)
str_view(string2)
str_view(string3)

Exercise: Create the string It's Thursday. What happens if you put the string inside single quotes? Double quotes?

# Your code

Because " and ' are special characters in the creation of strings, R offers another way to put them inside a string. We can escape these special characters by putting a \ in front of them:

string1 <- "This is a string with \"double quotes\""
string2 <- "This is a string with \'single quotes\'"
str_view(string1)
str_view(string2)

Given that \ is a special character, how can we put the \ character in strings? We have to escape it with \\.

Exercise: Create the string C:\Users. What happens when you don’t escape the \?

# Your code

Other special characters include:

  • \t (Creates a tab)
  • \n (Creates a newline)

Both can be useful in plots to more neatly arrange text.

string1 <- "Record temp:\t102"
string2 <- "Record temp:\n102"

str_view(string1)
str_view(string2)

Can we get str_view() to show the tab instead of {\t}? We can use the html argument to have the string displayed as if on a webpage:

str_view(string1, html = TRUE)

Often we will want to create new strings within data frames. We can use str_c() or str_glue(), both of which are vectorized functions (meaning they take vectors as inputs and provide vectors as outputs - can be used within mutate()):

  • With str_c() the strings to be combined are all separate arguments separated by commas.
  • With str_glue() the desired string is written as a template with variable names inside curly braces {}.
df <- tibble(
    first_name = c("Arya", "Olenna", "Tyrion", "Melisandre"),
    last_name = c("Stark", "Tyrell", "Lannister", NA)
)
df

df |>
    mutate(
        full_name1 = str_c(first_name, " ", last_name),
        full_name2 = str_glue("{first_name} {last_name}")
    )

Exercise: In the following data frame, create a full date string in month-day-year format using both str_c() and str_glue().

df_dates <- tibble(
    year = c(2000, 2001, 2002),
    month = c("Jan", "Feb", "Mar"),
    day = c(3, 4, 5)
)

Extracting information from strings

The str_length() counts the number of characters in a string.

comments <- tibble(
    name = c("Alice", "Bob"),
    comment = c("The essay was well organized around the core message and had good transitions.", "Good job!")
)

comments |>
    mutate(
        comment_length = str_length(comment)
    )

The str_sub() function gets a substring of a string. The 2nd and 3rd arguments indicate the beginning and ending position to extract.

  • Negative positions indicate the position from the end of the word. (e.g., -3 indicates “3rd letter from the end”)
  • Specifying a position that goes beyond the word won’t result in an error. str_sub() will just go as far as possible.
x <- c("Apple", "Banana", "Pear")

str_sub(x, start = 1, end = 3)
str_sub(x, start = -3, end = -1)
str_sub(x, start = 2, end = -1)
str_sub("a", start = 1, end = 15)

Exercise: Using str_sub(), create a new variable with only the middle letter of each word in the data frame below. (Challenge: How would you handle words with an even number of letters?)

df <- tibble(
    word_id = 1:3,
    word = c("replace", "match", "pattern")
)

Finding patterns in strings with regular expressions

Suppose that you’re exploring text data looking for places where people describe happiness. There are many ways to search. We could search for the word “happy” but that excludes “happiness” so we might search for “happi”.

Regular expressions (regex) are a powerful language for describing patterns within strings.

. . .

data(fruit)
data(words)
data(sentences)

We can use str_view() with the pattern argument to see what parts of a string match the regex supplied in the pattern argument. (Matches are enclosed in <>.)

str_view(fruit, "berry")

Essentials of forming a regex

  • Letters and numbers in a regex are matched exactly and are called literal characters.
  • Most punctuation characters, like ., +, *, [, ], and ?, have special meanings and are called metacharacters.
  • Quantifiers come after a regex and control how many times a pattern can match:
    • ?: match the preceding pattern 0 or 1 times
    • +: match the preceding pattern at least once
    • *: match the preceding pattern at least 0 times (any number of times)

. . .

Exercise: Before running the code below, predict what matches will be made. Run the code to check your guesses. Note that in all regex’s below the ?, +, * applies to the b only (not the a).

str_view(c("a", "ab", "abb"), "ab?")
str_view(c("a", "ab", "abb"), "ab+")
str_view(c("a", "ab", "abb"), "ab*")
  • We can match any of a set of characters with [] (called a character class), e.g., [abcd] matches “a”, “b”, “c”, or “d”.
    • We can invert the match by starting with ^: [^abcd] matches anything except “a”, “b”, “c”, or “d”.
# Match words that have vowel-x-vowel
str_view(words, "[aeiou]x[aeiou]")

# Match words that have not_vowel-y-not_vowel
str_view(words, "[^aeiou]y[^aeiou]")

Exercise Using the words data, find words that have two vowels in a row followed by an “m”.

# Your code
  • The alternation operator | can be read just like the logical operator | (“OR”) to pick between one or more alternative patterns. e.g., apple|banana searches for “apple” or “banana”.
str_view(fruit, "apple|melon|nut")

Exercise: Using the fruit data, find fruits that have a repeated vowel (“aa”, “ee”, “ii”, “oo”, or “uu”.)

# Your code
  • The ^ operator indicates the beginning of a string, and the $ operator indicates the end of a string. e.g., ^a matches strings that start with “a”, and a$ matches words that end with “a”.
  • Parentheses group together parts of a regular expression that should be taken as a bundle. (Much like parentheses in arithmetic statements.)
    • e.g., ab+ is a little confusing. Does it match “ab” one or more times? Or does it match “a” first, then just “b” one or more times? (The latter, as we saw in an earlier example.) We can be very explicit and use a(b)+.

Exercise: Using the words data, find (1) words that start with “y” and (2) words that don’t start with “y”.

# Your code

Exploring stringr functions

Read in the “Dear Abby” data underlying The Pudding’s 30 Years of American Anxieties article.

posts <- read_csv("https://raw.githubusercontent.com/the-pudding/data/master/dearabby/raw_da_qs.csv")

Take a couple minutes to scroll through the 30 Years of American Anxieties article to get ideas for themes that you might want to search for using regular expressions.


The following are core stringr functions that use regular expressions:

  • str_view() - View the first occurrence in a string that matches the regex
  • str_count() - Count the number of times a regex matches within a string
  • str_detect() - Determine if (TRUE/FALSE) the regex is found within string
  • str_subset() - Return subset of strings that match the regex
  • str_extract(), str_extract_all() - Return portion of each string that matches the regex. str_extract() extracts the first instance of the match. str_extract_all() extracts all matches.
  • str_replace(), str_replace_all() - Replace portion of string that matches the regex with something else. str_replace() replaces the first instance of the match. str_replace_all() replaces all instances of the match.
  • str_remove(), str_remove_all() - Removes the portion of the string that matches the pattern. Equivalent to str_replace(x, "THE REGEX PATTERN", "")

Exercise: Starting from str_count(), explore each of these functions by pulling up the function documentation page and reading through the arguments. Try out each function using the posts data.

Solutions

Creating strings

Solution 
x <- "It's Thursday" # We need double quotes because of the apostrophe
x
x <- 'It's Thursday'
x <- "C:\\Users"
str_view(x)
# \U is the start of special escape characters for Unicode characters
# The \U is expected to be followed by certain types of letters and numbers--like \U0928
x <- "C:\Users"
df_dates <- tibble(
    year = c(2000, 2001, 2002),
    month = c("Jan", "Feb", "Mar"),
    day = c(3, 4, 5)
)

df_dates |>
    mutate(
        date1 = str_c(month, "-", day, "-", year),
        date2 = str_glue("{month}-{day}-{year}")
    )

Extracting information from strings

Solution 
df <- tibble(
    word_id = 1:3,
    word = c("replace", "match", "pattern")
)

df |>
    mutate(
        word_length = str_length(word),
        middle_pos = ceiling(word_length/2),
        middle_letter = str_sub(word, middle_pos, middle_pos)
    )

Finding patterns in strings with regular expressions

Solution 
# This regex finds "a" then "b" at most once (can't have 2 or more b's in a row)
str_view(c("a", "ab", "abb"), "ab?")
# There has to be an "a" followed by at least one b
# This is why the first string "a" isn't matched
str_view(c("a", "ab", "abb"), "ab+")
# There must be an "a" and then any number of b's (including zero)
str_view(c("a", "ab", "abb"), "ab*")
str_view(words, "[aeiou][aeiou]m")
str_view(fruit, "aa|ee|ii|oo|uu")
# Words that start with y
str_view(words, "^y")

# Words that don't start with y
str_view(words, "^[^y]")