---
title: "HW6: Data Wrangling + TidyTuesday"
execute: 
  warning: false
  code: true
fig-height: 3
fig-width: 5
fig-align: center
code-fold: false
number-sections: false
---

::: {.callout-caution title="Learning Goals"}
- Sharpen your data wrangling and data viz skills by practicing on a different dataset.
- Engage with the larger data science community by working on a recent TidyTuesday dataset.
:::


## Kiva



[Kiva](https://www.kiva.org/) is a non-profit organization that allows people from around the world to lend small amounts to others to start or grow a business, go to school, access clean energy, etc. Since its founding in 2005, more than \$1.2 billion in loans to over 3 million borrowers have been funded. In the following exercises, we'll examine some Kiva data from 2005-2012. 

### Exercise 1: Kiva partners

To begin, let's explore data on Kiva's *field partners*. These partners act as intermediaries between Kiva (the lenders) and borrowers. They evaluate borrower risk, post loan requests on Kiva, and process payments. Load data on the field partners below. A codebook with variable descriptions is [here](https://mac-stat.github.io/data/kiva_partners2_codebook.xlsx)

```{r warning = FALSE, message = FALSE}
# Load the tidyverse
library(tidyverse)

# Load the data
partners <- read_csv("https://mac-stat.github.io/data/kiva_partners2.csv")
```

#### Part a

Let's get to know the data.

```{r}
# Calculate the lowest, median, and highest total amount raised by any partner


# Identify the 6 partners that have raised the highest total amount
# Show just the partner names, countries, and total amount raised


# Show the names of the partners in Bolivia
# (Don't include any other variables, just the partner names)


```

#### Part b

Create a new table with only five columns:

-   `countries.region`
-   `total_partners` = total number of partners per region
-   `total_loans` = total number of loans posted per region
-   `total_amount` = total amount raised per region
-   `average_loan` = average loan size per loan posted in each region (calculated as total amount raised per region / total number of loans posted per region)

Print the entire table, sorted from high to low with respect to `total_amount` raised. Note that

-   Your table should have 7 rows and 5 columns.
-   Your first row should have `countries.region = Asia`, `total_partners = 40`, `total_loans = 133060`, `total_amount = 84816225`, `average_loan = 637`.

```{r}

```

#### Part c

Identify two things that you learned from the table in Part b. Just pick whatever you found most interesting.

::: {.callout-note title="Answer"}

PLACE YOUR ANSWER IN THIS CALLOUT BOX

:::

#### Part d

Draw a map that includes a dot for each of Kiva's `partners`. Color the dot corresponding to the total amount raised by the partner. NOTE: It's easier to do this on a static map, than with leaflet.

```{r warning = FALSE, message = FALSE, fig.width = 8}
library(rnaturalearth)
library(mosaic)

# Get a background map of the entire world
# world_boundaries <- ___

# Plot the partner locations on the background map
# ggplot(___) + 
#   geom___() + 
#   geom_point(
#     data = ___,
#     aes(x = ___, y = ___, color = ___)
#   ) +
#   theme_map()
```




### Exercise 2: Kiva loans (Part 1)

The `loans` data contains information on a sample of 10,000 individual loans to borrowers:

```{r}
# a random sample of 10,000 loans
loans <- read_csv("https://mac-stat.github.io/data/kiva_loans_small.csv")
```

View the `loans` table, browse through some of the data, and check out the [codebook](https://mac-stat.github.io/data/kiva_loans_codebook.xlsx). Before working with this data, we have to do some pre-processing / wrangling. Take the following steps and store the results as `loans_2`.

1.  Only keep the loans that have a positive `funded_amount`.

2.  The information about *when* a loan request was posted is separated out into different fields for year, month, day, hour, etc. Combine some of this information into a single variable that records the exact posting time. Do this in three steps. NOTE: You'll get some warning messages, but these are not errors.

    -   Define a new variable which records the exact date a loan request was posted by *pasting* together the year, month, and date of the request, separated by hyphens: `post_dt = paste(posted_yr, posted_mo, posted_day, sep = '-')`
    -   Define a new variable which records the exact time of day a loan request was posted by *pasting* together the hour and minute of the request, separated by a colon:\
        `post_time = paste(posted_hr, posted_min, posted_sec, sep = ':')`.
    -   Define a new variable which combines the exact date and time of day a loan request was posted:\
        `post_date = ymd_hms(paste(post_dt, post_time, sep = ' '))`

3.  Similar to `post_date`, define a new variable called `fund_date` that reports the exact date and time at which each loan was *funded* (not *posted*).

4.  Define a new variable called `days_to_fund = difftime(fund_date, post_date, units = "days")` which records the number of days between the time a loan was posted and the time it was funded.

5.  Get information about the `countries.region` for the partner of each loan from the `partners` dataset. NOTE: `partners` and `loans` have many variable names in common. To avoid conflicts, simplify the `partners` dataset to just 2 columns before connecting it to `loans`.

6.  Keep only the following columns: `loan_id, status, funded_amount, paid_amount, sector, countries.region, location.country, lat, lon, partner_id, post_date, fund_date, days_to_fund`

```{r}
# Define loans_2


# Confirm that loans_2 has 9884 rows and 13 columns


```


### Exercise 3: Kiva loans (Part 2)

#### Part a

```{r}
# Show the top 5 countries by number of loans


# Show the top 5 countries by total funded loan amount


```

#### Part b

Plot the mean loan size in each sector (y-axis) vs the number of loans in each sector (x-axis). Represent each sector by its name (text), not a point. HINT: You'll have to calculate the mean loan size and number of loans in each sector before you can plot them.



### Optional Exercise: funding time

NOTE: This exercise won't be graded, but it is strongly recommended as additional practice for the quiz.

How many days does it take borrowers to get a loan funded? Let's explore the `days_to_fund` variable in the `loans_2`.

#### Part a

Construct a univariate visualization of `days_to_fund`.

```{r}

```

#### Part b

Construct a plot of `days_to_fund` (y-axis) vs `funded_amount` (x-axis) for each loan in `loans_2`.

```{r}

```

#### Part c

Construct a plot of `days_to_fund` (y-axis) vs `funded_amount` (x-axis) in each `countries.region`. Don't represent each loan on this plot. Rather, include curves to represent the trends in each region.

```{r}

```

#### Part d

Summarize, in words, some takeaway messages from parts a-c about how long it takes to get a loan funded.



### Optional Exercise: loan status

NOTE: This exercise won't be graded, but it is highly recommended as additional practice for the quiz.

The `status` variable in `loans_2` indicates the status of a loan:

```{r}
# Check out a table summary after you've defined loans_2
# loans_2 %>% 
#   count(status)
```

Let's focus here on the patterns in loans that were paid back and those that were defaulted, i.e. not paid back.

#### Part a

Define a new dataset, `defaults`, that only includes loans that were either defaulted or paid.

```{r}
# Define defaults

# Confirm that defaults has 7010 rows and 13 columns

```

#### Part b

Using `defaults`, construct a visualization of the relationship between the funded amount and status of a loan.

```{r}

```

#### Part c

Define a new dataset with four columns:

-   `partner_id`
-   number of defaulted loans through that partner
-   number of loans completely paid back through that partner
-   percentage of loans defaulted

Sort your table from highest default percentage to lowest, and print out only those with at least a 50% default percentage. HINT: You'll have to reshape the data in this process.

```{r}

```

#### Part d

Provide some take-away messages about loan defaults using your results in parts a-d.





## TidyTuesday

Tidy Tuesday is a weekly data project put on by some folks from the R Data Science community. Each week, a different data set is posted and people around the world wrangle and visualize that data. According to the organizers, "The intent of Tidy Tuesday is to provide a safe and supportive forum for individuals to practice their wrangling and data visualization skills independent of drawing conclusions."


The goals of this TidyTuesday are:

1.  Practice generating questions. You have to decide what to ask and how to answer it with a graphic.
2.  Practice identifying what viz and wrangling tools are useful for addressing your questions.
4.  Get a sense of the broader data science community. Check out what people share out on [X/Twitter #TidyTuesday](https://x.com/search?q=%23TidyTuesday). Maybe even share your own #TidyTuesday work on social media. Recent Mac alum [Erin Franke (X/Twitter)](https://x.com/efranke7282) has an inspiring account! Scrolling through, you'll notice the trajectory of her work, starting from COMP/STAT 112 to today. Very cool.



### Exercise 4

Go to [TidyTuesday](https://github.com/rfordatascience/tidytuesday). Pick a dataset that was posted in the **last 4 weeks**. Here, include:

- A short (\~2 sentence) written description of your data. This should include: 
  - the original data source, ie, where did TidyTuesday get the data from?
  - units of observation, ie, what are you analyzing?, and 
  - the data size, ie, how many data points do you have? how many variables are measured on each data point?.

::: {.callout-note title="Answer"}

PLACE YOUR ANSWER IN THIS CALLOUT BOX

:::

- Code to import and examine the basic properties of your chosen dataset. This code must support the facts you cited in your short written description.

```{r}

```

### Exercise 5

In the 3 sections below (Viz 1, 2, 3), construct 3 separate graphs that tell a *connected* story about the data you chose.  

**Before each viz:**

-   write a simple but specific research question you're trying to address with the viz.
-   write a 2-4 sentence summary of what you learn from the viz. This should connect back to your research question!

**After each viz:**

-   Comment on at least 2 effective aspects of the viz (consider the effective visualization principles).
-   Comment on at least 2 aspects of the visualization that could be improved. Perhaps these are aspects that you don't know how to implement yet but wish you could update it.

**Make sure each viz:**

-   has meaningful axis labels and legend titles
-   has a figure caption (fig.cap)
-   uses alt text (fig.alt)
-   uses a more color-blind friendly color palette


**Tips:**

-   Start with some questions in mind of what you want to learn.
-   Start with a simple viz (viz 1), and build this up into something multivariate (viz 3).
-   Reflect on each viz -- what new questions do you have after checking out the viz? Let these questions guide your next viz.

**Expectations:**

- You should be wrangling data throughout your visual analysis. For example, perhaps you'll need to wrangle the data before plotting it. Perhaps wrangling / summaries will help address some follow-up questions you have from your viz. 
- At least some of this wrangling should demonstrate the combination of multiple verbs, i.e. span multiple lines.

#### Viz 1

::: {.callout-note title="Research Question"}

PLACE YOUR RESEARCH QUESTION IN THIS CALLOUT BOX

:::

```{r}

```

::: {.callout-note title="Summary"}

PLACE YOUR SUMMARY IN THIS CALLOUT BOX

:::

#### Viz 2

::: {.callout-note title="Research Question"}

PLACE YOUR RESEARCH QUESTION IN THIS CALLOUT BOX

:::

```{r}

```

::: {.callout-note title="Summary"}

PLACE YOUR SUMMARY IN THIS CALLOUT BOX

:::

#### Viz 3

::: {.callout-note title="Research Question"}

PLACE YOUR RESEARCH QUESTION IN THIS CALLOUT BOX

:::

```{r}

```

::: {.callout-note title="Summary"}

PLACE YOUR SUMMARY IN THIS CALLOUT BOX

:::



## Finalize Work

Congratulation 🎉.  You're done with the homework.  See the instruction at top of the Homework Assignments page for how to submit.
