An oversimplified categorization:

• Data science produces insights
• Machine learning produces predictions
• Artificial intelligence produces actions

(From David Robinson)

So, how do we get insights?

(Image credit: Hadley Wickham)

Pull if you haven't already and fire up RStudio.

1. Look left.
2. Look right.
3. Pick a partner (groups of three are fine).
4. Give them a friendly, professional smile.
5. Maybe a handshake too.

We'd like to fit a smoother to each listing to long-term trend. We can fit a single one like this:

model_data <- prices %>%       # extract a single listing's 
    filter(listing_id == 5506) # worth of data

# fit a loess model, the span is a hyperparameter, a bit 
# like lambda in LASSO
single_model <-  loess(price_per ~ as.numeric(date),  
                       data = model_data, 
                       span = .5)

model_preds <- broom::augment(single_model, model_data) 
model_preds %>% head()

##   listing_id       date price_per  .fitted  .se.fit     .resid
## 1       5506 2017-09-05      72.5 73.73994 2.042531 -1.2399385
## 2       5506 2017-09-04      72.5 73.66160 1.984215 -1.1615956
## 3       5506 2017-09-03      72.5 73.58551 1.927151 -1.0855140
## 4       5506 2017-09-02      72.5 73.51158 1.871376 -1.0115812
## 5       5506 2017-09-01      72.5 73.43968 1.816931 -0.9396847
## 6       5506 2017-08-31      72.5 73.36971 1.763855 -0.8697121

model_container <- ????
for(id in unique(prices$listing_id)){
    model <- prices %>% 
        filter(listing_id == id) %>% 
        loess(price_per ~ as.numeric(date),
              data = .,
              span = .25)
    
    model_container %>% update(model) # ?????
    .
    .
    .
}

my_list <- list('To', 'boldly', 'go', 'where', 'no', 'man', 'has', 'gone', 'before')

length_list <- list()
i           <- 1
for(word in my_list){
    length_list[i] <- nchar(word)
    i              <- i + 1
}

length_list %>% unlist() # just for display

## [1] 2 6 2 5 2 3 3 4 6

my_list <- list('To', 'boldly', 'go', 'where', 'no', 'man', 'has', 'gone', 'before')

length_list <- map(my_list, nchar) # or my_list %>% map(nchar)

length_list %>% unlist() # just for display

## [1] 2 6 2 5 2 3 3 4 6

purrr::map() applies a function (nchar) to each entry of the original list.

Use map to apply the same operation to a list of objects. Use reduce to combine them all together.

# product of the number of letters in each word
my_list %>% 
    map(nchar) %>% 
    reduce(`*`) # `*` is the product function: `*`(a,b) = a*b

## [1] 51840

Use lists of data frames (and data frames of lists) to organize your work.

Data frames are the fundamental unit of data science.

Usually their columns are atomic vectors of integers, doubles, dates, characters, or booleans. E.g.

## # A tibble: 6 x 3
##   listing_id       date price_per
##        <int>     <date>     <dbl>
## 1    3075044 2017-08-22      32.5
## 2    3075044 2017-08-21      32.5
## 3    3075044 2017-08-20      32.5
## 4    3075044 2017-08-19      37.5
## 5    3075044 2017-08-18      37.5
## 6    3075044 2017-08-17      32.5

But this is somewhat inflexible. What about more complex objects? Lists can hold anything…

prices_nested <- prices %>% 
    tidyr::nest(-listing_id)

# view the data types of the columns
map(prices_nested, class) %>% unlist()

## listing_id       data 
##  "integer"     "list"

prices_nested %>% head()

## # A tibble: 6 x 2
##   listing_id               data
##        <int>             <list>
## 1    3075044 <tibble [359 x 2]>
## 2       6976 <tibble [319 x 2]>
## 3    7651065 <tibble [334 x 2]>
## 4    5706985 <tibble [344 x 2]>
## 5    2843445 <tibble [365 x 2]>
## 6     753446 <tibble [347 x 2]>

prices_nested$data[[1]] # get the first item of the list

## # A tibble: 359 x 2
##          date price_per
##        <date>     <dbl>
##  1 2017-08-22      32.5
##  2 2017-08-21      32.5
##  3 2017-08-20      32.5
##  4 2017-08-19      37.5
##  5 2017-08-18      37.5
##  6 2017-08-17      32.5
##  7 2017-08-16      32.5
##  8 2017-08-15      32.5
##  9 2017-08-14      32.5
## 10 2017-08-13      32.5
## # ... with 349 more rows

Now do the same thing, but assign the result to a new list column of prices_nested.

prices_nested %>% 
    mutate(highest_price = map_dbl(data, get_biggest_price))

## # A tibble: 5 x 3
##   listing_id               data highest_price
##        <int>             <list>         <dbl>
## 1    3075044 <tibble [359 x 2]>      37.50000
## 2       6976 <tibble [319 x 2]>      32.50000
## 3    7651065 <tibble [334 x 2]>      39.50000
## 4    5706985 <tibble [344 x 2]>      66.66667
## 5    2843445 <tibble [365 x 2]>      37.50000

We know how to:

1. Nest a data frame, creating a list of data frames.
2. Use map to apply a function to every element of a list.
3. Apply a model function to an individual data frame (from Session 3).

Looks like it's time for…

my_loess <- function(data, span){
    loess(price_per ~ as.numeric(date),
    data = data, 
    span = span)
}

prices_nested <- prices %>% 
    nest(-listing_id) 

prices_modeled <- prices_nested %>% 
    mutate(model = map(data, my_loess, span = .25))

prices_with_preds <- prices_modeled %>% 
    mutate(preds = map2(model, data, augment))

prices_unnested <- prices_with_preds %>% 
    unnest(preds)

my_loess <- function(data, span){
    loess(price_per ~ as.numeric(date),
    data = data, 
    span = span)
}

prices_unnested <- prices %>% 
    nest(-listing_id) %>% 
    mutate(model = map(data, my_loess, span = .25),
           preds = map2(model, data, augment)) %>% 
    unnest(preds)

• Remove the seasonal (long term) and weekly (short term) trends, leaving the April bump.
• Separate out individual listings using k-means.

• Data science is nonlinear – learn to love the cycle.
• Modeling isn't always the final step – sometimes it's a cleaning step for the analysis you really want to do.
• The tidyverse set of packages gives you a powerful toolbox for accomplishing complex tasks with simple code.

• Illustrative vignettes for purrr and broom.
• Jenny Bryan's Stat 545 course website has excellent coverage of all our R topics, including purrr.
• Our K-means example borrowed a lot from this broom vignette.

• R For Data Science is a book by Garrett Grolemund and Hadley Wickham that goes deep into a complete suite of tools for the data science pipeline.

• Automating long data science pipelines with GNU make.
• Literate programming with RMarkdown. These slides, as well as the "pretty" lecture notes, were made in R!
• Interactive data applications with Shiny
• Develop your own R packages.
• Structure of the R language – check out Advanced R.