Research Notebook: Acquiring NCAA Baseball Stats with R

by Bill Petti
November 29, 2016

It’s true, you too can pull college stats into R! (via Minda Haas Kuhlmann)

In the latest release of my baseballr package I’ve included a function that makes it easy for a user to pull data from the NCAA’s website for baseball teams across the three major divisions (I, II, III).

The function, ncaa_scrape, requires the user to pass the function three parameters:

school_id: Numerical code used by the NCAA for each school
year: A four-digit year
type: Whether to pull data for batters or pitchers

For the latter two, the inputs are easy. The issue for most will be knowing what the school_id is that the NCAA website uses for the school they are interested in. To help, I decided to include a massive lookup table in the package so a user could easily identify the necessary school_id.

I thought it would be helpful to walk through how I built that file and then show how to use the ncaa_scrape function to start acquiring actual statistics.

Building a lookup table for all NCAA schools

Let’s load the following R packages:

require(baseballr)
require(dplyr)
require(rvest)
require(xml2)

The first thing we need to do is build a function that allows us to obtain the numerical codes that the NCAA uses for each of the conferences across their three divisions.

If you look at the url structure used at the site, there is a parameter–conf_id–that allows you to view conferences and the teams associated with each conference individually:

http://stats.ncaa.org/team/inst_team_list?academic_year=2016&conf_id=-1&division=1&sport_code=MBA

When set to -1 the site will show all schools in a given division. The only list I could find that details each conference is a javascript drop down that becomes available when you hover over “ALL TEAMS.”

Let’s write a function to grab the urls and url names for all conferences from the site:

conference_table <- function(year, div) {
  url <- paste0("http://stats.ncaa.org/team/inst_team_list?academic_year=", year, "&conf_id=-1&division=", div, "&sport_code=MBA")
  read <- read_html(url)
  links <- html_nodes(read, "a") %>%
    html_attr("href")
  link_names <- html_nodes(read, "a") %>%
    html_text()
  table <- as.data.frame(cbind(link_names, links))
  names(table) <- c("conference", "links")
  table$conference <- as.character(table$conference)
  links_conferences <- table %>%
    filter(grepl("changeConference", links))
  conference_ids <- sub("\\).*", "", sub(".*\\(", "", links_conferences$links))
  conference_ids <- as.data.frame(conference_ids)
  names(conference_ids) <- "conference_id"
  table <- cbind(links_conferences, conference_ids)
  table <- table %>%
    mutate(year = year, division = div, conference_id = as.numeric(as.character(conference_id))) %>%
    select(year, division, everything(), -links) %>% .[-1,]
  table
}

The function needs the user to pass two values: a four-digit year and the division they are interested in (either 1, 2, or 3).

The function takes those inputs and constructs the url based on the inputs. After the html is read and parsed we can extract the names of the conferences and then the links attached to each conference (link_names and links, respectively, in the function).

Now, this is pulling all the links available on the page, so the next step is to combine these two sets of data into a single table and then filter out irrelevant links. Links that include “changeConference” will have the numerical conference codes that we are looking for. After a little string manipulation we will be left with a table that looks like this (for division 1 in 2016):

Now, in some cases the composition of conferences can change from prior years. In order to account for this we can loop over each year and division combination to ensure that we know which schools were members of which conference each year.

A Hardball Times Update

by RJ McDaniel

Goodbye for now.

To do that we construct a table with all year/division combinations that we are interested in:

year <- c(2013, 2014, 2015, 2016)
division <- c(1,2,3)
div_yr <- expand.grid(year, division)

Then, we group the div_yr table by year and division (Var1 and Var2 respectively) and use the do function to loop over all the possible combinations and combine the results into a single table:

conference_code_lu <- div_yr %>% 
  group_by(Var1, Var2) %>%
  do(conference_table(.$Var1, .$Var2))

The result will look like this:

Next, we need another function that will scrape the teams and the team urls within each conference. You’ll notice it looks very similar to the conference_table function:

teams <- function(year, conference, div) {
    url <- paste0("http://stats.ncaa.org/team/inst_team_list?academic_year=", year, "&conf_id=", conference, "&division=", div, "&sport_code=MBA")
    read <- read_html(url)
    links <- html_nodes(read, "a") %>%
      html_attr("href")
    link_names <- html_nodes(read, "a") %>%
      html_text()
    table <- as.data.frame(cbind(link_names, links))
    table$links <- as.character(table$links)
    table$link_names <- as.character(table$link_names)
    table <- table %>%
      filter(grepl("team", links)) %>%
      filter(!grepl("inst_team", links)) %>%
      filter(!grepl("schedule", links))
    table$links <- gsub("/team/", "", table$links)
    table$links <- sub("/.*", "", table$links)
    table$year <- year
    table$division <- div
    table$conference_id <- conference
    names(table) <- c("school", "school_id", "year", "division", "conference_id")
    table
}

The major difference is this function also takes the conference_id we just acquired and customizes the url so that we are only scraping the available data for that conference. That allows us to ensure that teams are associated with the right conferences in a given year.

Armed with this function, we can now loop over our conference_code_lu table and apply the new function to all the possible values:

master_ncaa_team_lu <- conference_code_lu %>% 
  group_by(year, division, conference, conference_id) %>% 
  do(teams(.$year, .$conference_id, .$division)) %>%
  ungroup() %>%
  select(school, conference, everything()) %>%
  mutate(school_id = as.numeric(school_id)) %>%
  arrange(school)

The result is a tidy lookup table that includes each school, their conference, both their school and conference IDs, as well as the year and the division:

Acquiring NCAA Team Statistics

Now that we have our lookup table, acquiring actual NCAA baseball statistics is a piece of cake.

Let’s say we want all batter statistics for Vanderbilt from the 2015 season. To acquire the data you simply feed the ncaa_scrape function the relevant information:

ncaa_scrape(736, 2015, "batting")

Want statistics for Vanderbilt’s pitchers? No problem, just change the type parameter to “pitching”:

> ncaa_scrape(736, 2015, "batting") %>%
+     select(year:SlgPct)
   year     school   conference division Jersey            Player Yr Pos GP GS    BA OBPct
1  2015 Vanderbilt Southeastern        1     43      Wiel, Zander Jr INF 72 71 0.316 0.406
2  2015 Vanderbilt Southeastern        1     20   Reynolds, Bryan So  OF 72 71 0.318 0.388
3  2015 Vanderbilt Southeastern        1     10      Toffey, Will Fr INF 71 67 0.294 0.380
4  2015 Vanderbilt Southeastern        1      8    Wiseman, Rhett Jr  OF 71 71 0.317 0.415
5  2015 Vanderbilt Southeastern        1      7   Swanson, Dansby Jr INF 71 71 0.335 0.423
6  2015 Vanderbilt Southeastern        1      1       Coleman, Ro So INF 66 53 0.295 0.402
7  2015 Vanderbilt Southeastern        1      2   Campbell, Tyler Jr INF 63 61 0.229 0.297
8  2015 Vanderbilt Southeastern        1      3    Kendall, Jeren Fr  OF 60 56 0.281 0.394
9  2015 Vanderbilt Southeastern        1     25     Ellison, Karl So   C 45 43 0.215 0.291
10 2015 Vanderbilt Southeastern        1     18    Rodgers, Nolan So  OF 44 17 0.247 0.396
11 2015 Vanderbilt Southeastern        1      5      Delay, Jason So   C 37 29 0.283 0.373
12 2015 Vanderbilt Southeastern        1     44      Wright, Kyle Fr   P 32  6    NA    NA
13 2015 Vanderbilt Southeastern        1     22   Pfeifer, Philip Jr   P 29 10    NA    NA
14 2015 Vanderbilt Southeastern        1     16      Murfee, Penn Fr INF 29 16 0.256 0.330
15 2015 Vanderbilt Southeastern        1     15    Fulmer, Carson Jr   P 26 25    NA    NA
16 2015 Vanderbilt Southeastern        1     35       Bowden, Ben So   P 26  0    NA    NA
17 2015 Vanderbilt Southeastern        1     24 Sheffield, Jordan Fr   P 22  6    NA    NA
18 2015 Vanderbilt Southeastern        1     21 Kilichowski, John So   P 21 14    NA    NA
19 2015 Vanderbilt Southeastern        1     13   Buehler, Walker Jr   P 18 18    NA    NA
20 2015 Vanderbilt Southeastern        1     55      Green, Tyler Fr INF 17  3 0.238 0.304
21 2015 Vanderbilt Southeastern        1     11      Sabino, Liam Fr INF 17  3 0.240 0.231
22 2015 Vanderbilt Southeastern        1     39       Smith, Kyle Jr  OF 16  0 0.062 0.211
23 2015 Vanderbilt Southeastern        1     40    Snider, Collin Fr   P 16  0    NA    NA
24 2015 Vanderbilt Southeastern        1     31     Johnson, Ryan Fr   P 15  5    NA    NA
25 2015 Vanderbilt Southeastern        1     45   Ferguson, Tyler Jr   P 15  3    NA    NA
26 2015 Vanderbilt Southeastern        1     36   McCarty, Aubrey Fr INF 10  0 0.167 0.167
27 2015 Vanderbilt Southeastern        1      6     Parker, Drake Fr  OF  8  0 0.200 0.200
28 2015 Vanderbilt Southeastern        1     28  Ruppenthal, Matt Fr   P  8  0    NA    NA
29 2015 Vanderbilt Southeastern        1     19       Mundy, Joey Fr INF  7  0 0.333 0.429
30 2015 Vanderbilt Southeastern        1     33    Chari, Tristan Fr   C  6  0 0.500 0.600
31 2015 Vanderbilt Southeastern        1     23     Abraham, Joey Fr   P  6  0    NA    NA
32 2015 Vanderbilt Southeastern        1      -            Totals  -   -  -  - 0.290 0.379
33 2015 Vanderbilt Southeastern        1      -   Opponent Totals  -   -  -  - 0.216 0.314
   SlgPct
1   0.571
2   0.462
3   0.420
4   0.566
5   0.623
6   0.394
7   0.286
8   0.530
9   0.282
10  0.288
11  0.394
12     NA
13     NA
14  0.385
15     NA
16     NA
17     NA
18     NA
19     NA
20  0.238
21  0.320
22  0.125
23     NA
24     NA
25     NA
26  0.167
27  0.200
28     NA
29  0.333
30  0.500
31     NA
32  0.455
33  0.300

Acquiring the pitching data is just as easy:

> ncaa_scrape(736, 2015, "pitching") %>%
+     select(year:ERA)
   year     school   conference division Jersey             Player Yr Pos GP App GS  ERA
1  2015 Vanderbilt Southeastern        1     43       Wiel, Zander Jr INF 72  NA NA   NA
2  2015 Vanderbilt Southeastern        1     20    Reynolds, Bryan So  OF 72  NA NA   NA
3  2015 Vanderbilt Southeastern        1     10       Toffey, Will Fr INF 71  NA NA   NA
4  2015 Vanderbilt Southeastern        1      8     Wiseman, Rhett Jr  OF 71  NA NA   NA
5  2015 Vanderbilt Southeastern        1      7    Swanson, Dansby Jr INF 71  NA NA   NA
6  2015 Vanderbilt Southeastern        1      1        Coleman, Ro So INF 66  NA NA   NA
7  2015 Vanderbilt Southeastern        1      2    Campbell, Tyler Jr INF 63  NA NA   NA
8  2015 Vanderbilt Southeastern        1      3     Kendall, Jeren Fr  OF 60  NA NA   NA
9  2015 Vanderbilt Southeastern        1     25      Ellison, Karl So   C 45  NA NA   NA
10 2015 Vanderbilt Southeastern        1     18     Rodgers, Nolan So  OF 44  NA NA   NA
11 2015 Vanderbilt Southeastern        1      5       Delay, Jason So   C 37  NA NA   NA
12 2015 Vanderbilt Southeastern        1     44       Wright, Kyle Fr   P 32  29  3 1.23
13 2015 Vanderbilt Southeastern        1     22    Pfeifer, Philip Jr   P 29  27  9 3.55
14 2015 Vanderbilt Southeastern        1     16       Murfee, Penn Fr INF 29  NA NA   NA
15 2015 Vanderbilt Southeastern        1     15     Fulmer, Carson Jr   P 26  19 19 1.83
16 2015 Vanderbilt Southeastern        1     35        Bowden, Ben So   P 26  26 NA 2.89
17 2015 Vanderbilt Southeastern        1     24  Sheffield, Jordan Fr   P 22  22  6 2.85
18 2015 Vanderbilt Southeastern        1     21  Kilichowski, John So   P 21  17 10 2.83
19 2015 Vanderbilt Southeastern        1     13    Buehler, Walker Jr   P 18  16 16 2.95
20 2015 Vanderbilt Southeastern        1     55       Green, Tyler Fr INF 17  NA NA   NA
21 2015 Vanderbilt Southeastern        1     11       Sabino, Liam Fr INF 17  NA NA   NA
22 2015 Vanderbilt Southeastern        1     39        Smith, Kyle Jr  OF 16  NA NA   NA
23 2015 Vanderbilt Southeastern        1     40     Snider, Collin Fr   P 16  16 NA 2.41
24 2015 Vanderbilt Southeastern        1     31      Johnson, Ryan Fr   P 15  15  5 3.02
25 2015 Vanderbilt Southeastern        1     45    Ferguson, Tyler Jr   P 15  15  3 6.30
26 2015 Vanderbilt Southeastern        1     36    McCarty, Aubrey Fr INF 10   5 NA 3.00
27 2015 Vanderbilt Southeastern        1      6      Parker, Drake Fr  OF  8  NA NA   NA
28 2015 Vanderbilt Southeastern        1     28   Ruppenthal, Matt Fr   P  8   8 NA 2.70
29 2015 Vanderbilt Southeastern        1     19        Mundy, Joey Fr INF  7  NA NA   NA
30 2015 Vanderbilt Southeastern        1     33     Chari, Tristan Fr   C  6  NA NA   NA
31 2015 Vanderbilt Southeastern        1     23      Abraham, Joey Fr   P  6   6 NA 7.94
32 2015 Vanderbilt Southeastern        1     32      Stone, Hayden So   P  3   3  1 9.64
33 2015 Vanderbilt Southeastern        1     34 Spagnuolo, Brendan Fr   P  1   1 NA 9.00
34 2015 Vanderbilt Southeastern        1      -             Totals  -   -  -  NA NA 2.84
35 2015 Vanderbilt Southeastern        1      -    Opponent Totals  -   -  -  NA NA 5.75

You can also acquire data from the same school across multiple years. You need to use both the ncaa_scrape function as well as the dplyr package.

You first create a table with each year you want and the school_id for the school. This can be built with the expand.grid function. Then, that table is grouped by the two relevant variables and fed into the do function. This will take the ncaa_scrape function and apply it to all relevant combinations of year and school_id found in the table you just made with expand.grid:

year <- c(2014, 2015, 2016)
school_id <- 736
division <- 1
v_table <- expand.grid(school_id, year)
v_table %>%
  group_by(Var1, Var2) %>%
  do(ncaa_scrape(.$Var1, .$Var2, "pitching"))
|=================================================================|100% ~0 s remaining     Source: local data frame [100 x 43]
Groups: Var1, Var2 [3]

    Var1  Var2  year     school   conference division Jersey          Player    Yr   Pos
   <dbl> <dbl> <dbl>      <chr>        <chr>    <dbl>  <chr>           <chr> <chr> <chr>
1    736  2014  2014 Vanderbilt Southeastern        1      3    Conde, Vince    Jr   INF
2    736  2014  2014 Vanderbilt Southeastern        1     20 Reynolds, Bryan    Fr    OF
3    736  2014  2014 Vanderbilt Southeastern        1      7 Swanson, Dansby    So   INF
4    736  2014  2014 Vanderbilt Southeastern        1     10   Norwood, John    Jr    OF
5    736  2014  2014 Vanderbilt Southeastern        1     43    Wiel, Zander    So   INF
6    736  2014  2014 Vanderbilt Southeastern        1      8  Wiseman, Rhett    So    OF
7    736  2014  2014 Vanderbilt Southeastern        1      9  Turner, Xavier    So   INF
8    736  2014  2014 Vanderbilt Southeastern        1     18  Rodgers, Nolan    Fr   INF
9    736  2014  2014 Vanderbilt Southeastern        1      1     Coleman, Ro    Fr   INF
10   736  2014  2014 Vanderbilt Southeastern        1      5    Delay, Jason    Fr     C
# ... with 90 more rows, and 33 more variables: GP <chr>, App <int>, GS <int>, ERA <dbl>,
#   IP <dbl>, H <int>, R <int>, ER <int>, BB <int>, SO <int>, SHO <int>, BF <chr>,
#   `P-OAB` <chr>, `2B-A` <int>, `3B-A` <int>, Bk <int>, `HR-A` <int>, WP <int>, HB <int>,
#   IBB <int>, `Inh Run` <int>, `Inh Run Score` <int>, SHA <int>, SFA <int>,
#   Pitches <chr>, GO <int>, FO <int>, W <int>, L <int>, SV <int>, KL <int>, teamid <dbl>,
#   conference_id <dbl>

The end result is a dataframe with all three years of data combined together.

You could also include a series of school_ids in that table that you feed into the do(ncaa_scrape) function if you wanted to pull multiple schools at once. The example below will provide data for both Florida and Vanderbilt from 2014-2016:

year <- c(2014, 2015, 2016)
school_id <- c(235, 736)
v_table <- expand.grid(school_id, year)
v_table %>%
  group_by(Var1, Var2) %>%
  do(ncaa_scrape(.$Var1, .$Var2, "pitching"))
|=================================================================|100% ~0 s remaining     Source: local data frame [197 x 43]
Groups: Var1, Var2 [6]

    Var1  Var2  year  school   conference division Jersey          Player    Yr   Pos
   <dbl> <dbl> <dbl>   <chr>        <chr>    <dbl>  <chr>           <chr> <chr> <chr>
1    235  2014  2014 Florida Southeastern        1     12  Martin, Richie    So   INF
2    235  2014  2014 Florida Southeastern        1     17  Gushue, Taylor    Jr     C
3    235  2014  2014 Florida Southeastern        1      2  Turgeon, Casey    Jr   INF
4    235  2014  2014 Florida Southeastern        1     23     Reed, Buddy    Fr    OF
5    235  2014  2014 Florida Southeastern        1     20   Alonso, Peter    Fr   INF
6    235  2014  2014 Florida Southeastern        1     33 Mattson, Braden    Jr     C
7    235  2014  2014 Florida Southeastern        1      5    Powers, Zack    Jr   INF
8    235  2014  2014 Florida Southeastern        1     66    Larson, Ryan    Fr    OF
9    235  2014  2014 Florida Southeastern        1     16  Shafer, Justin    Jr    OF
10   235  2014  2014 Florida Southeastern        1     11    Tobias, Josh    Jr   INF
# ... with 187 more rows, and 33 more variables: GP <chr>, App <int>, GS <int>, ERA <dbl>,
#   IP <dbl>, H <int>, R <int>, ER <int>, BB <int>, SO <int>, SHO <int>, BF <chr>,
#   `P-OAB` <chr>, `2B-A` <int>, `3B-A` <int>, Bk <int>, `HR-A` <int>, WP <int>, HB <int>,
#   IBB <int>, `Inh Run` <int>, `Inh Run Score` <int>, SHA <int>, SFA <int>,
#   Pitches <chr>, GO <int>, FO <int>, W <int>, L <int>, SV <int>, KL <int>, teamid <dbl>,
#   conference_id <dbl>

References & Resources

Code available in my GitHub repo

3 Comments

Oldest

Newest Most Voted

Inline Feedbacks

View all comments

Graham

7 years ago

How would you scrape all the teams at once?

Reply to Graham

I often get errors when doing more than 2 teams at once

Quick comment on R: the dplyr function slice() allows you to select rows of a data frame similar to select() with columns. For instance, slice(-1) removes the first row.

BAL	CHW	LAA
BOS	CLE	OAK
NYY	DET	SEA
TBR	KCR	TEX
TOR	MIN	HOU

ATL	CHC*	ARI
MIA	CIN	COL
WSN	MIL	LAD
NYM*	PIT	SDP*
PHI	STL	SFG