Oliver is not standing still
Let me say it at the start, the numbers have changed since last week. Some are up, probably more are down, but overall I am confident that they are better.
Oliver has been public for almost two years: 2009 at FanGraphs, and then in 2010 coming here to The Hardball Times. Even now I am still constantly looking at the code and trying to work out new theoretical models that will give both a more accurate and more detailed profile of a player. I confess I am the type of person who is more comfortable staring at a database, letting the numbers speak to me, than forcing myself to put my thoughts on figurative paper in the form of an article such as this.
When the 2011 version of the THT projections and its accompanying Fantasy Price Guides were rolled out on Dec. 20, I followed with my own piece which mentioned upgrades made since the 2010 release, particularly using a player’s exact birth date to determining a decimal age, which is then used in calculating and applying aging curves, and regressing each player to a mean determined by his age, level and position played in the previous season.
So how do we know if the 2011 THT Forecasts are any better than the ones published last year? Last month Mitchel Lichtman, at The Book blog, reviewed the accuracy of various 2010 projections. Oliver shows up in comment No. 8, and I admit I was disappointed with the results. At the time I had just concluded my own tests of the 2011 engine, generating projections based on past season data and comparing those results to other projections published in those years. Tom Tango, also of The Book, offered to run his own independent tests on my data set, and publish the results, whether good or bad, to avoid appearances of bias on my part.
Tango published his results two weeks ago, and in more detail in parts two and three, in which I was very pleased to see a much stronger performance from Oliver. Tango details various tests which arguably show Oliver an overall strong second to Chone, and in the lead with players having no major league experience. Despite the good showing, I spent a few days studying the details, looking for any weaknesses. Going back to my code, I made an adjustment in comparing each player’s performance in each minor league to his performance in the majors, and then readjusted the amount of regression for each stat category to minimize the total error of the resulting projection.
Now I can say that I am excited about the results, showing the newest Oliver to be even stronger in comparison to other projections.
Please read Tango’s posts for a detailed comparison of the five projection systems, while here I’ll add my own tests after the newest revisions.
Mean error is the sum of all the errors, weighted by plate appearances. A positive (high) error will be canceled out by a negative (low) error, so a value of zero means the numbers were all properly centered on the mean.
wOBA BA OB SA BH HR BB SO
Chone 0.006 0.002 0.005 0.005 0.003 0.002 0.006 -0.003
Marcel 0.008 0.007 0.007 0.016 0.006 0.002 0.000 -0.003
Oliver 0.000 0.001 -0.003 0.004 -0.001 0.001 0.002 -0.003
Pecota 0.004 0.002 0.003 0.007 0.002 0.002 0.007 -0.005
Zips 0.006 0.002 0.005 0.005 0.003 0.001 0.005 -0.004
Then I looked at the distribution of the errors, in players with 350 or more plate appearances each season. ‘Good’ is when the projection was within .010 wOBA of the actual, while very high or low was when the projections missed by more than .0325. Oliver has a slight lead in the percentage of good projections, but tends to under rather than over estimate players (which I will be looking into).
vlow low good high vhigh
Chone 0.120 0.214 0.270 0.238 0.159
Marcel 0.101 0.207 0.263 0.248 0.164
Oliver 0.163 0.240 0.273 0.207 0.118
Pecota 0.132 0.219 0.265 0.231 0.153
Zips 0.125 0.208 0.261 0.241 0.166
Even after studying error rates, what many fantasy users want to know is how well a projection system does at getting the players in the proper order, and how many times can it deliver the best projection.
These are the results of each projection head to head with Oliver. Following Tango’s criteria, a winning projection must be more than .010 wOBA better than the loser, and within .040 of the actual. A tie is when the projections are within .010 of each other. No decisions are when both projections were off by more than .040. The percentages are per player.
For example, from 2007 to 2010, 2,347 players that received projections from both Chone and Oliver. For 81 players both were crap, missing the actual performance by at least .040 wOBA. For 1,144 players, either Chone or Oliver had a projection within .040 of the actual, and within .010 of each other. A tie. For 479 players, Chone had a projection within .040 of actual and at least .010 better than Oliver, while 643 times Oliver had a projection within .040 of actual and at least .010 better than Chone. Of the 2,347 players, Chone and Oliver tied on 48.7 percent. Chone had the clearly better projection for 20.4 percent of the players, and Oliver was clearly better 27.4 percent of the time.
Number W L T ND Other% Oliver% Tie%
Chone 2347 479 643 1144 81 0.204 0.274 0.487
Marcel 2068 401 635 903 129 0.194 0.307 0.437
Pecota 2310 510 672 1033 95 0.221 0.291 0.447
Zips 2348 451 620 1165 112 0.192 0.264 0.496
Here are the number of times each produced the best projection for a player that was also within .040 wOBA of the actual, without regard to the margin of victory. If all the differences between the projections were due to random variance, each would be expected to have the best projection on 20 percent of the players. Oliver delivered the best projection for 36.6%.
2007 2008 2009 2010 Total Pct
Chone 39 36 36 37 148 0.140
Marcel 34 31 43 25 133 0.126
Oliver 95 102 82 108 387 0.366
Pecota 30 51 74 62 217 0.205
Zips 43 60 23 45 171 0.162
Finally, the number of clearly best projections—same as above, but the gap between the best and second best projection was larger than .010 wOBA. Smaller numbers, but a similar distribution.
2007 2008 2009 2010 Total Pct
Chone 4 2 3 5 14 0.140
Marcel 2 4 4 2 16 0.160
Oliver 13 5 3 12 33 0.330
Pecota 8 7 8 3 26 0.260
Zips 2 2 4 3 11 0.110
Oliver has shown a deal of improvement from a year ago, and even in the past week after further adjustments. Still, I’m not satisfied to leave it alone. All this has been done with season statistics. I have just begun to research, but have yet to implement, batted ball analysis from play by play descriptions. How often does a batter hit the ball in the air to the outfield? What percent of the time does he pull the ball? Park adjusted, what percent of flies to each position are home runs? What’s the relationship between a pitcher’s groundball rate and his home run per airball rate?
There are many more questions such as these that can lead to a better understanding of how players will perform in different ballparks and and at different levels of competition. Closer to being published are detailed defensive counting stats and ratings for differing skills at each position, an injury database, transactions history, and enhanced biographical data.
Oliver is not standing still.