Oliver is not standing still

Let me say it at the start, the numbers have changed since last week. Some are up, probably more are down, but overall I am confident that they are better.

Oliver has been public for almost two years: 2009 at FanGraphs, and then in 2010 coming here to The Hardball Times. Even now I am still constantly looking at the code and trying to work out new theoretical models that will give both a more accurate and more detailed profile of a player. I confess I am the type of person who is more comfortable staring at a database, letting the numbers speak to me, than forcing myself to put my thoughts on figurative paper in the form of an article such as this.

When the 2011 version of the THT projections and its accompanying Fantasy Price Guides were rolled out on Dec. 20, I followed with my own piece which mentioned upgrades made since the 2010 release, particularly using a player’s exact birth date to determining a decimal age, which is then used in calculating and applying aging curves, and regressing each player to a mean determined by his age, level and position played in the previous season.

So how do we know if the 2011 THT Forecasts are any better than the ones published last year? Last month Mitchel Lichtman, at The Book blog, reviewed the accuracy of various 2010 projections. Oliver shows up in comment No. 8, and I admit I was disappointed with the results. At the time I had just concluded my own tests of the 2011 engine, generating projections based on past season data and comparing those results to other projections published in those years. Tom Tango, also of The Book, offered to run his own independent tests on my data set, and publish the results, whether good or bad, to avoid appearances of bias on my part.

Tango published his results two weeks ago, and in more detail in parts two and three, in which I was very pleased to see a much stronger performance from Oliver. Tango details various tests which arguably show Oliver an overall strong second to Chone, and in the lead with players having no major league experience. Despite the good showing, I spent a few days studying the details, looking for any weaknesses. Going back to my code, I made an adjustment in comparing each player’s performance in each minor league to his performance in the majors, and then readjusted the amount of regression for each stat category to minimize the total error of the resulting projection.

Now I can say that I am excited about the results, showing the newest Oliver to be even stronger in comparison to other projections.

Please read Tango’s posts for a detailed comparison of the five projection systems, while here I’ll add my own tests after the newest revisions.

Mean error is the sum of all the errors, weighted by plate appearances. A positive (high) error will be canceled out by a negative (low) error, so a value of zero means the numbers were all properly centered on the mean.

         wOBA     BA     OB     SA     BH     HR     BB     SO
Chone   0.006  0.002  0.005  0.005  0.003  0.002  0.006 -0.003
Marcel  0.008  0.007  0.007  0.016  0.006  0.002  0.000 -0.003
Oliver  0.000  0.001 -0.003  0.004 -0.001  0.001  0.002 -0.003
Pecota  0.004  0.002  0.003  0.007  0.002  0.002  0.007 -0.005
Zips    0.006  0.002  0.005  0.005  0.003  0.001  0.005 -0.004

Then I looked at the distribution of the errors, in players with 350 or more plate appearances each season. ‘Good’ is when the projection was within .010 wOBA of the actual, while very high or low was when the projections missed by more than .0325. Oliver has a slight lead in the percentage of good projections, but tends to under rather than over estimate players (which I will be looking into).

          vlow    low   good   high  vhigh
Chone    0.120  0.214  0.270  0.238  0.159
Marcel   0.101  0.207  0.263  0.248  0.164
Oliver   0.163  0.240  0.273  0.207  0.118
Pecota   0.132  0.219  0.265  0.231  0.153
Zips     0.125  0.208  0.261  0.241  0.166

Even after studying error rates, what many fantasy users want to know is how well a projection system does at getting the players in the proper order, and how many times can it deliver the best projection.

These are the results of each projection head to head with Oliver. Following Tango’s criteria, a winning projection must be more than .010 wOBA better than the loser, and within .040 of the actual. A tie is when the projections are within .010 of each other. No decisions are when both projections were off by more than .040. The percentages are per player.

For example, from 2007 to 2010, 2,347 players that received projections from both Chone and Oliver. For 81 players both were crap, missing the actual performance by at least .040 wOBA. For 1,144 players, either Chone or Oliver had a projection within .040 of the actual, and within .010 of each other. A tie. For 479 players, Chone had a projection within .040 of actual and at least .010 better than Oliver, while 643 times Oliver had a projection within .040 of actual and at least .010 better than Chone. Of the 2,347 players, Chone and Oliver tied on 48.7 percent. Chone had the clearly better projection for 20.4 percent of the players, and Oliver was clearly better 27.4 percent of the time.

         Number    W     L     T   ND  Other% Oliver%    Tie%
Chone     2347   479   643  1144   81   0.204   0.274   0.487
Marcel    2068   401   635   903  129   0.194   0.307   0.437
Pecota    2310   510   672  1033   95   0.221   0.291   0.447
Zips      2348   451   620  1165  112   0.192   0.264   0.496

Here are the number of times each produced the best projection for a player that was also within .040 wOBA of the actual, without regard to the margin of victory. If all the differences between the projections were due to random variance, each would be expected to have the best projection on 20 percent of the players. Oliver delivered the best projection for 36.6%.

	      
          2007  2008  2009  2010  Total    Pct		  
Chone       39    36    36    37    148  0.140
Marcel      34    31    43    25    133  0.126
Oliver      95   102    82   108    387  0.366
Pecota      30    51    74    62    217  0.205
Zips        43    60    23    45    171  0.162

Finally, the number of clearly best projections—same as above, but the gap between the best and second best projection was larger than .010 wOBA. Smaller numbers, but a similar distribution.

          2007  2008  2009  2010  Total    Pct		  
Chone        4     2     3     5     14  0.140
Marcel       2     4     4     2     16  0.160
Oliver      13     5     3    12     33  0.330
Pecota       8     7     8     3     26  0.260
Zips         2     2     4     3     11  0.110

Oliver has shown a deal of improvement from a year ago, and even in the past week after further adjustments. Still, I’m not satisfied to leave it alone. All this has been done with season statistics. I have just begun to research, but have yet to implement, batted ball analysis from play by play descriptions. How often does a batter hit the ball in the air to the outfield? What percent of the time does he pull the ball? Park adjusted, what percent of flies to each position are home runs? What’s the relationship between a pitcher’s groundball rate and his home run per airball rate?

There are many more questions such as these that can lead to a better understanding of how players will perform in different ballparks and and at different levels of competition. Closer to being published are detailed defensive counting stats and ratings for differing skills at each position, an injury database, transactions history, and enhanced biographical data.

A Hardball Times Update
Goodbye for now.

Oliver is not standing still.


In addition to writing for The Hardball Times, Brian has written for FanGraphs, consulted for a Major League Baseball team and invented the Oliver projection system. Follow him on Twitter @blcartwright.

Comments are closed.