So how did THT projections do?

Last February, we released our first real pre-season book, The Hardball Times Season Preview 2008. Though I am especially proud of the fantastic team essays and player commentary in the book, I know that a lot of people’s interest lies in the projections published in the Season Preview, and now that the season is over, some of you have probably been wondering how they stacked up against other major systems.

Well, I’ve gathered the numbers, and I can tell you right off the bat that we did pretty well. Today, we’ll compare The Hardball Times projections to the three best publicly-available projection systems I know of: Baseball Prospectus’s PECOTA, Sean Smith’s CHONE, and Dan Szymborski’s ZiPS. All these systems are designed by very smart people, and if we can measure up, I’ll be very happy.

The versions I used of these projections were those published by early February, which is when the Season Preview was released and fantasy drafts generally start happening.

Let’s start with hitter projections. I selected all players with at least 200 plate appearances in 2008 that were projected by all four systems, a total of 329. I could have tried to do something with every player in the major leagues, but that would have involved a lot more work and probably changed the results very little.

The first, most basic, and perhaps most important test is to find each system’s average error for those players. To do that, I compared each player’s projected OPS to his actual OPS, adjusting the numbers so that averages were equal. So if we had three players with OPSs of .800, .850, and .900, and their projections had been .825, .875, and .925, the projections would each be adjusted down 25 points, as they are on average 25 points higher, and the average error would be 0. Here are the four systems ranked by average error:

THT, .051
CHONE, .053
PECOTA, .055
ZiPS, .061

We start off on a good note for THT, which had the most accurate hitter projections of the year. Now what happens if we restrict our sample to the hitters the systems disagreed on most? I calculated the coefficient of variation for each player’s projections, and selected the 65 hitters (or about 20 percent) with the most varied projections. Here’s how the systems stacked up:

CHONE, .061
PECOTA, .062
THT, .064
ZiPS, .071

CHONE wins this round, though the top-three systems are very close. ZiPS lags clearly behind. The player the systems most disagreed on, by the way, was Blake Dewitt. Here were his projections:

CHONE: .656 OPS
PECOTA: .698 OPS
THT: .625 OPS
ZiPS: .556 OPS

ZiPS is the big outlier here, and Dewitt ended up outperforming everyone’s expectations, posting a .728 OPS on the year.

We can also split players up by age. What if we look at how the projection systems did predicting the numbers for young and old hitters? Note that young hitters are defined here as those 24 years or younger in 2008, while old hitters are 34 years old or more. Both categories encompass a little less than 20 percent of qualified hitters.

Here are the average errors for young hitters:

CHONE, .050
THT, .054
PECOTA, .059
ZiPS, .064

CHONE wins another category, though I’m quite happy to come in front of PECOTA and ZiPS in projecting this difficult-to-gauge group. What if we look at the old guys?

CHONE, .041
THT, .041
PECOTA, .043
ZiPS, .043

A Hardball Times Update
Goodbye for now.

As group, older hitters were much easier to project, and THT and CHONE end up at the top.

Let us now turn our attention to pitchers. The process I followed for pitchers was much the same as what I did for hitters, except instead of OPS I used ERA, and my minimum baseline was 50 innings pitched in 2008. Let’s start off with the overall results:

PECOTA, .620
THT, .629
CHONE, .665
ZiPS, .676

PECOTA has always been known for being particularly good with pitchers, and this year was no exception, as it ever-so-slightly beat out THT for first place. Something crazy, however, happens if we look only at the 55 (roughly 20 percent of the whole sample) pitchers on whom the systems disagreed the most:

PECOTA, .786
ZiPS, .849
THT, .888
CHONE, .905

PECOTA really shines here, and I’m going to examine these results in more details to see just where our system can be improved. Jonathan Papelbon, by the way, was the pitcher on which the systems disagreed most, with his projections ranging from very good (THT) to otherworldly (ZiPS).

Let’s take a look at young and old pitchers now, with the definitions of young and old slightly tweaked so that each group includes about 20 percent of all qualified pitchers. Young pitchers are those who are 23 years of age or young; older pitchers must be at least 35. Let’s start with the young:

THT, .698
ZiPS, .736
PECOTA, .736
CHONE, .745

I’m happy to say that THT did the best out of these systems in projecting young pitchers, since that is a group that is generally very difficult to project. Here are the results for older pitchers:

PECOTA, .518
THT, .524
CHONE, .563
ZiPS, .596

And here we see another slight victory for PECOTA.

Overall, I am very happy with the results for THT. We did better than anyone else in projecting hitters, and were a close second with pitchers. Does that necessarily make our system better than any other? I don’t think so, but we definitely are at least as good as any other projection system.

Moreover, our goal is not just to correctly project a player’s OPS or ERA—though that’s part of it—but also to give you numbers that other systems don’t even bother with, like projected defensive ratings based on play-by-play data, projected depth charts, and fantasy dollar values.

If those are things that interest you, or if you just want to read great writing from the best baseball bloggers on the internet, make sure to pre-order The Hardball Times Season Preview 2009 today.


1 Comment
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Fake Chanel Bags
8 years ago

Around the bend, somebody’s there to boost our spirits. On that toughest mile, just when we think that we’ve hit a wall, someone will be there to cheer us on and pick us up if we fall.