Beating the Odds: When Teams Outperform Their Projections

Projection systems have a purpose in baseball. They’re the most measured form of expectation heading into a season. They set a baseline for how good or bad a player or team will be, simultaneously pointing out possible strengths and weaknesses.

Predicting performance in baseball isn’t a perfect science. It isn’t meant to be. The process is tailored to accuracy. This involves a variety of methods, including adjustments brought on by changing factors of performance, regression to the mean, and accounting for age. What projections don’t account for are the actual changes a player made. Maybe a player adjusted his swing, molding him into an entirely different hitter than he was before (i.e. Max Muncy). Maybe a pitcher made a change to his repertoire or release point, making a lot of the previous data on him irrelevant (i.e. Lucas Giolito).

Players like these perhaps make up the hardest part of projecting performance, for a variety of reasons. A lot of times, their changes go unnoticed before the season. Most of the time, there is a low level of reliability in the thought of these changes showing up in actual results.

Thinking deeper, these players that blew past their projections can still be a contributing factor into the analysis of player development. For example, heading into the 2017 season, Cardinals outfielder Jose Martinez was projected to put up a .311 wOBA (per Steamer), below the major league average, and with his lackluster defensive skills, making him appear to be a below-replacement-level hitter. But while spending his age-26 and age-27 seasons with the Royals Triple-A affiliate, the Cardinals noticed something that suggested he could perform at a high level in the majors.

Cardinals officials first eyed Martinez a year earlier, when he was swinging his way to a Pacific Coast League batting crown for Kansas City’s Triple-A affiliate in Omaha. He hit .384/.461/.563 that season, his 12th in pro ball without a big-league taste. The breakout year failed to make a major prospect out of Martinez, who by that point was 26, had cycled through three organizations, undergone three knee surgeries and never hit for power. Back home in Venezuela that winter, he wondered what he still had to prove.”

After a few adjustments, Martinez went on to post a .379 wOBA at the major league level for the 2017 season, ranking in the top eight percent of major league hitters and making him one of the biggest over-performers relative to his preseason projections that season.

So who deserves the credit for Martinez’s surprising season? Obviously Martinez himself gets a ton of credit for executing the changes necessary for him to perform, but one would think a fair amount of recognition should go to the player development and scouting staff for the Cardinals for the assistance in making his impressive season possible.

Measuring the skills of an organization’s player development system is tricky. Sure, the ones that are good at what they do are well known, while some of the bad ones may be pointed out from time to time. But everything in between lurks in the unknown. Quantifying the results in this area is tough, but a method might lie in preseason projections.

One of the more prominent projection systems out there is the well-known Steamer projections. They have been putting out preseason projections for both hitters and pitchers dating all the way back to the 2010 season. The accuracy of their method is strong and through recent history has only been getting better.

For my analysis, I’ll be using preseason wOBA projections for hitters and SIERA projections for pitchers and comparing their projections to their in-season results in those respective metrics, all data coming from Steamer. To keep recency into play, I’ll only be going back to the 2015 season, hopefully giving a good view of the current landscape of baseball while keeping the sample size ideal.

The goal here will be to determine the level at which teams over-perform their preseason projections on a player-by-player basis. The thought heading into this investigation would be that teams with good player development systems would over-perform their preseason projections more often, while teams with bad ones would have their players under-perform their projections more often.

First, to set the basis, here are the hitters with at least 200 plate appearances in a season dating back to 2015 who have over-performed relative to their projections the most.

A noticeable trend among these 20 players is their respective ages. Thirteen of them are 25 or younger, suggesting that over-performance in projections can be most common in players with little to no data in the majors. There are a few notable breakouts here, such as Max Muncy, Marwin Gonzalez, and Zack Cozart, while a couple of them ran BABIP-fueled campaigns to success, like Sandy Leon, Andres Blanco, and Ryan Raburn.

Looking to the opposite end of the spectrum, here are the 20 hitters who under-performed their projections the most.

The most common trend among these hitters once again is age, as 11 of the 20 hitters were over the age of 30. Hitters like Chris Davis, Prince Fielder, Miguel Cabrera, and Jose Bautista followed up season(s) of high-level offensive production with plummeting performances when the aging curve hit them. Then there are prospects like Lewis Brinson and Garrett Hampson, who couldn’t reach their expectations once arriving in the big leagues.

With the biggest over-performances in SIERA among pitchers since 2015, it’s mostly relievers. You would have to get by 59 relievers in that time to find the biggest over-performance by a starting pitcher, that being 2019 Lucas Giolito. Pitching in smaller sample sizes, relievers can be really volatile and hard to project. There is no clear distinction with this group, as most of them are just pitchers who put up really good seasons.

For the most under-performing pitchers, it’s a bit of a mixed bag. In terms of age, it’s a pretty even distribution. This array of players probably speaks to volatility also, with most of these pitchers being projected at the least to have acceptable seasons before proceeding to fall apart for one reason or another.

Piecing all of this together can give a good understanding of what teams have their players over- or under-perform their projections the most, ideally giving a useful measurement of the success of an organization’s player development staff. Initially, the plan to compile this data simply was going to be to find the average of each team’s difference in projections and results, separated by hitters and pitchers. That plan was foiled though, as it didn’t take long to realize the consistency of league-wide differences between projections and results varied year to year to a strong extent, likely caused by the league-wide offensive variation resulting from changes to the makeup of the baseball.

An adjustment was needed for this effect, so I found the z-score of the projections-results difference for each player compared to their respective season, ultimately converting their differences into a plus stat.

Now after adding context to the data and finding the average differences in wOBA and SIERA for each team, there is a final calculation.

Averaging these two tables gives the final product of an attempt to measure the success level of a player development staff.

Could the top team have been any less surprising? The Astros, well known for their success in scouting and player development, have had their players outperform their projections more than any other organization’s players over the last five seasons, with instances like Gerrit Cole, Wade Miley, Dallas Keuchel, Yordan Alvarez, and Marwin Gonzalez leading the way, among many others. Other organizations ranking towards the top are familiar when the topic turns to good player development systems, including the Rays, Dodgers, Indians, and Yankees.

Occupying spots at the bottom are the White Sox, Cubs, Orioles, Rockies, and Phillies. It is worth mentioning that the Orioles, the fifth most under-performing team in terms of projections, have outperformed both their wOBA and SIERA projections more this season than any of the last four. In 2019, only eight teams have had their hitters out-perform their wOBA projections more, and only seven teams have had their pitchers out-perform their SIERA projections more. This comes right after the hiring of Mike Elias and Sig Mejdal, two key members of the Astros brain trust during their rebuild and success, to help replace the former front-office regime.

The big question is, how can this information be utilized going forward? To answer, there is recent reasonable evidence to suggest this data can be factored into future projections. This capability lies in the fact that performance relative to projections can be predictive on a team basis, showing a low level of correlation.

The idea behind factoring these numbers into future projections would be operating under the thought that a team’s player development system has an impact on how a single player performs. This obviously isn’t the case 100 percent of the time, but if it can create more predictiveness in projections, it’s useful. By using team projection performance data from the past two seasons (more recency causes more reliability) and adding the factor into a player’s projections (using Steamer again), it is shown there is indeed an increase in correlation with projections and results for both hitters and pitchers. (The sample here is all hitters with at least 200 plate appearances and pitchers with at least 40 innings pitched in 2019.)

Factoring an organization’s track-record of player development through its performance relative to projections into a single player’s projections never came to mind. Once it did, the idea of doing so initially appeared dubious to me, mostly due to an array of unknown inputs.

But what flipped my thoughts on this was the discovery of some level of year-to-year correlation of team performance against their projections. With consistency existing, it wasn’t crazy to think this data could be used as another input into a players’ projection.

There are still some issues with how projection vs. result can be used to measure a player development system’s success. The main concern is how to deal with front offices that have had recent turnover in this department. But for teams with longevity in their track records, there definitely is usefulness in this practice.

Player development is becoming more and more of a mainstay in this current age of baseball as technology and data collecting continue to grow. With top teams like the Astros, Dodgers, Rays, Yankees, and Indians continuing to fill their rosters with players who exceed expectations, it seems increasingly important to have data to measure and predict with.

References & Resources

• Credit to Kyle Boddy, for giving me this idea through a tweet of his
• This obviously couldn’t have been done without the extensive work Steamer Projections does and the past projection data they have on their website
• FanGraphs Splits Leaderboard for results data