Why the Pyramid Rating System Changed to a Grading System

Very few in the sabermetric community, if anyone, view WAR as an absolute value. However, the myth that WAR is an absolute value has been one of the toughest things to dispel among the non-sabermetric group. And while my overall sentiment is very much in the sabermetric crowd, I can sympathize with the people who don’t fully understand how the stat works and therefore question a lot of WAR’s validity. How does someone with 6.0 WAR compare to someone with 4.0 WAR? How does that comparison differ from that of a 4-WAR player to a 2-WAR player? What about negative WAR players: How is it even possible for a player to subtract wins from a team?

While a sabermetrician could rattle off answers to all of these questions, explaining how it all works to someone who doesn’t understand WAR can be problematic, if not impossible. To do that, you have to get into the math behind WAR, math that is difficult to understand even for people in the sabermetric community.

Complicating the situation is the need to treat these estimated values as absolutes at some point to make calculations and draw conclusions. In a lot of cases, having to do this can contradict the idea behind WAR and lead to outcomes that aren’t as conclusive as one might think — or just flat out inaccurate.

The original concept behind the Pyramid Rating System went to great lengths to ensure that players across major league history were treated fairly by strictly limiting comparisons to counterparts while ensuring each era is represented proportionally among the all-time greats. But one thing the original Pyramid Rating System did not do was make it any easier to discern what the real difference was between a 5.0 WAR player and a 4.7 WAR player. Instead, it merely shifted the debate between differing between a 5.0 WAR player and a 4.7 player to one that tried to understand the difference between a 62-grade player and a 57-grade one.

Because the system had no way of discerning the real difference between a great season and a good one, this would occasionally lead to issues in the ranking process where Player A was interpreted as having a better season than Player B even though the two seasons were almost equal. That’s because the math behind the ranking process treated the values as absolutes, which in turned contradicted the idea of both WAR and the Pyramid Rating System that viewed the numbers as estimated values and not hard facts.

Switching over to a grading system not only takes into account this estimated value quirk, but it also puts in far better perspective the difference between WAR for players.

The old system, as seen above, ranked every season 1-70,000+ and plotted them along a curve. The new model keeps this approach, but rather than giving a hard ranking for players, it places them into grading buckets.

The idea behind this is twofold. One is to give people a better understanding of how different WAR seasons compare against each other.

As you can see from the chart above, the relationship between WAR and the letter grade is very correlated, with A++ seasons standing up far above the rest for both pitchers and hitters, and a steady decline in WAR for all grades A and below. (It’s worth noting with pitching that relievers are in the same grade as starters, which explains why WAR is more depressed for pitchers relative to hitters for grades A and lower.)

But in terms of ease of explanation, it’s easy to imagine an A++ player being the equivalent of a class valedictorian or an MVP Cy Young Award winner. The rest of the starters typically fall somewhere in the D+ to A+ range, while the ranks with grades of D, F, and incomplete are made up mostly of bench-warmers, middle relievers, and severely underperforming starters.

The second reason behind this change is to look at WAR and stats in general as a sample size with a margin for error, rather than a 100% accurate assessment of talent. Though the grading system on the surface seems to be more limiting in what it can account for, it still leaves room for players to fall into one of 14 talent categories. Even when looking at the game’s all-time best, no two players have the same grading breakdown.

We can also see the magnitude with which the Pyramid Rating System treats an A++ season, with Mike Trout already rating as the 14th best player in major league history on the strength of five A++ seasons. Only six players have as many, and all are in the top 10 all time according to the Pyramid Rating System. It’s also worth noting Babe Ruth having 23 graded seasons listed despite playing only 22 seasons in the majors. That’s because Ruth’s 1918 and 1919 seasons are graded separately as both batting and pitching seasons. This change was made to account for the arrival of Shohei Ohtani.

But even without getting into how Barry Bonds’ 2002 season compares against Ruth’s 1927 season, or how Rogers Horsnby’s 1924 season matches up against Ted Williams’ 1941 season, we still can make very definitive and accurate conclusions about where players rank all time. Instead of trying to figure out how to adjust WAR and account for era adjustments across different eras, the Pyramid Rating System has all of this built right into a grading system that represents each era proportionally as you go up and down the all-time rankings.

By grouping each player’s season into a grading bucket, we can gain more insight into a player’s relative greatness, rather than trying to measure the perhaps indiscernible difference between a 5.8 WAR season and a 5.3 WAR season.

This system also matches up more to how WAR should be interpreted, which is an estimate of a player’s value. By saying X number of players are rated as a B+, the Pyramid Rating System is saying there isn’t any inarguable conclusion you can draw as to who the statistically better or worse player is in this group. Did one player face tougher pitchers relative to another? Was another player in a better position to duck unfavorable pitching match-ups because his team had a better bench? Did one player play hurt for part of the year, affecting his stats? All of these factors can impact a player’s performance, but none of them has anything to do with a player’s ability.

The overall effect of these ancillary factors is likely minimal, but they will still show up in a player’s performance. Attempting to analyze these factors isn’t likely to yield anything of substance concerning how any given player compares to another, although it may steer further away from the reality of how the two compare if you attempt to dial in on a 0.1 difference in WAR.

However, drawing conclusions from these sometimes minor differences is precisely what the Pyramid Rating System was doing at times when trying to rank players against each other. It was always assuming seasons fall along that curve. But by grouping players into grading buckets, there is still a clear difference between a great season and a lousy season, regardless of how many or how few seasons are being looked at during the ranking process.

The greatest potential in this grade change is to make sabermetrics more palatable for not just the critical “old-school” fan but also the non-fan. A full explanation of WAR can take many minutes — and, to a non-fan, might be impossible to understand regardless. But someone who has never seen a baseball game in their life will be able to identify that an A+ player is tremendous, while a B player is good, and a C level one is about average.

The end goal of the Pyramid Rating System — examining how good a player is by taking in a variety of factors at once and outputting them into a value stat — remains the same with this update. One still can quickly identify the best players in history, as well as the oddballs and the overlooked players. But with this update, the information is now presented in a way that feels more innate and more natural to interpret, giving a better sense of accuracy in ways that overanalyzing the minutiae of WAR sometimes can miss.

Those interested in downloading the entire database to see how each season in major league history rates, along with where players rank all time, can do so by clicking here.