The Hall of Fame’s Bullpen Problem
What exactly makes a reliever Hall worthy? (via Alex Kim, Terren Peterson, Dirk Hansen & Howell Media Solutions)
Editor’s Note: This is the third post of “Hall of Fame Week!” For more info, click here.
The Hall of Fame tends to bring about a lot of arguing. Arguments over players like Jim Rice (“the fear”) and Bert Blyleven (strikeouts are awesome!) went on for some time, as did the squabbling over Jack Morris and the importance of pitcher wins. Then we harangued each other about steroid users and the character clause. Now, over the next few years, the big Hall of Fame argy-bargy will be about relief pitching.
The debuts of Trevor Hoffman and Billy Wagner on the Hall of Fame ballot this year are about to expose the fact that the baseball community still has no idea what to make of relievers. They are a relatively new phenomenon, by baseball standards (40 or 50 years old), and their role and usage have changed a lot even within that short lifespan. As a result, modern-style relief pitchers didn’t appear regularly on the Hall of Fame ballot until the 2000s.
As we stand today, only five pitchers have been elected to the Hall primarily for their efforts out of the bullpen. (Compare this to 62 starting pitchers and 61 outfielders.) To this point, voters have been able to get away with treating reliever candidates on a case-by-case basis (though it has produced erratic results). But with two of the relief corps’s strongest candidates yet embarking on their Hall of Fame campaigns in 2016, we need a reliable way to compare Wagner to Hoffman, those two to ballot holdover Lee Smith, and those three to the five standing honorees. Above all, it’s time we established a fair Hall of Fame standard against which to measure relievers’ careers.
But first, let’s get one thing out of the way: Relief pitchers belong in the Hall of Fame. Reliever is a distinct and established position in today’s game, albeit a less valuable one, and the Hall doesn’t adequately capture the full story of baseball without including them. Keeping Hoffman or Wagner out of the Hall because you don’t like the position they played is about as fair as barring entry to Edgar Martinez for being a designated hitter. Some positions inevitably will be worth more than others, which is why voters should enshrine the best players at each position, rather than comparing relievers to starters, left fielders to catchers, or apples to oranges.
Leaving that debate for another day, the search is on for the best statistics to use to evaluate relievers. We can’t use traditional stats like wins or saves—artificial constructs that assign pitchers credit or blame for patterns of use that are out of their control. But we also can’t use more accepted advanced metrics like WAR. Because of how little they play, relievers, no matter how good they are, never have the chance to sniff the high win-contribution totals amassed by elite starters and position players.
At most positions, if a player has more than 60–70 career WAR, he’s in Hall of Fame territory, but not relievers. Is their magic number closer to 40? Thirty? Twenty? (This is an important question to answer for one of the more popular sabermetric Hall of Fame barometers, Jay Jaffe’s/Sports Illustrated’s JAWS, which relies on comparing candidates to their enshrined brethren at the same position.) The small sample size of relievers in the Hall makes the ideal positional reference point uncertain.
For the record, the average fWAR of the five bullpen inductees is 33.4. But we can’t in good conscience use that number as a benchmark; it’s skewed too high by the exceptional circumstances of the five. One, Dennis Eckersley, earned 41.4 of his career 61.8 fWAR as a starting pitcher before he was converted to relief. The others all played part or most of their careers as multi-inning closers,\.
| Pitcher | IP in Relief | Relief Appearances | Average Relief IP/G |
| Hoyt Wilhelm | 1,872.3 | 1,018 | 1.84 |
| Rollie Fingers | 1,505.7 | 907 | 1.66 |
| Rich Gossage | 1,556.7 | 965 | 1.61 |
| Bruce Sutter | 1,042.0 | 661 | 1.58 |
In essence, guys like Gossage, Fingers, and Wilhelm got into the Hall of Fame throwing 150 to 200 percent more career innings than modern closers do. Of the four, Sutter was the closest to a modern reliever — he never started a game, for instance, unlike the other three — yet he still pitched an average of 1.58 innings per appearance. That’s a chance to pick up 1.58 times more WAR than modern relievers do thanks solely to managerial discretion.
Clearly, one-inning closers are still uncharted territory for the Hall of Fame. That’s bad news for Hoffman (1.05 IP/G) and Wagner (1.06 IP/G), who are likely to be unfairly compared to their more heavily used predecessors, despite the fact that on an inning-by-inning basis, they were actually far superior. There’s an easy way to quantify this, of course: Look at WAR per inning pitched. To make the numbers easier to conceptualize, let’s scale it to WAR per 200 innings.
| Pitcher | Career fWAR | fWAR as Reliever | IP in Relief | Relief fWAR/200 |
| Dennis Eckersley | 61.8 | 20.4 | 807.3 | 5.1 |
| Rich Gossage | 31.1 | 28.8 | 1,556.7 | 3.7 |
| Bruce Sutter | 19.2 | 19.2 | 1,042.0 | 3.7 |
| Rollie Fingers | 27.4 | 25.9 | 1,505.7 | 3.4 |
| Hoyt Wilhelm | 27.3 | 19.4 | 1,872.3 | 2.1 |
That’s an average of 3.6 fWAR per 200 innings among current Hall of Fame relievers. Now let’s see how the three relief pitchers on the 2016 ballot stack up.
| Pitcher | Career fWAR | fWAR as Reliever | IP in Relief | Relief fWAR/200 |
| Billy Wagner | 24.2 | 24.2 | 903.0 | 5.4 |
| Trevor Hoffman | 26.1 | 26.1 | 1,089.3 | 4.8 |
| Lee Smith | 26.6 | 25.8 | 1,252.3 | 4.1 |
Those are significantly better numbers than four of the five current Hall of Famers. On this scale, only Eckersley was a better closer than Hoffman and Smith, and Wagner tops them all.
Still not convinced? There are other advanced metrics that do proper justice to relievers. Win probability, for instance, is an even more precise way to get at dominance, drilling down to the level of individual plays elicited by these bullpen aces. Two analogous stats on this front are context-neutral wins (a.k.a. WPA/LI) for a player’s effect on win expectancy and REW (RE24 converted to a wins scale) for its effect on run expectancy. Win-probability stats only go back to 1974, so data for Fingers (debuted in 1968) and Gossage (debuted in 1972) are incomplete, and Wilhelm (retired in 1972) is missing entirely, but here are the data we do have.
| Pitcher | WPA/LI in Relief | REW in Relief |
| Rich Gossage | 17.67 | 20.96 |
| Billy Wagner | 18.03 | 19.91 |
| Trevor Hoffman | 17.50 | 18.84 |
| Lee Smith | 13.88 | 17.79 |
| Dennis Eckersley | 13.29 | 13.59 |
| Rollie Fingers | 9.80 | 12.72 |
| Bruce Sutter | 12.69 | 12.10 |
The four current Hall of Famers average 13.36 WPA/LI and 14.84 REW. As for the hopefuls? Well, it’s decisive. Again, the modern closers far surpass all but one of their decorated peers—this time Gossage, who bests everyone in REW but still trails Wagner and effectively ties Hoffman in WPA/LI.
The verdict is clear on the preponderance of evidence: Pound-for-pound, Hoffman, Wagner, and even the much-maligned Lee Smith were far more dominant than four of the five current Hall of Famers. This isn’t to say Wilhelm et al. aren’t worthy—they do deserve some extra credit for their durability and the value some of them brought as starters. But as pure relievers, the current ballot’s trio were more fearsome. They should be immortalized in the Hall, low-end WARs and all.
And leading the pack overall: Not 600-save club member and AC/DC fan Trevor Hoffman, but scorching southpaw Billy “The Kid” Wagner. Throw out saves (though Wagner still has 422, good for fifth all time), and it’s hard to make a case for Hoffman over the less heralded lefty. Wagner’s 2.31 ERA and 2.73 FIP far outstrip Hoffman’s (2.87 and 3.08). He is second all time among relievers in both RE24 and WPA/LI (Hoffman is fourth in both). Among pitchers with at least 500 innings pitched, Wagner’s 1.00 WHIP is the second-lowest in baseball history. His strikeout rate of 33.2 percent stands alone as the highest.
Wagner, simply put, is the best reliever ever to appear on a Hall of Fame ballot. Of course, this won’t be true in three years’ time, when Mariano Rivera makes what will surely be a brief appearance on the ballot, as he is the best reliever in history by virtually every measure (his 652 saves, 6.4 fWAR per 200 innings, 34.38 WPA/LI, and 34.75 REW are all the best marks ever amassed out of a bullpen).
Rivera will be elected easily when he’s eligible in 2019, and rightfully so, but hopefully his specter won’t keep others out in the meantime. Wagner may not be Rivera, but using that as a reason to keep him out of the Hall makes about as much sense as saying Tim Raines is unworthy because he’s not Rickey Henderson.
If you can only find room on your ballot for one reliever, make it Wagner. But measuring by more precise yardsticks, all three closers on the 2016 ballot clear the bar that voters have set for relief pitchers: an average of 3.6 WAR/200, 13.36 WPA/LI, and 14.84 REW, to go along with enough total value (a floor that has been set around 20 fWAR based on modern bullpen usage) to guarantee some longevity along with that dominance. You can quibble with where those thresholds should be set, but by precedent alone, Wagner, Hoffman, and Smith would all be credits to their position’s standing in the Hall of Fame.
Well done. Wagner it is.
“At most positions, if a player has more than 60–70 career WAR, he’s in Hall of Fame territory, but not relievers. Is their magic number closer to 40? Thirty? Twenty? ”
More to the point, does WAR truly capture their value?
Obviously not. Clearly there is a serious problem if there is general recognition that a different WAR standard has to be used for relief pitchers getting into the Hall of Fame.
I think this is one of the most thoughtful explorations of relief pitcher’s place in the Hall of Fame I’ve ever read (and I’ve spent a lot of time trying to find analysis on this subject.) I feel this should be mandatory reading for all HOF voters.
Fair enough. Though I’m a big hall guy. I’m predicting Hoffman gets over 50% but Wagner checks in at less than 20% making it a tough slog for him.
And fact of unless these guys get in over the next 3 years the looming candidacy of Rivera will hurt them.
For a few years after Rivera is elected voters will compare them to Rivera and say, “these guys are far lesser players”.
Add the fact that Sutter is already viewed as a mistake and it’s easy to see saves being the benchmark for a while to come.
Which would be like measuring outfielders against Wills Mays
FYI, Baseball Reference has WPA stats for the entire WPA era.
Echoing The Dude above in that this was great, thoughtful analysis. Thank you for writing and sharing.
I was surprised to see Wilhelm so low in WAR. That’s because of the differences in fWAR (27.3) and rWAR (50.1). In his case, it comes down to beating his FIP (2.52 ERA, 3.06 FIP) considerably by limiting hits on balls in play (.245). I don’t mean to restart the debate on which method is best generally, but in this case it is obvious that fWAR underrates Wilhelm, since we’ve known that knuckleballers have consistently low BABIP since Voros first published his DIPS research.
Using the current HOF relievers to establish precedent would lead to a flood of closers in the hall. In my opinion Sutter and Fingers are mistakes and Eckersley is a hybrid. That leaves Gossage as a fair standard by the metric chosen. While I was OK with Gossage going in, this question gave me pause: Was he more important to the late 70s/early 80s Yankees than Ron Guidry? I think it’s much more likely that Guidry could have pitched fewer innings and dominated late in the ballgame like Gossage than Gossage could have been as effective starting as Guidry.
Seconded. Very well put, Rally. Steve Treder’s work here many years ago on translation of starting pitching performance to relief pitching performance provides a basis for your views on Gossage/Guidry.
Actually for about 1/2 season Guidry was moved to the pen when Gossage was hurt and Guidry did quite well.
I love sabermetrics, but on relief pitchers it has to go deeper, As sabr people say you can’t judge on small sample sizes. Closers, especially today are a small sample size. So too totally judge a closer a few other things must be taken into consideration. The purpose of the closer is also to be the man at biggest times. At the biggest times (post season)Billy Wagner was so far beyond horrid it can’t be ignored. Add to that his 2005 melt down in back to back games against Houston that cost Phillies a playoff appearance. A year later against the Cardinals, his manager did everything not to use him in the last playoff game. He was great at times but to be a Hall of Famer at limited position he needed to be great when great was needed.
Talk about small sample size. You are taking a couple of bad games Wagner had-admittedly important games-and using it to judge his career? The problem with this is you get into the issue of what is an important game. It can’t only be a couple of late-season games. I’m sure Wagner also saved some important games; albeit maybe not direct win-or-go home games. What about the games he saved to get the Astros in position for the playoffs; those games weren’t important? It just seems very unfair to pick a couple of games from his career. I can think of two playoff games that Mariano Rivera blew-Game 5 in the 1997 ALDS and, of course, Game 7 of the 2001 World Series.
Hold on. So you start off the post admitting to the problem of small sample sizes… and then blast Wagner for a handful of anecdotal games you remember.
Bold strategy, sir.
While I agree that Wagner was better than Hoffman, I don’t think it’s fair to compare rate stats between the 70’s relievers and the modern closers. Clearly, it’s easier to be dominant, and throw all-out, knowing that you’re only going one inning, versus having to maybe go two or three innings. The 70’s relievers also came into the game with runners on far more often, necessitating pitching from the stretch.
I think it really comes down to how many relievers ought there be in the Hall. Currently here are 73 starters (counting Eck as a starter), 4 relievers and between 16 and 24 at each position. Of course relief pitching is a relatively modern phenomenon, so you obviously wouldn’t want 16 relievers, unless you really like Sparky Lyle or Doug Jones as candidates. Since Wilhelm is the earliest reliever with any support, we can put estimate the reliever era to be 1960-present, which covers about half of Wilhelm’s career. That is just about 40% of baseball history, which translates to 6-10 relievers meeting the standards of other positions.
Now its pretty clear that relievers are less valuable than other players by virtue of playing less. That would make me err on the lower end of that range, which seems to be 6-7 relievers. From looking over the statistics, it seems to me there are a few tiers to look at.
The first is the pair of Rivera and Wilhelm, who were clearly the best of their respective ‘genres’ of relief pitching. If you think any relievers ought to be in the Hall, its these two.
Gossage has the next tier all to himself. He makes three
The next tier would be Fingers, Hoffman, and Wagner. I think that this is the borderline tier. If you think that there ought to be more relievers in the Hall of Fame than just the best of the best, the guys in here are your best choices. If we add in all three, that puts us right at 6, which is what I estimated to be the ‘proper’ number of reliever. Mentally, these guys do seem about equal with Andre Dawson in terms of dominance/goodness relative to the position, and he seems to be the best candidate for a borderline player.
Sutter, Smith, and Eck are in the next tier with Joe Nathan, and, surprisingly, Tom Henke. Eck has the part of his career as a very good starter which pushes him over the line, but the other four seem to fall short.
After looking this over, it seems to me that Wagner and Hoffman are solid, if unspectacular candidates, Sutter was probably a mistake, but not a really bad one, and Smith would be a similarly underwhelming choice. Oddly enough, this seems to fit in pretty well with the collective Hall of Fame wisdom, which is surprising considering the less than good track records at the other positions.
The article mentions comparing apples and oranges as something to avoid. Comparing Wagner to Gossage without somehow factoring in Innings Pitched (903 to 1556.7) *is* comparing apples and oranges. I like the approach in general, but feel it is missing that element.
Agreed- the one inning “closer” is a different animal than the multi-inning “fireman”. Rivera, Hoffman, Wagner, etc, should be considered on their own merits against each other, not against Gossage and his gang. Comparing closers to firemen in a HOF discussion is like comparing third basemen to shortstops…
One inning closers have a much easier job than earlier closers. Saying that Wagner’s numbers are better than Wilhelm’s or Fingers is like comparing Doc Gooden’s first inning numbers to Tom Seaver’s total numbers. I hate to say but the only way for HOF voters to rate closers is by using their own sense of how good they really are. I’d vote for only Wilhelm, Gossage, Fingers & Rivera at this time.
I think you’re being too hard on relievers when you use WPA/LI and REW. They should get some credit for leveraged use, by virtue of being good enough to pick in those high-leverage situations.
But on the other hand, if a reliever doesn’t match up with other Hall of Famers even by raw WPA — giving them 100% credit for leverage, probably too much — then how do you really make the case?
Rivera 57
(Randy Johnson 53)
(Schilling 35)
Hoffman 34
Gossage 32
(Tim Hudson 30)
Wagner 29
(Troy Percival 24)
Smith 21
(Todd Jones 19)
Sutter 18
Compare Helton 53, Olerud 33, Burrell 19.
One inning closers: Hoffman and Wagner were clearly one inning closers, but Hoffman did make 9 appearances of 4 or more innings. He also threw 4 innings once. I haven’t looked very closely at Wagner. Both of them retired with about 50 more innings pitched than appearances.
Compare that with a current guy like Craig Kimbrel who has fewer innings pitched than appearances in each of his seasons. He will surely retire with fewer innings than appearances. I wonder what the voters will feel about that?
Will Dan Quisenberry’s career be re evaluted?
Why is WHIP hardly considered? Looks to me like that is one reason Lee Smith doesn’t get in (and I tend to sgree).
Also, I like what Jaack wrote, and it seems to me it should be Mo, Hoffman, Wagner, and not much after that for quite some time.
Great article. It seems strange to give rate stats (WPA/LI or WAR/IP) primary importance when WAR is the gold standard when talking about Hall candidacy for non-relievers
Raw WPA (http://www.baseball-reference.com/leaders/wpa_def_career.shtml) seems like a great first thing to look at to me. It suggest that Rivera is an inner-circle Hall of Famer (something almost everybody agrees with, unless you just don’t like the idea of relievers in the Hall). Raw WPA also suggests that Hoffman might come out ahead of Wagner (something many could agree with).
Using WPA also reflects why teams want to use great pitchers in a relief role (apart from differences in endurance). It’s a good idea to have a great pitcher on the mound when the game is on the line — why not use a the stat that captures that idea so perfectly?
While raw WPA is quite noisy, the closers we’re talking about had long careers. And even if their totals are still noisy, giving relievers credit for coming through when it counts seems like a just way of evaluating a reliever’s success. That’s what they signed up for.
The problem with only using raw WPA is that is doesn’t reflect luck. It should be combined with FIP- (or something like that) in some way.
I am no fan of the yankees, but to read Marc Schneider write that “Mariano Rivera blew Game 7 of the 2001 World Series,” is a bit too much to take. The Closer known as Mariano induced Luis Gonzalez to induce an infield pop-up that would have been caught had the manager, Joe Torre, not drawn the infield in. The Closer did his job. If anyone should be questioned for the outcome of that game, I suggest that person be Joe Torre. (And how about IBB, called for by the manager, being assigned to the poor pitcher? Manager gives batter free pass and when he scores it bloats the pitcher’s ERA! This is then used against him when it comes time for arbitration…)
This vividly illustrates the problem with “hits” allowed by pitchers. I watched a Cardinals game recently in which the third baseman, Carpenter, air mailed a throw to the first baseman which allowed a run to score, which was considered “unearned.” Later that inning the right fielder, Heyward, “lost” a ball that should have been caught, which allowed two runners to score, and those runs were “earned” because it was scored as a “hit.” The pitcher now shows an extra hit and two “earned” runs allowed when the inning should have been over. Mr. Schneider would blame the pitcher for “not doing his job.” Obviously, with anything less than a strikeout, or a ball hit right at a fielder, The Closer, Mariano Rivera, would not have done his job.
In 1969 Bob Veale’s BABIP was .343. Of the nine most often used pitchers the next highest BABIP belonged to Dock Ellis at .299. The Pirates DEFENSE allowed a BABIP of .295 that season. While Bob “did not do his job” Joe Gibbon did his because the Pirates DEFENSE allowed a BABIP of only .229. This shows that Joe was obviously a much better pitcher than Bob that year. Have I got that right, Mr. Schneider?
Since baseball is a team game, I advocate doing away with individual pitcher hits, charging them to the team, just as some advocate having a team error. If the pitcher is not responsible for what happens after the ball is put into play, then why charge him with allowing a hit? Either Voros McCracken was on to something, or his theory is full of beans. Which is it, Mr. Schneider?
BTW, in 1964 Tommie Sisk, pitching for the Pirates, shows a BABIP of .407! Next was Don Schwall at .325 in 50 innings. The Pirates DEFENSE allowed a BABIP of .297. Tommie only pitched in 61 innings, so the sample size is much less than Bob Veale, who pitched 226 innings in ’69. Does a pitcher ever pitch enough innings in a year to have his BABIP taken seriously? The only thing that counts is TEAM BABIP!
I am reminded of the time Jose Canseco allowed a ball to hit him in the head and bounce over the fence for one of those things “chicks love.” The box score shows the pitcher “allowed” a home run because he “did not do his job.”
Yeah, sometimes pitchers get screwed by bad defense. And sometimes they get saved by bad defense. You say the pitchers should not be held responsible for their teammates’ bad defense, but what if a fielder makes a truly spectacular play that really “should not have been made”? Should that out not be recorded?
I concede your point about Joe Torre’s blunder in GAme 7. But, remember, Rivera had given up the tying run before Gonzalez’s bloop hit. Now, granted, some of that was due to his own error, but he did give up a couple of normal hits. In fact, while I agree it was a mistake to play the infield in, it would not have been in if the winning run had not already been on third base. So, the closer DID NOT do his job. The point I was trying to make, which you seem to have missed, is not that Mariano Rivera was not a great closer-obviously he was-but that even the best closers blow saves and, with respect to Hoffman, it’s unfair to take a couple of blown saves in important games and make that the measure of his career. Instead, you decided to attack a strawman argument that I didn’t make. The rest of your bloviation has nothing to do with anything I said and I don’t even know what point you are trying to make. If you are going to attack someone, please attack only what they actually said.
I am saying exactly what I said, Barney. A pitcher should not take the blame for something out of his control. Mariano Rivera should not take the blame for “blowing” a save in Game 7 of the 2001 WS because he did not “do his job.” Baseball is a team game and as such if any blame is given, it should be given to the “yankees” as a team because the manager, Joe Torre, a great manager (and ballplayer, who is HOF worthy on just what he did as a catcher alone since a catcher should be judged differently because of the demands of the position), is part of a TEAM! Joe’s decision, which can, and has been, debated, did not work out in favor of the bad guys. If the outcome had been a strike out of Gonzalez most would not remember the manager pulling the infield in and Rivera would be hailed as a “hero” who had “done his job.” What, exactly, is a pitcher’s job? His job is to get outs, and for that he needs fielders who can make plays. Or is his job to strike out every batter? Is a pitcher a failure because he allows a batter to hit a bloop that is not caught? I think not. The yankees as a team “blew” that game, not The Closer!
Ralph Terry “blew” the 7th game of the 1960 WS because he allowed a game, and series, winning HOME RUN, hit by Bill Mazeroski, a great day in the history of baseball, because a pitcher’s job in that case is to NOT allow a walk-off ball the chicks love. To blame The Closer for allowing a bloop that ordinarily could have been caught even by the worst fielding SS in the history of MLB, Derek Jeter, is, as George H. W. Bushwhacker would say, “Beyond the pale.”
And yes, Barn, that out should be recorded because it is the job of a fielder to make ALL THE PLAYS, from the routine to the most difficult. The pitcher should not take the blame when a fielder does not do HIS JOB! Every pitcher is only as good as his fielders.
I recall reading that the manager of the Braves, Fredi Gonzalez, posted a sign for the players to read in the tunnel on the way to the field which read, “Defense wins games.” If anyone disagrees with that statement I suggest he go to B-Ref and check out the “DefEff” TEAM ratings throughout the seasons. The better teams are always near the top. For example, the Amazin’ Mets led the NL in “DefEff” and the team they beat, the Atlanta Braves, who won the Western Division (What were they doing in that division?) were right behind them in second place. As a famous song says, “There’s a reason…”
It seems like you are rewarding Billy Wagner for retiring while he was still at his peak. It seems to me that his having 20% less career IP than Hoffman should probably ding him a bit. If you subtract Hoffman’s last four seasons you get to about the same IP as Wagner and I bet that Hoffman would probably have better rate stats. Since Hoffman produced positive WAR during those last four seasons, why should Wagner get the nod over him?
Im not sure its as simple as just lopping off the end of Hoffman’s career.
Wagner was unquestionably more dominant with a career K/9 of 11.92 vs Hoffman’s 9.36
That being said, Hoffman is certainly more deserving than Sutter.
There is clearly a serious problem with relief pitchers and WAR. According to this according, radically different criteria for WAR have to be applied for relief pitchers in terms evaluating whether they should be in the Hall of Fame, and I think everyone would agree with that. But the entire idea of WAR is supposed to be that it’s a universal measure of value for any position. This is a contradiction. The method of calculating WAR for relief pitchers has to be changed significantly so that this mismatch isn’t so glaring.