Using WPA to grade bullpen management, part two

In my last article, I outlined a method of analyzing bullpens by making a small tweak to the inside of the WPA Clutch metric. To quickly recap, Clutch gives the difference between how a player performed and how that player would have performed in a luck-neutral context-independent environment. Clutch was designed for batters who cannot choose when and where to bat, so the first term has a pLI denominator that puts every batter on an equal footing regarding the number of high leverage plate appearances they participated in.

But since we’re measuring relievers, who don’t get equal amounts of leverage by design, the pLI denominator excises some pretty useful information. By stripping away the pLI denominator from the first term, the Clutch metric changes what the stat is measuring. By taking it out, we arrive at a number that is a sum of how a player performed in the clutch added to how efficiently a manager deployed his relievers. To put it in mathematical terms:


If we want to remove player effects and isolate the manager’s contribution, all it takes is some simple algebra.


Which finally leaves us with this.


It’s really quite simple. WPA gives a total picture of a player’s contributions to a team’s win. Subtracting context-independent WPA/LI leaves behind only the effects from the timing of game events. Subtracting away Clutch from that total pares WPA down even further, so all that remains is how a manager helped or hurt his team with his bullpen management decisions.

And now, of course, the fun part. Here’s 2012’s leaderboard.

Team Manager WPA
Orioles 5.06
Athletics 4.73
Cubs 4.51
Angels 4.09
Padres 4.08
Blue Jays 3.45
Giants 3.06
Braves 2.65
Mets 2.43
Royals 2.25
Cardinals 2.24
Mariners 1.90
Reds 1.75
Rangers 1.53
Rockies 1.52
Marlins 1.50
Rays 1.49
Yankees 1.46
Twins 1.42
Tigers 1.20
White Sox 1.17
Pirates 1.04
Astros 0.80
Indians 0.54
Diamondbacks 0.42
Phillies 0.37
Brewers -0.41
Dodgers -0.46
Red Sox -0.73
Nationals -1.01

I find this absolutely fascinating. Two teams that surprised everyone by making the playoffs are ranked one and two, with bullpen management responsible for five wins over the course of the season. Keep in mind that this doesn’t mean that Buck Showalter and Bob Melvin were five wins better than the average manager. Zero wins would mean that a manager’s bullpen management was no better than drawing names out of a hat, and as you might imagine, most managers are better than that. The average, as counted from the beginning of WPA data on Fangraphs, is a hair shy of two wins.


So what does it all mean? Managers who use their bullpens efficiently generally add somewhere around two or three wins to a team’s season total, as compared to an average two-win manager. By the same coin, managers with inefficient bullpen management can cost their teams a win or two by using the wrong pitchers in critical situations.

References & Resources
All data from Fangraphs (1974-2012).

My full data set is here. It contains WPA, Clutch and Manager figures for every team from 1974 to the present.

Newest Most Voted
Inline Feedbacks
View all comments
Ian R.
10 years ago

I’m genuinely surprised by the evenness of the results, especially at either end of the curve. I’d always believed that a bad manager could cost his team more wins than a good manager could add, but it looks as though (at least in terms of bullpen management) that’s not really the case.

10 years ago

Interesting idea, but I think small sample size quirks are skewing the results. Take the Blue Jays, for example. Applying the WPA – WPA/pLI formula to each individual pitcher gives us this top five:

0.72   David Carpenter
0.66   David Pauley
0.50   Casey Janssen
0.34   Darren Oliver
0.32   Bobby Korecky

So John Farrell, at +3.45 overall, somehow got half of that from Carpenter, Pauley, and Korecky, who together pitched ten innings and weren’t any good when doing so?

Also, 8th on the list is Ryota Igarashi (+0.25), which is just a cruel joke for anyone who sat through his appearances this year.

If we remove pitchers with less than 20 relief IP, the Toronto total goes from 3.45 to 0.44. That makes much more sense to me both in process and results.

10 years ago

I am guessing, but I would assume that Farrell used them in low leverage situations (ie: blowouts) where their lack of talent would not effect the game as much. The pLI of these players other than Janssen appear very low. Further, while he didnt have to use them at all, he saves his other bullpen arms for closer more meaningful situations.

Craig Burley
10 years ago

Right Darren, but who you use in a blowout doesn’t matter. There is no win impact at all from who you use in a blowout going either way (although guys like Pauley and Carpenter are bad enough to make any lead unsafe).

10 years ago

LOL Don Mattingly and Davey Johnson!

10 years ago

Really nice analysis if the results hold up re the complaint about the individual results. 

Re Donny, he had avg in 2011, DrB.  But yeah, lol on 2012.

Wow, Bochy shows up really nicely in the data.  First year with Pads and Giants, he had a very low win, but for Pads afterward, he was much above average, and over 8 in one season.  He has not done great wiith the Giants, but he has at least been above avg, over two, other than his first season.  I will need to add it up, but he has been above average much of his managerial career, and in total much over, which dovetails nicely with his record in one run games, where he has been much above average as well.  It would be good confirmation for this analysis if they appear to be coincident.

10 years ago

Oh, and LOL Bobby Valentine too!

10 years ago

Do the same managers do well every year?

If so, this is exciting.

If not it does paint a picture for a season as RBIs do, but it may not be useful in evaluating managers.

10 years ago

I have trouble assigning these scores to the managers because over the course of the season anagers use Prior information about each pitcher, collect New Information, and combine the prior and new information to use in the next game. The list appears to give high or low scores to teams that were presented with lots of New Information (A’s, Nationals), and near zero scores for teams that were presented with less New Information and just stuck with a formula that worked (Rays, Yankees). I don’t think that managers should be penalized for sticking with a formula that works.

10 years ago

Dan, this is really interesting. What you’re capturing however, is how well the pitcher pitched to the leverage of the situation.  You could claim that that is the manager’s job, but it seems to me you could more easily claim that that was the pitcher’s job.

In other words, haven’t you really just designed a different way of defining clutch?  Why assign the difference between this and the other definition of clutch to the manager?

Jorgenswest points to the next level of analysis, I think. If there is some predictability from year to year, particularly across different bullpens, than you’re probably onto something significant.

Jonathan Hale
10 years ago

“So John Farrell, at +3.45 overall, somehow got half of that from Carpenter, Pauley, and Korecky, who together pitched ten innings and weren’t any good when doing so?”

I think this is because using this technique, credit is given for good pitchers pitching in high leverage situations as well as vice versa (since – WPA/LI will be larger when LI is less than 0). Also, when LI is approaching zero (as for Carpenter) then the values will get really large.

Doesn’t it make also sense that one more side of the coin is much more important? I don’t think it matters much how effective a manager is at getting September callups into blowouts, and this would penalize a manager for giving a good pitcher some work in one.

Pizza Cutter
10 years ago

I would love to see whether the results for managers showed any consistency over time.