Are saves predictable?

I’ve heard people talking this year—even some experts— about how a guy like Joakim Soria is a good closer to draft because the Royals are improving, or how Todd Jones is a good pick because of how stacked the Detroit offense has become. For me, I need some real evidence before I make assertions like this. So, let’s run some tests to see if saves actually are predictable.

I see two necessary steps to take if we are going to predict saves. We first need to predict how many save opportunities a team will produce for its closer. Then, we need to predict how often the closer will convert these opportunities into actual saves.

Predicting save opportunities

If we’re trying to predict saves, first we need to predict save opportunities. If a pitcher isn’t given the chance to save games, then he won’t accumulate many saves, will he? Once we so that, (if we are able to predict save opportunities, that is), then we can predict how often closers convert these opportunities.

Let’s run some simple regressions analysis to try to predict save opportunities. For each of our tests, we’ll use team data (as opposed to individual player data) from 2004-2007. First, we’ll check if a team’s wins have any bearing on the save opportunities it produces.

Wins on Save Opportunities
R Square: 0.15
Adjusted R Square: 0.14
P-value: 1.2E-05
Level of Significance: 1%

As you see, there is some correlation between wins and saves, but it is pretty weak. It can be inferred that wins are only marginally effective in predicting save opportunities.

Maybe a team’s offense is what does it, as the people who claim Todd Jones will pick up more saves believe. Let’s run the test on teams’ runs scored and save opportunities.

Runs Produced on Save Opportunities
R Square: 0.007
Adjusted R Square: -0.001
P-value: 0.35
Level of Significance: Not significant

Wow. There seems to be zero connection between a team’s offensive production and the save opportunities it produces.

Let’s see if the number of runs a team’s pitching allows has anything to do with save opportunities.

Runs Allowed on Save Opportunities
R Square: 0.06
Adjusted R Square: 0.06
P-value: 0.006
Level of Significance: 1%

Better than offensive runs, but worse than wins.

I’ll spare you 18 more sets of regressions analysis results, but I ran these same types of tests using a multitude of factors. For example, I thought maybe teams that hit a lot of singles and steal a lot of bases (in an attempt to represent “small-ball” clubs) might be better at producing save opportunities. After tons of tests like this, wins remained the best way to go.

Predicting save conversion percentage

While we didn’t get the best results when we tried to predict save opportunities, we do have something to work with. Let’s now see how well we can predict save conversion percentage. Save conversion percentage simply measures how often a pitcher turns a save opportunity into an actual save. Here is the formula: Saves/(Saves+Blown Saves).

Let’s run some more regressions tests, this time using individual players from 2004-2007. We’ll use ERA as the independent variable and save percentage as the dependent variable. We’ll include all players who received at least 25 save opportunities in a given year.

ERA on Save Percentage
R Square: 0.34
Adjusted R Square: 0.33
P-value: 5.02E-10
Level of Significance: 1%

We definitely have significant results, but the relationship between the two is still pretty weak. It’s better than when we tried to predict raw save opportunities, but still not very good.

The big question

The big question is, are saves really predictable? Does team performance or player performance have anything to do with saves? These results seem to indicate that both do play a role, but that unexplained variance plays a much larger one.

It can be surmised from today’s tests that saves are not, in reality, predictable. Consider that the results are only mediocre when we have all of the data in front of us. How do you think the tests would turn out when we’re trying to predict future results? To do this, we’d need to predict how many games a team will win in the coming year. Advanced systems can do fairly well at this, but we couldn’t even predict save opportunities well when we had the exact number of wins to work with. It would be even more difficult trying to use projected wins.

The same thing goes for save conversion percentage. Our results were only okay when we had the pitcher’s precise ERA in front of us. ERA, as you’re aware, is prone to severe fluctuation, often times by no fault of the pitcher. Using a pitcher’s predicted ERA in place of actual ERA, I’m certain the results would be significantly worse.

Put the two together, and you just have a mess. We would be trying to predict two things that were relatively unpredictable to begin with using two other predicted stats, one of which is relatively unpredictable in itself. To me, and hopefully now to you, it just doesn’t seem feasible to predict saves.

Draft plan

So where does that leave us? Well, while we can’t predict saves, knowing that we can’t is a valuable piece of information. It will help us make informed decisions during our draft and prevent us from making bad decisions.

What I left out of the tests on save opportunities was all the pitchers who didn’t actually get any opportunities. I’m sure this isn’t news to you, but the most valuable means of accumulating saves in a fantasy league is by taking players who will get opportunities. Therefore, the most important component in drafting for saves is determining opportunity.

The second most important component is drafting skill. A large portion of closers lose their jobs every year. Those who have good skills—like a Joe Nathan or a J.J. Putz or a Jonathan Papelbon—tend to keep their jobs longer. You might have a guy like B.J. Ryan get injured, but I’m sure it would have been a surprise to everyone if the Blue Jays all of a sudden promoted Jason Frasor to closer while Ryan was healthy.

Don’t take this as my blessing to go out and draft Nathan or Putz, because I would never do that in a mixed league. I’ll discuss that more in my next article, about my strategy for closers.

Concluding thoughts

What we learned today is that saves are not predictable and should not be chased in fantasy leagues. The best bets at closer are those pitchers who have a firm hold on the job and who have skills to back it up.

Be sure to come back tomorrow as I talk about the strategy I use for dealing with closers and saves.