Evaluating catchers: Quantifying the framing pitches skill

Catchers’ ability at framing pitches is a skill that, since the advent of PITCHf/x data, seems so close to being understood and measured. Several analysts have given first passes at the issue (Dan Turkenkopf in 2008, Matthew Carruth and Bill Letson in 2010).

Following is another contribution on the subject, which will give a rough estimate of the runs a catcher might save in a season thanks to this particular ability. A file with catchers’ ranking is available at the end of the article.

Definition and problems

The framing skill will be measured here as the increased (or decreased) probability of a pitch to be called a strike due to the presence of a particular catcher behind the plate. Obviously it is not easy to isolate the receiver effect from the rest of factors influencing the final umpire’s decision. Otherwise, a framing measure would have been around for some time now.

Several components concur and interact in the outcome of a pitch not swung at. John Walsh has shown how umpires tend to be “compassionate”, give the benefit of doubt to the batter on two-strike counts, while helping the pitchers when they are one ball away from issuing a walk. Other than their merciful nature, umpires are individuals and each has his own peculiar strike zone. Some of them are known for being pitcher friendly, while others are particularly stingy at calling strikes.

Pitchers also have different abilities of getting a favorable call on borderline pitches. Tom Glavine, for one, was able to expand the strike zone inch after inch (at least that was his reputation). Also, different kind of pitches, due to their particular movement, have unequal chances of being declared strikes.

Finally it is possible that the batters themselves can influence the call, because of their stance or their established reputation as selective hitters or free swingers.

Tools and technique

Since the nature of the subject is full of complications, this analysis focuses on a selected part of the problem. Only outside pitches to right-handed batters will be considered. Let’s go step by step in selecting our subset of data.

Pitches are selected such as their height is unlikely to influence the strike call. Mike Fast has recently shown that nearly every pitch delivered between 2.2 and 3.0 feet of height is called for a strike if it’s thrown to the fat part of the plate. The charts below show the chance of pitches at that height of being called strikes, as a function of their horizontal location.

Focusing our attention to the outside part of the plate (thick blue line in the left chart), we see the probability of a called strike has sort of a sigmoidal trend: It’s constantly close to 100 percent in the fat part of the plate, and constantly close to zero from two feet outside on. The uncertainty is in the zone in between, which has been zoomed and marked with a thick red line in the right chart: Here is where the factors mentioned before, including the catcher’s skill, come into play. Thus pitches (every one recorded by PITCHf/x from 2007 to the end of May 2011) in that zone are considered for this analysis.

To deal with pitcher-catcher simultaneous effects on wild pitches and passed balls, Tom Tango has widely used a technique he dubbed With-Or-Without-You, shortened to WOWY. In a previous article I proposed an advanced statistical technique which makes possible to take care of more-than-two-ways interactions. In that case it was possible to measure the base stealing game giving appropriate credit to pitchers, catchers and baserunners.

Here the multilevel logistic regression with crossed random effects (that’s the name of the technique for those who want to check) allows us to apportion the difference in called strikes probability among umpires, pitchers, catchers and batters.

Results

Before ranking the catchers, it’s useful to check if the estimated effects of other (more known) factors make sense. According to the model, the chance of a borderline pitch to be called a strike decreases of about 20 percent for every inch it gets further from home plate. So far so good.

Below is an estimate of the change in probability according to the ball-strike count (The 0-0 count is set as the reference).

count  prob change
 0-1       -16
 0-2       -19
 1-0         4
 1-1        -7
 1-2       -12
 2-0         8
 2-1        -4
 2-2       -11
 3-0         2
 3-1        -7
 3-2        -7

The results are quite in line with the previous knowledge, as the player who is behind in the count usually gets the favorable call. The 3-0 value is probably lower than one would expect after having read Walsh’s work.

The umpire ranking resembles the one by Jeff Zimmermann at Beyond the Boxscore”>Jeff Zimmermann at Beyond the Boxscore (circa 2010), with Brian O’Nora, Brian Runge and Wally Bell outlined as pitcher friendly and Paul Schrieber, Gerry Davis and Adrian Johnson among the stingiest.

Regarding pitchers, I did not know who to expect at the top and at the bottom of the list. Since Tom Glavine is always cited as a pitcher who could get the call on borderline pitches, it’s reassuring to find his name in the above-average part of the ranking (though not at the extreme).

The most likely to get a questionable call their way are, according to the model, Francisco Cordero, Jake Peavy and Chris Young. The least likely, on the other side, are Max Scherzer, Jonathon Niese and Jonny Venters.

What to expect in the batters’ table? A great hitter like Albert Pujols being among the ones getting the benefit of doubt should not be a surprise, perpetrating a decades long tradition best symbolized by Hall of Fame umpire Bill Klem’s address to a rookie pitcher: “Son, when you throw a strike, Mr. (Rogers) Hornsby will let you know it”.

Finding a selective hitter like Marco Scutaro on one side (getting the benign call) and noted free swingers like Vladimir Guerrero and Jeff Francoeur on the other is also expected (however there doesn’t seem to be a correlation between hacking tendencies and rank on the list).

Since the model seems to give sensible results for the other factors, it’s quite likely that what we will see for the catchers is a decent approximation of the truth as well.

Catchers

Since logistic regression results might not be of immediate interpretation, what you will find in the Excel file below has gone through a sort of translation to make it easier to understand.

Thus the four sheets in the Excel file (80 Kb) report the difference in probability due to the particular player (or umpire) involved.

The magnitude of variation is highest for pitchers and lowest for batters, with umpires and catchers in between and quite close between them.

It makes sense that batters are less likely to influence the call since, other than their stance, there should be nothing they can do to play a role in the outcome. It also makes sense that the pitchers have the most power on the call, as they are the ones delivering the ball and imparting the (possibly deceiving) trajectory.

The fact that umpires and catchers have a similar range of variation implies that playing with a receiver who is good at framing pitches is the equivalent of having a pitcher friendly umpire calling the game.

Games won thanks to framing pitches

Let’s do some back-of-the-envelope calculations. A top catcher at framing pitches, such as Russell Martin, improves the chances of a borderline pitch to be called a strike of roughly 20 percent. Since the difference in run value between a ball and a strike has been estimated around 0.13 runs (see the References and Resources section at the end), a skilled catcher might be worth 0.026 runs on a single borderline pitch.

In the data used for this analysis, each game contributes on average four pitches. That is, there are four borderline pitches per game on the outside corner. Let’s conservatively guesstimate a total of eight when we add the inside part and the upper and lower border of the strike zone. Then divide by two, to apportion the borderline pitches between the two teams playing the game. That makes four uncertain pitches a game where the catcher can make the difference.

Martin caught 97 games in 2010. Multiply that by four pitches and by 0.026 runs and you get 10 runs in a limited number of games.
According to this analysis the top catchers can win a ballgame per season (even playing fewer than 100 games) only with the skill of framing pitches.

If you think that’s a lot, I’m with you.

Anyway, let’s look at that from a different perspective. Please re-read the last sentence of the previous section. The fact that umpires and catchers have a similar range of variation implies that playing with a receiver who is good at framing pitches is the equivalent of having a pitcher friendly umpire calling the game. Now, suppose you are allowed to have every game called by an average umpire when your team is at the plate and by the most pitcher-riendly umpire when your team is pitching. Does an extra win per season seem an acceptable effect of having such an advantage?

Reliability, assumptions and limitations

Replicating the analysis on two subsets of the data (even-numbered caught pitches and odd-numbered caught pitches for every catcher) we get pretty stable results: The correlations of the catchers’ effect calculated in the two subsets of pitches is a solid 0.76.
The pitchers’ effect shows a similar stability (0.75), while the batters’ expected effect returns a lower correlation (0.40). Finally, the umpire effect lends the highest correlation (0.89) between the two subgroups, a good indicator of individual consistency.

As it always happens when building models, some simplifications have been made.

Here the “compassionate effect” is modeled as constant among umpires, while probably each man in blue has his own peculiar way of expanding and shrinking the zone according to the count. Also, while the probabilities are adjusted by the type of the pitch, again it is assumed that umpires react uniformly to the different behavior of fastballs, curves, and the rest.

Similarly the catchers’ skill is measured across the board; in other words, the model does not consider the possibility that one backstopper can be awful at framing fastballs but very competent on offspeed pitches. Plus, when doing the back-of-the-envelope calculation of saved runs, we implicitly assumed that catchers have equal skills on pitches outside and inside, high or low.

While the aforementioned assumptions do somewhat simplify the reality, it would be very difficult and computing intensive to add those individual variations in a model, as the one used for this article runs for hours.

Final remarks

According to the analysis presented here, the best catchers at framing pitches can add something like one or two wins per season, which is the equivalent of trading Alex Rodriguez’s 2010 bat (.270/.341/.506, 30 HR, 125 RBI, 19 runs above replacement) with Alex Rios’ lumber in the same year (.284/.334/.457, 11 HR, 45 RBI, nine runs above replacement).

The number could even be a conservative estimate. In fact, as soon as a pitcher realizes his catcher gives him an edge on borderline pitches, he should immediately begin to exploit the advantage.

If the magnitude of the framing effect measured in this study is confirmed, major league teams should not neglect this factor when they go hunting for a catcher in the market, especially those with pitching staffs that make their living on the black.

References & Resources
PITCHf/x data by MLBAM, corrected using the author’s algorithm.

Dan Turkenkopf on the value of switching a ball to a strike.

John Walsh on pitches run value.

Craig Burley on the benefit of throwing a strike, by count.