Defensive Regression Analysis: Part Three

Editor’s Note: Last year, Michael Humphreys introduced a revolutionary new fielding statistic, called Defensive Regression Analysis (DRA), which represents an entirely new way of thinking about fielding stats. DRA uses stats that are available throughout baseball history so it can be used to evaluate fielders of any era. We consider it a significant improvement over fielding Win Shares.

The original DRA article (pdf) submitted to Baseball Primer is now available. Also, Web Archive has the original Primer articles, with the correct formatting — Parts One, Two, and Three.

In this series of three articles, Michael will explain DRA, use it to evaluate major league fielders from 2001-2003, and compare it to zone-based systems such as Zone Rating and Ultimate Zone Rating in order to verify its accuracy.

The first and second articles have already been presented. A complete document of this work is now available.

Complete DRA Test Results for 2001-03

So that you won’t be overwhelmed by a mass of numbers, I’ll show results separately position-by-position, explain the deletions and make other comments below. The “check” comment indicates a greater-than-12-run difference between UZR and DRA (as calculated by the Excel spreadsheet, without rounding); the word “dra” indicates that non-UZR zone-based information supports DRA more than UZR; “pk” flags a park effect in Colorado, which DM says depresses out-conversion rates at each outfield position by about 24 plays per season. (DRA currently has no park factors, and I personally believe they’re not worth the trouble in evaluating fielders except in extreme cases, such as left field in Fenway.)

A. Shortstop

Pos  Yrs  Last          UZR  DRA   ZR  DFT 
 6        Aurilia         1   -9    5    5 
 6    3   Cabrera        11   13    5   13
 6        Clayton        15   -4    4    5  Check  dra
 6        Cruz           -5  -10    3   -4 
 6        Furcal          7    5   -5   -4 
 6        Garciaparra     9   12   -3   -7 
 6        Gonzalez, A.    2    2    3   -6 
 6    3   Gonzalez, A.S.  8    8    5   10 
 6        Guillen         3   -3    1  -11 
 6        Guzman        -16   -2  -12   -5  Check  dra
 6        Hernandez      16   10    8    7 
 6        Jeter         -25  -22  -19  -20 
 6        Ordonez        -1   -1    4    6 
 6    3   Renteria        7    7    5   -3 
 6    3   Rodriguez       9   -5   12   10 
 6    3   Rollins       -11    3    2   -1  Check  dra
 6    3   Tejada         -1    6   -8  -11 
 6        Vizquel         8   -4   -4    9 
 6        Wilson         -8    3   -1    2 

          Avg             1    0    0    0    
          Std            11    9    7    9    
          corr w/UZR   1.00 0.58 0.69 0.62

UZR rates Royce Clayton as one of the two best shortstops of the period. He is never mentioned in 2001-03 DM Gold Glove reviews. The negative inference from just that data source is that he was either average or below. DM specifically says in its team evaluation in 1999 and 2000 that Royce was average. DRA, DFT and ZR are all closer to this evaluation.

Guzman is trickier to evaluate. He is the worst shortstop under UZR (other than every sabermetrician’s “favorite” fielder, Jeter). He’s not mentioned in the 2001-03 Gold Glove essays, but all we can conclude from that is that he was average or below. DM says nothing about Guzman in 2000, but in 1999 they describe him as “spectacular” but “inconsistent,” due to his “botching” the easy plays and making too many errors. His error rates actually went down during the 2001-03 period, and his stolen base data doesn’t suggest a decrease in his speed. As reported by Raindrops, PMR gives him a 2003 per-162-game rating of -6; that’s a little difficult to interpret, because it includes fielding fly balls, which UZR, ZR, DM (and DRA) ignore. (I don’t know about DFT.) Guzman’s 2004 PMR ground ball fielding rating (which is what UZR and DRA focus on) is only -5 plays, or -3 runs, in a season in which PMR shows league-average shortstops underperforming multi-year baseline data, so Guzman was average or perhaps even slightly above-average in 2004. ZR probably rates him low because the ZR “zones” exclude a lot of the field, so that a low-range, sure-handed fielder will look better than a rangier fielder who “botches” the routine plays. I believe the evidence overall suggests that Guzman was probably a little below average in 2001-03, but not a downright poor fielder.

The UZR A-Rod rating is more accurate than the DRA rating. Without going into all the detail, DM consistently identifies A-Rod as someone with average-to-slightly-better-than-average range and good hands. (He does have notably good error rates.) My best guess as to how DM evaluates range and surehandedness is that range includes reaching a batted ball, even if you drop it for an error. Thus, a fielder such as A-Rod with slightly better than average range and great hands will make several plays more than average overall. This is reflected in his high ZR. Solid, sure-handed fielders have higher relative ZR than UZR (or DRA).

Jimmy Rollins is never mentioned by DM, but has a +3 runs-per-162-game PMR rating, and is -4 plays under PMR in 2004 (separate ground ball rating unavailable). ZR rates him +2 for his Full Seasons in 2001-02. I think the non-UZR consensus is closer to DRA than UZR.

I have a theory that UZR mismatches in the infield are somehow due to the calculation of the effect of errors. UZR tracks them and calculates them separately from plays made. Tangotiger reports on his website that allowing a runner to reach base on an error is basically no worse than not making the play at all. Errors should simply be treated the same as plays not made. My best guess is that UZR somehow overrates errors, so that surehanded infielders rate higher and error-prone infielders rate lower. Recall the +39 Rey Ordonez rating for 1999, in which he made only four errors.

If you delete Clayton, Guzman and Rollins from the sample, you get the following summary results at shortstop (edited sample size of 16 players):

Shortstop (16)  UZR   DRA    ZR   DFT
Avg               2     1     1     0
Std              10     9     7     9
corr w/UZR     1.00  0.75  0.69  0.65

DFT does well, in some ways better than ZR, because it shows a better standard deviation.

B. Second Base

Pos  Yrs  Last          UZR  DRA   ZR  DFT  
 4        Alomar        -13  -13   -9    2  
 4    3   Anderson       -4    2    8   -9  
 4        Biggio         -1  -18  -13  -14  check  dra
 4    3   Boone          14    9    4    7  
 4    3   Castillo        0    1    6   -2  
 4        Grudzielanek   11    4    5   -2  
 4    3   Kennedy        21   15   13   11  
 4        Kent            7    4    2   12  
 4        Rivas         -20  -22  -14  -19  
 4    3   Soriano        -4  -15   -8  -11  
 4        Vidro           1    0   -5   -5  
 4        Vina            4  -10    4   -6  
 4        Walker         -9   -9   -5  -12  
 4        Young         -11   -8    5   -3  

          avg             0   -4    0   -4    
          std            11   11    8    9    
          corr w/UZR   1.00 0.83 0.66 0.71

The decision to delete Biggio is difficult. Let’s look at the complete 2001 DM analysis regarding Biggio:

This former Gold Glover missed the last two months of the 2000 season with a knee injury that required surgery. In January, his general manager warned that Biggio’s range and baserunning ability would most likely be limited, especially early in the year. Those comments proved to be accurate, as Biggio’s range was far below its previous level and he stole only seven bases, down from 50 only three years ago. His baserunning instincts are still good, so he was a little above average in that regard, but nowhere near the Excellent level he sustained before he hurt his knee. DM, “2001 Gold Glove Review.”

I’m not certain whether “Excellent” refers to his baserunning or his fielding, but grammatically it refers to baserunning, and DM provides such ratings (“Excellent,” “Very Good,” etc.) for baserunning separately from hitting and fielding. DM says Biggio had a “classic year” either in 1999 or 2000, without mentioning fielding in particular, so I have no particular DM information about what Biggio’s “previous [pre-2001] level” of fielding performance was.

Just for the sake of argument, let’s assume DM rated Biggio’s fielding as “Excellent” at one point during his career. One of the difficulties of relying on DM is that they seem to think that overall fielder ratings should balance out range and sure-handedness; one of their essays asks whether one should prefer ranginess or sure-handedness. No balancing act is appropriate—you simply need to look at the number of plays made given zone-identified opportunities, and when I look to DM for evidence about UZR, I focus on the range information and then adjust up or down for the (usually) relatively tiny variance in error rates. Biggio did always have low error rates at second, so perhaps an Excellent rating could reflect DM’s tendency to value sure-handedness for its own sake. Anyway, my November 2003 DRA article has a long discussion about Biggio, who has poor DRA ratings during his early seasons after making the historic (first ever?) shift from catcher to second base, eventually has a good DRA rating or two, then has terrible DRA ratings after his injury in 2000. I also quote Biggio’s admission that switching from catcher to second was the “hardest thing” he ever did in his life. My take is that with his speed (and perhaps agility) gone, he could no longer field second base effectively. Even his ZR numbers (which, remember, overemphasize surehandedness and underemphasize range) are quite low. Deleting Biggio from the sample helps DRA, but also ZR and DFT.

A Hardball Times Update
Goodbye for now.

DM identifies Vina in 2001 as being the best National League second baseman, with “above average” range. In 2002 they say his range declined “noticeably” to “near average.” That sounds like a +10 to +15 in 2001 and a +5 in 2002, for what would be an average rating between +5 and +10. UZR is clearly correct.

Second base summary results, with Biggio deleted.

Second Base (13)  UZR   DRA    ZR   DFT
Avg                 0    -3     1    -3
Std                11    11     8     9
corr w/UZR       1.00  0.88  0.73  0.74

C. Third Base

Pos  Yrs  Last          UZR  DRA   ZR  DFT  
 5        Alfonzo        -3    2    2    0  
 5        Batista        -3    3   -4   10  
 5        Bell           28   10    9    7  check dra
 5        Beltre         16    1    5  -10  check
 5        Castilla        1   -4    2   -8  
 5    3   Chavez         17   12    8   12  
 5        Cirillo        25    9    9   12  check dra
 5        Glaus         -10   -2   -1   -4  
 5    3   Koskie         11    9    7   11  
 5        Lowell         -6   -3   -2   14  
 5        Ramirez       -25    0   -5   -9  check dra
 5        Rolen          18   13    6   10 
 5        Ventura        19   10    7    3 

          Avg             7    5    3    4  
          Std            15    6    5    9  
          corr w/UZR   1.00 0.77 0.94 0.43

During his Full Seasons in 2001 and 2002, David Bell was probably about as good as Scott Rolen was, on average, during his 2001 and 2003 Full Seasons, that is, about +15 runs per season, on average. He was not 10 to 15 runs better than Rolen. DM says Rolen was “amazing” in 2001, and elsewhere says he was approximately +40 plays, or about +30 runs. That I would consider the absolute upper bound for single-season performance at third. DM notes that Rolen fell to just about average performance in 2003. Thus an average +15 rating or so; UZR gives him a +18 rating. DM never describes Bell in those terms, but does say he was second or third best in the American League in 2001 and second best in 2002. He’s clearly a +10 to +15 guy.

A similar problem occurs with Cirillo. DM says he was the second or third best in the National League in 2001; UZR gives him a +35 runs saved rating. DM doesn’t mention Cirillo at all in the 2002 Gold Glove essay. Assuming he was average then, he looks to be about +5, maybe +10 on average over those two seasons.

UZR is probably right about Beltre, and his rating is the only infield DRA rating I’m really unhappy about. DM does not mention Beltre in the 2002 Gold Glove essay, but it does consider him for its “Gold Glove” for 2003 due to his “good” range. His fielding drew the “very good” and “excellent” adjectives in 1999 and 2000.

Ramirez is probably not a good fielder, but he is almost certainly not the disaster that UZR says he is. There is no DM commentary, but his 2003 PMR (at a 162-game pace) is only -6 runs. His ZR over his 2001-02 Full Seasons is -5. His provisional 2004 PMR is about +4 runs, but that is against a base that seems to rate most third basemen in 2004 as above average against a multi-year baseline.

Deleting Bell, Cirillo and Ramirez, here are the summary results at third:

Third Base (10)  UZR   DRA    ZR   DFT
avg                6     4     3     4
std               11     6     4     9
corr w/UZR      1.00  0.80  0.89  0.11

DRA’s standard deviation is a little low, but ZR’s is even smaller.

D. Center Field

Pos  Yrs  Last          UZR  DRA   ZR  DFT  
 8    3   Beltran         6    6    8    9  
 8    3   Cameron        28   24   10   12  
 8        Damon          13   -1    5    4  check
 8        Edmonds        -4    5    4   10   
 8        Erstad         42   36   13   20   
 8    3   Finley        -16   -3   -3    2  check  dra
 8    3   Hunter          8    0    4    3  
 8    3   Jones          15   24   -1   17 
 8    3   Pierre         -1   -4    2   -2  pk
 8        Wells          -2  -14    5    0  check
 8        Williams      -20  -11  -14  -11  
 8        Wilson         -4  -26   -9   -6  check  pk

          Avg             5    3    2    5  
          Std            17   18    8    9  
          corr w/UZR   1.00 0.81 0.78 0.81

DM describes Damon as the #4 2002 American League centerfielder and in 2003 as “close” to being as good as Torii Hunter, who is described as one of the best centerfielders in either league. UZR is right; DRA isn’t.

UZR gives Finley a -40 runs allowed rating in 2003. Minus forty runs. Though DM concedes that “age caught up with” Finley in 2003, it gives no indication that he was anything close to being that bad. DM rates his range “above average” in 2002, ignores him in their 2001 Gold Glove essay, and says he was “basically average” in 1999-2000. PMR gives him below-average ratings in 2003 (-13 runs per 162 games) and approximately -13 runs as well in 2004. My interpretation of all of this information is that Finley was probably about -15 runs in 2003, perhaps +10 runs in 2002 and +0 in 2001, so between -5 and 0 for 2001-03. I believe that DRA, ZR and DFT are closer to DM’s perception.

Pierre and Wilson ratings are shown, but I will delete both of them for the Denver park effect described above. DM doesn’t discuss Wells in 2002, but says he was “close” to being as good as Hunter in 2003. So DRA is clearly wrong.

Erstad’s UZR and DRA ratings are quite close, so Erstad is not flagged under the “AED” test, but it’s a fair question whether both ratings are so screwy that they should be deleted. DM describes Erstad as its “Gold Glove” centerfielder in 2001 and “Defensive Player of the Year” in 2002. Furthermore, based on the DM quotation in Part II.C referred to in the Rey Ordonez discussion, Erstad may in fact have made close to 60 marginal plays in 2002. Arguably that would results in a +30 average runs-saved rating over two years. Since DRA isn’t too far from that, I left it in.

My best guess for the sometimes extreme UZR ratings in the outfield is that the park effects used under UZR may be excessive.

Deleting Finley, Pierre and Wilson, we get the following summary results:

Center Field (9)  UZR   DRA    ZR   DFT
Avg                10     8     4     7
Std                18    17     8     9
corr w/UZR       1.00  0.86  0.78  0.83

E. Corner Outfielders

Left field and right field are combined in this chart, because the sample of players who played full-time at just one of those positions is so small.

Pos  Yrs  Last          UZR  DRA   ZR  DFT
 9    3   Abreu          -7   -4    5   -6 
 9        Burnitz       -15    0   -4    1  check dra
 9    3   Green         -13    9    5    6  check
 9        Guerrero       16    0    4   -6  check
 9    3   Ordonez        -7    1    5    2
 9    3   Sosa           -6   -2   -1   -4
 9    3   Suzuki          7   15   -2   12 
 7    3   Anderson       -2    4    6    6 
 7        Bonds          -8   -6   -7   -4
 7    3   Burrell       -11  -12    2    0
 7    3   Gonzalez       11    3    5    5
 7        Jones, C.       2   -5    2   -6
 7        Jones, J.      14   12    9    5
 7    3   Lee             4    0   10   -3

          Avg            -1    1    3    1  
          Std            10    7    5    6  
          corr w uzr   1.00 0.42 0.45 0.12

Once again, right field gives DRA (but also ZR and DFT) trouble. Shawn Green is the only player, in either the 1999-2001 or 2001-2003 DRA tests, whom DRA rates as clearly above average who is pretty clearly below average. DM says there is a slight park effect favoring outfielders at Dodger Stadium, but I don’t think that can explain the error. All other “misses” generally involve the failure to identify a good fielder by rating him only average. (No good fielder is rated poorly by DRA in either test.) Green actually won a “real” Gold Glove in 1999 without DM second-guessing it, yet (A) DM never mentions Green during 2001-03 Gold Glove essays, (B) the PMR per-162-game rating for 2003 is -23 runs, and (C) the Dodgers moved him to first base in 2004. UZR is probably right about this one.

According to DM, Guerrero had “great range” in 2001, but wasn’t mentioned thereafter, as injuries reduced his range. Nevertheless, the UZR rating is closer than the DRA rating.

Jeromy Burnitz is another tricky case, similar to Biggio, whose deletion from the sample someone could take issue with. DM has nothing to say about him. Raindrops does not provide a PMR rating for 2003. But Burnitz’ team moved him to center for about half his playing time in 2004. He was well below average there (probably about -10 runs for less than half a season’s worth of play) and about -7 runs in right field (also part time), but both those outcomes may be due to the unusual Denver park effect and are in any event are derived from samples far too small to draw any conclusions from. Burnitz maintained average or well-above average Range Factors (ugh) during every one of his five Full Seasons in right field (including four Full Seasons with Milwaukee and one Full Season with the New York Mets), and actually played centerfield for 20 or more games for three seasons with three teams (Milwaukee, LA and Denver). That doesn’t sound like a right fielder who costs his team 15 runs per season.

Deleting Burnitz (and leaving the misses in):

Corner Outfield (13)  UZR   DRA    ZR   DFT
Avg                     0     1     3     1
Std                    10     7     4     6
corr w uzr           1.00  0.44  0.35  0.13

DRA doesn’t do too well, but the problem is limited to right field and DRA anyway clearly outperforms ZR and DFT. By the way, the UZR ratings in the outfield do not include “Arm Ratings.” I thought DFT ratings might include the effect of outfielder assists, but the UZR/DFT correlation did not improve by including Arm Rating in UZR.

F. First Base

Pos  Yrs  Last          UZR  DRA   ZR  DFT  
 3    3   Bagwell         7    0   -6   -5  
 3        Casey           5  -11   14   -3  check
 3    3   Delgado        -2    1   -1   -3  
 3    3   Helton         22   11    7   16  
 3        Konerko       -10   -6   -3  -11  
 3    3   Lee, D.         9    8   12    7  
 3    3   Lee, T.        11    8   13    8  
 3        Martinez       14    5    8    9  
 3    3   Mientkiewicz   10    6    7   12  
 3    3   Olerud          0    8    2    2  
 3    3   Sexson         -5    9    1   15  check
 3        Thome         -14  -10    5   -6  
 3        Young          10    0   12    5  

          Avg             4    2    6    4 
          Std            10    7    6    8 
          corr w uzr   1.00 0.56 0.50 0.64

Not much to say here, except that DM, PMR and ZR are clearly more consistent with UZR than DRA in evaluating Casey and Sexson. As mentioned before, the problem is that we can’t know how many ground balls are fielded by first basemen. In the November 2003 article I argued that Bill James was wrong, and that estimating unassisted putouts by first basemen is not worth the effort, because too many pop-ups and short fly outs pollute the total. I’ve since come around to Bill’s point of view and adapted his methods to DRA.

Editor’s Note: This concludes Michael Humphreys’s series on Defensive Regresssion Analysis.

References & Resources
I’d like to thank Dick Cramer for his support in the past, Mitchel Lichtman for creating UZR, and baseball analyst, Tangotiger, for making detailed UZR output available in a convenient form. I’d especially like to thank the folks at Retrosheet:

The information used here was obtained free of charge from and is copyrighted by Retrosheet. Interested parties may contact Retrosheet at 20 Sunset Rd., Newark, DE 19711.

There is one more absolutely necessary acknowledgement: my own fallibility. In creating DRA and tracking the results of other fielding systems, I had to do a tremendous amount of cutting and pasting and hand-coding of data. I have done my best, but I’m sure there are some errors, though I don’t believe any of them are significant.

I look forward to hearing from you. Don’t hesitate to e-mail with questions, criticisms and corrections.


Comments are closed.