Is seeing believing?

A month ago, Greg Rybarczyk, the developer of the HitTracker website, published Seeing is Believing here at THT and made a convincing argument that the future of sabermetrics is in observational analysis. His article contained his usual meticulous research and insightful observations. I hope I am summarizing his argument correctly: that traditional stats fail to give a complete picture of what is happening on the field and need to be supplemented with additional data that can be gathered only by first-hand observation of the games. His definition of observation included both human observation and observation through the innovative use of technology such as the SportVision PITCHf/x system.

Two sentences in Greg’s article particularly intrigued me, as he uses them for the crux of his later arguments. In the section called “Limitations of Existing Systems,” Greg states: “The landing point of a fly ball can vary enormously due to the effects of wind, temperature, and altitude, so by itself, HITf/x will never be able to predict the landing point of a fly ball with any greater precision than we get today with the conceptual ‘defensive zones.'”

In the following sentence he says we’ll have a similar problem providing useful information on ground balls. My first thought on reading these sentences was “Do we have any idea what the current level of precision is on locating hit balls?” It seems this issue has never been investigated. So I did.

Two commercial concerns, Baseball Info Solutions (BIS) and STATS Inc., track hit balls as part of their voluminous baseball data-gathering on behalf of various clients. STATS told me that their hit ball data is gathered by a contract employee in the press box, and cross checked with a second employee scoring the game from video. Any discrepancies are resolved by going back to the video or adding data from other employees who were gathering the data from the game for special projects. As far as I could determine, BIS has a similar system of cross checks to ensure the accuracy demanded by its paying clients.

Rybarczyk tracks the landing location of all home runs at his HitTracker web site. Last year, he began tracking some in-play balls as well. His database for the hit balls of Torii Hunter and Andruw Jones was the basis for his article “Of Home Runs and Free Agents” in the 2008 Hardball Times Annual. He uses careful observation from commercial video and detailed models of each stadium to ensure the accuracy that is the basis of his Website’s reputation.

These three data sources give the distance to the nearest foot. Greg and BIS give the angle to the nearest degree. STATS places each hit in a zone of approximately 4 degrees. For computational purposes, I gave each hit in a zone the angular measurement of the center of the zone.

MLB’s Gameday also tracks hit balls for use in its Gameday graphics and hit ball charts. It, too, uses a contract employee in the press box to gather the raw data. But the end use of graphical representation for entertainment purposes doesn’t require the same system of cross checks for accuracy. Cory Schwartz, MLB.com’s director of stats, says that the contract employee’s training includes methods and emphasis on obtaining the best data possible, but Schwartz realizes the limitations of having only a single uncorroborated source.

The data is recorded in an XY coordinate system because it integrates with the graphics, but this is also a source for error during the translation to a angle-distance format based on feet and degrees. For hit balls that are actual hits, the system records where a ball is picked up by a player rather than where it hits the ground. Even though this differs from other hit ball location systems, this was a conscious decision on what would be a most accurate graphical representation of the play for Gameday viewers.

I already had the Gameday hit locations for the last three years integrated with my Retrosheet database. Greg provided his spreadsheet for the hit locations for Hunter and Jones. STATS graciously provided the same data for research purposes. I paid a nominal fee for the BIS data. I now had four independent observation sources for the locations of the 947 hit balls of those two players.

Well, not quite. Because home runs are problematic and because of Gameday’s unique method of recording the locations of hits, I decided to limit the data to balls in the field of play that were fielded for outs. This still left 568 data points that could be compared from each source.

Let me emphasize that none of the analysis that follows will help us know the actual landing locations of any hit ball or which of the four sources has the most accurate data. We will never know precise hit locations until we have chips in the ball or triangulated data from cameras covering the whole field.

What this analysis can give us is an idea of the confidence we can have in human observation as a source for hit ball locations. If independent trained observers are in close agreement, then we can have more confidence in their observations. The greater the difference in their observations, the less confidence we have. With that caveat, we have table 1.

Table 1. Hit Ball Location Standard Deviation Between Sources
First Source    Second Source    Distance SD feet   Vector SD degrees
BIS               GREG                 10.29            2.22
BIS               STATS                10.97            2.94
GREG              STATS                11.95            2.67
MLB               BIS                  12.72            3.04
MLB               GREG                 13.37            3.12
MLB               STATS                14.10            3.64

Depending on your expectations and point of view, these could be good numbers or bad numbers. That the two observers who are closest in agreement can’t place the ball within 10 feet of each other more than 68 percent of the time may be a little discouraging to some.

That BIS and STATS would disagree on what zone a ball is in more than 32 percent of the time might make some fielding analysts pause. They will pause even longer when they investigate further and find out that the actual zone agreement is only 46.4 percent. However, that Gameday is not further off the mark on this subset of data might inspire analysts to find new uses for its hit location data.

Many of you may just find this data confusing. What if we asked the question a different way? What if we guessed the most likely spot for the actual hit ball location and asked, “What is the minimum in distance and degrees that has the two best observers within that error range at least 95 percent of the time?” Isn’t that what we really want to know for the basis of a fielding metric?

Let’s take the two observers in closest agreement, BIS and Greg, split the difference between them and call that the best guess of the actual hit location. What is the minimum distance and degrees that will have 95 percent of both Greg’s and BIS’ observations included? The answer is +-18 feet and +-4 degrees. That’s a pretty big area. It is two whole zones in width.

Perhaps we can make an improvement by adding the third expert observer, STATS, and establishing the best guess of the actual hit location as the average of all three? On The Book blog Tom Tango has suggested this “wisdom of the crowds” method of multiple observers recording the data. Let’s try it and see what happens. The answer is +-22 feet and +- 6 degrees.

Why did the error get larger? That’s not what the wisdom of the crowds would predict. Actually, it’s entirely logical and points out the fallacy of the “wisdom of the crowds” theory. Adding an additional observer to the best two will always make the error greater. Adding an observer can only move the original average further away from one of the two “best” observers, increasing the total error.

To make an improvement, the third observer’s data points would have to be closer to the original best guess of the actual hit location, but if they were, they would also be closer to one or the other of the original two “best” observers, which is a contradiction because we defined the original two “best” observers as the ones whose data was closest. It doesn’t matter if you have three observers or 3,000, the composite data will never have any less error than that of the two closest. Having many observers is only useful for finding those two best observers.

It turns out that +-18 feet and +-4 degrees is the best we can do for these four observers and given the redundancy built into STATS and BIS, Greg’s thoroughness, and the high motivation for accuracy of all three sources, it probably is very close to the best we can expect for any human observers. Whether HITf/x will be able to be more precise remains to be seen since the system is not yet a reality.

For those of you not familiar with the proposed HITf/x system, it would use the same hardware as the existing PITCHf/x system and basically the same software, but modified to track the outgoing hit ball instead of the incoming pitched ball. It would provide a direct measure of the speed off the bat, the initial vertical angle of the hit ball, and the initial horizontal angle. The landing location of the hit ball would have to be computed from these initial inputs since the existing cameras do not cover the entire field.

It is the precision of this computed landing location that Greg was skeptical of in his article. But since it is still in the conceptual stage and since SportVision has been unusually open to suggestions as to how it will be structured, we have an extraordinary opportunity to make HITf/x as good as it can be. That means incorporating Greg’s suggestions for including wind, temperature and altitude factors.

HITf/x also would have some additional benefits for fielding analysis. Accurate speed off the bat data is useful for determining whether one pitcher’s hit balls are easier to field than another’s. During this study I also found that there is not a consensus as to whether a hit ball is a fly, a liner or even a ground ball. An objective definition of fly balls and line drives should be possible using the initial vertical angle and the speed off the bat as parameters.

Given the lack of precision of our current data, HITf/x certainly deserves a chance.

References & Resources
My thanks to Cory Schwartz at MLB.com, Greg Rybarczyk at HitTracker.com, Marv White at Sportvision, Damon Lichtenwalner at Baseball Information Systems and Jeff Chernow at STATS Inc. for help in preparing this article. And thanks to Retrosheet for freely providing the data that make any baseball statistical research possible.