How do we know we’re right?

I enjoy sticking my nose where I haven’t been asked to, and I also enjoy not having to develop my own concepts for my column, therefore, I think you guys can anticipate what time it is. That’s right, it’s time for unsolicited commentary on the Card Runners-rooted discussion du jour from the other Derek!

Seriously, I’d like to touch on some of the more abstract issues raised around identifying and claiming success for identifying breakouts that were raised in Derek Carty’s post, and the comments section thereof, last week.

The role of intuition in identifying breakouts

Both Eric Kesselman and Chris Liss imply that deeply engaged baseball or fantasy baseball fans are subject to strong gut instincts about players that are difficult to put in the context of objective analysis. Somewhat surprisingly, I actually agree. By definition, a break out is a player who is redefining his past profile by making a quantum leap in terms of translating his skillset into production. So, while it is entirely reasonable to expect to be able to study development patterns and elemental metrics to identify those who are more likely to “break out,” the spirit of what is happening in a break out – a player doing something he’s never done before – is difficult to predict by betting on pattern and trends alone.

But, before proceeding, it is important to ask what our intuition about certain players really is. In the outset of Blink, Malcolm Gladwell tells the story of an art curator who is unsure of the authenticity of a sculpture that would be highly valuable were it to be determined authentic. At this point, an accomplished artistic scholar views the piece and despite performing no tests, and without giving detailed reasons, determines the piece a fake. The owner of the sculpture brings in experts, and basically enacts the full CSI treatment, which generally points to the piece being authentic. Subsequently, new information is unearthed confirming the work as a fraud. (Forgive me if my retelling of this anecdote is a bit off, as it was many years ago that I read the book and have just realized, after attempting to reread it, that I must have leant it to somebody who never returned it.)

The point Gladwell makes, and the predominant, recurring theme of the book, is that people are actually rather good at “thin slicing” very complicated, multi-faceted sets of information. I think this notion is generally applicable to the discussion at hand, but it does come with many caveats.

For one, every Tom, Dick and Harry is not qualified to have these insightful intuitions. If Eric Kesselman has a strong intuition about a poker hand, or Chris Liss has a strong intuition about Ricky Romero, these are things that shouldn’t be causally dismissed just because the objective evidence supporting the intuition is not entirely robust enough to support the claim. But, if some fly-by-night fan, analyst or leaguemate determines so-and-so is in for a sophomore slump, or Joe Prospect “has all the tools,” these are not real intuitions. They’re the messages inside fortune cookies.

Second, these intuitions are relatively rare, very strong and their originators are uncharacteristically casual and committed to their prognostications. By general pre-ranking consensus, I over-drafted Nelson Cruz in just about every league I could. You know what? I didn’t hesitate on that selection or experience buyer’s guilt or remorse at all. There were other players who I rostered many times, but most of those were value plays. Nellie Cruz was me thin slicing. I simply drafted Cruz expecting him to blow many players drafted before him right out of the water. (Pro-rated, he has!)

Third, we must be honest with ourselves about these intuitions. We can’t fall victim to confirmation bias by constructing a revisionist history and we must recognize that these intuitions are not always foolproof. I have a few pitchers I feel this way about every year. This past year, one of those players was Jonathan Sanchez. Was I right? Kind of. Meanwhile, I was able to nab David Price in a few leagues too. Did I have the magic feeling about Price? No, I just thought that he had an extremely high ceiling, making him a wise gamble at the prices I paid. Last year, I owned Josh Johnson in every league I played in. But I’m far from perfect. I’ll be the first one to tell you about the season (or two) I walked around with my chest out thinking I got over on everybody, and couldn’t wait to see the league scratch their heads as they watched my John Patterson mow down the National League.

Fourth, as both Derek and Mike Podhorzer note, we must examine our successes and determine whether they are really out successes within the context of our predictions. Right now, Cruz is hitting .330 and benefiting from stratospheric .370-ish BABIP. Cruz may well indeed go on to finish the season hitting comfortably above .300, but as high as I was on him, I don’t think I could honestly take credit for predicting that. I say that for two reasons. One, we don’t know if it is legitimate. He had a curiously low BABIP last year, and is sitting on a curiously high BABIP this year, so it is not really even clear who the real Nelson Cruz is as a hitter for average. Two, I wasn’t anticipating a batting average above .300. When drafting him, I just simply said to myself, this is going to go 35/30, maybe even 40/30.

Under the specific conditions above, I’m willing to respect and consider the divine intuition phenomenon, and not dismiss is outright simply because it may be difficult to present fully within the framework of a “business case.”

How do we know when we’re right?

When can we legitimately claim victory when it comes to predicting a break out? This is a very difficult question. In the comments section of Derek’s article, Mike Podhorzer says:

Tell me why you expected a player to perform the way he has that proves you “right” and then I will determine if you deserve credit or not based on your answer.

But, even then, how do we know? Many voices in the fantasy community predicted this year would mark a break out in the power department for Billy Butler. What was the foundation of this hypothesis?

Well, it sort of looked like this; Butler is beginning to enter his mid-20s and has showed promise and progress thus far in his development as a real threat at the plate. Last year, he had a fine season and hit a respectable 21 homers while posting a solid but unspectacular 11.8 HR/FB. He did hit 51 doubles, though, and it is reasonable to project that, at his age, Butler becomes stronger and converts more of those doubles to homers in 2010. He seemed to really find his power stroke in the second half of 2009 and we should expect this trend to continue and intensify in 2010. Sound about right?

Well, what happened? Butler is hitting the ball almost exactly the same as he was last year. As I write this, his 2010 BABIP is identical to his 2009 mark. His LD/GB/FB distribution is within 1 percent of 2009 rates for each batted-ball type, and his HR/FB ratio is down by about a third.

Let’s imagine that the 30-homer pace Butler prediction came true, though. How would we know we were correct even if the performance fit our hypothesis? Basically, what would have happened would be the same thing that has happened, but in reverse. His batted-ball type distribution stays largely the same, as does his BABIP, but his HR/FB go up from about 12 to 16, instead of down to 8. Basically, with the exception of one column in a spreadsheet evolution, stagnation and devolution would look pretty similar in Butler’s case. Further, we know this key column is prone to yearly swings. So, if Butler did appear to take this step forward, how are we to know that he won’t take a step back the following year? At what point can you say this is the real Butler and I predicted this?

It’s very difficult to make these determinations as players are developing, which is when break outs are most likely. We have come to know who Albert Pujols really is, but the inverse side of that certainty is that we also know that Pujols is extremely unlikely to drastically over-perform or under-perform consensus expectations, which is what having a break out is really all about.

My point here is that as the body of evidence mounts that enables us to more confidently ascribe any seasonal performance to a player’s “true talent” the less likely that player becomes to “break out.” So, when we predict something and it comes true, we can’t be so sure it wasn’t due to luck. And, when we finally have enough information to establish a solid baseline for our predictions, the players have aged to the point where breakouts are unlikely.

My approach when it comes to giving credit for predicting breakouts is a bit simpler. I don’t really stress individual cases much at all. I determine a fantasy prognosticator’s aptitude the same way I would a weatherman’s (meteorologists? Are the TV weathermen actually meteorologists or just the TV-friendly spokespeople for the real scientists in the back room?). I look at somebody’s track record. I presume that if your reasoning is sound, you will make more picks that turn out correctly. Sure, some might turn out correctly for the wrong reasons, but so too will some come out incorrectly despite being equally well thought out. It’s certainly interesting to hear any particular analyst’s in-depth opinions on the players they are particularly high or low on, but what really matters over the course of time is track record.