How do we know we’re right?
I enjoy sticking my nose where I haven’t been asked to, and I also enjoy not having to develop my own concepts for my column, therefore, I think you guys can anticipate what time it is. That’s right, it’s time for unsolicited commentary on the Card Runners-rooted discussion du jour from the other Derek!
Seriously, I’d like to touch on some of the more abstract issues raised around identifying and claiming success for identifying breakouts that were raised in Derek Carty’s post, and the comments section thereof, last week.
The role of intuition in identifying breakouts
Both Eric Kesselman and Chris Liss imply that deeply engaged baseball or fantasy baseball fans are subject to strong gut instincts about players that are difficult to put in the context of objective analysis. Somewhat surprisingly, I actually agree. By definition, a break out is a player who is redefining his past profile by making a quantum leap in terms of translating his skillset into production. So, while it is entirely reasonable to expect to be able to study development patterns and elemental metrics to identify those who are more likely to “break out,” the spirit of what is happening in a break out – a player doing something he’s never done before – is difficult to predict by betting on pattern and trends alone.
But, before proceeding, it is important to ask what our intuition about certain players really is. In the outset of Blink, Malcolm Gladwell tells the story of an art curator who is unsure of the authenticity of a sculpture that would be highly valuable were it to be determined authentic. At this point, an accomplished artistic scholar views the piece and despite performing no tests, and without giving detailed reasons, determines the piece a fake. The owner of the sculpture brings in experts, and basically enacts the full CSI treatment, which generally points to the piece being authentic. Subsequently, new information is unearthed confirming the work as a fraud. (Forgive me if my retelling of this anecdote is a bit off, as it was many years ago that I read the book and have just realized, after attempting to reread it, that I must have leant it to somebody who never returned it.)
The point Gladwell makes, and the predominant, recurring theme of the book, is that people are actually rather good at “thin slicing” very complicated, multi-faceted sets of information. I think this notion is generally applicable to the discussion at hand, but it does come with many caveats.
For one, every Tom, Dick and Harry is not qualified to have these insightful intuitions. If Eric Kesselman has a strong intuition about a poker hand, or Chris Liss has a strong intuition about Ricky Romero, these are things that shouldn’t be causally dismissed just because the objective evidence supporting the intuition is not entirely robust enough to support the claim. But, if some fly-by-night fan, analyst or leaguemate determines so-and-so is in for a sophomore slump, or Joe Prospect “has all the tools,” these are not real intuitions. They’re the messages inside fortune cookies.
Second, these intuitions are relatively rare, very strong and their originators are uncharacteristically casual and committed to their prognostications. By general pre-ranking consensus, I over-drafted Nelson Cruz in just about every league I could. You know what? I didn’t hesitate on that selection or experience buyer’s guilt or remorse at all. There were other players who I rostered many times, but most of those were value plays. Nellie Cruz was me thin slicing. I simply drafted Cruz expecting him to blow many players drafted before him right out of the water. (Pro-rated, he has!)
Third, we must be honest with ourselves about these intuitions. We can’t fall victim to confirmation bias by constructing a revisionist history and we must recognize that these intuitions are not always foolproof. I have a few pitchers I feel this way about every year. This past year, one of those players was Jonathan Sanchez. Was I right? Kind of. Meanwhile, I was able to nab David Price in a few leagues too. Did I have the magic feeling about Price? No, I just thought that he had an extremely high ceiling, making him a wise gamble at the prices I paid. Last year, I owned Josh Johnson in every league I played in. But I’m far from perfect. I’ll be the first one to tell you about the season (or two) I walked around with my chest out thinking I got over on everybody, and couldn’t wait to see the league scratch their heads as they watched my John Patterson mow down the National League.
Fourth, as both Derek and Mike Podhorzer note, we must examine our successes and determine whether they are really out successes within the context of our predictions. Right now, Cruz is hitting .330 and benefiting from stratospheric .370-ish BABIP. Cruz may well indeed go on to finish the season hitting comfortably above .300, but as high as I was on him, I don’t think I could honestly take credit for predicting that. I say that for two reasons. One, we don’t know if it is legitimate. He had a curiously low BABIP last year, and is sitting on a curiously high BABIP this year, so it is not really even clear who the real Nelson Cruz is as a hitter for average. Two, I wasn’t anticipating a batting average above .300. When drafting him, I just simply said to myself, this is going to go 35/30, maybe even 40/30.
Under the specific conditions above, I’m willing to respect and consider the divine intuition phenomenon, and not dismiss is outright simply because it may be difficult to present fully within the framework of a “business case.”
How do we know when we’re right?
When can we legitimately claim victory when it comes to predicting a break out? This is a very difficult question. In the comments section of Derek’s article, Mike Podhorzer says:
Tell me why you expected a player to perform the way he has that proves you “right” and then I will determine if you deserve credit or not based on your answer.
But, even then, how do we know? Many voices in the fantasy community predicted this year would mark a break out in the power department for Billy Butler. What was the foundation of this hypothesis?
Well, it sort of looked like this; Butler is beginning to enter his mid-20s and has showed promise and progress thus far in his development as a real threat at the plate. Last year, he had a fine season and hit a respectable 21 homers while posting a solid but unspectacular 11.8 HR/FB. He did hit 51 doubles, though, and it is reasonable to project that, at his age, Butler becomes stronger and converts more of those doubles to homers in 2010. He seemed to really find his power stroke in the second half of 2009 and we should expect this trend to continue and intensify in 2010. Sound about right?
Well, what happened? Butler is hitting the ball almost exactly the same as he was last year. As I write this, his 2010 BABIP is identical to his 2009 mark. His LD/GB/FB distribution is within 1 percent of 2009 rates for each batted-ball type, and his HR/FB ratio is down by about a third.
Let’s imagine that the 30-homer pace Butler prediction came true, though. How would we know we were correct even if the performance fit our hypothesis? Basically, what would have happened would be the same thing that has happened, but in reverse. His batted-ball type distribution stays largely the same, as does his BABIP, but his HR/FB go up from about 12 to 16, instead of down to 8. Basically, with the exception of one column in a spreadsheet evolution, stagnation and devolution would look pretty similar in Butler’s case. Further, we know this key column is prone to yearly swings. So, if Butler did appear to take this step forward, how are we to know that he won’t take a step back the following year? At what point can you say this is the real Butler and I predicted this?
It’s very difficult to make these determinations as players are developing, which is when break outs are most likely. We have come to know who Albert Pujols really is, but the inverse side of that certainty is that we also know that Pujols is extremely unlikely to drastically over-perform or under-perform consensus expectations, which is what having a break out is really all about.
My point here is that as the body of evidence mounts that enables us to more confidently ascribe any seasonal performance to a player’s “true talent” the less likely that player becomes to “break out.” So, when we predict something and it comes true, we can’t be so sure it wasn’t due to luck. And, when we finally have enough information to establish a solid baseline for our predictions, the players have aged to the point where breakouts are unlikely.
My approach when it comes to giving credit for predicting breakouts is a bit simpler. I don’t really stress individual cases much at all. I determine a fantasy prognosticator’s aptitude the same way I would a weatherman’s (meteorologists? Are the TV weathermen actually meteorologists or just the TV-friendly spokespeople for the real scientists in the back room?). I look at somebody’s track record. I presume that if your reasoning is sound, you will make more picks that turn out correctly. Sure, some might turn out correctly for the wrong reasons, but so too will some come out incorrectly despite being equally well thought out. It’s certainly interesting to hear any particular analyst’s in-depth opinions on the players they are particularly high or low on, but what really matters over the course of time is track record.
Nice read Derek, this whole discussion is right up my alley.
However, I still continue to struggle with understanding what “gut instinct” and “intuition” actually means in the context of projecting breakouts. There has to be something that leads to your belief that a certain player will breakout. And that something is your basis, not gut instinct or intuition. Those two terms simply represent a feeling, which alone cannot possibly be a reason for expecting a breakout.
Isn’t that feeling going to be caused by something you see or hear? Assuming so, then that something you see or hear becomes your explanation, not some meaningless gut instinct or intuition.
Even in your own examples, Jonathan Sanchez has had a great strikeout rate and some bad BABIP luck in the past. Your opinion about David Price’s “extremely high ceiling” is due to his minor league performance, #1 overall pick status, and glowing scouting report at the time. You knew he was supposed to be great. In just a couple of sentences, I explained how anyone could have projected breakouts for these two pitchers, and they had nothing to do with gut instinct or intuition.
I think the problem is that lots of people like to call it that to sound smart and pretend they are the only ones with this opinion since they see things with their own eyes that no one else does. This is just not the case though as there is always something concrete that you could point to that is really behind your breakout picks.
“In just a couple of sentences, I explained how anyone could have projected breakouts for these two pitchers, and they had nothing to do with gut instinct or intuition.”
I’m not sure you really did though. Yes, Jonathan Sanchez has had a great strikeout rate and some bad BABIP luck in the past, but that doesn’t explain what would make someone think that this year wouldn’t be more of the same. And yes, David Price had great minor league performance, he was a #1 pick, and had glowing scouting reports, but that doesn’t explain why someone might pick THIS season for him to break out.
Simple answer here Matt, and that is that the person projecting the breakout is completely guessing. He has no idea if it will be this year or next or never. He will simply pick a guy to breakout based on some supporting evidence. So there really is no reason why it would be THIS year, and whoever it is who says he knows is lying to you and fooling himself.
Cruz isn’t a great example. He hasn’t been a top 50 player this year. He’s basically returned fair value given his draft position. Let’s not discount injury risk.
Here is the thing- I do not believe we can take any credit for predicting an “individual” breakout. We do not, in the industry, make predictions about one player we make predictions about many. you may be right about nelson cruz but wrong about gordon beckham.
your process of identifying these guys may be very valuable and rooted in strong analysis. but the fact is that you take 20 guys, make predictions and some will be right and some will be wrong. which ones you are right or wrong about is solely a matter of chance and nothing more.
you cannot take credit for an individual prediction. what you can take credit for is a process that is better than everyone else’s, like Mike Podhorzer last year going 42-0 in the Tango forecasters challenge. he beat every other site measured in head to head battles. but that only means his process was strong, it does not mean in the case of an individual player or even five that he gets credit for that individual player’s performance.
This is a perfect example of the Texas Sharpshooter fallacy in action.
*Isn’t that feeling going to be caused by something you see or hear? Assuming so, then that something you see or hear becomes your explanation, not some meaningless gut instinct or intuition.*
Yes, it is. But the feeling is how what one has seen and heard is communicated to the conscious mind. In other words, there are a lot of variables – past performance, age, health, team context, historical baseball context, personal memories of similar players in similar situations, visual observation of players, etc.
It’s quite easy to list a whole bunch of factors that go into valuing a player. K/9, GB/FB, velocity, scouting reports, etc. But it’s hard to aggregate each factor in the right proportion. How do we evaluate a 93 mph fastball with a decent 83 mph change and an above average curve, in a pitcher with a 7.1 K/9 at age 25 in the AL East with a 1:1 GB/FB ratio who many scouts think has room to get better, who you saw dominate a good Red Sox lineup, but he had some elbow soreness, etc. What weight do we give to each factor? It’s very complex to evaluate all of these things in combination. But as Derek says, we’ve been down this road many times, and something (we might not be able to identify it consciously) jumps out at us and says: “BUY!:” or “AVOID!”) Yes, we can make a conscious case based on the positives, or the negatives, but there are pros and cons for every player at his market price. It’s the feeling that tells us: “You like this player for some reason. Buy him.”
Again, we can weigh pros and cons all day, but in combination, it’s very complex, and sometimes the conscious mind cannot easily apportion the right weight to each factor. The feeling about a player is the communication from your experienced brain that according to its algorithm, this player is a great value.
This is only worthwhile if you have a ton of experience studying and observing players and looking at relevant indicators. But it does sometimes happen, and I’ve learned to trust it.