Monday, September 7, 2009

Bad hockey predictions: Black swans, hindsight bias, and why the Leafs could win the Cup.

(Editor's note: This is one of those annoying serious posts, but if you stick with it I promise to slip in a "Kyle Wellwood is fat" joke somewhere. If not, I'll be back with some fun stuff later in the week.)

The new season is almost here, and that means it's time for one of hockey's annual traditions -- terrible, terrible predictions.

One example: Inspired by a recent brilliant post on Puck Daddy, the folks over at Mile High Hockey spent some time digging through ESPN's hockey archives, and found out that, predictably, ESPN is awful at making predictions.

None of their small army of so-called experts successfully picked the Penguins as Cup winners, and most ended up picking champs who didn't even end up winning a single round.

The experts: Not just wrong, but lazy too
This isn't a surprise. Hockey predictions are always awful. Ask a typical hockey expert to predict the coming season and chances are they'll take last year's standings, move a few teams up or down one spot, and call that "analysis".

And they'll be wrong. Not just a little bit wrong, but terribly, hilariously, "not even in the ballpark" wrong.

This time last year, everyone agreed that the Habs would win the Stanley Cup. The Stars and Avs would contend in the West, the Blues would challenge for last place overall, and the Bruins would struggle to make the playoffs.

Everybody agreed on this. Everybody was wrong. And this happens every single year.

It would be tempting to point at this as evidence that so-called hockey experts are really just frauds who know as much (or less) as you or I. But there's actually more to it than that. Hockey experts aren't frauds. They're just human.

Human are terrible at making predictions
Find any prediction from any time on any subject -- sports, politics, economics -- and it's likely that it turned out to wrong.

A year before the US election, most experts were arguing over whether Hillary Clinton would beat Rudy Guliani. Find any resource that does financial analysis and you'll find plenty of experts who swore the economy could never tank, right up until the economy tanked. Find any prediction ever made about what the world would be like in the future, then look outside your window and count how many flying cars you see.

We love to make predictions about the future. And when we do, we're almost always wrong. What's going on?

There's just some basic math at play here that's hard to overcome. Take any system with even the slightest bit of complexity, start predicting the possible states, and you pretty quickly find that you're dealing with some pretty large numbers -- large enough that predicting anything with much accuracy becomes near impossible.

Now nobody looks at a set of sports predictions and expects the expert to be exactly right. But there are so many unknowns and moving parts that even being vaguely, sort-of, quasi-right ends up being incredibly unlikely.

So it's not that we're bad at predicting complex things because we're dumb. We're bad at it because, mathematically, it's almost impossible. But we don't seem to know that. Plenty of psychological experiments have shown that when you ask people to make predictions, and then ask them to rate their confidence in their predictions, they always miss by a mile.

Put another way: we're not just bad at making predictions, we're bad at predicting how bad our predictions will be. No matter how many times we're wrong, we always think we're going to be right next time. And we never are.

Excuse me sure, I think your bias is showing
Here's the good news: We're terrible at predicting, but we're great at rationalizing.

There's a well known psychological phenomenon called the hindsight bias. Basically, even though we're terrible at predicting what will happen, we're great at fooling ourselves after the fact. We have a built-in ability to pretend that things were predictable all along.

Of course the Habs crashed and burned last year -- look at all the holes in their lineup! Of course the Bruins challenged for first overall -- look at all that young talent! Of course the Stars missed the playoffs -- everyone knew Avery would destroy that dressing room!

Those same experts who couldn't predict the future are more than willing to accurately predict that past.

And it gets worse.

Black swans
In probability, the "black swan" theory was presented by Nassim Nicholas Taleb in his book of the same name. (The book, by the way, is highly recommended if you're really interested in mathematical probability, economics, and authors going on and on about how wonderful they are until you want to hunt them down and slap them.)

Taleb's "black swans" are events that have enormous impact, were unpredictable, and are considered incredibly rare. Examples include 9/11, the rise of the internet, and various stock market meltdowns. Nobody sees them coming, even though everyone agrees that, in hindsight, we should have (there's that bias again).

While not on the same level of importance as world wars and global catastrophes, the sports world is filled with black swans. Kurt Warner. Len Bias. Tom Brady. Mark Prior. All were black swans of the sports world, good and bad.

Hockey has plenty of examples. An undrafted and unwanted Martin St. Louis winning an MVP, Art Ross and Stanley Cup? Black swan. The Red Wings drafting future hall-of-famers in the final rounds of back-to-back drafts? Black swan. Leeman for Gilmour? Black swan. Kyle Wellwood doing a situp? Big time black swan.

What will this year's black swans be? Maybe Jonas Gustavsson wins the Vezina. Maybe Luke Schenn regresses and gets sent to the minors. Maybe Jason Blake works in a pass or two.

Nobody knows. But those black swans are out there, for more than a few teams. They're going to change everything, and they'll throw all the expert analysis right out the window.

So what does all this have to do with anything?
Nobody's arguing that it's impossible to predict anything about the sports world, or that the results we'll see will be determined solely by random chance. Some players and teams are better than others, and they'll probably have better results. That's common sense. Sometimes, things really do work out the way you'd expect.

But there will be surprises. There will be injuries, and blockbuster trades, and guys who come out of nowhere to become stars, and hot shot prospects that turn out to be duds. Virtually every team will have a few black swans of its own that will throw all the conventional wisdom out the window.

All of us -- experts, bloggers, the guy next to you in the bar -- are going to be wrong. By a mile. This season is going to end up looking nothing like what any of us predict.

And when it's all over, we're going to look back and pretend we knew it all along. After all, we're only human.

So here's the bottom line: Ignore the predictions. Sit back and enjoy the ride. And if every expert in the world is predicting that your team will come in dead last this year, don't worry.

They may be right. But don't bet on it.




19 comments:

  1. This is why it's better to make predictions with a combination of blind hope and cyncism: Blackhawks to the finals thanks to Hossa, and either the Caps or the Pens to win depending on whether Bettman's coin toss comes up heads or tails.

    I enjoy the serious posts because, suprisingly, Leafs analysis does not trickle down to Virginia.

    ReplyDelete
  2. You're absolutely right about predictions. Back in April I wrote in my blog that the Cubs would beat the Rays in the World Series, with the Mets and Indians making the playoffs. And when I write my NFL preview this week, I'm sure I will be embarassed when I look back at the end of the season.

    ReplyDelete
  3. Kyle Wellwood doing a situp? Big time black swan.

    I found waldo!

    ReplyDelete
  4. I predict Kyle Wellwood will take on a Tim Horton's all by himself and win handily.

    Also at some point the Rangers will cause me to have an aneurysm.

    ReplyDelete
  5. Last year I was successful in predicting the Bruins to come first, Recchi to have 60 points (he finished with 61) and for the Sens to suck and miss the playoffs.

    But I also predicted a Montreal-Calgary final, Tucker would rebound points-wise, and Kessel wouldn't get a lot of points.

    The thing with experts predictions, is that they don't want to be seen as going against what everyone else says and make bad predictions. So if they make a prediction everyone else makes and it's wrong, it's no big deal, because everyone was wrong.

    ReplyDelete
  6. Great post - Hockey "experts" are pretty much all idiots. Like the Buffalo New's Bucky Gleason. All these guys do is guess.

    My guess is that the Sabres make the Eastern Conference finals this year, and beat the Leafs, then smoke Dallas in the Stanley Cup finals....

    Statistically, that is probably just as likely as a Pens/Detroit repeat. Or more likely.

    ReplyDelete
  7. is this your way of dealing with the leafs being horrible this season? ie. there's always a chance they could perform well...

    ReplyDelete
  8. The best part about this otherwise terribly hockey-less time of the year is that right now, the Leafs have exactly the same chance of winning the Cup this season as every other team.

    I predict that Leafs bashers will laugh at me for clinging to this, and I further predict that I won't give a rat's posterior.

    ReplyDelete
  9. I predict nightly sellouts!

    ReplyDelete
  10. This post smells of "I wonder why I hate making predictions? I'm going to figure out why." I only got as far as the question myself and so I haven't posted in a week.

    ReplyDelete
  11. this basically reads like a Leafs fan therapy session. what does it say about the organization if our only hope comes from the chance of a statistical outlier. i suppose for now, that's all we have

    ReplyDelete
  12. what does it say about the organization if our only hope comes from the chance of a statistical outlier

    That the team isn't very good?

    Which we already knew. I'm not trying to break any scoops there.

    ReplyDelete
  13. We spent so much time on Kyle Wellwood jokes last night that we never got to the "Black Swan" thing, because I honestly believe last year's Bruins team fit the bill. Everything just went remarkably right in Boston, until the playoffs. I can't see that happening again.

    ReplyDelete
  14. BesterThanYesterdaySeptember 8, 2009 at 2:17 PM

    Anyone else surprised TSN has the Sens finishing 24th overall? I'd dance a jig if it happened, but that seems low...

    Ah well, time for more beer.

    ReplyDelete
  15. Ah, DGB, I knew you were going to say all that.

    ReplyDelete
  16. I met Don Cherry at a commercial he was shooting in September before the season Mario's Penguins won the Cup for the first time. A friend and I went up and knocked on his trailer and he came to the door. We asked him who he thought would win and he accurately predicted the Pens. I don't recall anyone else predicting it. Sometimes people are right, but maybe not enough to be statistically significant.

    ReplyDelete
  17. i actually enjoyed this as a thoughtful piece of hockey blogging. i love the humour stuff as much as anyone but i can appreciate a well reasoned argument like this. thanks for a breath of fresh air from the typical prediction bullshit. for a funny alternative take on predictions check out the triple deke's sega 94 preview of the wings roster.

    ReplyDelete