The Terrible Secret of Nate Silver

Ok, so I’m going to say upfront that I like Nate Silver.  I read Five Thirty Eight.  I definitely would have read Burrito Bracket had I been aware of it.  He seems like a cool guy.  For those of you who were not interested in the future of the country back in October, Nate Silver is the stat-head who launched the poll aggregating website Five Thirty Eight.  Given the infantile way most website and media outlets measured polls (yes, RealClearPolitics, that means you), Five Thirty Eight was a breath of fresh air.  Silver did a solid job analyzing polls and conveying to non-math oriented people (ie just about everyone) what a poll’s data actually means.  So what’s the problem?  Everybody. Loves. Him.  They can’t stop raving about him, like he’s the ayatollah of poll aggregation.

He built an interesting website, give the man his credit.  This has gone way too far, though.  Much like Jack Donaghy, I think it’s hard to call the man a genius based on this work (Burrito Bracket, however, is another matter altogether).  But that’s exactly what’s going on.  Everybody treats his website like it’s gospel, or the latest dailysalad post, as though he can do no wrong.  Really, he’s the Bill Simmons of electoral projections.  Yeah, he’s still cool, but the hype is just too much.  And it’s just getting worse.  The other day, he projected the Minnesota recount down to the individual vote (Franken by 27).  Recently he signed a lucrative two book deal.  He was named one of the sexiest men of the year.  Just last week Rachel Maddow claimed that Silver was the best prognosticator ever, that he exactly nailed the election results, and that everyone in America should buy him a beer (all 300 million of us).  But really?  This should be a pretty easy claim to verify.

Actual Results:

Nate Silver’s Projections:

So that’s pretty close, right?  I guess Ms. Maddow and I have a different definition of what “exact” means, though.  While I would wholeheartedly endorse ignoring Indiana and Omaha out of general principle (let’s face it, the world would be much better off), the fact is that Silver did not get those electoral votes right.

So what is The Terrible Secret of Nate Silver?  Many people seem to be only loosely familiar with Silver’s work before he became the holy electoral oracle.  He used to write primarily about baseball and was definitely at the cutting edge of baseball analysis.  Working for Baseball Prospectus, Silver developed the groundbreaking PECOTA system of player forecasting.  The simplified explanation is that PECOTA would take a player’s track record of performance, physical attributes, etc and project out a career track heavily influenced by similar players in baseball history.  It was numeric, dynamic, and based on solid methodology.  I ate that shit up.  And it ruined my fantasy baseball teams for years because Nate Silver was wrong.

PECOTA was dangerous because it really wasn’t that accurate, but it gave off the pretense of certainty in the inherently uncertain world of projection.  It featured all sorts of really cool charts and percentages, but at the end of the day, while it may be mostly right, has enough screw ups to make one leery.  In 2005, it projected the eventual World Series Champion Chicago White Sox to win 72 games.  PECOTA also kept telling me that Jeremy Bonderman was going to win a Cy Young.  Or what about the cavalcade of young hitters that were on the verge of establishing themselves as superstars?  Hee Seop Choi.  The Royals dynamic duo of Alex Gordon and Billy Butler.  Even Wily Mo Pena, for whom PECOTA did accurately project a breakout year in 2004, but then kept on seeing promise as he proceeded to regress over the next few years.  Jeremy Hermida.  Jeremy Reed.  Josh Phelps.  Read what was said about him in the 2004 edition of Baseball Prospectus:

It’s been a fitful start to a career, but it’s only the dead calm before he starts terrorizing bleacher denizens.  Phelps is a devoted student of the art of hitting, and a dead pull hitter fully primed to kill any kind of offering at the plate.  A full, healthy season for Phelps will mean more than 30 bombs.  That isn’t a case of trying to whistle our way past the old cover curse, either [Phelps was on the previous year’s cover -LP].. c’mon, Richard Hidalgo turned out OK [he wasn’t -LP].

Phelps hit 17 homers in 2004 and hasn’t broken double digits since then.  The point is, though, that all of these guys ended up on my teams and they were all huge drags on my lineup.  And what about the future studs downplayed by PECOTA?  Guys like Brian Roberts, Hanley Ramirez or, my personal favorite, Michael Young:

In a year when Young’s offensive production toko a huge step forward, his fielding took a step backward, raising questions about his reputation in some circles as a world-beating second baseman.  Young’s offensive spike came almost entirely from an uptick in singles–his isolated power rose only slightly, and his walk rate actually dropped–so there’s a good chance he’ll give some of it back this year.

Sounds like Young is a guy to sell high on, right?  Well, since those words were written, Michael Young has not missed an All-Star team.  But you know what team he missed?  The 2005 Tri-City Tycoons, that’s who.  And it’s all because PECOTA scared me away from him.  Thanks, Nate Silver!

Most egregious, though, was his undying faith in Eric Chavez, perhaps the most disappointing player of the young century (apologies to Toe Nash).  Year after year, he’d sell me on the idea that this would finally be Chavez’s breakout year.  He’d finally put together two superstar halves and really make a run at that MVP award.  And year after year, I’d overdraft Chavez.  Silver, in 2005:

I’m completely befuddled here. Chavez was my MVP pick prior to the start of the season, and he has no obviously exploitable problems in his profile, except for his persistent problems against lefties (Chavez isn’t hitting lefties this year, but he isn’t hitting righties either). The base hits at least have come back during the past ten days or so and it’s tempting to wonder whether there was some kind of undiagnosed injury problem at the start of the season. This exercise is not kind to Chavez, lopping a full 25 points off his long-term EqA projection, but I’m going to go on record as saying that I think he deserves the benefit of the doubt, and will be fine long-term.

And what was the long-term?  Chavez has never made an All-Star team and now, at age 30, is injury-plagued and nearly washed up.

So can we all finally admit that Nate Silver is not infallible?

To end on a positive note, here’s one projection that he did nail, for both politics and baseball in 2006:

PECOTA projects that Mr. Frist’s presidential aspirations will go the way of Sammy Sosa’s late career.

Indeed.