XKCD’s take on the accuracy of polling

By now, you probably know about Nate Silver, the guy behind the New York Times’s Five Thirty Eight blog whose in-depth analysis of opinion polling data produced an electoral map that has correctly predicted the outcome of the presidential race in 49 of 50 states, and likely Florida as well if Obama maintains his slim lead there.

He’s being heralded as a wizard by Democratic supporters after being vilified as a left-wing hack by Republicans in the weeks before the election because he predicted a strong possibility of Obama winning enough swing states to take the election.

Silver is a very smart man, and his model has shown to be correct in multiple elections now. He’s no wizard, just someone who took in a lot of data and applied a formula to it (using yet more data) that made projections with it that turned out to be correct. Few things are polled as much as U.S. presidential elections, and Silver had a lot of data to work with, particularly in those swing states. Simply averaging all those polls together would have given a pretty good, and other projections based on multiple polls produced nearly identical maps. Silver’s model was more complex, counting for pollsters’ biases by looking at their accuracy in previous elections.

Silver got some things wrong. He called the North Dakota senate race “safe Republican”, but it went to the Democrat in a squeaker. Silver could easily blame the lack of polling in that state – only one published poll in the final two weeks of the election campaign. In next-door Montana, where Democrat Jon Tester (first elected during the Democratic wave of 2006) was widely expected to lose his seat, Silver’s model ignored the fact that polls consistently showed Tester in the lead, and instead used “state fundamentals” numbers (i.e. that the state is very red) to push the race in the “lean Republican” category. No doubt he’ll take a look at that when revising the model.

(I’ll give you a moment to note the irony that where Silver was wrong, it was that he leaned too far toward Republicans.)

And Silver got maybe a bit too defensive when people criticized his work, leading to a poorly-thought-out bet with Joe Scarborough that got him into trouble with the Times.

But what’s important here is that Silver has shown that in general, elections with a high amount of public polling from different sources can get a very accurate view of how people are going to vote. Biases, like the inability to reach people with cellphones, or the type of people who respond to pollsters, or the strength of get-out-the-vote efforts, can be counted for quantitatively and compensated.

Mocking the mockers

Silver’s critics acted, it seemed, not because they disagreed with his methodology, but because they disagreed with its results. Projecting a 90% chance of an Obama win didn’t fit in with people’s gut feelings that the race was too close to call. Even though it didn’t take an expert to see that Romney’s path to 270 electoral votes was much more difficult than Obama’s for what were generally agreed to be the swing states up for grabs, pundits refused to accept that the race was all but over.

So many on the conservative side made their own predictions.

There was George Will predicting a huge electoral college victory for Romney. There was The Weekly Standard predicting a Romney win mainly because it felt Republican voters were more motivated than Democratic ones. There was that guy behind the website Unskewed Polls that said Silver is “extremely biased” toward Democrats and had to eat crow when not only did the expected big Romney win not arrive, but those projections he attacked turned out to be right.

There were more reasoned criticisms of the models used by Silver and others. This piece in the National Review complained about the weights Silver gave polls, suggesting they were entirely subjective (they weren’t), and probably deserves better than me pointing out that the same writer called Florida, Ohio, Colorado, Iowa, New Hampshire, Virginia and Wisconsin for Romney, and all seven of those states (if we assume Florida doesn’t flip) went to Obama.

And there was Karl Rove famously questioning the Fox News election desk after it declared Obama the winner, saying the vote in Ohio wasn’t clear yet, leading to Megyn Kelly confronting their supernerds and getting them to explain why they made their call, and plenty of mockery of the news network for going through the five stages of grief.

Shades of 2004

Even though it was clear by about 11pm that Obama was going to win the election, and networks started calling it around 11:15, there was defiance of yet more math by Romney’s supporters. Even Romney’s staff had apparently deluded themselves. Romney himself refused to concede at first, the story went, and didn’t prepare a concession speech, resulting in a delay before he gave it.

While this was happening, I thought back to 2004. The election counting went well into the night, but eventually George Bush was declared the winner after taking the state of Ohio by about 120,000 votes. His Democratic opponent, John Kerry, wouldn’t concede until the next day, and many hard-core Democrats still believe Kerry should have won that state and some secret trickery prevented it. To many Democrats, Bush’s unpopularity was so obvious, so widespread that a landslide Kerry win was simply inevitable.

The echo chamber of the right

This reminded me of that year because it wasn’t just a question of right-wingers being disconnected from reality. It was about how everyone seems to have constructed their own reality based on facts they choose to listen to.

I think of people like Ezra Levant, who despite billing himself as a free-thinking voice seemed to regurgitate just about every Republican talking point, even on election night. The things he retweeted on his Twitter account all went in the same direction: the polls are wrong because Romney will win. The day before the election, he discussed on his show how a CNN poll showing a dead heat was obviously biased toward Obama because it oversampled Democrats. He and his guest concluded that the number of Democrats who would vote in the election was overestimated based solely on their gut feeling that enthusiasm for Obama must be lower than it was in 2008.

Levant predicted “a big Romney win“, four years after having predicted “McCain will win“. Levant’s electoral map showed seven states – Colorado, Iowa, Wisconsin, Ohio, Virginia, New Hampshire and Florida – all going for Romney when they all went for Obama. Online polls on the Sun News website show that voters believed that Romney was more likely to win, by a huge margin.

When Levant turned out to be wrong, again, rather than take the Nate Silver approach and adjust his baseline, he went into an apparent on-air depression, saying America is now doomed and Canada should look to other countries to do trade with. He concluded that America has changed and that they’re all just freeloaders who want government handouts. When I pointed out how his prediction differed from those “biased” polls he complained about so much, he accused me of not having “courage” for not making a public prediction on TV, as if my lack of public prediction somehow made him less wrong.

What’s ironic about all this is that this is the exact same kind of stuff that Sun News itself accuses its competitors of: sitting in an echo chamber, listening only to people who agree with them, and putting out journalism that is biased toward their personal views because they block out people they don’t like. Levant and colleague Brian Lilley sometimes have a point when they criticize other journalists for their often liberal views about things, but they’re hardly setting an example when they go even further in the other direction, no matter how much they might pat themselves on the back for being honest about it.

The election result on Tuesday provided rare hard evidence that the echo chamber Levant and other conservatives choose to live in has led them to beliefs that are quantifiably wrong. Unfortunately, their response seems to have been to dismiss this like one would a failed bet on a hockey game, and move on without questioning what led them to incorrect conclusions.

The echo chamber of the left

Before this blog post gets passed around on Rabble.ca as if it’s the greatest thing in the world, I’ll point out once again that the echo chamber isn’t restricted to Republicans, or even Americans. A quick trip to Facebook and I saw plenty of anti-Republican ridiculousness passed around by people who should know better. Posts about how Romney’s slogan was taken from the KKK (it wasn’t). Exaggerations of a case where a single miscalibrated voting screen caused an input for Obama to be recorded as an input for Romney. In general, anything that made Romney and the Republicans look bad, or Obama and the Democrats look good, got lots of attention and very little verification.

We know that Obama would do well in Canada, and certainly in this part of it. But people with only friends in Montreal might be left with the impression that everyone loves Obama, and that only the really stupid and the truly evil support Romney and the “Rapepublicans”. That’s about as large a brush to paint on Romney supporters as Romney himself painted on the 47% of Americans he called freeloaders because they don’t pay federal income taxes.

And before you start thinking that left-wingers are just better at math and science than the right, I’ll remind you that when I tried to use a scientific approach to measure the size of a large protest over tuition, my numbers were challenged by people on the left. Why? Because the total number that came out of the process didn’t match their gut feeling of how large the crowd must have been.

How we can fix it

The polling part is easy. Don’t overanalyze. People broke down polls and picked at the ones that didn’t show things the way they wanted. They adjusted the assumptions, convinced that the pollsters have some vested interest in being incorrect and that their amateur fiddling would correct their internal biases. Silver’s model doesn’t involve tinkering with individual polls, it simply adjusts for known quantifiable data (including things like how well those pollsters did in the last election). There is still plenty to disagree on or debate about polling methodology, but people on both sides have to acknowledge that when they start playing with the numbers themselves they tend to inject bias more than they remove it.

For the rest, it means being more critical of stuff you hear that you and your friends agree with, and being more receptive to those you don’t. It means, for those on the left, spending less time accepting at face value what’s said by people who worked on the NDP campaign in the last election, and more time listening to reasonable people whose political views you don’t like.

I’d say start watching Sun News, but it’s hard to take them seriously. Maybe start off with some light conservatism, like Andrew Coyne.

And to those on the right, well if the re-election of Barack Obama didn’t provide enough proof that your assumptions are wrong, I honestly don’t know what to say.