Nice inauguration yesterday, wasn’t it? I’m sure the exit pollsters of Edison Media Research and Mitofsky International thought so — nice enough, anyway, to overshadow the 77-page report that the two firms released Wednesday detailing the failure last November of the Edison/Mitofsky Election System that generated fatally flawed exit poll data for National Election Pool members (the broadcast networks, CNN, Fox News, and the AP) and subscribers (most other large media organizations) suggesting that John Kerry was heading for victory.
The pollsters have good reason to be embarrassed; their post-election spin, always dubious, is fully derailed. For months, their line has been that their data was just fine, and it was misinterpreted by the nefarious bloggers who broke the embargo by posting leaked data that was incomplete. “I’m not designing polls for some blogger who doesn’t even understand how to read the data,” Edison vice-president and co-founder Joe Lenski told the Los Angeles Times on November 4, adding that “it’s like if you were graded by your readers on the first draft of your article.” Warren Mitofsky emailed Keith Olbermann of MSNBC in late November to tell him that “leakers were reading complex displays intended for trained statisticians. The leakers did not understand what they were reading and the bloggers did not know they were getting misinformation.” The data, said Mitofsky, was like “the score at half time at a football game.”
But Edison/Mitofsky’s data was flawed long after half time. As late as 7:33 PM on election night, NEP members and subscribers were sent “weighted” data showing that Kerry was winning by 9 points among women and losing by 4 points among men; the next afternoon, the data was revised to show Kerry leading by only 3 points among women and losing by 11 points among men. When this was pointed out, Mitofsky stuck to his guns, firing off an angry email to Mickey Kaus of Slate in which he insisted that “the only ones confused were the leakers and the bloggers” — a strange assertion, given that anyone watching the news on November 2 could see that everyone on camera thought Kerry was winning.
So what happened?
THE REPORT (AVAILABLE IN .pdf format here) is written to accentuate the positive — for example, that no media organization called a state the wrong way (a goal that could be accomplished, if news organizations didn’t need or want timely data, by eliminating exit polling altogether and waiting for the official count). But the inescapable conclusion that the report dances around is that a pro-Kerry bias in the data resulted from Edison/Mitofsky’s over-reliance on poorly-trained young liberal activists to gather data.
There was no problem with the precincts that Edison/Mitofsky chose to survey; comparing those precincts’ actual vote totals to the actual vote totals of the states shows that they should have generated a good, balanced sample. (Putting paid to conspiracy theories about electronic-ballot rigging, the report also shows that “precincts with touch screens and optical scan voting have essentially the same error rates as those using punch card systems.”) Instead, the problem — apart from a few technical glitches with the NEP computer system that showed up on election day — was “Within-Precinct Error” (WPE), mistakes in data-gathering made on the ground.
The report breaks down WPE rates according to various factors, based on a telephone poll of interviewers. Various immutable facts of an exit-pollster’s life increased WPE — things like bad weather, laws keeping pollsters a certain distance from polling places, and uncooperative or meddlesome election officials. More interesting are the characteristics of the interviewers themselves that correlated to high WPE rates. For some reason, Edison/Mitofsky didn’t ask interviewers about their political leanings, but there are certain inferences that can be made.
Younger interviewers’ data had much higher WPE rates than older interviewers’ — a big problem, since 35% of interviewers were 24 or younger and 50% were 34 or younger.
One of the highest WPE rates came in the data collected by those interviewers with post-graduate degrees. This is a group that leans to the left on average. (According to Gallup, Bush won among those with college degrees and among those with some college, but lost among those with high school diplomas or less and among those with post graduate degrees; the most highly-educated tend to be Democrats, but the average Republican is more educated than the average Democrat.) Many interviewers got the job through recommendations from their college professors (speaking of left-leaning demographic groups); others found the job via listings on craigslist.com, a site heavily frequented by urban hipsters. Even assuming there was no deliberate fudging of the numbers going on — something I’m perfectly willing to assume — there’s little doubt that many of these interviewers had a demeanor that absolutely screamed liberal. Small wonder that Kerry voters would be more likely to talk to them.
Of course, experience and training should teach pollsters how to overcome that problem. But fully 77% of interviewers had never worked as exit poll interviewers, and most worked alone. Their training, such as it was, consisted of a twenty-minute phone call and an interviewer manual sent by Fed-Ex along with the polling materials.
Edison/Mitofsky plan a number of corrections to their system for future elections, most importantly adjusting their training and recruitment methods. Their petulance over the past few months, and their attempt to bury this report, suggest that an attitude adjustment might also be called for. If exit pollsters are to serve their clients well, good data-collection is crucial — but so is humility and honesty.