POL: Will the perils of polling be revealed

[albionmoonlight]

Something that has been discussed a little bit in the leadup to this election how it is getting more and more difficult to conduct good polls. I won't rehash the arguments here, but you can see how it was much easier to get a representative cross-section of people in the 123 area code or 12345 zip code back when every house had one land line with no caller-id or call block or cell phones or VoIP or etc. etc. etc. Some, though not most, think that we may reach a point where traditional polling methods simply cannot get a proper cross-section of potential voters.

So, does anyone think that this election will be the one to show just how wrong the polls are and how a new polling methodology is needed? Or, will the map on Wednesday morning look pretty much like the map the pollsters predicted on Monday night?

[JonInMiddleGA]

Originally Posted by albionmoonlight

Or, will the map on Wednesday morning look pretty much like the map the pollsters predicted on Monday night?

I would think it would be pretty close, say, within double the margin of error.
There'll probably be 1 or 2 that are big flips, but if they only miss a couple of the ones outside marginX2, that would qualify to me as "pretty much like".

[kcchief19]

As is usual, I think the only perils revealed will be the peril of reading polls the wrong way. Will the Tennessee Senate race be a 12-point win for Corker that an aberrant poll showed over the weekend? I highly doubt it. I largely agree with JIMGA -- it will be pretty close to predicted, with one or two surprises and the toss-ups being toss-ups.

The problem with polls isn't necessarily the polls themselves but how people use the polls. Basing on one poll -- even if it's an exit poll -- is a faulty mechanism to a certain degree. There will be single polls tonight that are wrong -- but as an aggregate, I think they will be highly accurate. Anybody who wants to find fault with polls will be able to because they will be able to point to one single poll that was way off and ignore the aggregate.

I don't think we're to the point where ne need a new polling methodology. You can still get good samples, it just takes longer. I know some pollsters have started using online polls -- scientific online polls, not What's America Barbecuing polls on USA Today -- but they still have some limitations based on demographic skew.

[Solecismic]

Hopefully, they've fixed the problems with the exit polls. In 2004, the polling companies hired a lot of young people to ask the questions. They felt more comfortable interviewing younger voters, who skewed the results toward the Dems. And fueled the fire for many tin-foil-hat wearers.

Turnout in NH is apparently between 2002's off-presidential election record and 1998's decent turnout. I'm sticking with D+5 in the senate, +32 in the house.

[Klinglerware]

Originally Posted by Solecismic

Hopefully, they've fixed the problems with the exit polls. In 2004, the polling companies hired a lot of young people to ask the questions. They felt more comfortable interviewing younger voters, who skewed the results toward the Dems. And fueled the fire for many tin-foil-hat wearers.

Turnout in NH is apparently between 2002's off-presidential election record and 1998's decent turnout. I'm sticking with D+5 in the senate, +32 in the house.

Having worked in the business, the demand characteristics generated by the exit poll worker has limited effect, since questionnaires are administered in paper & pencil format rather than orally. However, you are correct that there is a somewhat consistent bias in exit poll results in favor of the democrats. During my time with the operation, we could never figure out what was causing this. The prevailing theory at the time was that people who were more likely to understand at least the basics of polling and were more likely to agree to respond to the poll, tended to be more highly educated types who also tended to vote Democratic. We could never prove this statistically, though.

The democratic bias has been well known by the operators for years, and is corrected for as a matter of course, but only later on in the day when there is more data to correct for it. The leaked afternoon raw exit poll data (typically devoid of the necessary weighting and readjustment, at that point in time) has really hurt the credibility of the operators in recent cycles...

Finally, I would agree with the posters who argue that there is not too much methodologically wrong with polling itself (both regular and exit polls), just that there are too many people who don't understand their limitations or how they should be interpreted.

[cuervo72]

There's a silent film joke in here somewhere.

[kcchief19]

My impression is that the polling data I saw last night was very well done, at worst within the margin of error. The Missouri senate race was 49-48 in the exit poll I saw, and it's currently 49.5 to 47.4. The tally on the Missouri stem cell initiative was spot on. Polls also showed Webb and Tester with thin margins.

I think the worst performance I've seen thus far was on the Tennessee race, which I think had Corker with a 7-point lead. That proved to be a bit excessive but even that wasn't far outside the MOE.

[MrBigglesworth]

I think the post topic is more on the calling polls than the exit polls. But most of the calling polls seemed to be pretty good, no major race that I know of was picked incorrectly by the pollsters.

[albionmoonlight]

I was talking about calling polls and not exit polls (though that information about the "raw numbers" being leaked in 2004 was interesting and explains a lot).

And, to me, I think that the calling polls got it right. By getting it "wrong" I meant something like Harold Ford winning by 6 points in Tennessee, or DeWine winning Ohio by 5 points. Something that would indicate that we really just had no idea what people were going to do.

Nothing like that happened.

[QuikSand]

Seems to me that if you were folling the national election picture via a responsible poll-gathering site like elctoral-vote.com, then almost nothing that happened on election night came as a surprise. That site, by collecting advance poll data, and using some common sense when to reject an outlier that deviated from an overall trend, did a superb job of watching any movement, and in the end gave a nearly pinpoint prediction of how things were going to go.

I don't claim to know how much expertise goes into the "poll watching" business... but to me, after the last two national elections, it's become pretty clear that if done well, it has a very good sense of where things are going. Both in 2004 and 2006, election night unfolded almost precisely as predicted by astute advance poll watchers.

[Butter]

Originally Posted by QuikSand

Seems to me that if you were folling the national election picture via a responsible poll-gathering site like elctoral-vote.com, then almost nothing that happened on election night came as a surprise. That site, by collecting advance poll data, and using some common sense when to reject an outlier that deviated from an overall trend, did a superb job of watching any movement, and in the end gave a nearly pinpoint prediction of how things were going to go.

I don't claim to know how much expertise goes into the "poll swtching" business... but to me, after the last two national elections, it's become pretty clear that if done well, it has a very good sense of where things are going. Both in 2004 and 2006, election night unfolded almost precisely as predicted by astute advance poll watchers.

You are correct. Thanks to electoral-vote.com, I knew ahead of time that it would likely all come down to Virginia... I was surprised only perhaps by Montana being as close as it was, thinking that the late close polling was a bit overstated. Turns out it wasn't.

[Klinglerware]

Originally Posted by albionmoonlight

I was talking about calling polls and not exit polls (though that information about the "raw numbers" being leaked in 2004 was interesting and explains a lot).

And, to me, I think that the calling polls got it right. By getting it "wrong" I meant something like Harold Ford winning by 6 points in Tennessee, or DeWine winning Ohio by 5 points. Something that would indicate that we really just had no idea what people were going to do.

Nothing like that happened.

I agree--and I also agree with the subsequent posts about electoral-vote.com et al's use of multiple polls to get at convergence. That site did a better job this year...

The polling firms are very sophisticated and are well aware of the issues moving forward, such as the move towards cell-phone only and voip. Demographic weighting could be a partial solution to the problem, but I'm sure others are out there or will be devised...