Best views, weather, etc. How to test them 👓 SC, Ala. sites look back Betty Ford honored
WASHINGTON
Barack Obama

Gallup identifies flaws in 2012 election polls

Martha T. Moore
USA TODAY
President Obama and Mitt Romney talk in the Oval Office  on Nov. 29, 2012, in Washington.
  • Gallup says several factors led to overstatement of Romney support
  • Group plans to test new polling in 2013 elections
  • USA TODAY had polling partnership with Gallup in 2012

WASHINGTON – Pollsters at Gallup said Tuesday they have identified flawed methods that contributed to their incorrect prediction that Mitt Romney would win the 2012 presidential election, but they are still working to determine how to better identify who is likely to vote.

The survey firm undertook a far-reaching review of its operations after its surveys came up short in the election: Gallup's final pre-election estimate showed Romney with 49% support to Obama's 48%, with a margin of error of +/-2%. Most polls estimated Obama would win the popular vote by 1 percentage point. Obama won the popular vote by 3.85 points.

In pre-election polling, Gallup consistently showed Romney with a 3-percentage point lead over Obama. When Gallup switched to surveying only "likely voters," Romney's edge increased to 4 percentage points.

Gallup, with researchers from the University of Michigan, will experiment with ways to better identify likely voters in surveys during the 2013 governor's races in New Jersey and Virginia. Gallup asks seven questions in its phone surveys to determine whether people are likely to vote – a questionnaire that may rely too much on past voting and on how much "thought" voters have given to the election, Gallup Poll editor in chief Frank Newport said. Though all polling outfits showed an increase of support for Romney among likely voters vs. registered voters, Gallup's bump for Romney was the most extreme. "We really are re-evaluating that from square one," Newport said.

In a six-month postmortem review, Gallup determined that part of the poll's overstatement of Romney support arose from too few phone interviews in the Eastern and Pacific time zones, overstating the white vote through a flawed procedure for racial identification, and relying on listed landline phone numbers:

•Time zones: In its telephone surveys, Gallup requires a set number of phone interviews per region but not per time zone. In the Midwestern and Southern region, Gallup says, the Eastern time zone was underrepresented, and the Pacific time zone was underrepresented in the Western region. That led to low-ball calculations of Obama's support.

•Identifying race and ethnicity: Gallup found that asking voters to identify their race and ethnicity through a series of questions, rather than letting them choose from a list, led to overstatement of Native American identity -- which inadvertently increases the impact of the white vote on the overall survey results. Gallup now asks respondents to choose from a list, similar to the procedure used by the Census.

•Calling listed landline phone numbers: Gallup surveys consist of half cellphone interviews and half landline interviews, but the review found that relying on listed phone numbers for landline calls — instead of random-digit-dialing method -- resulted in an over-representation of older Republican voters

Each of the methodological flaws was small in itself, Newport said, "but they are significant enough that we think they made a difference in our overall estimate of who was going to win the presidency last fall.''

USA TODAY partnered with Gallup throughout the 2012 election, a relationship that ended this year. "USA TODAY had a long and successful relationship with Gallup. Any time we hear there may be problems with information we rely on, we are concerned,'' said David Callaway, USA TODAY editor in chief. "But we trust Gallup, like us, conducted its polls with the utmost integrity.''

Gallup is one of the best known names in opinion surveys, and political polls are "the public face of survey research,'' said Michael Traugott of the University of Michigan, a leader of the Gallup review. Though Gallup surveys have far wider value than predicting the outcome of a presidential election, "confidence in the method and in the entire industry are related to how well pre-election polls do.''

Gallup's problems with accurately representing the racial and ethnic distribution of voters were pegged a year ago by Mark Blumenthal, founder of Pollster.com, writing in The Huffington Post. The methodological problems are specific to Gallup, but the challenge of correctly identifying likely voters is relevant to all pollsters, Blumenthal says. He says it is unclear whether future opinion surveys will be able to accommodate a new level of voter contact and sophisticated data analysis that the Obama campaign brought to bear in the 2012 election. "The hard part for everyone here is to try to disentangle what was unique to this election and what will be different next time,'' he said.

Featured Weekly Ad