The Public Policy Polling controversy

Public Policy Polling's methodology is under attack. But since PPP is prolific, cheap and accurate, criticism is unlikely to dent it

- Harry J Enten
- theguardian.com, Wednesday 25 September 2013 14.13 EDT
- Jump to comments (…)

Public Policy Polling (PPP) and Nate Cohn are involved in the Nerd Fight of 2013. For those who don't know, Cohn has pointed out that PPP hadn't fully revealed its true weighting methodology and were potentially weighting their results towards the polling averages. I strongly suggest you read Cohn's arguments and PPP's Tom Jensen's rebuttals, if you haven't already.

I'd weigh in on the methodological debate myself, though I think Cohn's and Jensen's views speak for themselves. The question that I want to answer here is whether there are any long-term ramifications for PPP. The answer, as far as I can tell, is that there won't be. Why?

1. PPP's sponsors are sticking by them

To conduct polling, you need money. PPP's main political world patron is the liberal blog Daily Kos. Despite harangues from Cohn and Nate Silver, Kos' leader, Markos Moulitsas, took to Twitter to defend PPP in the aftermath of Cohn's first article. David Nir, who runs the indispensable Daily Kos elections blog, called Cohn's article "a bit curious".

My own conversations with Daily Kos writers reveal that they believe that Cohn is engaged in a vendetta. They believe PPP is accurate, and they think all pollsters engage in unusual weighting techniques. PPP is only being attacked because it releases a lot more of its internal data into the public sphere, which allows people to see how it comes to the numbers it does.

2. People see PPP as accurate

Quotes put together by Mark Blumenthal are telling. Accuracy is what people care about. Most people don't care if you only interview tourists in SoHo to figure out who will win a US election, even if that is terrible methodology. Thus, Cohn's arguments about how PPP is accurate, which are quite clever, just don't matter to a lot of people.

What does matter is that Mark Blumenthal's readjustment of Nate Silver's 2010 pollster rankings to ensure no penalty for not following AAPOR disclosure requirements found that PPP ranked sixth out of 30 major pollsters. Silver's analysis of the 2010 election put PPP in the middle of the pack for accuracy. Steve Singiser's 2012 pollster overview had PPP fifth out of 17 very active pollsters.

I tend to think that accuracy is overrated, as my chart on special elections over the past ten years shows. I get the feeling though that most give credit to PPP just for trying, even if they are worse than the average pollster in special elections.

3. PPP is cheap

This is an extension of points one and two to a degree. When you're cheap, people are willing to put up with a lot of stuff they may not otherwise. PPP, which uses recorded instead of live voices in their calls, costs about 4% the amount of most polls you see television networks sponsor.

The fact that PPP is cheap allows it to produce a lot of polls. Over 18% of the polls done last cycle were PPP surveys. They can poll races that few care about. They can poll Ryan Braun's favorability in Wisconsin. In other words, they can flood the zone. They can get their brand out, so that even if their percentage of polls that get news stories is lower than other pollsters, the absolute number is higher.

In the three days since Cohn's first article on PPP, the news archive site News Library shows that PPP has been cited 70 times by newspapers and television stations throughout the country. That's actually more than in the same period during August.

4. Rasmussen still gets cited

Rasmussen Reports didn't do a very good job in 2010 or 2012. Most people don't think their methodology is exactly top-notch. Yet, they still produce polls, and people still talk about them. They got 41 hits on News Library in the same period PPP had 70. That's despite producing fewer state data than PPP.

If an inaccurate pollster with a methodology that some deplore continues to make the news, then I just don't see how one that many people view as accurate, even if dubious is going away anytime soon.

5. The aggregators who used to use PPP are still going to use it

Blumenthal's HuffPollster, Drew Linzer, Real Clear Politics, Silver, and Sam Wang produced polling aggregates that became must-reads for many political junkies in 2012. Had anyone pledged not to use PPP in 2014 or 2016, it would be big news. It turns out that none of them is eliminating PPP from their datasets.

Linzer and Wang have actually defended PPP's methods. Silver has been harshest about PPP, though PPP is still going into his models. Blumenthal has been more even-handed, but will continue to put PPP in the HuffPollster aggregates. Real Clear hasn't said anything; but they continue to put PPP surveys into their database.

They all seem to think that PPP adds more than it detracts from their aggregates. Indeed, therein lies the rub. No one thinks PPP's data is phony, even if people believe it to be massaged at times. Even PPP's most ardent critics, like my friend Republican pollster Logan Dobson, will cite PPP; especially when it shows a result they like.

Conclusion

Many professional pollsters have long thought that Public Policy Polling's methods were suspect. Nate Cohn's recent expose confirms for many their suspicions. That said, in a world in which polling data can be hard to come by, there's no real sign that usage of PPP's data has or will slow, as long as their polls continue to be seen as accurate.