Kudos to the editors of this blog for passing the saltshaker. StPetePolls and SaintPetersBlog are affiliated – loosely affiliated, but affiliated nonetheless – so it could have been awkward to do a true arm’s length assessment.
I’m happy, therefore, to take a stab at it.
First, good move in balancing parties by what can reasonably be expected to be a +3 GOP advantage. That will cover a lot of sins in a poll, as party is currently the strongest anchor in any poll measuring this gubernatorial race. With so many pollsters getting this basic fact wrong, it was a breath of fresh air to see this vital element handled well.
Second, large numbers are great – and this one has a sample size of over 3,000 – but it is vital to understand that large numbers do not necessarily cover methodology sins – they can often compound them.
Third, combined methodology of robo-calls and emails can create problems if not handled correctly. And in this example, the emails came from a proprietary voter file, so it is fair to assume that people who volunteer their email addresses either through campaigns or to the local supervisors are more likely to be engaged voters, campaign volunteers or activists. The other problem – as has been discussed in this blog – comes with blending an audio poll that says the word, “libertarian” in conjunction with Adrian Wyllie and an Email poll that either shows the ballot correctly (“LPF”) or incorrectly with the word “Libertarian” in order to be consistent. This is the kind of Hobson’s choice that anyone using a combined methodology faces and, as best I can tell, there is no right or wrong answer. But it is at least a small problem.
Fourth, the lack of cell phones, in my estimation, will cost a half-grain.
Finally, the methodology says in two separate places that this is a survey of registered voters. This is an off-year election where turnout will likely hover around 50% (it was 47% in 2006 and 49% in 2010, the last two off-year elections.) Polling “registered” voters is not the same as polling “likely” or “probable” voters as they are very different (and only partly fixed by weighting) and this oversight costs the poll another half-grain.
In total, the balance by age, race, gender and party look solid and the “Libertarian” thing isn’t that big of deal. But no cell phones and not screening for (or pre-selecting) likely voters means this poll should be taken with a grain of salt – but just a grain. The totals are consistent with other well-balanced polls and, based on what I am seeing, is probably a fair indication of where this race sits today.
What is the SPB Salt Shaker Scale? It’s our new measurement guide to reading polls. These are the ratings:
- A full shaker: There are enough problems with the methodology to warrant serious concerns.
- A few grains: There are some concerns with how this poll was conducted and/or with the Demographic make-up to raise some issues.
- A single grain: There are only a few concerns with how the poll was conducted.
- No salt needed: Solid pollster, solid methodology and the numbers look balanced.