Mark Blumenthal of The Huffington Post, perhaps the best public resource on the intricacies of polling, has taken an inside look into the nitty-gritty details of Gallup's polling. The piece is a must read for polling junkies, but the short story is that defensible methodological choices lead Gallup to under sample non-white voters and produce marginally more GOP-friendly results than other pollsters. Given the stark disparity between white and non-white voting preferences, even modest changes in the demographic composition of a poll can make the difference between a tight race and a clear advantage for one side. Blumenthal is digging deep: he goes into the raw data and carefully examines Gallup’s samples with the help of Gallup’s pollsters, who generously answer the questions he raises along the way.
How should these insights influence our reading of the polls? For most poll watchers, Blumenthal's piece should be yet another reminder of the virtues of polling averages, which happen to be remarkably accurate despite their simplicity. Every poll has quirks, and few pollsters are transparent enough to judge their methodological peculiarities. However, the oddities of individual firms and samples tend to balance out in the aggregate and, consequently, the polling average works pretty well.
While some are likely to spin Blumenthal’s piece into “Gallup is wrong,” it is important to emphasize that Gallup’s cooperation and transparency enhances the credibility of their polling. I’ll take a transparent pollster that shows its work every time, even if it has a discernible house effect. Poll watchers can make adjustments for known house effects with an identifiable cause, but too many pollsters release top-line numbers without an account of the underlying details. Gallup’s willingness to answer questions, release raw data, and report detailed demographic breakdowns is admirable and the organization’s numbers should not be discounted, especially given their sample sizes.
Instead, Gallup’s numbers can be yielded with confidence, but in context of the differences that distinguish their numbers. Most polls of registered voters generally assume a more diverse electorate than Gallup and a stronger Obama showing among African Americans. At this stage, no one can predict the composition of the electorate with confidence, so there is no reason to declare any pollster “wrong” on account of their demographic breakdowns, at least at this stage. Instead of declaring Gallup wrong, one can say that the race is tied among registered voters, if whites are 77 percent of registered voters. Gallup’s transparency and large samples allow us to make that claim with relatively high confidence, even if there is cause to believe that they under-sample the non-white vote. With opaque pollsters, we cant qualify their findings, which detracts from our ability to gauge the state of the race.
The difficulty of polling non-white voters is cause to consider the white subsamples of polls. As noted in the Blumenthal piece, pollsters struggle to contact blacks and Hispanics for a variety of reasons, including youth, a greater number of households without a phone, lower educational attainment, and a higher number who speak English as a second language. Although whites are far from homogenous, many of the problems associated with low response rate groups are less acute among white voters. Better still, focusing narrowly on white voters controls for changes in the composition of the electorate, contributing to better comparisons between polls given the difficulty of gauging expected turnout five months before an election.
Since May 1, all but two polls give Obama between 38 and 40 percent of the white vote. One of the two polls, Fox, did not push undecided voters and probably would have produced a 38-40 percent figure if they had done so. At approximately 39 percent of the white vote, Obama is far from assured of reelection. In this range, realistically high or low estimates of non-white turnout and/or support for Obama could deliver the presidency to either party. If Obama can pick up additional white support, Romney would start needing gains among Latinos or surprisingly low non-white turnout. Conversely, Obama’s chances would be jeopardized by additional losses among white voters.
In an era of low response rates and an increasingly diverse electorate, polling is becoming more and more difficult. To account for these challenges, well-intended pollsters will make defensible but different choices. Gallup’s choices clearly yield a less diverse electorate than other pollsters, but Gallup’s transparency should be lauded and their polling should not be discounted. Instead, one should continue to follow polling averages and view Gallup’s top-line numbers in the context of their underlying choices.