ELECTION COVERAGE

Little Margin for Error
Patrick Moynihan and Laura Backstrom, Fordham University

The polling community may have become accustomed to politicians dismissing the relevance of the polls in decision-making (most notable in recent memory being Governor Schwarzenegger’s statements at the GOP Convention), but this campaign season criticism seemed especially fierce from both the right (e.g., Republican challenges to polling by the Minnesota Star Tribune and the L.A. Times) and the left (e.g., MoveOn.org’s full page advertisement in the New York Times criticizing the Gallup Organization). Coupled with these accusations, suspicion about the veracity of the polls continued throughout the campaign season as results seemed to vary within and between organizations – followed by notable stories in prominent publications as the New York Times and the San Francisco Chronicle (among others) which critically reviewed vagrancies in polling operations. These are certainly not positive developments for cultivating a credible professional image in the public, but as the coin of the realm in polling is accuracy, it’s reasonable to move beyond the recent spate of claims about the industry and review actual survey organization performances in the national and state pre-election polls.

Of the seven national pollster vote projections listed by www.pollingreport.com, five out of the seven were correct in projecting Bush as the winner (see Table 1). Two called the election accurately for Bush with a three percentage-point margin (Battleground: Tarrance by 3.4 points and Pew Research Center by 3.0 points) while three others had Bush in the lead (TIPP by 2.1 points, Harris by a single point, and Zogby by 0.3). Two organizations inaccurately called the contest (Gallup as a push and Battleground: Lake had Kerry with a 2.1 point lead). By this simple but straightforward measure, these projections were fairly accurate and consistent as they were all within 5 points of each other, and if you take away Battleground: Lake all of these projections had the margin from a push to +3.4 points. Simple counts of correct predictions from the final 14 trial heat polls showed a similar degree of accuracy (with 10 showing Bush with a with a positive margin).

Another simple strategy to assess the accuracy of these polls is to take the arithmetic averages of candidate support. For the pollster vote projections, we find average support for Bush at 49.8% and 48.7% for Kerry – a percentage-point gap of 1.1. This closely mirrors the election results: Bush at 51.1% and Kerry at 48% for a 3.1-point margin. Across the final trial heats, we find averages of 48.5% for Bush and 47% for Kerry for a 1.5-point margin – again closely following the actual election results.

When a similar but more comprehensive tack is taken with the state-level polls, we find some interesting patterns. (These analyses are based on the 1203 state-level polls found at www.dalythoughts.com, which named Kerry as the Democratic candidate and had data for each candidate. See Table 2.) Looking across the polls fielded during the entire campaign across the 50 states, we find that 23% (278 of 1203) of the individual polls had results in the wrong direction or as a push. Given that we are including polls from many months back, this is a pretty impressive outcome. As expected, the vast majority (238) of these ultimately inaccurate polls were concentrated in 10 closely contested states: Florida, Iowa, Michigan, Minnesota, New Hampshire, New Jersey, New Mexico, Ohio, Pennsylvania, and Wisconsin. In fact, 28 states showed no polls in the opposite direction of the electoral outcome (or even a push) at any point over the course of the campaign.

According to these data, the average pre-election Bush-Kerry gap was in the wrong direction in only Wisconsin where the pre-election polls averaged a slight edge for Bush, but Kerry managed a victory of less than one point. (In three states – Iowa, New Hampshire, and New Mexico – the average pre-election polls showed a push whereas in the actual election the difference was about one point in each.)

When we look at the differences between the electoral outcome and the average pre-election margins by state, we find only eight states with differences of five points or more (Alabama, Arizona, Hawaii, Idaho, Kansas, Oklahoma, Vermont, and West Virginia). In each of these cases, we find the average pre-election gap between Bush and Kerry closer than the electoral outcome – but, interestingly, each case was a state where the electoral outcome margin was at least nine percentage-points. So the state-level pre-election polls were least precise in those states that were not hotly contested.

Another general trend was that undecideds and others missing from the pre-election polls did not break evenly, as Bush averaged over four percentage-points less in state pre-election polls than on November 2nd and Kerry about three points less. Even still, simple correlations between state electoral outcomes and state pre-election poll averages for each candidate’s level of support were almost perfect: +0.97 for the Bush vote and +0.98 for the Kerry vote.

While we find this all to be strong evidence of high quality work executed by the pre-election pollsters, we offer a few cautionary notes in thinking about how these results might be interpreted by the public. As for the simple counts, it seems possible that some individuals use such a method to assess poll accuracy – that is, to keep a (mental) tally of results in favor of one candidate versus the other. But should the fact that 70% of the national polls were the right direction be trumpeted – or the fact that over one-quarter of these final predictions were wrong be lamented? How do members of the public define such results? As for the national polling average margins of 1.1 and 1.5 points, it seems that members of the public are rarely exposed to poll averages, so it might not be a fair assessment of the accuracy of polls from the standpoint of the public. Lastly (but related to the first two caveats), while we present results across a variety of polls, we wonder if people look at even a fraction of the available polling data. This seems unlikely given the search costs required to compile this information on one’s own and that network media – which are a main source of such data for the public – are unlikely to represent polling data from competing organizations.

In an extremely close race, the pollsters were as accurate as can be expected within the fundamental limitations of polling. But such points are often lost on the public. Part of the problem is not the polls themselves, but the nature of the contest itself. The outcome of this election was, in many ways, unpollable. For pollsters to be considered accurate in a tight election by the public, they have to be dead-on with their numbers. It is not enough to be within a couple of points of the outcome as the outcome was a couple of points. Others have made the point that polling is a blunt instrument not necessarily appropriate for the delicate surgery of narrow margins. Consumers of polls might not be satisfied with the pre-election poll results because they could not give a clear and right answer, even if they were technically accurate and did the best they could in predicting an unknown population parameter.

The reasons for public skepticism of polling are many and extend beyond this election. A fundamental cause of confusion and frustration with the polls is a lack of understanding of basic statistical and methodological principles. But in thinking about the public’s perspective of the election polls, it might be worth noting the expectation of polls to provide a clear-cut winner and that for the public the value of the polls rests on their predictive function, rather than just merely being fairly accurate and technically sound in a tight race. This year these expectations were unfulfilled.

Presently, pollsters face a challenge in demonstrating their worth to the public. The polling industry has a unique contribution to make to political decision-making and social policy. If members of the polling community are to continue to serve the public by providing information to allow them to make informed decisions, then asserting credibility to the public is more than a mere public relations concern.

Patrick Moynihan is assistant professor of sociology at Fordham University and Laura Backstrom is an undergraduate student majoring in sociology. For comments or questions, they can be reached at pmoynihan@fordham.edu and backstrom@fordham.edu.

Click here Table 2: Accuracy of State Pre-Election Polling Averages.