For pollsters, 1999 was a very short year. Of course, 1998 was extra-long: it stretched right into the first two months of 1999, with near continuous polling about the Clinton/Lewinsky scandal, and with poll results under never-ending attack. The 2000 polling year began before 1999 was even half over. And I can safely predict that in 2000, polls will be attacked again.
But we shouldn't be surprised to see our polls under attack. Indeed, polls will face three serious challenges. First: polls have become, in effect, "players" in the political process. Second: we can't change that fact, so we need to be even more open about what we do. And third: we can't ask stupid questions any more.
Let me explain what I mean when I say we are "players." I work in the media polling world, and for better or worse, we are the first line of defense for public opinion research. We make the research visible to the public, and that has been both a great strength and the trigger for much of the criticism.
Media polls have been important tools in allowing journalists to establish and maintain objectivity. When Fortune magazine began working with Elmo Roper on the Fortune survey, in 1935, its editor wrote (in the third person) that he had "no preference as to the facts he hopes to discover. He prefers no particular outcome. He is quite as willing to publish the answers that upset his apple cart of preconceptions as to public the answer that bear him out."
We all know that media polls can do that. Witness how many journalists had their "apple cart of predispositions" upset by the polls they reported during the 1998's Monica Lewinsky scandal. They reported them avidly and often, even though it was apparent that many didn't really believe - or didn't want to believe - the results.
I contend that news reports of public opinion "democratize" information: by which I mean that information about the public that was previously accessible only to the power elite - to politicians and decision-makers - is now available to the public at large. As a result, many more individuals can choose to act on or ignore public opinion information than ever before.
Hamilton Jordan claimed media polls "changed the rules." He was chief of staff under President Carter; and in the early 1980's Jordan and I were on a panel together at Emory University, where he took issue with me, CBS News, and the New York Times, for conducting and making public our poll results. I asked him whether he had ever learned anything new about public opinion from us - anything that he had not already heard from Carter's own pollster, Pat Caddell. And Jordan responded, "Of course not." Obviously, we had obtained much of the same information that Pat Caddell had obtained, only we had made it available to a wider audience. That's why Jordan accused us of "changing the rules."
Well, the fact is: the rules have changed. And paradoxically, it is our success that has made us so vulnerable to attack.
Over the past 50 or 60 years, polling and polls have become generally accepted as newsworthy information. And there is a tacit understanding everywhere that polls are in fact accurate indicators of public opinion. If they were not perceived that way, no one would bother to attack them!
We can be proud of this success, but it has come at a price. One of our core beliefs has been shattered. We no longer have the luxury of believing that the world views public opinion research - the polls - as a neutral search for truth. We may choose to believe that what we do is impartial, but much of the country does not see it that way.
The politicization of polls is not new. Ever since public opinion results began making their way to the public directly - mainly through news reports - they have been attacked by people whose interests are served by rejecting the findings. There is a long history of objections from the power elite.
Back in 1824, voters in many states did not have a direct voice in the selection of presidential electors. But poll counts began appearing in the partisan papers - poll counts that suggested that the public might not agree with the legislature - their political leaders - in their choice of electors.
In some cases, counts of candidate support were taken at public meetings. In others, books were opened for people to register their preference. Some newspapers praised the technique. Niles Weekly Register said of a count taken at a public meeting, "This is something new; but an excellent plan of obtaining the sense of the people." Another paper said about the poll book, "We would recommend to our fellow citizens throughout the Union this mode of ascertaining the sentiments of the people. Let the political managers at Washington and elsewhere know the people's will and, if that is not to decide the question, why let the people know it."
But criticism was just as quick in coming, according to recent research done by Keating Holland of CNN. He scoured through old newspapers in the Library of Congress, and found that polls were praised mainly by fans of those politicians who were leading the polls: Andrew Jackson and John Quincy Adams. But the polls were criticized by opponents, especially by the Federalists, who - unknown to them, of course - were about to lose the 1824 election and, soon after, to disappear as a party.
The Federalists attacked the poll book. How did they attack the reports? It almost sounds like vote fraud - or poll fraud. Their criticisms: Bad sample and bad response. The book, they claimed, contained votes of minors, of nonresidents, and some people have erased their names! And only a sixth of the eligible voters participated!
In 1896, the Chicago Democratic party railed against a poll conducted by the independent newspaper, The Chicago Record. They called it "a scheme - one of the fraud and debauchery, the first step to do away with popular elections under the law and place the molding of public opinion on the hands of millionaires and corporations.
What had the Record done? It had sent postcard ballots to every registered Chicago voter, and to a sample of one in 10 voters in eight surrounding states. The Record mailed a total of 833,277 postcard ballots, at a cost of $60,000. By the way, they got back some 240,000 of those sample ballots. The Record found that Republican William McKinley was far ahead of the Democrat, William Jennings Bryan. McKinley won; and in Chicago, the Record's pre-election poll results came within four one-hundredths of a percent of the actual election-day tally.
The U.S. Post Office has recently issued "The 1940's" in its series of stamps honoring the 10 decades of the Twentieth Century. And among the dozen or so iconic images of the '40's that are commemorated there is Harry Truman, the morning after his 1948 victory, holding up that Chicago Tribune with the eight-column headline: "Dewey Defeats Truman."
Now, Harry Truman had a real complaint about the polls. He called polls "sleeping polls" because, he said, they were like "sleeping pills: designed to lull the voters into sleeping on Election Day." Intelligent voters, according to Truman, were "not being fooled. They know that sleeping polls are bad for the system. They affect the minds. An overdose can be fatal."
Of course, poll bashing in contemporary presidential campaigns is common, but lately it has increased as a proportion of campaign content. If you examine election data bases compiled by The Annenberg School at the University of Pennsylvania, you would find that in 1988, Michael Dukakis referred to polls in just under 20 percent of all the speeches he made in the fall campaign. He said things like "pollsters don't vote, people do."
George Bush, Who was leading in that election, referred to polls only a third as often as Dukakis. But in 1992, when Bush was trailing, he poll-bashed in more than 30 speeches: the equivalent of one in every four times he spoke publicly.
And in 1996, Bob Dole talked about the polls even more frequently: in one third of all his speeches.
Perhaps the epitome of poll-bashing was reached in early 1999. The San Francisco Chronicle headlined a story about Henry Hyde and the Republican House impeachment managers, explaining why they did not prevail in the Senate: The headline read: "Managers Blame Opinion Polls."
Those who attack pubic polls these days treat them as players in the political process. The short version of this is apparent in what the Dole campaign said in 1996, when it accused media polls of interviewing "too many Democrats."
And that sort of attitude is not confined to politicians. As part of a larger study, Bob Shapiro and Ron Hinckley have discovered that many reporters, in their stories, divide pollsters into "Republican" And Democratic" ones, balancing comments from each in order to appear objective and fair. Unfortunately, this practice suggests that all of us who do survey research have a political goal or orientation.
When people talk about "data" they should all be describing the same thing; data does not have a "Republican" or a "Democratic" interpretation. But when analysis is characterized that way, it reflects on all pollsters. In fact, Hinckley and Shapiro found just that when they looked at the variables that relate to the credibility of pollsters among the public - that is, to the belief that pollsters give "truthful information and honest opinions." They found that those variables are all political. Whether or not one is a strong partisan, a Clinton voter or a Dole voter, is what predicts one's position about the credibility of pollsters. Not demographic variables such as gender, age, or income, but political variables.
Hinckley and Shapiro's study was done in September 1998, when the direction of the public poll results were apparent, and the attacks from political leaders were among the fiercest they had been all year. But the perception of politicization - that there is politics in everything that pollsters do - had already filtered down to the public at large. What had once been the province of the candidates, of the people that we would consider players, is now part of the way the public overall views the polls.
Well, that's one challenge, and we will just have to live with it. The politicization of the way we work is a given for 2000 and beyond. But that's not the only challenge we face. There are two more. But at least we can do something about those two.
Pollsters must, for example, be up front in telling people about our work. Our professional organizations (and there are many, not just AAPOR) have strived for years fro greater disclosure in describing polling methods, saying what's important for public polls to release. The public is entitled to know things like: when the poll was conducted, how it was done, how many people were interviewed, what questions were asked, and so on.
The effort to open up the process in that way has been fairly successful. Even the Political Hotline, which reports polls on an almost daily basis, actually refuses to print results from some polls; and it labels others as small sample polls. In the past, the Hotline cautioned its readers about one-night surveys, but the events of the last year may have legitimized the overnight poll. In fast moving times, as 1998 might have been, any day-to-day change (or lack of it) was clearly news.
Disclosure is critical to the acceptance of public opinion polls. It makes clear what they are not: that they are not self-selected, that they are not sponsored by any party, that they are not (or in some cases that they are) restricted to only part of the public. But most important, disclosure indicates that the pollsters are not hiding anything.
In the twentieth century, disclosure has been critical in establishing the journalistic integrity of the opinion poll, but that same demand for journalistic integrity has generated unintended consequences: chiefly, it has opened the door to methodological attack. Among critics, the more information they have, the greater the likely scope of their criticism.
The terms that we use to understand what we do are the same terms that others can now use to attack us. Even people who have little learning and understanding can talk about bias in question wording (and sometimes they can even cite academic sources). They can raise concerns about question-order effects, and about response rates. Sometimes what appears as methodological criticism is really political criticism: If you're unhappy with the results, attack the process. If you don't like the message, kill the messenger.
We haven't yet figured out how to cope with these attacks, because there are some core truths to them. We must be concerned about question wording and order, and we have always worried about non-response. But better methods, fully disclosed, are still our best protection against unsubstantiated attacks. I mentioned earlier the Federalists' challenges to the poll book of 1824 - when on a sixth of all potential voters participated. That sure sounds like a response-rate issue to me! And they even made a how-the-poll-was-conducted critique: the poll book made its first appearance in a bar! How can you possibly think those respondents knew what they were doing?
We ourselves give the public some of the openings through which they can attack polls. Sometimes we ask questions that can create opinion where there is none. Sometimes we find ourselves in the midst of controversy because, in retrospect, we asked rather stupid questions, or because we interpreted the results stupidly.
I said "stupidly" was the third challenge. But just as full disclosure of methodologies can counter claims of bias, there is a way we can take control of this challenge to our intelligence.
We don't always do it. Sometimes we are just too full of ourselves. George Gallup once said, "We can try out any idea in the world!" He was right, but with that ability comes power: a power that media polls sometimes take too lightly. Hypothetical questions can sometimes take on an aura of reality.
Consider horserace questions. They are certainly hypothetical questions, along the lines of: "If the election were being held today…" Potential voters are often asked to take us into their decision-making process: "If the election were held today and if the following were the parties' nominees…"
But times change between the poll-taking and election day. George Bush trailed badly in 1987 against the then-most likely Democratic nominee: Mario Cuomo. And Bush had a dramatic lead over every possible Democratic opponent in 1991. (Scarcely any Democratic challengers could even be found, at the end of the Persian Gulf War, when Bush had an approval rating of almost 90 percent.) Of course, neither of those elections were held on the "today" when respondents were asked to imagine for whom they were likely to vote. And that is a key point to remember, as we poll for the 2000 election season.
Let me suggest another reason to be wary of relying on hypothetical constructions. In 1983 and 1984, in the months before the Democrats named Geraldine Ferraro as their nominee for vice president, much of the campaign in support of her was based on a hypothetical poll question that was asked several times in several ways, but boiled down to this: "If the Democratic party nominated a woman for vice president, would you be more likely or less likely to vote for the Democrat…?"
One of the more frequently reported results suggested that more than a quarter of the public would be more likely to vote for the Democratic candidate if a woman were on the ticket. But the party insiders who used news reports of those results to push a female candidacy forward apparently forgot to check on the respondents' party affiliations, or to analyze the partisan breakdowns of the respondents as a whole. Naturally, many if not most of those who said that putting a woman on the ticket would make them more likely to vote Democratic were already committed to voting Democratic.
The Ferraro candidacy was, in this respect anyway, based on bad analysis of the data. The overly high expectations later resulted in crushing defeat. Did members of the press learn from this? Some did; others didn't. And one reason why they didn't may have been that sometimes a story is too much fun to step back and evaluate what the polls are measuring.
In 1992, for example, well before the end of the primary season, the nominations of George Bush and Bill Clinton were easy to predict. So Ross Perot emerged as "the story." In California, Ohio, and New Jersey - states holding their primaries in early June - exit pollsters asked what would have happened if Ross Perot had been on the ballot that day. "The story" that emerged suggested that Perot was the "people's choice," at least that day. But he certainly wasn't the people's choice when it really mattered - on the day they actually had to vote for president.
In 1998, a congressional election year, election-related polling actually took a back seat to polling on a matter of ostensible grater importance. But again, "what if" questions dominated the polls.
IN the early months of 1998, we were asking such questions as: What if Bill Clinton had an affair with Monica Lewinsky? What if he lied about it? What if he committed perjury? What if he encouraged Monica Lewinsky to lie under oath? By the middle of the year, the questions ran to things like: What if Clinton had an affair with an intern? What if Clinton now publicly admits he did have an affair with Lewinsky? What if there is conclusive proof that Clinton lied under oath about having a sexual affair with Lewinsky?
And to most of these questions, whatever the "if" was, majorities of respondents claimed that, if it were true, then Bill Clinton should leave office. Of course, as more and more real information came out, the poll results took rather different directions. The what-if questions early in 1998 seemed to show support for Clinton's resignation and impeachment under certain circumstances; but that support went away as more and more Americans decided that in fact there had been a sexual relationship and that Clinton had, at the very least, misled and obfuscated under oath.
By the end of 1998, as the impeachment vote in the House of Representatives neared, polls asked what should happen if the president were impeached. There were wildly different responses - different from each other and different from opinions expressed after the fact. That alone should remind us that our assessment of what a hypothetical question means may not be the same as the public's assessment.
We demonstrated this last year. In a CBS News/New York Times poll conducted during the week preceding the House vote, half the sample was asked; "If the president is impeached, would it be better for the country if he resigns?" And 57 percent said "yes." The other half of the sample was asked: "If the House voted to send articles of impeachment to the Senate for a trial, would it be better for the country if the president resigns?" Only 40 percent said yeas to that. One reason for the difference is that, up until the time of the House's vote, nearly one-third of the public was under the mistaken impression that impeachment was the same thing as removal from the office, in which case it was logical to them that Bill Clinton should just leave quietly, instead of being dragged kicking and screaming perhaps, from the White House. So support for resignation was much higher in the first formulation than in the second, when the two-step nature of the process was made explicit.
Of course, after the impeachment vote was taken, still fewer people said that resignation would be better for the country. I think that because the issue was no longer hypothetical: Bill Clinton had been impeached, and the sky hadn't fallen. Only 31 percent said, then, that they wanted him to resign.
I'm sorry to say that the news we generate from asking what-if questions is too often news that we expect to generate. Whenever there's a military action somewhere, we already know that support for it will drop when casualties are hypothesized.
Still, media pollsters tend to focus too much on hypothetical questions, and even when we report the results with the necessary openness about our methods, that narrow focus can have negative consequences. Some have come to view legitimate polls by candidates and the media that ask hypothetical questions as analogous to "push polls," when campaigns (behind the mask of research) seek to plant negative opinions about a candidate into the minds of as many voters as possible. "Push polls" give voters negative and sometimes even false information in hypothetical question form. (A typical push-poll question takes this form: "If you knew about some nefarious activity by the candidate you currently support, would you still vote for him?")
Perhaps the saving grace in my three challenges - the politicization of the polls and pollsters, the need for openness, and the need to stop relying on easy but sometimes meaningless hypothetical questions - is that at their core, there is a truly democratic question - "Whose opinion matters?"
In 1824, polls showed - at least - that there was a discrepancy between public opinion and that of the political elite. Today, polls serve as a reminder that all information is democratic. That Fortune magazine editor who claimed, in 1935, the he wanted any kind of data, whether expected or not, also said that the information he gathered should be made available to all. That willingness, and the resulting democratization of information, is still key to the strength of polling.
When people attack polls, the only fix they typically offer is "Don't participate." And that is an anti-democratic fix, an elitist solution that, in today's world, indicates as surrender to the notion that only some people's opinions matter. Ironically, of course, non-participation works against the attackers' own agendas. They need polls to further their aims, since without some knowledge of what the public feels, they would be unable to develop efficient strategies. Most politicians and candidates simply couldn't function if non-participation in surveys were widespread.
Of course, if we take the question of "whose opinion matters?" seriously, we will have to work harder to reach people. We can't develop methods that deliberately leave some segments of the public out. The message that sends to people is very clear and very dangerous: that only some opinions matter, and that (their worst fears realized) we do just make up the rest.
And that would play right into the hands of those who hold with the academic, post-modernist critique of polls. To the post-modernists, poll majorities are not social but statistical. They're like a home movie, a floating sign, an image, not a reality. Internet polling, for example, as it's presently practiced, fulfills that post-modernist fantasy of polling as simulation. Murray Edelman, editorial director of Voter News Service, has described Internet polls as a "projection of a poll," using some people who have certain qualities (say, access to the Web and a willingness to sign up for surveys) to represent those who have only one or neither of those qualities.
I've spent a great deal of time here on the challenges we'll be facing in 2000 and beyond. But we mustn't lose sight of the fact that the polls wouldn't be subject to attack if they were not widely perceived to be accurate. In fact, so accurate are they perceived to be that reporters and politicians resort to citing them. And if there are no polls around, they will make something up that sounds like a poll!
In early June, Jim Lehrer (of all people!) said that the media, according to the polls, is viewed as poorly as lawyers, the Congress, and pornographers. Now, there's a lot of data out there about the public's level of trust in the media, in lawyers, and in Congress. And you can certainly quarrel about their relative rankings in public esteem. But there isn't too much poll data about pornographers. At least not in the Roper Center POLL data archive. And I certainly don't know of any poll where "the media" and "pornographers" make it on to the same list. (If anyone does, please forward the specifics - and the methodology - to me.)
But I don't think Lehrer was citing a specific poll, anyway. I think he was just repeating one of those things that flit through the public consciousness like an urban myth. He claimed that the media was held in as low regard as pornographers. Well, that might be true. Possibly it is true. But what's more important is that Jim Lehrer and at least some of his audience are now sure it must be true. Why? Because he talked about the polls.
The polls are the new expert in town. That's the good news, I suppose. But it's the direct consequence of polls having achieved success and accuracy. Polls give precision to opinion. "Counting" defines things. The reason particular race and ethnicity categories matter is because the Census defines them. Such numbers don't merely reflect reality - they create it. So in effect, polls create what public opinion is. The numbers in polls quantify politics in a deceptively precise way, and those numbers mean "objectivity" to journalists.
Numbers also imply expertness. Reporters have been seeking expertise - in themselves and in others - for a long time. And for that purpose polls work very well. Reporters rarely say "experts think such-and-such a think." And as Jim Beniger of USC recently pointed out to me, reporters never say "academics thing…" or "most professors say…" And yet we often hear reporters say "polls show…" or "the latest polls suggest…" How often does this happen? IN 1996 alone, the phrase "polls show" was used 111 times in CBS News television broadcasts. And other news organizations were at least as likely to use a similar construction.
But what do the polls show? For many reporters, what "polls say" is the launching pad for an analytic statement: "Polls show that the public no longer supports the bombing of Kosovo." Or "Polls show there is voter backlash on the character issue, but it's directed at the attackers, not at Mr. Clinton." In an interview, an introductory clause like that enables a reporter to appear as an expert and it is almost always used to position that ostensible expertise in opposition to the subject of his views. Typically, it takes this form: "Lamar Alexander: Polls show you stuck in sixth place. Why haven't all your efforts to win the presidential nomination been reflected in the polls?" Faced with a question like that, it's hard enough to frame a good answer, much less to challenge the underlying "expertise" associated with "the polls."
And after all, "the polls" are widely viewed as accurate reflections of public opinion. Or all the problems critics have with what we researchers do, that fact - that confidence in our fundamental accuracy - will help us get through in the next decade and meet the new challenges. If we're ever ignored - that's when we should start worrying!
We can handle the challenges. We continue to remind the public - nay, the whole world - about whose opinion really matters. The challenges that arise are almost all due to the fact that what we do is widely accepted as truth. And throughout human history, truth has always come under attack.
Despite ongoing questions about the credibility of the pollsters, and despite the attacks, the polls will continue to define politics. Polls both inform and elevate the level of public discussion. Polls are a way to hold up a mirror to the public, to enable individuals to understand where they fit into their own political system. And reporting public opinion polls tells readers and viewers that their opinions are important. That democracy matters. And that their opinion counts.
NYAAPOR 152 Madison Avenue, Suite 801
New York, NY 10016
Tel: (212) 684-0542
Fax: (212) 481-3071 info@nyaapor.org
