1997 - NEW YORK AAPOR OUTSTANDING ACHIEVEMENT AWARD
Public Opinion Research: An Honorable Profession, Warts And All
by Harry W. O'Neill
Those of us in survey research are in a fascinating and rewarding profession - an honorable profession of which we all should be proud. I would not have spent 40 years in it, and probably would have retired by now, had I not found it stimulating, challenging, of significant value and often just plain fun. We are in possession of a unique process for gathering information that can and does serve many useful purposes for a wide and growing array of users, for consumers and for the public at large. It is, however, a process that is easily misunderstood, misused, abused and overused. It is to some of these "warts" that this paper is directed.
Now some may take issue with focusing on problems rather than plaudits, with stressing negatives rather than the positives of our endeavors. However, I think it useful occasionally to take a look at our "warts," if for no other reason than to raise our consciousness and therefore cause us to strive to improve our performance. I do not mean to be a nattering nabob of negativism, but rather to be a constructive conveyor of criticism. So here goes - 10 warts.
1. Opinion polls have given respectability to uninformed opinion. We ask people about every conceivable issue and dutifully record their responses - whether or not they have ever heard about the issue or given it any thought prior to our interviewer bringing it to their attention. How much better to ask awareness first and ask opinion only of those with some awareness. But this, of course, takes time and money, both of which often are in short supply. Also, we have been known to probe a complex issue, such as the situation in the Middle East, with two questions, often overly simplified, on an omnibus survey and report the results as definitive public opinion - thereby making another serious contribution to the understanding of our country's foreign policy.
2. We have learned too well to be advocates rather than objective researchers. Now I have nothing against conducting research to be used for purposes of advocacy. I've done a lot of it and it can serve a legitimate and useful purpose. However, such research presents us with special responsibilities and obligations, which are not always in evidence.
It is tempting to serve our clients or their particular causes too well. For instance, if the National Endowment for the Arts retains us to show that the majority love the arts and are in favor of public funding of the arts, we might ask people if they agree or disagree with this statement: "An active arts program gives people an uplift from their everyday lives, contributes to a community's quality of life and gives creative people a socially acceptable outlet for their talents." However, if a politician who wants to demonstrate disapproval of public funding of the arts is our client, we might ask people if they agree or disagree with this statement: "Given a severe deficit and a lack of money for education in depressed areas, healthcare for the sick and infirm elderly, and school lunches for hungry children, public funds should be given to so-called artists who exhibit pornographic pictures and dance naked smeared with chocolate syrup." While our advocacy is usually more subtle that this, not infrequently it is just as biased.
Consider this actual example from a number of years ago. It was a poll on the "New Right." The poll claimed that conservatives were moving to the left toward more liberal positions, that the differences between liberals and conservatives are mostly differences of degree rather than direction. Among the several questions asked in this poll were these three:
Do you agree or disagree that the federal government ought to help people get medical care at low cost?
Do you agree or disagree that the federal government ought to see to it that everybody that wants a job can get one?
Do you agree or disagree that if blacks are not getting fair treatment in jobs and housing, the federal government should see to it that they do?
Guess what? Majorities of both liberals and conservatives agreed with each question -leading to the conclusion that "there are many supposedly 'new right' issues on which conservatives and liberals think alike. The way they think on many issues is more reminiscent of the New Deal liberal than of the 'new right' conservative."
At the request of the North American Newspaper Alliance, an equivalent sample was asked the same three questions, except "private enterprise" was substituted for "the federal government." And lo and behold majorities of both conservatives and liberals agreed with each question, proving, of course, that liberals are moving right towards more conservative positions.
The only valid conclusion to be drawn from either poll is that if the public perceives something as a problem and the public is offered a solution, the public probably will support the solution offered. No ideology is involved, only the desire to see pressing and persistent problems solved somehow by somebody. But this was not the publicly released message of the original poll, which was, I suspect, the message the originator of the poll wanted registered.
3. We mislead the public. This issue, of course, is related in many ways to the previous one, but more specifically goes to how surveys are released. Given the potential for advocacy research to influence public opinion and given its potential for misuse, not only must it be conducted with particular care, but we must insure that its public release is open and direct, clearly understandable, free of exaggerations, distortions and unsupported conclusions. Results must not be overinterpreted; results must be discussed within the specific context and time frame in which the questions were asked; extrapolation, if any, must be clearly indicated as such. Anything less falls short of our professional and ethical obligations to the public.
Recently MCI ran full-page ads touting a poll showing that "Americans voiced their disapproval about local phone monopoly access charges." The results of selected questions were shown. What was not shown is that over six persons in 10 said they had not heard, read, or seen anything about access charges and that seven in 10 were not aware of access charges. But nonetheless the public is told, in large headlines, that they disapprove of that which they don't know and what's more that nobody is listening to their disapproval.
A current issue is the TV rating system. The Valenti system, which is in place, is age-based. A suggested alternative system is content-based. A group supporting the Valenti age-based system released a survey that showed 83 percent of parents endorsed this system. The problem is that the survey asked only about this system; it offered no alternative. Two other studies - by the Media Studies Center and the National PTA - found the content-based system preferred over the age-based system by about eight in 10. Does this kind of situation not mislead the public?
Our responsibility here is twofold: (1) the proper design of the questionnaire - not, for example, to commit the sin of omission in leaving out a legitimate alternative and (2) controlling our client's releases to insure their accuracy or, when this fails, issuing a corrective release.
Do we mislead with sampling error, creating in the public's mind an image of being more scientific than is warranted? We have made sampling error a household term. No report of a survey in print or on TV, national or local, falls to give the sampling error (often inaccurately stated as a percent rather than in terms of percentage points) and, time or space permitting, the public is told that this is at the 95 percent confidence level. Wow! What precision! Not enough to put a man on the moon, but enough to give the aura of an exact science and, I suspect, leading the public to a false impression of a survey's soundness.
God forbid the public should ever find out that all sampling error and its 95 percent confidence interval really say is that if you conduct the same biased survey among the same unrepresentative sample 100 times you will get the same meaningless results 95 times within plus or minus some percentage points of the result of the particular survey being reported.
Now I will concede that informing the public that our surveys are susceptible to error has its value, but I'm afraid that our emphasis on sampling error - all too often in the absence of the mention of other sources of possible error - can be misleading.
4. We impose upon the public. We do this in several ways - apart from calling at dinner or during some more intimate moment - for example, by the sheer amount of research being conducted, by incomprehensible questions and by unconscionably long interviews.
Survey research may well have become excessive. Pick up any newspaper or magazine or listen to any news program and there is likely to be the report of a poll - ranging from rather important issues such as government spending to trivial issues such as whether lawn ornaments enhance a home's appearance. Just looking at political polls, it has been estimated that more than 400 national and state political polls were conducted for news organizations in the 1996 U.S. presidential campaign, and that, of course is in addition to local polls and the private polls at all levels. It's fair to ask if so much polling, besides trivializing all polling in the public's mind, is an undue burden.
We also impose by asking stupid, incomprehensible questions. Here is one of my favorite examples: It has been reported that in the sixth month of Clinton's presidency his popularity was weak and his outside consultants, including pollster Stanley Greenberg, were afraid that some of the economic policies being advanced would do further damage. The consultants were against both the House-passed BTU tax and the Senate-passed gas tax. In a poll, Greenberg asked about three options, including asking for respondent support or opposition to the option of dropping both energy taxes. This question was presented to the respondents:
"President Clinton may announce important changes in his economic plan in order to address the slow economy, which he believes is not creating jobs fast enough. This is the proposal. First, eliminate any new taxes on the middle class in order to boost consumer confidence. That means completely eliminating the energy tax. All new taxes will be paid by people making over $140,000. People earning under $30,000 will get tax relief. Second, Clinton will reduce slightly the deficit reduction goal from $500 billion to $480 billion over five years in order to find new incentives to create jobs. The plan will allow tax incentives to small business and technology companies to create jobs once the House and Senate pass his revised economic plan within a month."
Of the respondents, 63 percent expressed support, 28 percent opposition. Do you really believe this? How many people do you think understood this 126-word question? And what do you think the effect of such drivel is on our image?
Then there is the imposition of the excessively long interview. We laboriously construct a 45-minute telephone questionnaire on oven cleaners wherein excited consumers are asked to rate five competitive products on 30 attributes using a 10-point scale, rotating everything possible to avoid position bias and testing five different price elasticity scenarios, while giving no thought to how much sense the whole exercise makes because the ensuing discriminant analysis is fun and highly profitable. The four-color perceptual maps that come out at the end are worthy of museum display and indicate no sign of the respondent fatigue that set in during the fourth rotation of the third product. And another bunch of consumers has become non-respondents if ever contacted again.
5. We too often allow our romance with technology to overshadow the art of asking questions. We surely ask job applicants for a research position about their computer skills, but I'd put these second to their analytic and writing abilities.
Now I wouldn't deny for a minute that technologically there is a lot of good stuff going on in survey research today. But even though technological advances are important, they create their own problems, such as the temptation to believe that putting your data through some fancy computer model will produce the best solution to the problem under study, thus relieving you of the necessity for applying sound judgment. This is like the bachelor who wanted the perfect mate and took the route of asking the computer to locate her. He told the computer he wanted someone who is small and cute, loves water sports and enjoys group activities. The computer processed these data and out came the solution: Marry a penguin.
I know of no computer program that will transform data from an inept questionnaire into a meaningful report.
6. We misuse and overuse focus groups. They look easy - any idiot can sit around a table and talk to people - and a couple of groups are a lot cheaper than a quantitative study. So why not go that route? After all the Wall Street Journal writes front-page stories about focus groups to tell us how Americans felt about some pretty important issues. One expert in this area tells us that "a focus group is no more a simple group discussion than a group therapy session is a simple group discussion." I strongly recommend that we don't forget this.
The next time a client tells you that focus groups are all that's needed (obviously the misuse of focus groups is always the client's fault, never ours), relate this little story: It has been reported that when the Taurus automobile was first tested in focus groups, it bombed. People called it a "jelly bean." The Edsel, however, was a focus group success. I strongly recommend that you don't forget that either.
7. We are not as concerned as we should be about refusal rates. As mentioned earlier, we sure are concerned about sampling error. We conduct a thoroughly well designed survey among the general public on a serious issue for a client who is going to use the results in a perfectly legitimate manner - nirvana. And with a straight face we report to our client that the sampling error is plus or minus 2.7 percentage points. Got that? Not 3 percentage points, but 2.7 percentage points. God, we're good - and our client can happily report this at the press conference where the surveys are released for all the world to marvel at.
But the dirty little secret that we've kept to ourselves is that the refusal rate for the survey was 58 percent. What's that do to the concept of sampling error? We are following our Commander-in-Chief's policy for the military: Don't ask, don't tell. That policy is no more ethical for us than it is for him.
We are losing a precious resource - our respondents. We have learned well how to more effectively disturb more people at an inconvenient time - and then blame them for being uncooperative - and we have learned how to bore them unmercifully and insult their intelligence when they do cooperate.
The respondent cooperation problem, of course, is not entirely of our own making. In addition to increased requests on our part for survey participation, the growth of telemarketing, even when carried out legitimately, competes for the public's time. Also, changing lifestyles - such as the increase in single person households and dual-income households - have resulted in many people having less discretionary time and jealously guarding what they have. But I believe we have made inadequate efforts to respond effectively to these changes, as well as to those behaviors of ours that we know play a role in reduced cooperation.
For example, we probably can reduce refusals somewhat by doing a better job of drawing a potential respondent in through an introduction that better explains what the interview is about and something of its purpose; by allowing interviewers to be more flexible in their initial contact with a potential respondent, rather than slavishly following a prepared script; by making an adequate number of callbacks; more effort to schedule interviewing times that are convenient for respondents; and by a continual effort to control questionnaire length.
However, at least for those of us in the commercial world, it's all too easy when under the dual constraints of time and budget to just continue to do things as before rather than reflect on and experiment with procedures that might improve cooperation rates, resulting in better research, and therefore having a positive financial impact in the long run, resulting in better profits.
8. We don't pay enough attention to our interviewers. This wart follows from the problem of refusal rates. Mentioning this will not endear me to our chief financial officer or to anyone who sees research as a business first and a profession second - certainly not the view of the pioneers in our business - George Gallup, Arch Crossley, Elmo Roper, Alfred Politz. These were men of substance and integrity who, while engaged in selling their services, nevertheless gave top priority to quality research.
Back to the interviewer, though I couldn't resist that pertinent digression. Let's not forget that for most people it is the interviewer who is the only visible aspect of the research process. It is the interviewer who must gain the public's participation in our research projects, and respondent cooperation depends heavily on the initial impression made by the interviewer. Too many tend to see the interviewers at the bottom of the food chain. But it is their skills - or lack thereof - that determine the quality of the data gathered -and thus the value of the whole research effort.
Sitting back comfortably in our offices, it's easy to forget the interviewer - particularly since, as is too often the case today, researchers have never had the experience. They want to go from Harvard to vice president, skipping a few important and instructive steps in between.
I suggest that we had better raise our level of concern about how we recruit, train, motivate, supervise and remunerate those who collect data.
My last two warts are of a different nature, are related and essentially pertain to the world of commercial research.
9. We have a tendency in the commercial survey world to call ourselves "suppliers" or "vendors" and also to allow our clients to do so. I have been voicing my opinions on this wart for some time. The world is full of suppliers and vendors - you call them, order the product, they deliver it. One customer is no different from the next. Professionals, however, have special knowledge, special skills and thoughtfully and differentially apply them to the particular needs and problems of each individual client. We are professionals - and when we think of ourselves otherwise our self-esteem suffers, and when we refer to ourselves otherwise our external image suffers. When clients regard us as suppliers, they treat us as suppliers. And if you have been in this business for any length of time, you should have experienced the difference in the relationship with a client who regards you as a supplier and one who regards you as a professional.
10. A mentality to which we unfortunately contribute is the lowest bid mentality on the part of clients. Obtaining the best value for the money spent certainly makes sense, thus competitive bidding seems quite logical on the surface, particularly as clients' budgets are being tightened. As with most business practices, however, it can be abused so that the lowest bid is not the best value, and the bidding process takes place on the supplier rather than the professional level. While much of this problem rests with the client, we contribute to it - both passively and actively. A tempting reaction to competitive bidding is simply to declare that it is not a valid approach for the purchase of professional services and that it is demeaning to our professional status. While
there is truth in this posture, it ignores the realities of the marketplace. Nevertheless, the system can be fair and equitable - and we have responsibilities to insure, as best as possible, that it is so. For example, don't hesitate to press the client for as much specificity as possible, giving you as firm a basis as possible for costing and also causing the client to think through the proposed project early on, which is not always the case.
Also, ask how many bids are being requested and from whom and don't hesitate to refuse to respond to a request that was sent out wholesale. This can arise from three causes: (1) A client who does not know who is and who is not qualified to bid on a given project -so qualified firms find themselves in competition with unqualified firms - an unfair situation because, if the client did not know who was qualified in the first place, the resulting proposals cannot be evaluated properly. Going along with this reinforces client incompetence. (2) A client requests multiple proposals for what is nothing more than a brain-picking exercise - in effect, the gathering of much valuable information at no cost under the guise of seeking competitive bids. Going along with this reinforces unethical client behavior. (3) A client who is absolutely a price buyer figures the more requests sent out, the more responses received and the greater the chance for an unusually low bid. Going along with this reinforces client stupidity.
Given an appropriate bidding situation, our responsibility is to bid honestly. Corner-cutting that might be invisible to the client, but that lowers your bid, affects quality and does us all a disservice, as does submitting a bid known to be low in the hope of obtaining more money once the study is awarded and under way.
Quite simply, the fairness and efficacy of the competitive bidding system depends on the knowledge and, most important, the integrity of the participants. We can control our own behavior better than we do and therefore have a positive impact on the behavior of the client world.
So there you are - 10 warts to think about. Deliberately, I have not presented them in any order of priority or made any suggestion regarding their prevalence, and I may have engaged in some exaggeration to make a point. Suffice it to say I believe they are all serious and all occur - and we all, including myself and my company, could do more to prevent their occurrence. I strongly encourage all of us to do so.
NYAAPOR 152 Madison Avenue, Suite 801
New York, NY 10016
Tel: (212) 684-0542
Fax: (212) 481-3071 info@nyaapor.org
