I want to start by telling you how pleased I am to be given this award. It is especially gratifying to be recognized by the people I have known well during most of the years of my professional life. I have worked with some of you. I have spoken before most of you. Even played with some of you. And I know this is hard to believe, I have even disagreed with some of you. And after all that, to be given this award, is very gratifying.
I want to start by thanking the people I worked most closely with all these years for helping me build a first class research organization at CBS News. What ever else may have happened over the years I am most proud to have worked in such an organization. For many years we had something I really liked. We had magnificent support from CBS starting with Frank Stanton, the CBS corporate president at the time and a former survey researcher, who set the tone for the quality he expected. And Dick Salant, who saw to it that we had not only the resources, but the clout to get good research on CBS's air and keep bad survey research off. Both Stanton and Salant had standards that they wanted us to live up to.
What we did all those years at CBS News was truly a team effort. We had more than our share of successes and recognition. We also had a good time doing it. In a few minutes I want to talk about what it takes to have a very good research organization.
But first and foremost, I want to start by thanking Murray Edelman. We started working together in 1964 at the Census Bureau. In 1978, after graduate school, Murray came out of an early retirement in Haight-Ashbury to join us full time at CBS. That was about the same time that Kathy Frankovic came along off the campus of the University of Vermont where she had been teaching. The first person I worked with at CBS was Jan Werner. He caused quite a stir when he wore his French Gendarme cape to corporate headquarters. There were other people running various parts of our research operation. Richie Silverman, who passed away much too young. Marty Plissner, who still calls me from his retirement in Washington to point out his latest polling disappointment. Dotty Lynch, who has a bigger political network than CBS.
Rob Farbman, and I shared an office after CBS along with the company Xerox machine. It was a small company. Very different than our days at CBS. And my partner in our current venture for CNN, Joe Lenski, who is always one step ahead of me, no matter what we are doing. There are many others who contributed to our work, and I will only get in trouble if I even try to start naming them. So to them I will just say thank you.
There is one person I consider a mentor. It is Joe Waksberg. Many of you don't know Joe. He is a member of AAPOR. He was Murray and my boss at the Census Bureau and he worked with us through every election we did together, often keeping us on the right track, even though our inclinations did not always agree with his - and sometimes he kept us from killing each other on election night. One of the best lessons I learned from Joe and others at the Census Bureau is that there can be significant progress and innovation when good people disagree. If you cannot defend your ideas in a hot debate with your peers you are probably on the wrong track.
Lastly, I want to thank the New York AAPOR Council for this honor.
I want to use my time tonight to talk about some areas of survey research that I have been thinking about for some time. For a variety of reasons each of the topics concerns me. I'm concerned because it affects the quality of the work we produce. I said I would say something more about what it takes to have a first class survey research operation. I will start there.
THE TEAM APPROACH
My conception of survey research is that it is an eclectic field. It takes many different skills to deal with the various activities necessary to produce first class survey research. You all know these different activities. It includes expertise from social scientists, statisticians, data processing experts, data collection specialists, administrators, subject matter experts and more. A survey requires everything from conceptualization of the problem, questionnaire writing, sampling and estimation, data processing, interviewing and supervision, statistical analysis, interpretation, report writing. Those are the highlights. Some of you may be saying, "I can do all those things." Personally, I doubt it.
As I see it, no one person I know possess all the various skills at a high enough level necessary to conduct a survey. It takes a team of people to encompass all the areas. Years ago I was offered a job at USIA. They told me that they used the team leader approach to research projects. A team leader was responsible for all the various activities that went into the conduct of a survey. I thought about the offer and turned it down. There were parts of the survey process that I did not feel competent to handle well. I still believe that the quality of survey research suffers when people with very strong skills in some areas attempt to take on other areas in a survey research program where their skills are not strong enough. The research also suffers when they minimize the importance of areas where they are not most skilled.
Let me put it another way: Every part of the survey process is equally important. Question writing and report writing are not more important than keying the data accurately or interviewing respondents. If you accept the statement that every part of the survey process is equally important then you have to be skeptical of a survey where some of the parts are done by people who are not fully equipped to perform a particular task in the survey process. What I am advocating is a survey organization where the tasks are divided among different specialists, rather than an organization that centralizes too many tasks in a single person. Survey research is a team activity.
SUBJECT MATTER EXPERTISE
Next I want to suggest to you that survey researchers do not have the subject matter expertise to adequately explore many social, economic, political or other subjects we research. Many of us can write good questions. What we do not have is the expertise to explore the various dimensions of a complex issue. I want to make a very strong distinction between being able to write questions and being an expert in the subject matter.
First of all, survey researchers do not usually know what all the dimensions of an issue are. Second, we do not know the nuance for each dimension that someone more expert in the subject matter might know. I always thought the polls we did for CBS and the New York Times were much better when reporters or producers who spent a lot of time covering a particular subject worked with us in shaping the questionnaire. It was not that we couldn't write the questions. We could. But it was much better once someone focused our thinking in the right direction.
In a recent survey I worked on I realized after the fact that I had not asked about a person's religion in the most meaningful way. I thought I knew how to ask for a person's religion. I have been doing it for years. Two sociologists, much more familiar with religious practice than I, spotted the problem immediately. Survey researchers need help with the subject matter. They are rarely expert enough on their own.
NON-RESPONSE
The most talked about problem in scientific survey research today is the low completion rate for surveys. In recent years there was an international conference dedicated to the problem. And while we still do not adequately understand the effect of low response on the quality of our survey data there are two things we can do to improve completion rates that I do not think are being done adequately.
Interviewing: The first thing we can do is a lot more research, training and supervision on the interviewing process. It seems to be an area that is systematically ignored, except by the people who work on field procedures. I cannot figure out why the rest of the research community takes so little interest in the caliber of the interviewing. Surely it deserves more attention, much more.
I was always struck by the completion rates the Census Bureau got for the Current Population Survey. They were about 95%. Also, when I looked at completion rates for the various interviewers working on the CBS/Times polls there were a couple of interviewers who regularly completed 90% of their attempted interviews. To this day, I don't know what it was about those interviewers that were responsible for their success. I know the Census Bureau had an intense interviewer-training program and close interviewer supervision.
The lesson seems clear. The quickest way to change the awful response rates we have been getting is to pay more attention to the interviewing process. Until we do we will have more conferences with more papers on the declining completion rates, but nothing in the way of improvements?
While I am at it, I want to make a short diversion. A low completion rate is not always a sign of unreliable data and a high completion rate is not necessarily a sign of high quality data. We need to develop independent measures that reflect the effect of the respondents missing from a survey. One way is to find correlates of key variables in a survey. The correlates will be useful for evaluating completion rates if there is reliable independent data available about them. At least that would provide a hint about whether the missing respondents were skewing the results of our survey. My only caution is that demographics are not necessarily good correlates of key survey data.
I remember a test we did years ago to see if there was a relationship between completion rate and survey error. This was in an exit poll. Exit polls are wonderful for this kind of testing. It is the only survey where one gets parameters shortly after the survey. We wanted to know the effect of questionnaire length on the completion rate. We also wanted to know if a precinct's completion rate was correlated with the error in the vote. We had three different length questionnaires. As expected, the shortest questionnaire had the highest completion rate and the longest one the lowest completion rate. The surprise turned out to be that there was a constant bias in the precincts that had the short questionnaire - the one with the best completion rate. In this case a high completion rate did not mean more reliable results.
Quality Control: I said there were two things that could be done to improve completion rates. This second suggestion is really much broader than just affecting response rates. It has to do with improving the quality of the survey. It is quality control.
The survey process is fraught with opportunities for error. Error in the training of interviewers. Error in the designation of the sample units. Error in the preparation of the data. Error in the coding of questions. Error in the tabulation or the estimation models. There are lots of places where it is possible to make errors. In my early days, an expenditure of approximately one-quarter of the research budget was spent on quality control. At least, that is what was being spent on large-scale surveys for the federal government. Is there anyone here whose organization allocates even 10 per cent of the survey budget to quality control? [Let the record show that no one responded.]
What I am trying to suggest is that a little more attention to some of the basics of conducting a survey could have a profound effect on completion rates.
REPORTING SAMPLING ERROR
I want to say a few words about reporting sampling error. A number of people who have spoken here have talked of not reporting sampling error because it was confusing all those dear mindless souls who listen to our results. They were concerned we would make people think that sampling error was the only error in the survey. I guess I am not too sympathetic with that point of view. I am pleased to say that I feel somewhat responsible for getting CBS and the New York Times to report sampling error as part of their poll stories, along with all the other disclosure information called for by the AAPOR Code. I am also pleased that many other news organizations report this information also.
Knowing the sampling error is necessary if one is going to know a meaningful change in some key statistic from an insignificant one. It also should mean that someone has conducted a probability sample at all the various stages of selection. Without probability sampling the concept of sampling error is irrelevant. All those quota methods within household and elsewhere in the midst of what are otherwise probability designs make the sampling error concept meaningless. Without some notion that unbiased probability survey estimates converge around a true value we could not take our 1,500 interviews and make general comments about the opinions of 200 million people.
To say that people will be misled into believing sampling error is the only error is disingenuous. To me those researchers who don't want to report the sampling error are saying that they don't understand the statistical methods underlying survey research. People who have no idea how to compute sampling error usually make this criticism about reporting sampling error.
The same people who make this criticism also seem to exhibit no understanding of the statistical theory underlying weighting of sample data. I have listened to more conversations that seem to imply weighting is like fine tuning an engine. You turn a screw here. You make an adjustment there. Pretty soon the demographics look just like the population. The right number of men and women, of blacks and whites, of young and old. To put it as plainly as I can, statistical theory is clearly something these people don't understand. They are startled to learn that this kind of "tinkering" can make their estimates of some key characteristics worse. They may be actually increasing the sampling error rather than reducing it.
Which brings us back to my first point. Survey research requires many skills. When I hear this comment about reporting sampling error all I think is they should work with a good statistician and they should listen to him or her, because it is not their skill.
Survey research has been very good to me. I have had wonderful opportunities, both in terms of whom I worked with and the things I worked on. I believe I worked on significant projects in my early days at the Minnesota Society for Crippled Children and Adults and the Census Bureau. CBS has given me enough joy for a lifetime. I would not trade a minute of it for something else.
Since then I have discovered the thrill of change through elections in emerging democracies in the Philippines, in Russia and in Mexico. After the election of Vicente Fox in Mexico in 2000, after the 70 years of one party rule, the streets in Mexico City were alive with celebration through the night. I don't remember anything to compare to it except the day when World War II ended. Contributing to that election opened a new chapter for me.
Changes along the lines of the things I talked about tonight I hope will make survey research even better in the future.
NYAAPOR 152 Madison Avenue, Suite 801
New York, NY 10016
Tel: (212) 684-0542
Fax: (212) 481-3071 info@nyaapor.org
