July, 1994 / August, 1994

The Polling Business

The Holocaust Poll Error: A Modern Cautionary Tale

By Everett Carll Ladd 

The Public Perspective Vol. 5, No. 5; Pg. 3

"Does it seem possible, or does it seem impossible to you that the Nazi extermination of the Jews never happened?"

This asking of a terribly confusing version of an important question about a major, tragic historical occurrence triggered a series of responses which tell a story important to all consumers of poll data. The above question was posed in a November 1992 survey done by the Roper Organization for the American Jewish Committee (AJC).

The initial error was innocent and accidental. A badly framed question slipped past the review process, was posed in the field, and having confused a significant number of people who responded to it, yielded entirely misleading results. Yet, more than a year passed, following the release of the survey data in April 1993, before a full correction was brought before the public. In the meantime, an entirely incorrect account -- suggesting that a large segment of the population entertains doubts as to whether the Holocaust actually occurred, with the appalling implication such denial would carry -- circulated widely. The whole story of this polling fiasco can now be told.

An Incorrect Conclusion Was Drawn

The "Double negative" construction in the question shown above -- is it "impossible" the Holocast "never happened" -- confused many respondents. Prior polling experience had suggested strongly that it would. [1] Yet when the results were released -- showing 22% taking the "possible that it never happened" position, and another 12% saying "don't know" -- there was an uproar. Could the anti-Semitic clap-trap of Hoalocaust denial actually be finding at least tentative approval in as much as a third of the US population?

Surveys researchers were distressed by the attention being given to answers to an obviously flawed question. [2] The Gallup Organization, not party to the original error, set about to collect pertinent new data. [3] Gallup posed the question as Roper had to half the sample, and posed a far less confusing version to the other half. Both halves were then asked an additional question designed to examine the level of doubt. This Gallup experiment showed the number of doubters to be small -- one fifth or less of the proportion that the flawed question had suggested. (See the chart on p. 4.)

Might the Holocaust Have Never Happened?

Asked the Wrong Way: "The term Holocaust usually refers to the killing of millions of Jews in Nazi death camps during World War II. Does it seem possible, or does it seem impossible to you that the Nazi extermination of the Jews never happened?"

 
The Gallup Experiment, Asked Two Alternative Right ways:

"The term Holocaust usually refers to the killing of millions of Jews in Nazi death camps during World War II. Do you doubt that the Holocaust actually happened, or not?"

"Just to clarify, in your opinion, did the holocaust definitely happen, probably happen, probably not happen, or definitely not happen?"

Note: The Gallup Organization conducted an experiment to explore what it thought was the confusing, and hence distorting nature of the Roper question. For half of the sample, Gallup posed an item which began: "The term Holocaust usually refers to the killing of millions of Jews in Nazi death camps during World War II," and then they asked the "double negative" version just as Roper had (wording on page 3). As in the first Roper survey, this muddled effort found a large proportion (about a third) seemingly not prepared to say the Holocaust was certain historical fact. But the other half of the sample was asked: "The term Holocaust usually refers to the killing of millions of Jews in Nazi death camps during World War II. Do you doubt that the Holocaust actually happened, or not?" Whereas only two-thirds had said the Holocaust was a certain event by the first asking, nearly nine in ten called it certain in this latter version (see above). Then, both halves of the sample were asked: "Just to clarify, in your opinion, did the holocaust definitely happen, probably happen, probably not happen, or definitely not happen?" Ninety-six percent of all respondents showed up in the "it happened" camp in this follow-up version.
Source: Survey by the Gallup Organization, January 15-17, 1994.

The Roper Organization had itself been active in corrective survey work. It did further polling of a preliminary sort using split samples, comparing responses to the flawed version to those of an improved one. Finally, agreement was reached with the AJC for a full-scale redoing of the study. Field work was completed last March. All of the questions in the original survey were repeated except for the offending item. It was replaced by a better version: "Does it seem possible to you that the Nazi extermination of the Jews never happened, or do you feel certain that it happened?" Now only 1% said it seemed possible that the Holocaust had never taken place! (See the chart on p. 3.)

Might the Holocaust Have Never Happened?

Asked the Wrong Way
November 1992
 
Question: "Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?"

The Roper Correction: 
Asked a Right Way
March 1994
 
Question: "Does it seem possible to you that the Nazi extermination of the Jews never happened, or do you feel certain that it happened?"

Source: Surveys by the Roper Organization.

It could be argued that between the first survey in November 1992 and the second Roper asking of a far better version in March 1994, that the reality of the Holocaust might actually have been drilled home to many more people. The motion picture Scindler's List had played to large audiences and had been nominated for the motion picture academy's best picture award. The Holocaust Museum had opened on April 26, 1993 in Washington, DC. In fact, though, the 1994 Roper survey found a level of historical knowledge bearing on the Holocaust virtually identical to that found in the 1992 study. The only 1994 question that got a response at variance with that of the 1992 poll was the one replacing the obviously flawed question on whether it was possible or impossible that the Holocaust never happened. (See the chart on p. 5.)

On Every Other Holocaust Question, the 1994 Roper Poll Got the Same distributions As Did the 1992 Poll

Shown here: Percentages answering four knowledge questions correctly
 
Question: "Who was the leader of Nazi Germany?"
Percent answering "Hitler"

 
Question: "In which country did the Nazis first come to power?"
Percent answering "Germany"

 
Question: "From what you know or have heard, what were Auschwitz, Dachau, and Treblinka?"
Percent answering "Concentration camps"

 
Question: "Many Jews in Europe were forced to wear a symbol on their clothes during the Second World War. What was it?"
Percent answering "Yellow star"

Anatomy of the Error

Burns W. Roper was deeply disturbed by his firm's error, from the moment it was brought to his attention at the time of the April 1993 release of the data. He told his peers at a spring 1994 meeting of the American Association for Public Opinion Research (AAPOR) what he had conveyed to the AJC nearly a year earlier. "We should never have approved the question, and we certainly never should have written it," Roper said. "This is not the note on which I wanted to conclude my 48-year career in the opinion research field," he went on, saying that the confused wording had produced results which "served to misinform the public, to scare the Jewish community needlessly, and to give aid and comfort to the neo-Nazis who have a commitment to Hoalocaust denial.[4]

If the original question was quickly seen by virtually the entire survey research community as badly flawed, and if the survey organization which made the mistake sought vigorously to provide the necessary correction, why was a full correction so long in coming? Unfortunately, the client resisted a vigorous public repudiation of the initial work and dragged its feet on new polling.

The AJC's director of research, David Singer, is quoted in The New York Times of May 20, 1994 as saying he was "chagrined" when he learned of the problems with the first Roper effort, and that data from the new (March 1994) survey had to be carefully reviewed because "Roper by its own admission provided us with flawed data and created problems the first time around." If Mr. Singer were in fact embarrassed by deficiencies of the first Roper question, why did it take him more than a year to speak publicly about the problem? Why did the AJC go so far as to urge Roper not to appear at the AAPOR session held a full year after the offending question was identified?

The AJC has commissioned Tom Smith of the National Opinion Research Center (and a contributing editor to Public Perspective magazine), to examine all the Roper data and other relevant survey findings. The report is scheduled for release in July. Smith's new efforts are welcome. Research that he has already done shows, however, that levels of anti-Semitism -- of which Holocaust denial would be a part -- are in decline in the US, not increasing. [5]

While further research relating to the impact of the double negative question is appropriate, it must be stated that no one is the survey research community views the original Roper question as anything other the confusing. The question has no defenders.

The lesson of the flawed Holocaust poll is an important one, reaching far beyond one badly framed question. We are reminded that survey results are often not viewed neutrally by individuals and organizations which sponsor or interpret the work. People care about the results. The "interested" character of findings creates a constant potential for survey distortions.

The most deeply troubling feature of the Holocaust poll misstep is that even when the survey organization involved fully recognized its error, felt terrible about it, and actively sought a remedy, it was 14 months (April 1993-June 1994) before a proper acknowledgement of the error was brought to the American people.

[1] Roper pointed this out to Public Perspective in June 1993. He cited examples from a March 1984 Annuals article, showing the enormous confusion that "double negatives" had produced in prior polling. See the Public Perspective, July/August 1993, p. 33.

[2] It should be noted that other questions in this Roper/AJC survey were well done and provided useful information on the extent of the public's knowledge about the Holocaust, not only in the United States but in Great Britain as well, where the same questions were posed. We published a summary of the historical knowledge data in the July/August 1993 issue of Public Perspective, pp. 31-34. On the whole, the Roper/AJC survey showed a public less knowledgeable than one might like it to be about the Holocaust as a major historical experience, but not in any sense less attentive to it than to other large historical events.

[3] David W. Moore and Frank Newport, "Misreading the Public: The Case of the Holocaust Poll," The Public Perspective, March/April 1994, pp. 28-29.

[4] John Kifner, "Pollster Finds Error on Holocaust Doubts," The New York Times, May 20, 1994.

[5] See Richard Morin, "Study Cites Drop in Anti-Semitism," The Washington Post, June 21, 1994.

E-mail Mr. Callahan at stat110@edcallahan.com with questions or comments about this web site or about the class itself.