Selective Reporting

Students often tell about predictions or premonitions (feelings that something bad will happen) coming true. Few can list any past premonitions that did not come true. The result is selective reporting : people seldom hear about the failed predictions, but predictions that come true are given lots of attention.

Students often tell me about the time when Granny woke up in the middle of the night, knowing some close relative had been injured, at the exact moment her granddaughter was breaking her leg 300 miles away (or whatever). Many families have such stories. There are at least three possible explanations for why such stories are so common.

What sort of story is common in many families?

1. A strange sort of information transfer takes place, perhaps involving quantum effects, causing a psychic reaction in the brain of a relative far away, or

2. People are having déjà vu sensations and confabulating, or

3. People are worrying and having premonitions all the time, and then they remember mostly the predictions that come true.

I lean toward explanation #3 with a bit of #2 mixed in. Here I would cite Dewey's Law of No Fun. It seems to me people want to believe in special psychic powers. That leads to a strong tendency to preserve and propagate stories of ESP. Little details creep in such as "Granny told everybody about it before it happened" which, if true, would be vitally important. But, lacking a time machine, there is no way of determining whether the details are true. Instead, our only data consists of retrospective self-reports, the weakest of all forms of scientific data.

What is "powerful magic" to prevent unusual premonitions from coming true?

To improve the quality of data and establish proof of an important premonition, I tell students to write down important premonitions along with a date, and file them away. Invariably (in my experience) unusual premonitions do not come true if they are written down. It is powerful magic for preventing undesirable events. But the magic only works on low-probability predictions, such as "sister will break her leg" or "my flight will crash."

The same magic does not work when guessing a baby's gender, or predicting when somebody will become pregnant. Those predictions sometimes come true. For example, one study showed that three "common myths" used to predict a baby's gender (the fetal heart rate test, the Chinese calendar test, and the Drano test) all made accurate predictions about half the time (Ostler & Sun, 1999).

What is the file drawer effect?

The problem of selective reporting is not limited to students reporting family legends; it is a major problem for researchers as well. An example is the file drawer effect. Results of experiments that come out "wrong" (do not meet expectations or produce negative results) tend to be forgotten. They are "left in a file drawer" and never published. On the other hand, results that confirm the bias of scientists or produce a positive effect in an experiment are more likely to be published. With time, this can easily create the impression of a powerful effect, because studies that fail to show the effect are never published.

How can the file drawer effect undermine the technique of meta-analysis?

File-drawer effects are one reason for skepticism about meta-analysis , a statistical technique that compares large numbers of studies on the same topic to determine the average effect. A meta-analysis assumes that positive and negative outcomes are equally likely to be published, but that is seldom the case. Critics have pointed out that all meta-analyses in psychology seem to produce positive results (supporting the existence of any hypothesized effect). That is probably because studies with negative results are seldom published, so when investigating an elusive and perhaps non-existent effect, the average effect found in a meta-analysis will be small but positive. Therefore meta-analysis is not appropriate for tiny or questionable effects such as tests of ESP It is more appropriate for estimating the average size of an effect that is well documented and known to exist.

What "literal" file drawer effect occurred in the laboratory of J.B. Rhine?

J.B. Rhine, the Duke ESP researcher, had a literal file drawer effect operating in his laboratory. Rhine believed strongly in ESP, and he was suspicious that subjects who produced very low scores on ESP tests were deliberately trying to frustrate him. Cromer (1993) reported that Rhine kept an envelope in his desk where he stored data sheets of subjects who produced "purposely low scores." Of course, subjects who produced high scores were counted! The net effect was to create an appearance of positive results over the long term.

What is "the single most powerful argument against the existence of ESP"? What is the "ganzfeld" experiment?

Perhaps the single most powerful argument against the existence of ESP is that no demonstrations of ESP have been repeated in independent laboratories by non-believers. As Padgett and Cody (1984) put it several decades ago, "The world is still waiting for a replicable demonstration of 'psi'."

Has there been any progress toward "proving ESP" since Padgett and Cody made their statement in 1984? Some psychologists believe that experiments using a ganzfeld provide evidence of ESP. In a ganzfeld experiment, one individual looks at a picture and tries to send it by telepathy to an individual surrounded by a blank, stimulus free environment (called a ganzfeld). So this is a form of remote viewing.

What would be an impressive confirmation of the ganzfeld effect?

Believers claim that a meta-analysis (combining the data from many ganzfeld experiments) shows a small but definite effect (Bem & Honorton, 1994). However, this could be the result of a file drawer effect, like any meta-analysis involving weak effects. A much more impressive demonstration would be replication of ganzfeld effect by experimenters who are non-believers, as suggested by Hansel. So far, non-believers have not replicated ganzfeld effects. No doubt other researchers will try to "nail down" this phenomenon, and time will tell whether it is genuine. As always, replication is the key to progress in science.


Write to Dr. Dewey at psywww@gmail.com.

Don't see what you need? Psych Web has over 1,000 pages, so it may be elsewhere on the site. Do a site-specific Google search using the box below.

Custom Search

Copyright © 2007 Russ Dewey