Self-Report Measures: Notoriously Unreliable

One particular type of operational definition is well known for its lack of reliability. This is the self-report measure. Self-report measures are operational definitions in which a person is asked to report his or her own behavior or mental contents. Self-report measures include questions like "How happy are you on a scale of 1 to 10?" or "Do you dream in color?" Most people will agree to answer such questions, but the results are not always dependable. In fact, self-report measures are notoriously inaccurate and unreliable .

What is wrong with self-report measures? What is the least accurate type?

The least accurate type of self-report measure is one that asks a person to look back in time and remember details of a behavior or experience: the retrospective self-report. This combines the uncertainty of self-report with the uncertainty of memory reconstruction. Even a person who is trying very hard to tell the truth cannot necessarily distinguish between a creative fabrication and a genuine memory. We will see evidence for this in at least three different places in this book: in the study of hypnosis, the study of memory, and the study of eyewitness testimony.

What did your author discover about retrospective self-reports of study time?

In graduate school, my professors emphasized the inaccuracy of retrospective self-report data. However, the truth did not come alive for me until I was doing research on student learning for my PhD thesis. In one study, I asked students to estimate (to the nearest half hour) the amount of time they spent studying the previous week's chapter. All my students agreed to answer this question. Nobody said "I don't remember" or "I cannot make that estimate." I assumed the information was accurate, and I prepared to use it in my research.

Then I lost some of the questionnaires. No problem: the students cheerfully filled out identical, duplicate questionnaires.

A day later I found the original questionnaires. I compared to two sets to see how the data compared. To my shock, the estimates of study time were completely different! A student who put "4 hours" on the first questionnaire might put "1 1/2 hours" on the second. These two answers came from the same student, asked about the same chapter (the previous week's assignment) on two successive days ! My operational definition of study time, based on the student's retrospective self-report, was clearly unreliable. Therefore it was not valid either; it was not measuring what I thought it measured.

What should one always do, if using self-report data?

If one must use self-report data, they should always be labeled as such. (The word "they" is used here because the word "data" is plural; "datum" is the singular form.) For example, if you ask people whether they are happy, you are gathering self-report data. The data might not be very accurate, so the researcher should caution readers by labeling this data as self-reported happiness. Indeed, this is a typical approach to the problem of measuring happiness, mentioned earlier. Researchers commonly use self-report data labeled as such.


Write to Dr. Dewey at psywww@gmail.com.

Don't see what you need? Psych Web has over 1,000 pages, so it may be elsewhere on the site. Do a site-specific Google search using the box below.

Custom Search

Copyright © 2007 Russ Dewey