Biased Samples in Questionnaire Research

Students commonly equate lack of bias with lack of intention to cheat. But the concepts are not the same. Lack of bias means that everybody in the target population has an equal opportunity to be sampled. Even a sincere, honest, person who has no intention of cheating can produce a biased sample by using a non-random sampling process.

Suppose you want to take a poll of your fellow students to find out what they think of some hot political issue. You cannot ask every single student for his or her opinion, so you decide to ask an unbiased sample of the student body for their opinions. Here are examples of the right way and wrong way to do this.

What is the right way and wrong way to obtain an unbiased sample?

Right way : Start with a list of every student in the school, then use a random numbers table from the back of a statistics textbooks to pick out 50 names. Try to get in touch with all 50 students, and collect data from each person.

Wrong way : Set up a table outside the student union and ask 50 students at random to fill out your questionnaire.

Why is the second method wrong? If you might select any student who walks by, you are not introducing any bias into the sample, right? Wrong! Such a sample would be far from random. If you set up a table outside the union every day at noon, your sample will be biased toward students who (1) walk by the union, (2) do not have a class at noon, and (3) are willing to stop and take the time to fill out a questionnaire or answer questions. This excludes all the students who are taking classes at some other location on campus, have a class at noon, or are too busy or hungry to stop and fill out a questionnaire.

How could the sample be biased toward people who like karate? What is this supposed to illustrate?

A sample obtained in this way is not a random sample of the student population. It is a biased sample, and there is no way to figure out all the possible ways it might be biased. Perhaps on that day there was a karate demonstration going on near the union, so the sample is biased toward people who like karate. There are millions of possibilities. Any time a sampling process is not truly random , it is subject to all sorts of biasing influences, including those that might not come easily to mind.

What is a "convenience" sample, and what might be a better label?

A non-random sample is sometimes called a convenience sample to indicate that only conveniently accessible subjects were interviewed or measured. That does not make it any more acceptable from a scientific perspective, although it alerts the educated consumer that the results of the research represent an unknown group. Probably the term "garbage sample" or at least "non-random sample" would be more informative than "convenience sample."

A large N helps the accuracy of a poll when a sample is truly a random subset of a target population. But it does no good if the sampling process is biased. You could administer a questionnaire to 1,000 students who walk by the Administration building or the union or the cafeteria, and the results would not tell you any more than if you asked 50, because there is no way to know what larger group this sample represents. The "convenience" sample differs from the target population (that one is trying to describe) in multiple, unknown ways. It represents itself.

For journalism, a sample that "represents itself" may be enough. A reporter might say, "We interviewed 5 students walking by the Adminstration building, and they all complained about Coach X." The target population is implicitly the whole student body, and the "story" is that students are disappointed in the coach. Nobody mistakes this for scientific research. But if you were asked to provide a scientific poll of public opinion, you would have to define the target population carefully and arrange for a random way to sample from the target population.

What is the only way to get a random or unbiased sample?

The bottom line is this: the only way to get a random sample is to use some mechanism to insure that every member of the population under study has an equal chance to be selected.


Write to Dr. Dewey at psywww@gmail.com.

Don't see what you need? Psych Web has over 1,000 pages, so it may be elsewhere on the site. Do a site-specific Google search using the box below.

Custom Search

Copyright © 2007 Russ Dewey