Non-Random Samples

In many cases, the sampling of people for a survey is not random. Consider a simple telephone survey. In order to be included in a telephone poll, people must have the following characteristics:

What are some characteristics of the "accessible population" in a telephone poll? How can this be a problem?

1. They must own phones.

2. They must have a listed number.

3. They must be home when called.

4. They must speak a language the caller can understand.

5. They must cooperate and answer questions.

The group actually contributing data is called the accessible population. The accessible population may be quite different from the target population, which might be "all citizens" or "all consumers."

To the extent an accessible population differs from a target population, a poll or survey may be misleading. This effect is completely independent of so-called "sampling error" and adds to it. Sometimes it is called nonsampling error to distinguish it from sampling error (Converse & Traugott, 1986), although it involves errors in the sampling process. A better name might be "errors due to non-random samples and other biasing effects."

Consider the type of poll where TV viewers vote by calling a 900 number. In the U.S., a 900 number costs money. Viewers may be asked to call one number to vote "yes," another to vote "no." This type of poll is meaningless for describing a target population, because the accessible population is biased by so many uncontrollable factors. The accessible population must consist of people who are...

1. Watching a particular channel & program

2. Watching it live, not tape-delayed or digitally delayed

3. Motivated enough to pay the $.50 or $1.00 it takes to make a 900 call

4. Close to a long-distance phone

5. Able to jot down the proper 900 number (usually two are used)

6. Able to get through on the telephone line

What is the problem with polls which invite viewers to vote by calling a telephone number?

Somebody could treat the results of a 900 number poll as a random sample (erroneously!) and calculate the expected sampling error. One might conclude that such a poll was accurate "to within 5 percentage points." But what population is described by the poll? What population would this sample represent? The population of "all people who meet the above criteria." They are not a random sample of any known population. Therefore one cannot assume the results of a 900 poll represent any group such as "all citizens" or "music fans" or even "all viewers of the program." The meaningfulness of such a poll is wrecked by the poorly defined non-random sample. Television networks like CNN acknowledge this fact by noting briefly that the results of their call-in polls are "not scientific"...but how many viewers know that "not scientific" in this case means "totally meaningless and very probably misleading"?

That does not matter to TV networks who that phone-in polls, because the polls make money. They purpose of the poll is not to obtain an accurate sample of some target population, it is to excite viewers and generate revenue. For that purpose, the polls are "reliable" even if they are not valid!


Write to Dr. Dewey at psywww@gmail.com.

Don't see what you need? Psych Web has over 1,000 pages, so it may be elsewhere on the site. Do a site-specific Google search using the box below.

Custom Search

Copyright © 2007 Russ Dewey