Questionnaires, Surveys, and Polls

One of the easiest forms of data collection is administering a questionnaire. The underlying process is very similar to taking a poll. Professionals usually conduct surveys and polls, while amateurs often administer questionnaires. Together, these forms of data collection are called survey research.

Questionnaire research is one of the most common types of research conducted by undergraduates. Business majors, sociology majors, political science majors, psychology majors, and many others might be asked to design and administer a questionnaire, at some point in their careers.

Critical thinking is especially necessary when dealing with questionnaire research. Questionnaires seem so simple, yet questionnaire research is potentially very weak. It relies on self-reports. The questions are often "made up" and never subjected to the type of validity testing used with standardized tests sold commercially.

Why is questionnaire research "potentially very weak"?

A questionnaire is always intended to tell researchers about some group, which we can call the target population. If the target population is small (for example, students in a classroom) you can give the questionnaire to every person in the target population.

In that case, you do not need to worry about whether your sample (the group contributing data) represents the target population. The sampled population and target population are the same.

However, if the target population is large–for example, if you wish to learn about the opinions of all the students at a large college–then you are probably limited to collecting data from a smaller sample of that population. Laws of probability insure that if you take a random sample, the sample will represent the larger population with a known level of accuracy.

In other words, the sample may not give you results that are perfectly accurate. But if it is a random sample of the target population, you can use statistics to indicate its level of precision.

The Importance of a Large N

The letter "N" is used to indicate the number of subjects contributing data to an experiment or opinion poll. N=500 means 500 subjects were used, or 500 people were polled. This number has a powerful influence on the precision of the results.

If you take a small sample, even a small random sample, you can get very unusual and misleading results. For example, if you drew the names of three students at random from a college registration list, you might come up with three students who were over six feet tall, by chance.

However, if you used a random sample of 100 students, your estimate would be more accurate. The average height of students in the sample would closely resemble the average height of the entire student body.

What is the possible effect of taking a small sample?

Why is it possible to say, with confidence, a sample of 100 will have an average height close to that of the entire student body? This is due to the so-called "Law of Large Numbers."

The larger the N, the more closely a random sample will approximate its parent population. Random variations go every which way. The larger the N, the more likely it is that variations will cancel each other out and leave you with an accurate average value.

Students sometimes ask how large an N is required before the results of a poll become trustworthy. The answer can be calculated. It depends on the level of precision desired and the amount of variation in the characteristic being measured.

A national opinion poll using a random sample of a 500 people should produce results that are very accurate, providing the sample is truly unbiased. A smaller sample of 25 or so may give representative results if there is not a great deal of underlying variation. Polls based on 5 or 10 people seldom tell you anything except the opinions of 5 or 10 people.

Biased Samples in Questionnaire Research

Students commonly equate lack of bias with lack of intention to cheat. But the concepts are not the same.

Bias means that not everybody in the target population had an equal opportunity to be sampled. Even a sincere, honest, person who has no intention of cheating can produce a biased sample by using a non-random sampling process.

Suppose you want to take a poll of your fellow students to find out what they think of some hot political issue. You cannot ask every single student for his or her opinion, so you decide to ask an unbiased sample of the student body for their opinions. Here are examples of the right way and wrong way to do this.

Right way: Start with a list of every student in the school, then use a random numbers table to pick out 50 names. Try to get in touch with all 50 students, so you collect data from each person.

Wrong way: Set up a table outside the student union and ask 50 students at random to fill out your questionnaire.

What is the right way and wrong way to obtain an unbiased sample?

Why is the second method wrong? If you might select any student who walks by, you are not introducing any bias into the sample, right? Wrong!

Such a sample would be far from random. If you set up a table outside the union every day at noon, your sample will be biased toward students who (1) walk by the union, (2) do not have a class at noon, and (3) are willing to stop and take the time to fill out a questionnaire or answer questions.

This excludes all the students who are taking classes at some other location on campus, have a class at noon, or are too busy or hungry to stop and fill out a questionnaire.

A sample obtained in this way is not a random sample of the student population. It is a biased sample, and there is no way to figure out all the possible ways it might be biased. Perhaps on that day there was a karate demonstration going on near the union, so the sample is biased toward people who like karate.

How could a sample be biased toward people who like karate? What is this supposed to illustrate?

There are millions of possibilities. Any time a sampling process is not truly random, it is subject to all sorts of biasing influences, including some you might never imagine. If you did not happen to know about the nearby karate demonstration, you might never realize your opportunistic sampling process was biased in such a peculiar way.

A non-random sample is sometimes called a convenience sample. This means only conveniently accessible subjects were interviewed or measured.

A convenience sample is not acceptable from a scientific perspective. However, that label can alert an informed consumer not to take the results too seriously.

What is a "convenience" sample?

A large N helps the accuracy of a poll when a sample is truly a random subset of a target population. But it does no good if the sampling process is biased.

You could administer a questionnaire to 1,000 students who walk by the Administration building or the union or the cafeteria. The results would not tell you any more than if you asked 50, because there is no way to know what larger group this sample represents.

The convenience sample differs from the target population (that one is trying to describe) in multiple, unknown ways. It represents no larger population; it represents itself.

For journalism, a sample that represents itself may be enough. A reporter might say, "We interviewed 5 students walking by the Administration building, and they all complained about Coach X." The story is that many students are disappointed in the coach.

Nobody mistakes this for scientific research. However, if you were asked to provide a scientific poll of public opinion, and you wanted accurate results, you would have to define the target population carefully. Then you would have to arrange to obtain (as nearly as possible) a random sample from the target population.

What is the only way to get a truly random or unbiased sample?

The bottom line is this: the only way to get a truly random sample is to use some procedure that insures every member of the target population equally likely to be sampled. Then the sample is random.

Sampling Error

You have probably seen sampling error reported in poll results. For example, a poll might show that candidate A has the support of 43% of the population "plus or minus 3%." The plus or minus 3% is the sampling error, often called the margin of error.

A poll result of "43% plus or minus 3%" usually means, "With 90% likelihood the true figure for the entire population is between 40% and 46%." That is based on the assumption of a truly random sample.

What is the "margin of error"? How is it usually expressed? What is random sampling?

The words truly random sample have a specific meaning. Every member of the target population had an equal likelihood of being sampled. That is a high bar to reach.

An example of a truly random sample is the selection of a lottery ticket drawn from a rotating barrel. The barrel mixes up all the tickets.

The result is that everybody who enters has an equal chance of winning, as long as each was allowed only one ticket. Another way of expressing this is to say the process is unbiased.

What is a truly random sample?

Any biasing effect can throw off the accuracy of a poll. If the lottery company gave away lots of tickets and somebody entered ten times, obviously have ten times more likelihood of winning than somebody who entered once.

Such effects are not expressed in the plus-or-minus margin of error. The plus-or-minus statistic assumes perfect random sampling of the target population, which is seldom found in polling or questionnaire research.

Non-Random Samples

In most cases, the sampling of people for a survey cannot be random. The target population may be well specified, like "all citizens," but there is no practical way to get a random sample of that population.

Any non-random sampling procedure (any procedure that does not give every member of the target population an equal chance to be sampled) introduces bias into the results. For example, to be included in a telephone poll, people must have the following characteristics:

1. They must own a phone.

2. They must have a listed number.

3. They must answer a call from somebody they do not recognize.

4. They must speak a language the poll-taker can understand and use.

5. They must cooperate and answer questions.

The group actually contributing data is called the accessible population. The accessible population may be quite different from the target population, whether it is "all citizens" or "all consumers" or "all students at this school."

What are some characteristics of the "accessible population" in a telephone poll? How can this be a problem?

To the extent an accessible population differs from a target population, a poll or survey may be misleading. This effect is completely independent of so-called "sampling error" and adds to it.

Sometimes errors due to biased samples are called nonsampling error, to distinguish them from the concept of sampling error, but that is potentially confusing (because a "nonsampling" error, defined this way, does involve a biased sample). I prefer the label my professors used: "sample/target differences." Those are differences between the sampled population and the target population.

What is nonsampling error, and what label might be better?

Consider the type of poll where TV viewers vote by calling a 900 number or sending a text message. In the U.S., a 900 number costs money, and texting a vote also costs a small amount.

Viewers may be asked to call one number to vote "yes," another to vote "no." This type of poll is meaningless for describing a target population, because the accessible (sampled) population is influenced by so many uncontrollable factors. The accessible population must consist of people who are...

1. Watching a particular channel or program

2. Watching it live, not delayed

3. Motivated enough to pay the $.50 or $1.00 or whatever the charge is for calling or texting

What is the problem with polls which invite viewers to vote by calling a telephone number?

Somebody could treat the results of a 900 number poll as a random sample (erroneously!) and calculate the expected sampling error. One might conclude that such a poll was accurate "to within 5 percentage points."

But what population is described by the poll? What population would this sample represent?

The sampled population is "all people who meet the above criteria." They are not a random sample of any known population. Therefore a computation of sampling error is inappropriate and meaningless.

One cannot assume the results of a 900 number or texting poll represents any group at all. The group responding to such a poll represents only itself.

That might be enough for the company doing the call-in poll; after all, they are making money on it. They might present the results with a brief note that call-in polls are "not scientific."

They could use stronger language. They could say, "the results of this poll are totally meaningless for representing any larger group, but they are the people who paid to make their opinions known."

Prev page | Page top | Chapter Contents | Next page

Write to Dr. Dewey at psywww@gmail.com.

Don't see what you need? Psych Web has over 1,000 pages, so it may be elsewhere on the site. Do a site-specific Google search using the box below.