Can a Single Online Respondent Pool Offer a Truly Representative Sample?

Online research programs can often benefit by building samples from multiple respondent pools. Achieving a truly representative sample is a difficult process for many reasons. When drawing from a single source, even if researchers were to use various verification methods, demographic quotas, and other strategies to create a presumably representative sample, the selection methods them- selves create qualitative differencesor allow them to develop over time. The same is true of the parameters under which the online community or respondent pool was formed (subject matter mix, activities, interaction opportunities, etc.). Each online community content site is unique, and members and visitors choose to participate because of the individual experience their preferred site provides. As such, the differences between each site start to solidify as site members share more and more similar experiences and differences within the site's community decrease. (Think, birds of a feather flock together.)

As such, researchers cannot safely assume that any given online respondent pool offers an accurate probability sample of the adult U.S. or Internet population. Consequently, both intrinsic (personality traits, values, locus of control, etc.) and extrinsic (panel tenure, survey participation rates, etc.) differences will contribute variations to responsemeasure distribution across respondent pools. To control distribution of intrinsic characteristics in the sample while randomizing extrinsic characteristics as much as possible, researchers might need to use random selection from multiple respondent pools.

The GfK Research Center for Excellence in New York performed a study to see how the distribution of intrinsic and extrinsic individual differences varied between respondent pools. Respondents were drawn from five different online resource pools, each using a different method to obtain survey respondents. A latent class regression method separated the respondents into five underlying consumer classes according to their Internetusage driver profiles.

Researchers then tested which of the intrinsic characteristics tended to appear within the different classes. No variable appeared in more than three classes. Furthermore, the concentration of each class varied considerably across the five respondent pools from which samples were drawn.

Within the classes themselves, variations appeared in their demographic distributions. One of the five experienced a significant skew based on gender, and two other classes exhibited variable age concentrations, with one skewed toward younger respondents and the other toward older ones.

Overall, GfK's study revealed numerous variations across different respondent resource pools. As their research continues, current findings suggest that researchers must be aware of these trends, especially in choosing their member acquisition and retention strategies and in determining which and how many respondent pools to draw from.

1. If one respondent pool is not sufficient, how many do you think you would have to draw from to get a truly representative sample? Why do you think that?
2. When creating a sample, how would you propose accounting for the types of extrinsic characteristics mentioned?