The problem in our initial discussion of A-grade insurance markets was that adverse selection led to non-randomness in the insurance pool: Although almost everyone was willing to pay the insurance premium that would have made zero expected profit for insurance companies with a randomly selected insurance pool, no one was willing to pay as higher cost students adversely selected into the pool. This kind of non-random selection is, however, not confined to insurance markets but lies at the heart of much that we see around us.2 (Both part A and part B of this exercise can be done without having done section B in the chapter.)
A. Consider the following examples and describe the non-random selection that can cause observers to reach the wrong conclusion just as insurance companies would charge the “wrong” premiums if they did not take into account the effect of non-random selection.
(a) Suppose I want to know the average weight of fish in a lake. So you take out a boat and fish with a net that has 1-inch holes. You fish all day, weight the fish, take the average and report back to me.
(b) A TV report tells us the following: A recent study revealed that people who eat broccoli twice a week live an average of 6 years longer than people who do not. The reporter concludes that eating broccoli increases live expectancy.
(c) A cigarette company commissions a study on the impact of smoking on fitness. To compare the average fitness of smokers to that of non-smokers, they recruit smokers and non-smokers at a fitness center. In particular, they recruit smokers from the aerobics program and they recruit non-smokers from a weight-loss class. They find the “surprising result” that that smokers are more fit than non-smokers?
(d) “Four out of five dentists” recommend a particular toothbrush — from a sample of dentists that are provided free dental products by the company that makes the toothbrushes.
(e) When surveyed after 1 year of buying and using a facial cream, 95% of women attest to its effectiveness at making their skin look younger.
(f) Children in private schools perform better than children in public schools. Thus, concludes an observer, private schools are better than public schools. (Careful: The selection bias may go in either direction!)
(g) A study compares the test scores of children from high income and low income households and demonstrates that children from high income households score significantly higher than children from low income households. An observer concludes that we can narrow this test score gap by re-distributing income from high income families to low income families.
B. It is often said that the “gold standard” of social science research is to have a randomized experiment where some subjects are assigned to the “treatment” group while others are randomly assigned to the “control” group. Here is an example: A school voucher program is limited to 1,000 voucher participants — but 2,000 families apply, with each having their child tested on a standard exam. The administrators of the program then randomly select 1,000 families that get the voucher — or the “treatment” — and treat the remaining 1,000 families as the “control” group. One year later, they test the children again and compare the change in average test scores of children from the two groups. They find that those who were randomly assigned to the “treatment” group have, on average, significantly higher test scores.
(a) Suppose that all 1,000 children in each group participated in the testing that led to the computation of average score changes for each group. Would you be comfortable concluding that it was likely that access to the voucher program caused an increase in student performance?
(b) Suppose that only 800 students in each group participated in the testing at the end of the first year of the program, but they were randomly selected within each group. Would your answer to (a) change if only the average change in test scores for these students were used?
(c) Suppose that families had a choice in terms of whether to participate in the testing at the end of the year. But families in the “treatment” group were told that the only way they can continue using the voucher for another year is to have their child tested; and families in the “control” group were told that some new slots in the voucher program would open up (because some of the voucher families have dropped out of the program) but the only way the families in the “control” group get another chance to be picked to receive a voucher is to have their child tested. In the “treatment” group, who do you think is more likely to self-select to have their child tested: Families that had a good experience with their voucher, or families that had a bad experience?
(d) In the “control” group, who do you think is more likely to self-select to have their child tested: Families that had a good experience the previous year outside the voucher program, or families that had a bad experience?
(e) Suppose again that 800 students from each group participated in the testing—but now you know about the incentives that families have for showing up to have their child tested. How does this affect your answer to (b)?
(f) From a researcher’s perspective, how can the non-random selection into testing be described as “adverse” selection that clouds what you can conclude from looking at average test score differences between the two groups? How is this example similar to part A(c)?