Suppose you are the Chief Marketing Officer for a retailer that has data on the home addresses of its 1,000,000 most active customers. You hope to determine whether sending out "20% off your entire purchase" coupons by mail will increase revenues.

You conjecture that customers who have access to this coupon will spend more in the store over the next year. However, skeptics in your company argue that the coupons will just allow customers to spend less on items they would have purchased anyway. This is a debate that an experiment can resolve.

In thinking about how large of an experiment you need to have enough statistical power, you realize that many of the customers you send the coupons to in the mail will not open the mail and so will not realize they received the coupon.

a. The CEO argues that to estimate the effects of the coupons on revenue, you should compare the difference in revenues from a) people you sent the coupons and who used them and b) people you sent the coupons but who did not use them. Write a response to your CEO: describe the flaw with this plan in language the CEO will understand, and advocate for your proposed experiment. [5 points]

b. To avoid the cost of sending out coupons you do not need to, you ask the data science team to plan an experiment just large enough (with just enough statistical power) to reliably detect a treatment effect if the true effect on those who open the mail and realize they have the coupon is a $2 increase in revenue over the next year. The data science team tells you that an experiment with 100,000 people in the treatment group (leaving the remaining 900,000 in the control group) will be well-powered to detect an overall difference between the entire treatment and control groups of $2 in revenue over the next year. To send out the minimum number of coupons required while still having enough statistical power to detect a $2 effect of opening the mail, should you send out fewer, the same number of, or more coupons than 100,000? Explain the intuition behind your answer. [5 points]

c. Make up a hypothetical set of three pairs of potential outcomes for three individuals who open the mail (Compliers) and three pairs of potential outcomes for three individuals who do not open the mail (Never-takers) such that the true ATE (the true average treatment effect across all six individuals) is negative but the true CACE (the true average treatment effect but just for the three compliers) is positive. (Hint: the way to do this is to ensure there is a large negative effect among your three never-takers.) Each outcome should be "purchases over the next year, in dollars." Each person will have one such potential outcome called "if opened mail" and one called "if did not open mail." (Note that in the real world we would only ever observe the "if did not open mail" outcomes for the never- takers, regardless of whether they were in the treatment or control groups. However, we can still imagine what their outcomes might have been if they had in fact received mail and opened it. The following subquestions will ask you why this "imaginary" set of outcomes could still be important to consider.) [5 points]

d. Give a concrete example in the context of this experiment about why the true ATE could be negative (if everyone were successfully treated and so knew they had the coupon) but the true CACE (so, the effect of knowing they have the coupon on those who know) positive. In other words, why could there be a negative true effect for never-takers (were we able to force them to take the treatment) but a positive true effect for compliers? [4 points]

Note that the true ATE and observed ITT are different when there is non- compliance. When there is non-compliance, an experiment will allow you to estimate the ITT, which is "driven by" the CACE among the compliers. However, just like we can conceptualize the potential outcomes noted above for never- takers, we can also still conceptualize what the treatment effect might be for never-takers. This is true even though in an experiment with non-compliance, the never-takers never drive the ITT since they don't get treatment even when in the treatment group. The "true ATE" is the average treatment effect across the compliers and never-takers, which includes this more hypothetical effect we can conceptualize for the never-takers. That is, if there would be an effect for the never-takers if we had somehow managed to force the never-takers to get treatment, than the ITT you observe and the more theoretical ATE could be different.

e. Suppose you estimate a positive ITT and hence a positive CACE on revenue in this experiment. The CEO tells you that this means the store should cut prices for all 1,000,000 of the most active customers by 20%, not only with coupons but automatically at registers, because the experiment has shown that, in a random sample of these customers, lowering prices by 20% with the coupons has increased revenues. Assume all 1,000,000 of the most active customers would be aware of these proposed automatic discounts. Do you agree or disagree with the CEO's interpretation of the experiment? Why? [4 points]

f. You seek to convince your company that these coupons increase in-store sales. You can't really estimate the compliance rate in this experiment, because you don't know how many people opened the mail. But, based on an educated guess, you assume that 50% of people opened the mail and received the coupon. Based on this estimated compliance rate, you estimate that the effect of opening the coupon is a CACE of an average increase in spending in-store of $10. The head of the data science team argues that your interpretation is flawed, as their data indicates that only about 5% of people open their mail. The data science team head argues that this means the coupons were likely less effective in stimulating in-store purchases than you have argued. How would you respond? In particular, grant the data science team's alternative assumption of a 5% open rate, and describe whether you agree with their interpretation or not. [5 points]

g. Things go wrong in experiments. In this case, you learn from the marketing team that the wrong postage was put on 25,000 of the letters your mail vendor sent to 100,000 of the people you intended to send the coupons to. No coupons arrived for those 25,000 people.

You do know which 25,000 people in the treatment group got the improper postage and therefore didn't receive the coupons. However, you have been unable to figure out what determines whether individuals got the right postage or not, so can't figure out who in the control group would not have received the coupons had they been sent them. The data science team argues the experiment now must be re-done from scratch. How would you respond? Is there a way to recover the average effect of sending a letter with correct postage, even though only 75% of your treatment group was sent the letter with correct postage? How? (In answering this question, ignore the question of whether people opened the letters. We are now trying to estimate just the average effect of sending a letter with correct postage.) [4 points]

h. You later learn what determined whether individuals got the wrong postage: everyone whose last name started with the letter A or B got the wrong postage and therefore didn't get the coupons. Only individuals in the treatment group whose last names began with the rest of the alphabet, C-Z, therefore actually got the letters. (E.g., Joffrey Baratheon, whose last name starts with the letter B, wouldn't be sent the letter if he was in the treatment group; but Jaime Lannister, whose last name starts with L, would get it.) With this new knowledge, you're now able to determine who in the control group would not have been sent a letter, either -- the As and the Bs. Now that you know this, would you change the analysis strategy you proposed in part (g)? How? [4 points]