You are an employee of a newspaper and are planning a randomized experiment to demonstrate to Apple that online advertising on your newspaper's website causes people to buy iPhones. Each site visitor is shown the ad campaign is exposed to $0.10 worth of advertising for iPhones -- that is, exposing a user to ads over the course of the week costs Apple $0.10 per person. You run a pretty popular newspaper, so there are 1,000,000 users available to be shown ads on your newspaper's website during the one-week advertising campaign. Apple truthfully indicates that they make a profit of $100 every time they sell an iPhone and that 0.5% of visitors to your newspaper's website buy an iPhone in a given week in general, in the absence of any advertising.

Question a) First, a simple economics question that is not about statistics. By how much does the ad campaign need to increase the probability of purchase in order to be "just worth it" and a break-even ROI? Define ROI as [expected additional profit made as a result of advertising to each person - cost of advertising to each person]/cost of advertising to each person X 100 %. "Break-even" ROI would mean 0% ROI.

Question b) You expect the true effect of your ads is an increase in iPhone sales of 0.4 percentage points. (For example, an increase from 0.5% conversion (the baseline) to 0.8% conversion is 0.8 - 0.5 = 0.3 percentage points.) If your expectation is correct, what would be Apple's ROI on the ads?

Question c) Again, assume the true effect is an increase in sales of 0.4 percentage points. You want to figure out whether this experiment will yield an answer that is precise enough that you'll be able to make Apple abandon their skepticism that the ads are not ROI positive. To simplify things, suppose you expect your experiment will give you exactly the right answer, of 0.4 percentage points. If users are split 50:50 between the treatment group (exposed to iPhone ads) and control group (exposed to unrelated advertising or nothing; something you can assume has no effect on iPhone sales), what will be the confidence interval of your estimate on whether people purchase the phone? Express this confidence interval in percentage point terms -- that is, what the confidence interval will be on the percentage point increase in the probability of purchasing an iPhone. (For example, an increase from 0.5% to 0.8% is 0.8 - 0.5 = 0.3 percentage points.) [8 points] Note: The standard error for a two-sample proportion test is (1 )( 1 1 + 1 2 ) where = 1+2 1+2 , where x and n refer to the number of "successes" (here, the total number of purchases) over the number of "trials" (here, the total number of site visits). Here, 1 and 2 would refer to the treatment and the control groups (order is irrelevant). You can think of p as the average proportion of site visitors who make a purchase in the entire sample, averaging across treatment and control. The length of each side of a 95% confidence interval is calculated by multiplying the standard error by 1.96. We would suggest using R as a calculator to make the math easier. Note: sqrt() is the function in R to compute a square root.

Question d) Take the confidence interval you found in part (c) and express it in ROI terms.

Question e) Is this confidence interval precise enough that you would recommend running this experiment? Why or why not?

Question f) In experiments, we don't need to randomly assign people to treatment and control groups 50/50. We can also randomly assign more people to one group than another, as long as to which group each person assigned remains random. Exploiting this fact, the Chairperson of Board of Directors at the newspaper, worried about potential loss of revenue by not advertising to the control group, says she is not willing to hold back a control group any larger than 0.2% of users and says you should run the experiment with a 0.2% control group (that is, with only 1,000,000 * 0.002 = 2,000 people in the control group). In ROI terms, what would be the confidence interval for this same experiment if only 0.2% of users were placed in the control group and the remaining 99.8% in the treatment group? Given this, would you recommend running this experiment? Why or why not?