RestaurantGrades (RG) is a restaurant review platform (similar to Yelp or TripAdvisor) with an impressive stock of online reviews written by ordinary restaurant-goers. RG has compelling data demonstrating that these reviews have an economically important influence on the restaurant choices people make, which is exciting for the executive team and has caught a lot of attention within the industry. However, doubts have been raised about the efficacy of their main source of revenue - selling ads to restaurants. To better understand this issue, they have run a randomized controlled trial with a control group and two treatment groups: one treatment to test the impact of their current ads on restaurant sales, and the other treatment to test the impact of an alternative ad design that they are considering switching to. They have given you the attached data, and asked you to help them interpret and act on the results. In particular, they want to understand whether their ads really work, and whether they should stick with their current design or switch to the alternative design. Background on Advertisements On RG, each restaurant has a profile page with operational information including its hours, phone number, and location, where RG users who have visited the restaurant can leave reviews for other users. Users can also discover and search for restaurants on the platform using filters, and make reservations and order food through a restaurant's profile page. The majority of RG's revenues stem from selling ads through its sales team, which cold-calls restaurants to try to convince them to advertise on the platform. Advertisements, labeled as sponsored search results, are placed in a separate section above the organic results for searches that users conduct. Packages of advertisements are purchased for about $300 per month, and advertisers are required to sign up for a one-year contract. While RG uses a search algorithm much like Google's that determines when and which ads are shown given a user's search for restaurants on the platform, restaurants have little say in what search terms will trigger their ads. However, they are guaranteed that their ads will be shown a minimum of 1,000 times per month to users. RG's current search algorithm shows ads for restaurants triggered by type of cuisine within a 0.5- mile radius of a user's search. For example, if a user searches for Italian restaurants in Harvard Square, the algorithm will choose two Italian restaurants within Harvard Square to advertise. The engineering team has run a variety of tests looking at how users respond to different types of ads in different searches, and designed an alternative search algorithm. Rather than choosing two restaurants by

cuisine, the alternative algorithm shows advertisements when a user searches for a specific restaurant

and selects two restaurants with similar ratings and hours. While they are reasonably satisfied with

how the current algorithm shows ads, they are open to the possibility that the alternative design may

be significantly better (or significantly worse) in providing benefit for their advertisers.

Experiment

For the experiment, RG randomly selected 30,000 restaurants that were active on their platform but

were not currently advertising, yielding a sample that is representative of their population of

restaurants in the US.

For the one-month duration of the experiment, 10,000 restaurants were randomly selected to receive

free ads using the current advertising approach, and another 10,000 restaurants were randomly

selected to receive the alternatively designed ads The main difference between these two treatment

groups is that the alternative design used a very different algorithm to decide when to deliver ads and

which ads to pair with each search, as described above. The rest of the 10,000 restaurants received no

advertisements. None of the restaurants were informed about the experiment or the advertisements.

The spreadsheet supplement for this exercise (HBS No. 916-702) contains a variety of outcomes

observed for these 30,000 restaurants during the one month of the experiment.

Spreadsheet Data

The spreadsheet supplement contains data for each restaurant in the experiment observed in the

month during the experiment. The unit of observation is a restaurant-month, so data of each restaurant

are in a single row. For example, in a row, pageviews means the number of unique visits on restaurant's

RG page during that month. The variables included are as follows:

Variables

Variable Name Definition

business_id the restaurant's unique identifier

treatment =0: in the control group

=1: in the first treatment group (ads of current design)

=2: in the second treatment group (ads of alternative design)

restaurant type =chain: chain restaurants

=independent: independent restaurants

pageviews # of visits to the restaurant's RG page per month

calls # of phone calls made from the restaurant's mobile RG page per month

reservations # of reservations made from the restaurant's RG page per month

Questions

1. Given that there is already a set of Restaurants on the platform that have bought Ads, do you need an experiment? Or can you simply compare the outcomes of Restaurants do and dont advertise?

2. In the exercise, which outcome is most useful to consider? Are there other outcome variables that you think would be useful to measure?

3. Should Restaurant Grades stick with their current design or switch to the alternative? How confident you are with your decision, and what are possible concerns you may have?