Rocket Fuel: Measuring the Effectiveness of Online Advertising1 "Experiment and learn the secret of successful advertising for humans and computers alike." CLARA DEMIDENKO, ROCKET FUEL It was a typical foggy morning in the San Francisco Bay Area in February 2016 as the low clouds that had been pushed in by the sea breeze the previous night were slowly starting to burn off. Tensions were running high in Rocket Fuel's Redwood City headquarters as Clara Demidenko, a technical account manager, and her team were putting the final touches on a presentation for an all-important client meeting. The client's representatives from TaskaBella Inc., a manufacturer of women's luxury accessories, were already late as their morning flight into nearby SFO was delayed due to the low visibility at the airport. The extra two hours came in handy for Demidenko to go over the presentation with her team one last time. Their objective was to convince TaskaBella that the pilot advertising campaign that Rocket Fuel had recently completed was effective and to present strong evidence that the return on investment (ROI) was positive. The stakes were high as TaskaBella was in the process of reallocating a significant portion of its advertising budget spent on handbags and related accessories and was concluding a series of pilot studies to decide how to move forward. Rocket Fuel Inc. (NASDAQ: FUEL) was founded in March 2008 with a vision of transforming the digital advertising industry through big data and artificial intelligence. At a time when realtime bidding (RTB) and ad exchanges were still nascent, Rocket Fuel quickly grew into the industry leader by bringing the precision and conclusiveness of science to the world of marketing to drive higher ROI. By 2016, Rocket Fuel has grown into a business with nearly half a billion 1 Certain names and figures have been altered in the case study to protect the identity of the entities involved. Associate Professor Zsolt Katona and Brian Bell (MBA'18) prepared this case as the basis for class discussion rather than to illustrate either effective or ineffective handling of an administrative situation. Copyright 2017 by University of California at Berkeley Haas School of Business. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, used in a spreadsheet, or transmitted in any form or by any means without the express written permission of the Berkeley-Haas Case Series. dollars in annual revenue. Rocket Fuel's full Programmatic Marketing Platform is designed to leverage artificial intelligence (AI) to determine what marketing actions to take with a particular person in a particular moment of time. Using Machine Learning (ML) techniques the platform continuously updates the features of its underlying model to improve campaign effectiveness in real time. Just six months before the February 2016 meeting, TaskaBella approached Rocket Fuel to run a trial campaign for a newly released handbag model. The two teams discussed the overall scope of the campaign with an objective of targeting around half-million consumers online with TaskaBella's handbag ads. Over half of the discussion was centered on measuring the success of the campaign. TaskaBella was mostly concerned about reaching sufficiently high levels of conversion rates so that advertising dollars led to enough purchases to justify the spend. Demidenko explained that CPMs (cost per thousand) vary and were affected by many factors, but with the target sites and population TaskaBella had in mind, they could expect to pay an average of around $9 for a thousand impressions. TaskaBella estimated that given the price tag of over $100 on the handbag and given the variable costs associated with a unit, a converting user is worth about $40.1 At first, the perspective numbers looked very promising to TaskaBella; based on data from previous campaigns run with other providers, they estimated that a CPM of around $10 would translate to a cost of roughly $10 to $15 per conversion. However, Adam Burrup, a TaskaBella digital marketing manager, pointed out that since the manufacturer already had a strong social media presence, many consumers may buy the new handbag as a result of word-of-mouth, after hearing about it from friends. Therefore, it would be important to know the extent to which conversions can be attributed to the advertising campaign. Demidenko agreed and pointed out that one of Rocket Fuel's primary objective in every campaign is to provide evidence that the ads do make a difference. They do so by carving out a small portion of users reached during the campaign to form a control group. Users in the control group will be shown a public service announcement (PSA), such as wildfire prevention or Mothers Against Drunk Driving (MADD), instead of the ad in the exact same size and position on the page (See examples of PSA's in Exhibit 1). By randomly selecting which user is in the control group and which user is exposed, they can then measure how much difference advertising makes. After some brief calculations, TaskaBella and Rocket Fuel settled on a control group size of four percent. While Rocket Fuel's platform automatically calculates the size of the control group based on the campaign's parameters, TaskaBella's representatives wanted to understand the factors that determine the resulting size. Keeping in mind that serving PSAs to the control group costs money and represent a lost opportunity to convert potential customers, TaskaBella wanted to avoid having too large of a control group. Demidenko had often experienced clients who wanted to reduce the control group sizesometimes even in the middle of the campaignand she was prepared to explain the reasons to avoid a too small control group. First, conversion rates in display advertising are typically low compared to other experimental settings requiring a sizable control group to detect lift. Second, even if advertising is effective, the lift might be small. If the control group is too small and the lift is not very high, the experiment will not be able to detect a 1 Worth of $40 per converted user is calculated by taking the incremental or marginal revenue ($100+) of each handbag sold less the incremental or marginal costs associated with each handbag sold (material, labor, sales commission, etc.). The incremental or variable costs do not include fixed costs such as advertising expense.. significant lift. Finally, a larger control group allows for a deeper analysis with potentially significant differences in different segments of the population. For example, Rocket Fuel's internal algorithms that decide on ad placement and bids leverage the control group at the individual level to identify factors that make an impression more or less likely to lead to a conversion. The trial campaign ran from November 2015 to February 2016. The ads were first loaded into Rocket Fuel's content delivery network by an operations associate, which enables the quick displaying of the ads anywhere in the world within milliseconds. When the user visits a publisher's page (like CNN.com or NYtimes.com), the publisher partners with a supply-side vendor to put the impression opportunity out to auction. This all happens millions of times per second across the dozens of such exchanges that Rocket Fuel is integrated with. Rocket Fuel receives the bid request and needs to respond within 100 milliseconds in order for its bid to qualify. Rocket Fuel receives a hashed cookie ID in the bid request, which allows it to anonymously identify the user in its system, along with many other parameters about the ad size and content. It then applies ML models to predict the probability that this user will take the required action or actions for each of its campaigns running at the time of the bid. For each campaign, users that have been assigned to the control group are not actually served a campaign ad, but a PSA. By the end of the campaign, roughly 14.5 million impressions were served at an average CPM of $9. Out of the approximately 590,000 users identified, about 15,000 bought the new handbag. Rocket Fuel assembled a user-by-user summary of the campaign results by exporting the appropriate fields from their database. A detailed description of the CSV file containing the data is given in Exhibit 2. At the end of the campaign, the Rocket Fuel team quickly started working on analyzing the results in detail. In preparation for the meeting, the team planned to use simple charts and basic statistical analysis to demonstrate the effectiveness of the campaign. Demidenko insisted that the Rocket Fuel team should present the results in a format that is digestible for not only engineers, but for all of TaskaBella's managers. While waiting for TaskaBella's representatives to arrive after the late landing at SFO, Demidenko discussed the main points of the presentation. The first part of the presentation covered the basic statistics about the campaign. Most importantly, the total cost of the advertising was compared to the value of conversions to get an idea about the profitability and the ROI. The next section of the presentation covered an analysis of the number of impressions users were exposed to and how these figures impacted the effectiveness of the ad. Finally, the last part of the presentation explored timing issues by analyzing the days and hours in a week and the changes in response to advertising. Exhibit 1 Public Service Announcement Examples from the Ad Council The Ad Council is a non-profit organization that produces, distributes and promotes campaigns about issues in the interest of the public, focusing on roughly 50 national campaigns at a time. The campaigns are initiated by government agencies, non-government organizations (NGOs), or other non-profits. The ad copies are created by volunteer talent from leading agencies and disseminated through donated ad space and airtime. Below are two examples of the Ad Council's national campaigns. Wildfire Prevention Texting and Driving Prevention Source: Ad Council (public domain usage). Exhibit 2 Description of CSV file data set Each row in the CSV file data set (rocketfuel_data.csv) represents a uniquely identified user in the ad campaign. For each user, the following six columns are provided. user id: Unique identifier of the user test: Whether the user was exposed to advertising or was in the control group. 1 if the user was exposed to the real ad, 0 if the user was in the control group and was shown a PSA. converted: Whether the user converted. 1 if the user bought the handbag during the campaign, 0 if not. tot_impr: The total number of ad impressions the user encountered. For users in the control group this counts the number of times they encountered the PSA. For exposed users it counts the number of times they were shown the ad. mode_impr_day: Shows the day of the week on which the user encountered the most number of impressions. 1 means Monday, 7 means Sunday. For example if a given user encountered 2 impressions on Mondays, 3 on Tuesdays, 7 on Wednesdays, 0 on Thursdays and, Fridays, 9 on Saturdays and 2 on Sundays, this column takes the value of 6 (Saturday). mode_impr_hour: Shows the hour of the day (0-23) in which the user encountered the most number of impressions.

Remember, you are writing this as a part of the management team to the CEO of the Corporation.

Question 1: State clearly what the main issue facing the directors of TaskaBella is for the meeting upcoming with Rocket Fuel;

Question 2: State clearly what you think is relevant and important information regarding setting up the controlled experiment.

Question 3: State clearly what you think might be issues regarding the use of an experiment to test the ad campaign's effectiveness. Be sure to support your answer with information contained in the case.

Question 4: Provide an interpretation of the summary data table in terms of what the data suggests regarding the ad campaign's effectiveness.

Question 5: Given your answers to questions 1 - 4, a brief recommendation to the Directors of TaskaBella regarding the effectiveness of the ad campaign. Be sure to use the interpretation of the data from question 4 to support your recommendation.