The objective was to convince TaskaBella that the pilot advertising campaign that Rocket Fuel had recently completed was effective and to present strong evidence that the return on investment (ROI) was positive. The stakes were high as TaskaBella was in the process of reallocating a significant portion of its advertising budget spent on handbags and related accessories and was concluding a series of pilot studies to decide how to move forward.

Rocket Fuel Inc. (NASDAQ: FUEL) was founded in March 2008 with a vision of transforming the digital advertising industry through big data and artificial intelligence. At a time when realtime bidding (RTB) and ad exchanges were still nascent, Rocket Fuel quickly grew into the industry leader by bringing the precision and conclusiveness of science to the world of marketing to drive higher ROI. By 2016, Rocket Fuel has grown into a business with nearly half a billion dollars in annual revenue. Rocket Fuels full Programmatic Marketing Platform is designed to leverage artificial intelligence (AI) to determine what marketing actions to take with a particular person in a particular moment of time. Using Machine Learning (ML) techniques the platform continuously updates the features of its underlying model to improve campaign effectiveness in real time.

Just six months before the February 2016 meeting, TaskaBella approached Rocket Fuel to run a trial campaign for a newly released handbag model. The two teams discussed the overall scope of the campaign with an objective of targeting around half-million consumers online with TaskaBellas handbag ads. Over half of the discussion was centered on measuring the success of the campaign. TaskaBella was mostly concerned about reaching sufficiently high levels of conversion rates so that advertising dollars led to enough purchases to justify the spend. Demidenko explained that CPMs (cost per thousand) vary and were affected by many factors, but with the target sites and population TaskaBella had in mind, they could expect to pay an average of around $9 for a thousand impressions. TaskaBella estimated that given the price tag of over $100 on the handbag and given the variable costs associated with a unit, a converting user is worth about $40.^[1]

At first, the perspective numbers looked very promising to TaskaBella; based on data from previous campaigns run with other providers, they estimated that a CPM of around $10 would translate to a cost of roughly $10 to $15 per conversion. However, Adam Burrup, a TaskaBella digital marketing manager, pointed out that since the manufacturer already had a strong social media presence, many consumers may buy the new handbag as a result of word-of-mouth, after hearing about it from friends. Therefore, it would be important to know the extent to which conversions can be attributed to the advertising campaign. Demidenko agreed and pointed out that one of Rocket Fuels primary objective in every campaign is to provide evidence that the ads do make a difference. They do so by carving out a small portion of users reached during the campaign to form a control group. Users in the control group will be shown a public service announcement (PSA), such as wildfire prevention or Mothers Against Drunk Driving (MADD), instead of the ad in the exact same size and position on the page (See examples of PSAs in Exhibit 1). By randomly selecting which user is in the control group and which user is exposed, they can then measure how much difference advertising makes.

After some brief calculations, TaskaBella and Rocket Fuel settled on a control group size of four percent. While Rocket Fuels platform automatically calculates the size of the control group based on the campaigns parameters, TaskaBellas representatives wanted to understand the factors that determine the resulting size. Keeping in mind that serving PSAs to the control group costs money and represent a lost opportunity to convert potential customers, TaskaBella wanted to avoid having too large of a control group. Demidenko had often experienced clients who wanted to reduce the control group sizesometimes even in the middle of the campaignand she was prepared to explain the reasons to avoid a too small control group. First, conversion rates in display advertising are typically low compared to other experimental settings requiring a sizable control group to detect lift. Second, even if advertising is effective, the lift might be small. If the control group is too small and the lift is not very high, the experiment will not be able to detect a significant lift. Finally, a larger control group allows for a deeper analysis with potentially significant differences in different segments of the population. For example, Rocket Fuels internal algorithms that decide on ad placement and bids leverage the control group at the individual level to identify factors that make an impression more or less likely to lead to a conversion.

The trial campaign ran from November 2015 to February 2016. The ads were first loaded into Rocket Fuels content delivery network by an operations associate, which enables the quick displaying of the ads anywhere in the world within milliseconds. When the user visits a publishers page (like CNN.com or NYtimes.com), the publisher partners with a supply-side vendor to put the impression opportunity out to auction. This all happens millions of times per second across the dozens of such exchanges that Rocket Fuel is integrated with. Rocket Fuel receives the bid request and needs to respond within 100 milliseconds in order for its bid to qualify. Rocket Fuel receives a hashed cookie ID in the bid request, which allows it to anonymously identify the user in its system, along with many other parameters about the ad size and content. It then applies ML models to predict the probability that this user will take the required action or actions for each of its campaigns running at the time of the bid. For each campaign, users that have been assigned to the control group are not actually served a campaign ad, but a PSA.

By the end of the campaign, roughly 14.5 million impressions were served at an average CPM of $9. Out of the approximately 590,000 users identified, about 15,000 bought the new handbag. Rocket Fuel assembled a user-by-user summary of the campaign results by exporting the appropriate fields from their database. A detailed description of the CSV file containing the data is given in Exhibit 2. At the end of the campaign, the Rocket Fuel team quickly started working on analyzing the results in detail. In preparation for the meeting, the team planned to use simple charts and basic statistical analysis to demonstrate the effectiveness of the campaign. Demidenko insisted that the Rocket Fuel team should present the results in a format that is digestible for not only engineers, but for all of TaskaBellas managers.

While waiting for TaskaBellas representatives to arrive after the late landing at SFO, Demidenko discussed the main points of the presentation. The first part of the presentation covered the basic statistics about the campaign. Most importantly, the total cost of the advertising was compared to the value of conversions to get an idea about the profitability and the ROI. The next section of the presentation covered an analysis of the number of impressions users were exposed to and how these figures impacted the effectiveness of the ad. Finally, the last part of the presentation explored timing issues by analyzing the days and hours in a week and the changes in response to advertising.

Case Discussion Questions

Was the advertising campaign effective? Did additional consumers convert as a result of the ad campaign?

Was the campaign profitable?

How much more money did TaskaBella make by running the campaign (excluding advertising costs)?

What was the cost of the campaign?

Calculate the ROI of the campaign. Was the campaign profitable?

What was the opportunity cost of including a control group; how much more could have TaskaBella made with a smaller control group or not having a control group at all?

How did the number of impressions seen by each user influence the effectiveness of advertising?

Create a chart of conversion rates as a function of the number of ads displayed to users. Plot conversion rates for those who were in the control group and for those who were exposed to the ad. Group together number of impressions as necessary to obtain a meaningful plot. (Conversion rate means the percentage of unique users who made a purchase.)

What can you infer from the charts? In what region is advertising most effective?

What do the above figures imply for the design of the next campaign assuming that consumer response would be similar?

How does consumer response to advertising vary on different days of the week and at different times of the day?

Create a chart with the conversion rates for the control group and the exposed group as a function of the day of week when they were shown the most impressions.

Create the same chart for hours within a day (excluding the period between midnight and 8 a.m.).

What days/hours is advertising most/least effective?

Exhibit 1 Public Service Announcement Examples from the Ad Council

The Ad Council is a non-profit organization that produces, distributes and promotes campaigns about issues in the interest of the public, focusing on roughly 50 national campaigns at a time. The campaigns are initiated by government agencies, non-government organizations (NGOs), or other non-profits. The ad copies are created by volunteer talent from leading agencies and disseminated through donated ad space and airtime. Below are two examples of the Ad Councils national campaigns.

Wildfire Prevention

Texting and Driving Awareness

Exhibit 2 Description of CSV file data set

Each row in the CSV file data set (rocketfuel_data.csv) represents a uniquely identified user in the ad campaign. For each user, the following six columns are provided.

user id: Unique identifier of the user

test: Whether the user was exposed to advertising or was in the control group. 1 if the user was exposed to the real ad, 0 if the user was in the control group and was shown a PSA.

converted: Whether the user converted. 1 if the user bought the handbag during the campaign, 0 if not.

tot_impr: The total number of ad impressions the user encountered. For users in the control group this counts the number of times they encountered the PSA.For exposed users it counts the number of times they were shown the ad.

mode_impr_day: Shows the day of the week on which the user encountered the most number of impressions. 1 means Monday, 7 means Sunday. For example if a given user encountered 2 impressions on Mondays, 3 on Tuesdays, 7 on Wednesdays, 0 on Thursdays and, Fridays, 9 on Saturdays and 2 on Sundays, this column takes the value of 6 (Saturday).

mode_impr_hour: Shows the hour of the day (0-23) in which the user encountered the most number of impressions.

[1] Worth of $40 per converted user is calculated by taking the incremental or marginal revenue ($100+) of each handbag sold less the incremental or marginal costs associated with each handbag sold (material, labor, sales commission, etc.). The incremental or variable costs do not include fixed costs such as advertising expense.