Steps to Completing an A/B Test Whether you're launching a new product, changing a website design, or testing a new price, an AB test can help you make a decision with confidence when you don't have a lot of data. Below is a useful, high level guide to designing and running an AB test. Select performance metric It's important to understand the metric used to evaluate the results of the test. Whether the goal is to increase sales, profit, conversion rate, etc., this should be specified at the upfront. Select experimental design Matched pair - when the sample size is small and/or the data is difficult to collect, a matched pair experiment should be used. Randomized design - when the sample size is large and the data is easy to collect, then a randomized experiment should be used. Randomized experiments are very common for web-based AB tests. Select experimental and control variables Select the treatment and control units Each individual in the test is considered a unit. The unit can be a person, store, etc. In a test, units are split into two groups, the treatment group and control group. Treatment and control units are compared against each other Determine sample size and test duration These contribute most directly to statistical significance. You can improve statistical significance by either increasing the sample size or test duration. Duration: Generally the duration of a test should be at least as long enough to capture a representative group. If users generally visit a store or website once a week, then the test duration should be a week. Experimental variable - The experimental, or treatment, variable, is the variable that is different between treatment and control units. Control Variables - The control variables are the variables that should remain constant between test and control groups. These ensure that the treatment and control groups are representative of each other and that the results will apply to the population. Clean and prepare data Clean and filter the data appropriately. This could mean filtering for the dates of the test, ensuring there are no duplicate records, removing records with incomplete data, etc. Remove outliers. Make sure that any approach to remove outliers is unbiased. Calculate lift Determine statistical significance Compare the average performance between the two groups Performing a t-test gives a p-value. If the p-value is below 0.05, the results are considered statistically significant. It can also be useful to understand the distribution of the performance of the units Use a paired t-test for matched pair experiments. Estimate impact of broad implementation In order to provide an expected impact of broad implementation of the treatment, apply the lift calculation to the entire population. Use an unpaired t-test for randomized experiments. Project Overview: You're a business analyst for Round Roasters, a coffee restaurant in the United States of America. The executive team conducted a market test with a new menu and needs to figure whether the new menu can drive enough sales to offset the cost of marketing the new menu. Your job is to analyze the A/B test and write up a recommendation to whether the Round Roasters chain should launch this new menu. How Do I Complete this Project? This project uses skills learned throughout the "A/B Testing \" course. To complete this project: Skills Required In order to complete this project, you must be able to: Cleanup, format, and blend a wide range of data sources Plan and analyze A/B tests Project Details: The Business Problem Round Roasters is an upscale coffee chain with locations in the western United States of America. The past few years have resulted in stagnant growth at the coffee chain, and a new management team was put in place to reignite growth at their stores. The first major growth initiative is to introduce gourmet sandwiches to the menu, along with limited wine offerings. The new management team believes that a television advertising campaign is crucial to drive people into the stores with these new offerings. However, the television campaign will require a significant boost in the company's marketing budget, with an unknown return on investment (ROI). Additionally, there is concern that current customers will not buy into the new menu offerings. To minimize risk, the management team decides to test the changes in two cities with new television advertising. Denver and Chicago cities were chosen to participate in this test because the stores in these two cities (or markets) perform similarly to all stores across the entire chain of stores; performance in these two markets would be a good proxy to predict how well the updated menu performs. The test ran for a period of 12 weeks (2016April29 to 2016July21) where five stores in each of the test markets offered the updated menu along with television advertising. The comparative period is the test period, but for last year (2015April29 to 2015 July21). You've been asked to analyze the results of the experiment to determine whether the menu changes should be applied to all stores. The predicted impact to profitability should be enough to justify the increased marketing budget: at least 18% increase in profit growth compared to the comparative period while compared to the control stores; otherwise known as incremental lift. In the data, profit is represented in the gross_margin variable. You have been able to gather three data files to use for your analysis: Transaction data for all stores from 2015-January-21 to 2016August-18 A listing of all Round Roasters stores A listing of the 10 stores (5 in each market) that were used as test markets. Steps to Success Step 1: Plan Your Analysis To perform the correct analysis, you will need to prepare a data set. Prior to rolling up your sleeves and preparing the data, it's a good idea to have a plan of what you need to do in order to prepare the correct data set. A good plan will help you with your analysis. Here are a few questions to get you started: What is the performance metric you'll use to evaluate the results of your test? What is the test period? At what level (day, week, month, etc.) should the data be aggregated? Step 2: Clean Up Your Data In this step, you should prepare the data for steps 3 and 4. You should aggregate the transaction data to the appropriate level and filter on the appropriate data ranges. You can assume that there is no missing, incomplete, duplicate, or dirty data. You're ready to move on to the next step when you have weekly transaction data for all stores. Step 3: Match Treatment and Control Units In this step, you should create the trend and seasonality variables, and use them along with you other control variable(s) to match two control units to each treatment unit. Treatment stores should be matched to control stores in the same region. Note: Calculate the number of transactions per store per week and use 12 periods to calculate trend and seasonality. Apart from trend and seasonality... What control variables should be considered? Note: Only consider variables in the RoundRoastersStore file. What is the correlation between your each potential control variable and your performance metric? What control variables will you use to match treatment and control stores? Step 4: Analysis and Writeup Conduct your A/B analysis and create a short report outlining your results and recommendations. A/B Test Guide The A/B test guide can help you map out the process you'll need to go through to complete an A/B test. This will be useful for the project, as well as any A/B tests you do in a professional setting. Data round-roaster-stores.csv This file contains store information for each Round Roaster store in the USA. treatment-stores.csv This file contains store information for each store that offered the new menu items. round-roaster-transactions.zip This file contains transaction level information for all of Round Roaster's stores Tips Aggregate and Export We recommend you save your aggregated transaction database as a separate file to further reduce your development time. You shouldn't need to re-aggregate the transaction database every time you want to test out a new workflow. Creating a week variable When calculating the week column, assign week 1 to the first week of the test period, so the week_begin variable for week 1 would be your test start date. This will cause weeks prior to the test period to be negative, which is perfectly fine. See here for an example of how to calculate the week variable using a formula tool in Alteryx. Using the Trend tool The trend tool is used to create trend and seasonality variables to use as control variables. To do this, you need at least 52 weeks of data, plus the number of weeks you select in the tool to calculate trend, before the beginning of the test start date. In lesson 4, you used 6 weeks to calculate the trend, so you needed 58 weeks prior to the test start date. For the project, you are asked to use 12 weeks to calculate trend, so you'll need 64 weeks of data prior to the test start date. Since the test lasts for 12 weeks, this means you'll need a total 76 weeks of data. Gross Margin Sales, not Total Sales Make sure to use gross margin sales in all lift calculations and not total sales. qattachments_796792eee44e0ecf521fde80964ea909164703d8 StoreID Sq_Ft AvgMonthSales Right_Name Phone Number 10018 1183 18000 Bellflower & Spring 562-420-1317 10068 1198 16000 Foothill & Boston, La Crescenta 818-541-7693 10118 1204 13000 Magic Mountain & Tourney 661-260-0844 10168 1195 19000 Alameda & Shelton (818) 557-6604 10218 1193 15000 Victoria Gardens 909-646-8562 10268 1191 25000 Azusa & Amar, West Covina 626-964-5145 10318 1147 16000 Chase & Avocado, El Cajon 619-444-5107 10368 1159 19000 Albertsons -- San Marcos #6708 (760) 736-8880 10418 1146 19000 Fashion Valley Mall -Lvl 2 Food Crt 619-299-2593 10468 1151 21000 Mira Mesa Mall (858) 586-6361 10518 1147 11000 Imperial & Main, El Centro (760) 352-3502 10568 1192 21000 Vons - Redlands #1734 909-793-0447 10618 1187 15000 Moreno Beach & Hwy 60 951-485-6878 11268 1328 12000 Safeway - Concord #1192 825-365-2710 11318 1319 24000 Grove & Center 510-886-9010 11368 1319 22000 Marina & I-880 (510) 357-6101 11418 1323 15000 Oakland City Center (510) 832-1767 11468 1325 19000 Donahue & Hwy 101 415-289-0743 11518 1306 35000 Homestead & Lawrence - Santa Clara408 2441421 11568 1305 21000 Lincoln & Minnesota Willow Glen (408) 298-5272 11618 1307 16000 Bellevue & Shaffer - Atwater 209-357-2166 11668 1345 10000 Fulton & Guerneville 707-544-4256 11718 1353 21000 Birdcage Walk - Citrus Heights (916) 728-1587 11768 1350 16000 International & Zinfandel - RC 916-852-1893 11818 1353 15000 Target Woodland T-2408 (530) 665-4148 11868 1352 12000 American River College 916-484-8328 12019 1588 11000 Safeway-Lake Grove #1047 503-675-2500 12069 1593 8000 Orenco Station 5036400740 12119 1594 19000 Piedmont (503) 283-0671 12169 1593 10000 Safeway - Portland #1612 503-528-0500 12219 1541 12000 13th & Alder 5413384100 12269 1586 16000 Safeway-La Grande #1827 541-663-2700 12286 1666 11000 3 Bellevue Cntr (425) 646-9602 12486 1663 22000 Westwood Village 206-937-1732 12536 1669 12000 Target Northgate T-1284 206-494-0897 12586 1678 10000 Safeway-Everett #1484 425-258-3552 12686 1654 13000 N. 30th & McCarver/Old Town (253) 573-1901 12736 1644 18000 Safeway - Aberdeen #1546 360-533-1955 12786 1659 16000 Valley Mall Pkwy & 5th St NE 509-886-4778 12836 1667 16000 Sprague & Pine 509-926-8497 1508 1457 12000 3219 8th Street SW - Altoona 5159670631 1542 1506 12000 Target Brookfield T-2780. 262-395-1625 1580 1568 14000 Target Cottage Grove T-662 651-458-8179 1630 1582 17000 Round Lake Blvd and Northdale 6127902009 1662 1471 11000 Arlington Heights & Golf Road 8476402779 1664 1475 11000 Barrington 8478428048 1675 1472 15000 Northwest Hwy & Elmhurst Rd 8472531188 1696 1471 10000 Higgins & Meacham 2242232528 1700 1465 15000 Roosevelt & Summit (224) 500-9575 1712 1456 19000 159th & LaGrange 708-403-1461 1762 1469 8000 ORD OHare Terminal 1, Baggage Cla773-686-6181 1807 1353 14000 Target Chesterfield T-1353 636-536-6207 Page 1 qattachments_796792eee44e0ecf521fde80964ea909164703d8 1857 1863 1964 2014 2114 2214 2264 2283 2288 2293 2301 2322 2333 2341 2383 2409 2468 2518 2568 2572 2652 2702 2752 2802 2852 2902 2952 3002 3052 3102 3152 3185 3235 3302 3352 6992 7037 7087 7162 7212 7284 7334 7384 7434 7484 7534 7584 7634 7720 7770 7811 8112 8162 1296 1363 1158 1152 1049 1033 1231 1393 1387 1394 1387 1390 1395 1400 1366 1501 1169 1168 1220 1230 1189 1190 1198 1142 1154 1170 1246 1322 1323 1330 1370 1583 1541 1669 1646 1455 1503 1507 1584 1580 1470 1473 1466 1455 1463 1465 1463 1479 1355 1354 1363 1155 1149 15000 Ramey-Price Cutter-Nixa #55 417-725-6166 11000 Dillons Store-Lawrence #98 785-842-2942 11000 Angel & Lucas 2145515427 16000 Teavana - Dallas Galleria 972-788-9340 20000 1960 & Champion Forest 2818930556 15000 1604 & Braun 2106889553 8000 Target Amarillo - T-221 806-358-4030 18000 64th & Ward 14000 S. Parker Rd & E. Quincy Ave 3036806304 11000 King Sooper Arvada # 55 303-424-0670 11000 Hampden & Santa Fe, Sheridan 3037611675 14000 King Soopers - Denver #1 8000 104th & Bannock, Northglenn 3032555628 11000 1352 College Ave - Boulder 3034430309 10000 King Soopers #77 Colorado Springs 719-488-2955 14000 Pocatello - Yellowstone and Alameda 2085891763 13000 Target Phoenix SW T-2149 623-687-2136 13000 Arizona Mills (480) 897-4804 13000 Safeway-Sedona #1207 928-282-0118 12000 Market Place at Journal Center (505) 823-9208 19000 Central & Slauson 323-521-1535 17000 Olympic & Main 310-458-2802 5000 Reseda & Nordhoff 8189931472 17000 Bonita Road & Plaza Bonita 619-476-6648 11000 Carmel Mountain (858) 675-0412 17000 Outlets at San Clemente 9494418952 17000 Von's - Lake Isabella #2413 17000 564 Market Street (415) 291-0584 21000 315 1/2 20th St. (510) 465-8292 10000 I-5 & Benjamin Holt 209-952-5709 9000 Hwy 70 & Hwy 20 - Marysville 530-749-2009 13000 Berg Pkwy & Hwy 99 - Canby 5033138798 14000 W. 11th & Acorn Park - Eugene 5413431041 19000 Northgate Mall (206) 306-8525 18000 Downtown Olympia (360) 753-7771 15000 Tenth & Locust 515-280-8200 16000 Southridge Mall - Lower Level 414-423-5323 13000 State & Lake (608) 294-8345 12000 Super Target Otsego ST-2456 763-252-1316 17000 Super Target Blaine ST-1832 763-354-1000 15000 Biesterfield & Wellington 8475933252 10000 Willow & Waukegan - Northbrook 847-205-2542 10000 North & 25th - Melrose Park 708-681-1731 17000 Lockport - 159th & Farrell 815-838-4973 15000 Super Target Warrenville ST-1903 630-961-4061 12000 Target Chicago T-2781 312-279-3340 13000 Target Chicago T-1879 773-579-2120 9000 State & Trainer 815/398-7374 9000 SR-K & Weldon Spring 636-329-0251 12000 Hwy 50 & Thompson 660-829-0296 10000 89th and Metcalf 913-642-2588 9000 Renner & Northstar 972-480-9504 18000 Target Dallas/Medallion T-55 214-361-2026 Page 2 qattachments_796792eee44e0ecf521fde80964ea909164703d8 8212 8262 8312 8362 8562 8717 8767 8817 8867 8917 8967 9017 9081 9188 9238 9288 9338 9388 9438 9488 9524 9589 9639 9768 9818 9868 9918 9968 1143 1150 1040 1034 1036 1397 1385 1390 1395 1405 1359 1368 1527 1176 1170 1173 1167 1177 1130 1220 1234 1263 1265 1193 1191 1185 1190 1182 8000 Super Target Arlington ST-1339 11000 I-35 & Western Center 19000 University & Kelvin 12000 Bay Area & Seawolf 9000 1604 & Blanco Road 13000 King Soopers-Broomfield, CO #86 20000 King Soopers-Littleton 122 14000 6th & Lincoln 12000 102nd & Grant-Thornton 13000 Hover & Trade Center Dr 17000 31st & Colorado-CO Springs 18000 Safeway-Grand Junction #1533 13000 Albertsons-Boise #161 10000 Paradise Valley Mall 12000 Stapley & McKellips, Mesa 8000 Camelback Inn-Scottsdale 1000 Safeway-Gilbert #1534 13000 Grand & Reems, Surprise 14000 Sunrise & Swan - Tucson 14000 Bashas - Sedona #28 11000 NM 528 and Rockaway, Rio Rancho 13000 Albertsons - Las Vegas #6029 11000 Sahara & Nellis, Las Vegas 17000 Hollywood & Highland 15000 3rd & Gage, Los Angeles 12000 Hermosa & 13th, Hermosa Beach 16000 Main & Hill 16000 Beach & Chapman Page 3 817-557-2177 817-232-9527 (713) 529-1571 281-488-9800 210-479-6907 303-439-0113 303-738-5700 303-831-4996 303-255-3069 303-774-0143 719-578-5412 970-248-9869 208-322-7744 (602) 765-3563 480-833-0056 480-948-1700 480-545-7555 623-584-5015 520-615-0881 928-282-5351 505 891-5343 702-889-2000 702-431-3044 323-962-7414 323-265-4124 310-406-8569 (310) 392-3559 714-890-4388 qattachments_796792eee44e0ecf521fde80964ea909164703d8 Street Combined 2890 N Bellflower Blvd, #A-1, The Los Altos Marketplace 3747 Foothill Boulevard, A 25349 Wayne Mills Place, Tourney Retail Plaza 1190 Alameda Avenue, Suite G-2 12466 North Main Street, Space 3340, Victoria Gardens 2320 S. Azusa Avenue 450 E. Chase Avenue 151 Woodland Pkwy 7007 Friars Road, 855A, Fashion Valley Mall - San Diego 8250 Mira Mesa Blvd., Suite H 202 S. Imperial Avenue, Valley Plaza 522 NORTH ORANGE 12751 Moreno Beach Drive 4309 Clayton Rd 22095 Center Street 1209 Marina Boulevard, Suite 1209 1200 Clay Street, #58 190 Donahue St, A, Marin Gateway Retail Center 3551 Homestead Road, Homestead Shopping Center 1396 Lincoln Avenue 261 E. Bellevue Road, Five Corner Crossings 2547 Guerneville Rd, 501, Raley's Fulton Marketplace 6121 Sunrise Boulevard 3195 Zinfandel Dr., 1, Capital Village 18455 County Rd 102 4700 College Oak Dr 17779 SW Lower Boones Ferry Rd 6221 NE Cornell Rd, Orenco Station Town Center 6003 NE MLK Drive 1100 NE Broadway 801 E. 13th Avenue, (previously Guidos Bar) 2111 Adams Ave 601 108th Avenue NE 9023 25th Ave SW, Westwood Village 302 NE Northgate Way 4128 Rucker Ave 2112 N. 30th Street, Ste. C 221 W Heron St 500 A Valley Mall Parkway, The Valley Mall Plaza 12328 E. Sprague Ave 3219 8th St. SW 12725 W Bluemound Rd 8655 E Point Douglas Rd S 13131 Riverdale Dr NW 1808 South Arlington Heights Road 101 W. Main St., Barrington Village Center 90 East Northwest Highway 1261 East Higgins Road 17 W 633 Roosevelt Road 15858 South LaGrange Road Circular Building #6 40 THF Blvd Page 4 qattachments_796792eee44e0ecf521fde80964ea909164703d8 400 Massey Road 1740 S Massachusetts St 2680 Lucas Road 13350 DALLAS PKWY, STE 2615, Dallas Galleria 5602 FM 1960 Rd W 9950 West Loop 1604 North 8201 I-40 W 12395 West 64th Avenue 13702 E Quincy Ave 9983 Wadsworth Pkwy 3500 A River Point Parkway 1331 N Speer Blvd 200 W. 104th Ave. 1352 College Avenue, 115 1070 W Baptist Rd 883 Yellowstone Avenue 9830 W Lower Buckeye 5000 Arizona Mills Circle, Suite #391 2300 SR-89A 7600 Jefferson NE Street, Suite #1 5857 S. Central Ave. 1705 Ocean Ave., Unit C 9020 N. Reseda, Unit A 3047 Bonita Road, 108 11885 Carmel Mountain, Ste #909 101 West Avenida Vista Hermosa, Suite 632, Plaza San Clemente 5610 Lake Isabella Blvd 564 Market Street 3 15 1/2 20th St 3011 West Benjamin Holt 904 E Street, D 1075 SW 1st Ave, Suite A 3110 W. 11th Avenue 401 NE Northgate Way 550 Capital Way South, Space C 1013 Locust Street, SWC of 1st floor, The Temple for Performing Arts 5300 S. 76th Street, Space K060, Southridge Mall 661 State Street 15800 87th St NE 1500 109th Ave NE 102 Biesterfield Rd 1036 Willow Rd. 2517 W. North Ave. 16527 W. 159th St. 28201 Diehl Rd 1101 W Jackson Blvd 4433 S Pulaski 6239 E. State Street 4581 Highway K, Laboure Centre 3040 W Broadway, Suite 100 8801 Metcalf 4151 East Renner Road 212 Medallion Ctr Page 5 qattachments_796792eee44e0ecf521fde80964ea909164703d8 1400 W Arbrook Blvd 3030 Western Center Blvd 2520 University Boulevard 515 Bay Area Blvd, Ste 100, Concorde Center - Phase II 1130 N Loop 1604 West, #101, The Village at Blanco 12167 Sheridan Blvd 7901 S Broadway 575 Lincoln St, Suite A, The Shops at 6th and Broadway 10005 Grant St, Unit F 1087 South Hover Street, Unit A, Shoppes at St. Vrain Center 3104 W. Colorado Avenue, N/A 2901 F Rd 10700 Ustick Road 4550 E Cactus, #F042 1229 E. McKellips Road, #102 5402 E Lincoln Dr 90 S Val Vista Dr 18885 Reems Road, #105 4700 East Sunrise Drive 160 Coffee Pot Drive 115 State Highway 528 NE 5975 W. Tropicana Ave 2585 S. Nellis Blvd., #1 6801 Hollywood Blvd, 257 3853 E. 3rd Street, E 1303 Hermosa Avenue 2671 Main Street 11931 Beach Blvd., Beach/Chapman Center Page 6 qattachments_796792eee44e0ecf521fde80964ea909164703d8 Street 1 2890 N Bellflower Blvd 3747 Foothill Boulevard 25349 Wayne Mills Place 1190 Alameda Avenue 12466 North Main Street 2320 S. Azusa Avenue 450 E. Chase Avenue 151 Woodland Pkwy 7007 Friars Road 8250 Mira Mesa Blvd. 202 S. Imperial Avenue 522 NORTH ORANGE 12751 Moreno Beach Drive 4309 Clayton Rd 22095 Center Street 1209 Marina Boulevard 1200 Clay Street 190 Donahue St 3551 Homestead Road 1396 Lincoln Avenue 261 E. Bellevue Road 2547 Guerneville Rd 6121 Sunrise Boulevard 3195 Zinfandel Dr. 18455 County Rd 102 4700 College Oak Dr 17779 SW Lower Boones Ferry Rd 6221 NE Cornell Rd 6003 NE MLK Drive 1100 NE Broadway 801 E. 13th Avenue 2111 Adams Ave 601 108th Avenue NE 9023 25th Ave SW 302 NE Northgate Way 4128 Rucker Ave 2112 N. 30th Street 221 W Heron St 500 A Valley Mall Parkway 12328 E. Sprague Ave 3219 8th St. SW 12725 W Bluemound Rd 8655 E Point Douglas Rd S 13131 Riverdale Dr NW 1808 South Arlington Heights Road 101 W. Main St. 90 East Northwest Highway 1261 East Higgins Road 17 W 633 Roosevelt Road 15858 South LaGrange Road Circular Building #6 40 THF Blvd Street 2 #A-1 A Street 3 The Los Altos Marketplace Tourney Retail Plaza Suite G-2 Space 3340 855A Suite H Victoria Gardens Fashion Valley Mall - San Diego Valley Plaza Suite 1209 #58 A Marin Gateway Retail Center Homestead Shopping Center Five Corner Crossings 501 Raley's Fulton Marketplace 1 Capital Village Orenco Station Town Center (previously Guidos Bar) Westwood Village Ste. C The Valley Mall Plaza Barrington Village Center Page 7 qattachments_796792eee44e0ecf521fde80964ea909164703d8 400 Massey Road 1740 S Massachusetts St 2680 Lucas Road 13350 DALLAS PKWY 5602 FM 1960 Rd W 9950 West Loop 1604 North 8201 I-40 W 12395 West 64th Avenue 13702 E Quincy Ave 9983 Wadsworth Pkwy 3500 A River Point Parkway 1331 N Speer Blvd 200 W. 104th Ave. 1352 College Avenue 1070 W Baptist Rd 883 Yellowstone Avenue 9830 W Lower Buckeye 5000 Arizona Mills Circle 2300 SR-89A 7600 Jefferson NE Street 5857 S. Central Ave. 1705 Ocean Ave. 9020 N. Reseda 3047 Bonita Road 11885 Carmel Mountain 101 West Avenida Vista Hermosa 5610 Lake Isabella Blvd 564 Market Street 3 15 1/2 20th St 3011 West Benjamin Holt 904 E Street 1075 SW 1st Ave 3110 W. 11th Avenue 401 NE Northgate Way 550 Capital Way South 1013 Locust Street 5300 S. 76th Street 661 State Street 15800 87th St NE 1500 109th Ave NE 102 Biesterfield Rd 1036 Willow Rd. 2517 W. North Ave. 16527 W. 159th St. 28201 Diehl Rd 1101 W Jackson Blvd 4433 S Pulaski 6239 E. State Street 4581 Highway K 3040 W Broadway 8801 Metcalf 4151 East Renner Road 212 Medallion Ctr STE 2615 Dallas Galleria 115 Suite #391 Suite #1 Unit C Unit A 108 Ste #909 Suite 632 Plaza San Clemente D Suite A Space C SWC of 1st floor The Temple for Performing Arts Space K060 Southridge Mall Laboure Centre Suite 100 Page 8 qattachments_796792eee44e0ecf521fde80964ea909164703d8 1400 W Arbrook Blvd 3030 Western Center Blvd 2520 University Boulevard 515 Bay Area Blvd 1130 N Loop 1604 West 12167 Sheridan Blvd 7901 S Broadway 575 Lincoln St 10005 Grant St, Unit F 1087 South Hover Street 3104 W. Colorado Avenue 2901 F Rd 10700 Ustick Road 4550 E Cactus 1229 E. McKellips Road 5402 E Lincoln Dr 90 S Val Vista Dr 18885 Reems Road 4700 East Sunrise Drive 160 Coffee Pot Drive 115 State Highway 528 NE 5975 W. Tropicana Ave 2585 S. Nellis Blvd. 6801 Hollywood Blvd 3853 E. 3rd Street 1303 Hermosa Avenue 2671 Main Street 11931 Beach Blvd. Ste 100 #101 Concorde Center - Phase II The Village at Blanco Suite A The Shops at 6th and Broadway Unit A Shoppes at St. Vrain Center N/A #F042 #102 #105 #1 257 E Beach/Chapman Center Page 9 qattachments_796792eee44e0ecf521fde80964ea909164703d8 City State Postal Code Region Country Long Beach CA 908151125 West US La Cresenta CA 912141700 West US Valencia CA 913551827 West US Burbank CA 915062806 West US Rancho CucamongCA 917398886 West US West Covina CA 917921511 West US El Cajon CA 920206409 West US San Marcos CA 920693010 West US San Diego CA 921081129 West US San Diego CA 921262624 West US El Centro CA 922432859 West US Redlands CA 923743208 West US Moreno Valley CA 925554411 West US Concord CA 945212842 West US Castro Valley CA 945466707 West US San Leandro CA 945773322 West US Oakland CA 946121424 West US Sausalito CA 949651250 West US Santa Clara CA 950515161 West US San Jose CA 951253010 West US Atwater CA 953012304 West US Santa Rosa CA 954014048 West US Citrus Heights CA 956106834 West US Rancho Cordova CA 956706376 West US Woodland CA 957769314 West US Sacramento CA 958414217 West US Lake Oswego OR 970355237 West US Hillsboro OR 971249074 West US Portland OR 972134035 West US Portland OR 972321232 West US Eugene OR 974013788 West US La Grande OR 978502921 West US Bellevue WA 980044383 West US Seattle WA 981060908 West US Seattle WA 981256047 West US Everett WA 982032211 West US Tacoma WA 984033363 West US Aberdeen WA 985206224 West US East Wenatchee WA 988023812 West US Spokane WA 992160720 West US Altoona IA 50009 Central US Brookfield WI 53005 Central US Cottage Grove MN 55016 Central US Anoka MN 55448 Central US Arlington Heights IL 60005 Central US Barrington IL 60010 Central US Mount Prospect IL 60056 Central US Schaumburg IL 60173 Central US Oakbrook Terrace IL 60181 Central US Orland Park IL 60462 Central US Chicago IL 60666 Central US Chesterfield MO 63005 Central US Page 10 Coordinates (33.8085823059082, -118.124931335449) (34.2375450134277, -118.26114654541) (34.4237022399902, -118.579261779785) (34.1625328063965, -118.314529418945) (34.1120910644531, -117.533462524414) (34.0354042053223, -117.911918640137) (32.7806205749512, -116.956497192383) (33.1403732299805, -117.138748168945) (32.7674598693848, -117.166549682617) (32.9138488769531, -117.145957946777) (32.7915458679199, -115.570175170898) (34.0601615905762, -117.181457519531) (33.9356384277344, -117.178985595703) (37.9660797119141, -121.991058349609) (37.6900939941406, -122.062675476074) (37.7116317749023, -122.162475585938) (37.8044395446777, -122.273910522461) (37.8731155395508, -122.510070800781) (37.3384552001953, -121.994857788086) (37.3037948608398, -121.897270202637) (37.3610649108887, -120.593872070313) (38.4532623291016, -122.771148681641) (38.6748657226563, -121.273056030273) (38.5842742919922, -121.284538269043) (38.670108795166, -121.724243164063) (38.6489906311035, -121.350402832031) (45.3982467651367, -122.74195098877) (45.5342102050781, -122.916580200195) (45.5666618347168, -122.661819458008) (45.5347633361816, -122.654541015625) (44.0456085205078, -123.07999420166) (45.3212585449219, -118.080947875977) (47.6159477233887, -122.196533203125) (47.5227203369141, -122.365577697754) (47.7090606689453, -122.325424194336) (47.9625473022461, -122.211883544922) (47.2746315002441, -122.464630126953) (46.9725341796875, -123.818519592285) (47.4106788635254, -120.290641784668) (47.6567153930664, -117.238510131836) (41.6446723937988, -93.5011215209961) (43.0362281799316, -88.070198059082) (44.8167343139648, -92.9303131103516) (45.2090263366699, -93.3584289550781) (42.052604675293, -87.9823760986328) (42.1540565490723, -88.1362915039063) (42.0633544921875, -87.9355773925781) (42.039363861084, -88.048828125) (41.8600807189941, -87.9739685058594) (41.603816986084, -87.8534317016602) (41.9791107177734, -87.9062652587891) (38.6674957275391, -90.5852813720703) qattachments_796792eee44e0ecf521fde80964ea909164703d8 Nixa Lawrence Lucas Dallas Houston San Antonio Amarillo Arvada Aurora Westminster Denver Denver Northglenn Boulder Colorado Springs Pocatello Phoenix Tempe Sedona Albuquerque Los Angeles Santa Monica Northridge Chula Vista San Diego San Clemente Lake Isabella San Francisco Oakland Stockton Marysville Canby Eugene Seattle Olympia Des Moines Greendale Madison Otsego Blaine Elk Grove Village Northbrook Melrose Park Lockport Warrenville Chicago Chicago Rockford O'Fallon Sedalia Overland Park Richardson Dallas MO KS TX TX TX TX TX CO CO CO CO CO CO CO CO ID AZ AZ AZ NM CA CA CA CA CA CA CA CA CA CA CA OR OR WA WA IA WI WI MN MN IL IL IL IL IL IL IL IL MO MO KS TX TX 65714 Central 66044 Central 75002 Central 75240 Central 77069 Central 78254 Central 79121 Central 80004 West 80015 West 80021 West 80110 West 80204 West 80234 West 80302 West 80921 West 83201 West 85003 West 85282 West 86336 West 87109 West 90001 West 90401 West 91324 West 91910 West 92128 West 92672 West 93240 West 94105 West 94612 West 95219 West 95901 West 97013 West 97402 West 98125 West 98501 West 503092805 Central 531291102 Central 537031049 Central 553306546 Central 554494670 Central 600073657 Central 600626820 Central 601601121 Central 604418000 Central 605553934 Central 606072905 Central 606324010 Central 611082514 Central 633688686 Central 653012487 Central 662122074 Central 750822810 Central 752141579 Central US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US US Page 11 (37.047737121582, -93.30322265625) (38.9528656005859, -95.2359008789063) (33.0859184265137, -96.6186065673828) (32.9306373596191, -96.8210983276367) (29.9798069000244, -95.5135650634766) (29.5347938537598, -98.6816482543945) (35.1865844726563, -101.930641174316) (39.8132095336914, -105.137313842773) (39.6378707885742, -104.828552246094) (39.8431854248047, -105.08194732666) (39.6528053283691, -105.012954711914) (39.7375144958496, -104.997352600098) (39.8847808837891, -104.99006652832) (40.007194519043, -105.275108337402) (39.0566444396973, -104.842697143555) (42.8913078308105, -112.451469421387) (33.4213676452637, -112.270614624023) (33.380973815918, -111.964515686035) (34.8646812438965, -111.800285339355) (35.1658782958984, -106.591079711914) (33.9878997802734, -118.257171630859) (34.0106582641602, -118.491561889648) (34.2348022460938, -118.535797119141) (32.6483001708984, -117.058700561523) (32.9814376831055, -117.075248718262) (33.4396171569824, -117.627723693848) (35.6177062988281, -118.477569580078) (37.7897186279297, -122.401138305664) (37.8079833984375, -122.265480041504) (38.0057220458984, -121.353210449219) (39.1450157165527, -121.591651916504) (45.2561988830566, -122.703552246094) (44.0476951599121, -123.138664245605) (47.7042083740234, -122.32544708252) (47.0433731079102, -122.901275634766) (41.5857887268066, -93.6304626464844) (42.9484405517578, -88.0046691894531) (43.0748023986816, -89.3969345092773) (45.277961730957, -93.563346862793) (45.1654930114746, -93.2333984375) (42.0038528442383, -88.0074691772461) (42.1066436767578, -87.8037185668945) (41.9083595275879, -87.866081237793) (41.5969886779785, -88.0225219726563) (41.8013534545898, -88.1690216064453) (41.8778305053711, -87.6546325683594) (41.8124656677246, -87.7224349975586) (42.2667121887207, -88.9948806762695) (38.719165802002, -90.7021255493164) (38.7086067199707, -93.2646484375) (38.9682731628418, -94.6669387817383) (33.0010566711426, -96.6452560424805) (32.8562088012695, -96.7547988891602) qattachments_796792eee44e0ecf521fde80964ea909164703d8 Arlington Fort Worth Houston Houston San Antonio Broomfield Littleton Denver Denver Longmont Colorado Springs Grand Junction Boise Phoenix Mesa Scottsdale Gilbert Surprise Tucson Sedona Rio Rancho Las Vegas Las Vegas Hollywood Los Angeles Hermosa Beach Santa Monica Stanton TX TX TX TX TX CO CO CO CO CO CO CO ID AZ AZ AZ AZ AZ AZ AZ NM NV NV CA CA CA CA CA 760154103 Central 761311336 Central 770053229 Central 770582666 Central 782484507 Central 800202459 West 801222718 West 802033469 West 802292033 West 805017961 West 809042041 West 815045440 West 837135103 West 850327711 West 852032725 West 852534118 West 852961376 West 853748689 West 857184535 West 863364505 West 871245979 West 891034814 West 891217548 West 900286136 West 900632403 West 902543560 West 904054001 West 906803613 West US US US US US US US US US US US US US US US US US US US US US US US US US US US US Page 12 (32.683837890625, -97.1360778808594) (32.8606147766113, -97.3160095214844) (29.7152824401855, -95.4171295166016) (29.5517807006836, -95.1261444091797) (29.6078052520752, -98.5096206665039) (39.9167976379395, -105.053596496582) (39.5729484558105, -104.988578796387) (39.7253952026367, -104.986465454102) (39.880786895752, -104.98397064209) (40.1472969055176, -105.131423950195) (38.8536911010742, -104.872215270996) (39.0908126831055, -108.513702392578) (43.6344871520996, -116.315902709961) (33.6019172668457, -111.983062744141) (33.450984954834, -111.803215026855) (33.5311241149902, -111.964935302734) (33.3488922119141, -111.756805419922) (33.6557884216309, -112.374099731445) (32.3066520690918, -110.891906738281) (34.8659133911133, -111.795234680176) (35.2600288391113, -106.637954711914) (36.0988960266113, -115.223892211914) (36.1430511474609, -115.066024780273) (34.1027717590332, -118.339248657227) (34.0336227416992, -118.184219360352) (33.8630561828613, -118.400413513184) (34.0011253356934, -118.482490539551) (33.7894058227539, -117.993301391602) qattachments_796792eee44e0ecf521fde80964ea909164703d8 Latitude Longitude Timezone 33.80858231 -118.1249313 Pacific Standard Time 34.23754501 -118.2611465 Pacific Standard Time 34.42370224 -118.5792618 Pacific Standard Time 34.16253281 -118.3145294 Pacific Standard Time 34.11209106 -117.5334625 Pacific Standard Time 34.03540421 -117.9119186 Pacific Standard Time 32.78062057 -116.9564972 Pacific Standard Time 33.14037323 -117.1387482 Pacific Standard Time 32.76745987 -117.1665497 Pacific Standard Time 32.91384888 -117.1459579 Pacific Standard Time 32.79154587 -115.5701752 Pacific Standard Time 34.06016159 -117.1814575 Pacific Standard Time 33.93563843 -117.1789856 Pacific Standard Time 37.96607971 -121.9910583 Pacific Standard Time 37.69009399 -122.0626755 Pacific Standard Time 37.71163177 -122.1624756 Pacific Standard Time 37.80443954 -122.2739105 Pacific Standard Time 37.87311554 -122.5100708 Pacific Standard Time 37.3384552 -121.9948578 Pacific Standard Time 37.30379486 -121.8972702 Pacific Standard Time 37.36106491 -120.5938721 Pacific Standard Time 38.45326233 -122.7711487 Pacific Standard Time 38.67486572 -121.273056 Pacific Standard Time 38.58427429 -121.2845383 Pacific Standard Time 38.6701088 -121.7242432 Pacific Standard Time 38.64899063 -121.3504028 Pacific Standard Time 45.39824677 -122.741951 Pacific Standard Time 45.53421021 -122.9165802 Pacific Standard Time 45.56666183 -122.6618195 Pacific Standard Time 45.53476334 -122.654541 Pacific Standard Time 44.04560852 -123.0799942 Pacific Standard Time 45.32125854 -118.0809479 Pacific Standard Time 47.61594772 -122.1965332 Pacific Standard Time 47.52272034 -122.3655777 Pacific Standard Time 47.70906067 -122.3254242 Pacific Standard Time 47.9625473 -122.2118835 Pacific Standard Time 47.2746315 -122.4646301 Pacific Standard Time 46.97253418 -123.8185196 Pacific Standard Time 47.41067886 -120.2906418 Pacific Standard Time 47.65671539 -117.2385101 Pacific Standard Time 41.64467239 -93.50112152 Central Standard Time 43.03622818 -88.07019806 Central Standard Time 44.81673431 -92.93031311 Central Standard Time 45.20902634 -93.35842896 Central Standard Time 42.05260468 -87.9823761 Central Standard Time 42.15405655 -88.1362915 Central Standard Time 42.06335449 -87.93557739 Central Standard Time 42.03936386 -88.04882813 Central Standard Time 41.86008072 -87.97396851 Central Standard Time 41.60381699 -87.8534317 Central Standard Time 41.97911072 -87.90626526 Central Standard Time 38.66749573 -90.58528137 Central Standard Time Page 13 Current Timezone Offset -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -360 -360 -360 -360 -360 -360 -360 -360 -360 -360 -360 -360 qattachments_796792eee44e0ecf521fde80964ea909164703d8 37.04773712 38.9528656 33.08591843 32.93063736 29.9798069 29.53479385 35.18658447 39.81320953 39.63787079 39.84318542 39.65280533 39.7375145 39.88478088 40.00719452 39.05664444 42.89130783 33.42136765 33.38097382 34.86468124 35.1658783 33.98789978 34.01065826 34.23480225 32.64830017 32.98143768 33.43961716 35.6177063 37.78971863 37.8079834 38.00572205 39.14501572 45.25619888 44.04769516 47.70420837 47.04337311 41.58578873 42.94844055 43.0748024 45.27796173 45.16549301 42.00385284 42.10664368 41.90835953 41.59698868 41.80135345 41.87783051 41.81246567 42.26671219 38.7191658 38.70860672 38.96827316 33.00105667 32.8562088 -93.30322266 Central Standard Time -95.23590088 Central Standard Time -96.61860657 Central Standard Time -96.82109833 Central Standard Time -95.51356506 Central Standard Time -98.68164825 Central Standard Time -101.9306412 Central Standard Time -105.1373138 Mountain Standard Time -104.8285522 Mountain Standard Time -105.0819473 Mountain Standard Time -105.0129547 Mountain Standard Time -104.9973526 Mountain Standard Time -104.9900665 Mountain Standard Time -105.2751083 Mountain Standard Time -104.8426971 Mountain Standard Time -112.4514694 Mountain Standard Time -112.2706146 US Mountain Standard Time -111.9645157 US Mountain Standard Time -111.8002853 US Mountain Standard Time -106.5910797 Mountain Standard Time -118.2571716 Pacific Standard Time -118.4915619 Pacific Standard Time -118.5357971 Pacific Standard Time -117.0587006 Pacific Standard Time -117.0752487 Pacific Standard Time -117.6277237 Pacific Standard Time -118.4775696 Pacific Standard Time -122.4011383 Pacific Standard Time -122.26548 Pacific Standard Time -121.3532104 Pacific Standard Time -121.5916519 Pacific Standard Time -122.7035522 Pacific Standard Time -123.1386642 Pacific Standard Time -122.3254471 Pacific Standard Time -122.9012756 Pacific Standard Time -93.63046265 Central Standard Time -88.00466919 Central Standard Time -89.39693451 Central Standard Time -93.56334686 Central Standard Time -93.23339844 Central Standard Time -88.00746918 Central Standard Time -87.80371857 Central Standard Time -87.86608124 Central Standard Time -88.02252197 Central Standard Time -88.16902161 Central Standard Time -87.65463257 Central Standard Time -87.722435 Central Standard Time -88.99488068 Central Standard Time -90.70212555 Central Standard Time -93.26464844 Central Standard Time -94.66693878 Central Standard Time -96.64525604 Central Standard Time -96.75479889 Central Standard Time Page 14 -360 -360 -360 -360 -360 -360 -360 -420 -420 -420 -420 -420 -420 -420 -420 -420 -420 -420 -420 -420 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -480 -360 -360 -360 -360 -360 -360 -360 -360 -360 -360 -360 -360 -360 -360 -360 -360 -360 -360 qattachments_796792eee44e0ecf521fde80964ea909164703d8 32.68383789 32.86061478 29.71528244 29.5517807 29.60780525 39.91679764 39.57294846 39.7253952 39.8807869 40.14729691 38.8536911 39.09081268 43.63448715 33.60191727 33.45098495 33.53112411 33.34889221 33.65578842 32.30665207 34.86591339 35.26002884 36.09889603 36.14305115 34.10277176 34.03362274 33.86305618 34.00112534 33.78940582 -97.13607788 Central Standard Time -97.31600952 Central Standard Time -95.41712952 Central Standard Time -95.12614441 Central Standard Time -98.50962067 Central Standard Time -105.0535965 Mountain Standard Time -104.9885788 Mountain Standard Time -104.9864655 Mountain Standard Time -104.9839706 Mountain Standard Time -105.131424 Mountain Standard Time -104.8722153 Mountain Standard Time -108.5137024 Mountain Standard Time -116.3159027 Mountain Standard Time -111.9830627 US Mountain Standard Time -111.803215 US Mountain Standard Time -111.9649353 US Mountain Standard Time -111.7568054 US Mountain Standard Time -112.3740997 US Mountain Standard Time -110.8919067 US Mountain Standard Time -111.7952347 US Mountain Standard Time -106.6379547 Mountain Standard Time -115.2238922 Pacific Standard Time -115.0660248 Pacific Standard Time -118.3392487 Pacific Standard Time -118.1842194 Pacific Standard Time -118.4004135 Pacific Standard Time -118.4824905 Pacific Standard Time -117.9933014 Pacific Standard Time Page 15 -360 -360 -360 -360 -360 -420 -420 -420 -420 -420 -420 -420 -420 -420 -420 -420 -420 -420 -420 -420 -420 -480 -480 -480 -480 -480 -480 -480 qattachments_796792eee44e0ecf521fde80964ea909164703d8 Olson Timezone GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago Page 16 qattachments_796792eee44e0ecf521fde80964ea909164703d8 GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-07:00 America/Denver GMT-07:00 America/Denver GMT-07:00 America/Denver GMT-07:00 America/Denver GMT-07:00 America/Denver GMT-07:00 America/Denver GMT-07:00 America/Denver GMT-07:00 America/Denver GMT-07:00 America/Denver GMT+000000 America/Phoenix GMT+000000 America/Phoenix GMT+000000 America/Phoenix GMT-07:00 America/Denver GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago Page 17 qattachments_796792eee44e0ecf521fde80964ea909164703d8 GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-06:00 America/Chicago GMT-07:00 America/Denver GMT-07:00 America/Denver GMT-07:00 America/Denver GMT-07:00 America/Denver GMT-07:00 America/Denver GMT-07:00 America/Denver GMT-07:00 America/Denver GMT-07:00 America/Denver GMT+000000 America/Phoenix GMT+000000 America/Phoenix GMT+000000 America/Phoenix GMT+000000 America/Phoenix GMT+000000 America/Phoenix GMT+000000 America/Phoenix GMT+000000 America/Phoenix GMT-07:00 America/Denver GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles GMT-08:00 America/Los_Angeles Page 18 Project: Analyzing a Market Test Step 1: Plan Your Analysis To perform the correct analysis, you will need to prepare a data set. (250 word limit) Answer the following questions to help you plan out your analysis: 1. What is the performance metric you'll use to evaluate the results of your test? 2. What is the test period? 3. At what level (day, week, month, etc.) should the data be aggregated? Step 2: Clean Up Your Data In this step, you should prepare the data for steps 3 and 4. You should aggregate the transaction data to the appropriate level and filter on the appropriate data ranges. You can assume that there is no missing, incomplete, duplicate, or dirty data. You're ready to move on to the next step when you have weekly transaction data for all stores. Step 3: Match Treatment and Control Units In this step, you should create the trend and seasonality variables, and use them along with you other control variable(s) to match two control units to each treatment unit. Note: Calculate the number of transactions per store per week to calculate trend and seasonality. Apart from trend and seasonality... 1. What control variables should be considered? Note: Only consider variables in the RoundRoastersStore file. 2. What is the correlation between your each potential control variable and your performance metric? 3. What control variables will you use to match treatment and control stores? 4. Please fill out the table below with your treatment and control stores pairs: Treatment Store Control Store 1 Control Store 2 Step 4: Analysis and Writeup Conduct your A/B analysis and create a short report outlining your results and recommendations. (250 words limit) Answer these questions. Be sure to include visualizations from your analysis: 1. What is your recommendation - Should the company roll out the updated menu to all stores? 2. What is the lift from the new menu for West and Central regions (include statistical significance)? 3. What is the lift from the new menu overall? qattachments_33d96525b6b98e4a4c430975261522cda042e9ff StoreID Sq_Ft AvgMonthSales Phone Number Street Combined 1664 1475 11000 8478428048 101 W. Main St., Barrington Village Center 1675 1472 15000 8472531188 90 East Northwest Highway 1696 1471 10000 2242232528 1261 East Higgins Road 1700 1465 15000 (224) 500-9575 17 W 633 Roosevelt Road 1712 1456 19000 708-403-1461 15858 South LaGrange Road 2288 1387 14000 3036806304 13702 E Quincy Ave 2293 1394 11000 303-424-0670 9983 Wadsworth Pkwy 2301 1387 11000 3037611675 3500 A River Point Parkway 2322 1390 14000 1331 N Speer Blvd 2341 1400 11000 3034430309 1352 College Avenue, 115 Page 1 qattachments_33d96525b6b98e4a4c430975261522cda042e9ff Street 1 Street 2 Street 3 101 W. Main St. Barrington Village Center 90 East Northwest Highway 1261 East Higgins Road 17 W 633 Roosevelt Road 15858 South LaGrange Road 13702 E Quincy Ave 9983 Wadsworth Pkwy 3500 A River Point Parkway 1331 N Speer Blvd 1352 College Avenue 115 Page 2 City Postal Code Barrington 60010 Mount Prospect 60056 Schaumburg 60173 Oakbrook Terrace 60181 Orland Park 60462 Aurora 80015 Westminster 80021 Denver 80110 Denver 80204 Boulder 80302 qattachments_33d96525b6b98e4a4c430975261522cda042e9ff Region Central Central Central Central Central West West West West West Country US US US US US US US US US US Coordinates Latitude Longitude (42.1540565490723, -88.1362915039063) 42.15405655 -88.1362915 (42.0633544921875, -87.9355773925781) 42.06335449 -87.93557739 (42.039363861084, -88.048828125) 42.03936386 -88.04882813 (41.8600807189941, -87.9739685058594) 41.86008072 -87.97396851 (41.603816986084, -87.8534317016602) 41.60381699 -87.8534317 (39.6378707885742, -104.828552246094) 39.63787079 -104.8285522 (39.8431854248047, -105.08194732666) 39.84318542 -105.0819473 (39.6528053283691, -105.012954711914) 39.65280533 -105.0129547 (39.7375144958496, -104.997352600098) 39.7375145 -104.9973526 (40.007194519043, -105.275108337402) 40.00719452 -105.2751083 Page 3 qattachments_33d96525b6b98e4a4c430975261522cda042e9ff Timezone Current Timezone Offset Olson Timezone Central Standard Time -360 GMT-06:00 America/Chicago Central Standard Time -360 GMT-06:00 America/Chicago Central Standard Time -360 GMT-06:00 America/Chicago Central Standard Time -360 GMT-06:00 America/Chicago Central Standard Time -360 GMT-06:00 America/Chicago Mountain Standard Time -420 GMT-07:00 America/Denver Mountain Standard Time -420 GMT-07:00 America/Denver Mountain Standard Time -420 GMT-07:00 America/Denver Mountain Standard Time -420 GMT-07:00 America/Denver Mountain Standard Time -420 GMT-07:00 America/Denver Page 4 qattachments_33d96525b6b98e4a4c430975261522cda042e9ff Name Right_State Barrington IL Northwest Hwy & Elmhurst Rd IL Higgins & Meacham IL Roosevelt & Summit IL 159th & LaGrange IL S. Parker Rd & E. Quincy Ave CO King Sooper Arvada # 55 CO Hampden & Santa Fe, Sheridan CO King Soopers - Denver #1 CO 1352 College Ave - Boulder CO Page 5