Test\tObjective: Impact\tof\tOffer,\tand\tmodel effectiveness\ton\tcampaign Performance Page\t2 Background The\tmanagement\tof\tApparel\tXYZ\thas\tbeen\tplanning\ton\tgrowing\tits\tbusiness\tover\tthe\tnext\tfew\tyears. The\tSeptember\tfloor-set\tis\tmen's\t&\twomen's\toutwear.\tThere\tare\tseveral\tfactors\tavailable\tfor\ttesting:\tas\tfor creatives,\tthe\tcampaign\twill\tdo\tseparate\tcreative\tpieces\tfor\tmen's\tand\twomen's\toutwear.\tThe\tcustomer pools\tcan\tbe\tsegmented\tby\teither\tA/B/C/D\tor\tby\ta\tranking\tmodel,\tand\tthe\tefficiency\tof\tthe\ttwo\tmodels\tcan be\ttested\tfor\tthis\tcampaign.\tTwo\talternative\toffers\tcan\talso\tbe\ta\tfactor\tfor\ttesting.\tIn\taddition,\tthere\tare three\tways\tof\tmedia\tto\tinteract\twith\tcustomers:\tby\temail\tor\tby\tdirect\tmails,\tor\tby\tthe\tcombination\tof\ttwo. Test\tStrategy According\tto\tthe\tabove\tbackground,\tthere\tare\thundres\tof\tcombination\tin\tterms\tof\ttest\tdesign.\tThere\tare\ttwo options\tin\tcreative\t(men\tcreative/women\tcreative),\ttwelve\tlayers\tof\tmodels\t(ABCD*High/Medium/Low), three\tlayers\tof\tmedia\t(Email/Direct\tMail/Combination\tof\tTwo)\tand\talso\ttwo\tlayers\tof\toffers\t(20%/up\tto\t30% off).\tWith\tthese\tpremises,\tat\tleast\ta\ttotal\tof\t2*12*3*2=144\ttest\tdesign\tcombinations\tare\tavailable. To\tdetermine\tthis,\twe\tconduct\ta\tqualitative\tassessment\tof\tall\tthe\tfactors\tbased\ton\ttheir\timpact\ton investment\tand\teasiness\tto\texecute\tfeatures,\tand\tthe\tresults\tare\tas\tfollows. Model\tEfficiency High\tImpact\ton\tInvestment Offer Media Hard\tto\tExecute Easy\tto\tExecute Creative Low\tImpact\ton\tInvestment Due\tto\tthe\tlimited\tbudget\tand\ttime,\tsince\tthe\tactual\tcampaign\troll-out\tis\tpreminent,\twe\twill\tmake\ta\tstrategic elimination\tof\ttesting. We\thave\tnarrowed\tdown\tour\toptions\tfor\ttesting\tto\tmodel\teffectiveness,\tcreative\tand\toffers.\tAlthough\tMedia is\trelatively\teasy\tto\texecute,\tit\talso\tis\trelatively\tlow\tin\tterms\tof\timpact\ton\tinvestment.\tTherefore\twe\thave eliminated\tthe\tfactor\tof\tmedia.\tWith\tnow\tonly\tthree\tfactors\tto\twork,\twe\twill\tconduct\tproforma analysis\tto project\tthe\tpossible\tfinancial\toutcomes\tof\teach\texecution\tso\tthat\twe\twill\tbe\table\tto\tgarner\tthe\tinformation as\tto\twhether\ta\tcertain\ttest\twill\tbe\tfinancially\trewarding. Page\t3 I. Proforma\tof\tNew\tA/B/C/D\tSegment\tVersus\tRankin\tModel\tSegments Segment test Control Test1 Test2 What do we test? Old A/B/C/D Seg New A/B/C/D Seg Ranking Model Seg Circulation/Impressions 50000 50,000 50,000 Response Rate 1.80% 2.40% 2.54% Buyers 900 1200 1270 Orders 925 1273 1293 Total Gross Revenue $36,325 $52,478 $56,875 Avg. Gross Rev./Order $39.27 $41.22 $43.99 Offer Discounts %% 20% 20% 20% Offer Discounts $$ $7,265.00 $10,495.60 $11,375.00 COGS @ 30% $10,897.50 $15,743.40 $17,062.50 Total Margin $$ $18,162.50 $26,239.00 $28,437.50 Direct Promotion Costs $30,000 $30,000 $30,000 Total Net Margin $$ (Net of Offer, COGS and Promotion Costs) ($11,837.50) ($3,761.00) ($1,562.50) ROI $1 Promo Costs = ($0.39) ($0.13) ($0.05) Diff -0.14% (70) (20) $ (4,397) ($2.8) ($879) ($1,319) ($2,199) $0 ($2,199) ($0.07) The\tNew\tmodels\thave\tdemonstrated\tstrong\timprovement\tagainst\tthe\tcontrol\tsegments,\trendering potential\thigh\timpact\ton\tinvestment. II. Proforma\tof\tWomen\tcreative\tVersus\tMen\tCreative Email Test creatives What do we test? Circulation/Impressions Click Rate Clickers Response Rate Buyers Orders Total Gross Revenue Avg. Gross Rev./Order Offer Discounts %% Offer Discounts $$ COGS @ 30% Total Margin $$ Direct Promotion Costs Control Test1 Test2 Diff Men/Women Women Creative Men Creative 50,000 50,000 50,000 5.70% 5.40% 4.80% 0.60% 2850 2,700 1,200 1,500 2.65% 2.45% 2.35% 0.10% 1325 1225 1175 50 1455 1245 1190 55 $63,325 $50,478 $45,478 $ 5,000 $43.52 $40.54 $38.22 $2.3 20% 20% 20% $12,665.00 $10,095.60 $9,095.60 $1,000 $18,997.50 $15,143.40 $13,643.40 $1,500 $31,662.50 $25,239.00 $22,739.00 $2,500 $35,000 $30,000 $30,000 $0 Total Net Margin $$ (Net of Offer, COGS and Promotion Costs) ($3,337.50) ROI $1 Promo Costs = ($0.10) ($4,761.00) ($0.16) ($7,261.00) ($0.24) $2,500 $0.08 In\tthe\tcreative\tproforma,\twe\tcan\tsee\tthat\tthe\tnew\ttests\tactually\tperform\tworse\tthan\tthe\tcontrol,\tmaking the\tcreative\ton\tthe\tlow\tspectrum\tof\timpact\ton\tinvestment.\tTherefore\twe\tconclude\tthat\tcreative\ttesting shouldn't\tbe\tincluded\tin\tthe\ttesting. III. Proforma\tof\t20%\tentire\tpurchase\tdiscount\tVersus\tup\tto\t30%\toff\tdiscount Page\t4 As\tfor\tthe\tup\tto\t30%\tdiscount,\tit\tis\tassumed\tthat\t60%\tof\tthe\tentire\tproduct\tis\tactually\t30%\toff,\ttherefore\tthe overall\taverage\tdiscount\tis\t30%*60%=18% Offer Test What do we test? Circulation/Impressions Click Rate Clickers Response Rate Buyers Orders Total Gross Revenue Avg. Gross Rev./Order Offer Discounts %% Offer Discounts $$ COGS @ 30% Total Margin $$ Direct Promotion Costs Control Test 1 Test 2 Diff No promotion entire 20% up to 30% 50,000 50,000 50,000 2.30% 5.50% 4.80% 0.70% 1150 2,750 1,200 1,550 1.25% 2.55% 2.50% 0.05% 625 1275 1250 25 725 1325 1315 10 $26,325 $55,345 $54,495 $ 850 36.31034483 $41.77 $41.44 $0.3 0% 20% 18% $0.00 $11,069.00 $9,809.10 $1,260 $7,897.50 $16,603.50 $16,348.50 $255 $18,427.50 $27,672.50 $28,337.40 ($665) $27,500 $30,000 $30,000 $0 Total Net Margin $$ (Net of Offer, COGS and Promotion Costs) ROI $1 Promo Costs = (9,072.5) ($0.33) ($2,327.50) ($0.08) ($1,662.60) ($0.06) ($665) ($0.02) In\tthis\tproforma,\twe\tcan\tsee\tthat\tcompared\tto\tthe\tcontrol\toption\tof\tno\tdiscount,\tboth\ttest\t1\tand\ttest\t2\thave better\tperformance\tby\thaving\ta\tbetter\tROI\tand\talso\tby\tgenerating\tmore\trevenue\tand\tmargin. Test\tDesign There\tare\ttwo\tfactors\tup\tfor\ttesting Segment/offer High ranking Medium ranking Low ranking A B C D A B C D A B C D Current offer 20% of entire purchase Up to 30% off Total Page\t5 For\tfirst\tlayer\tof\tsegment\tfactor,\twe\thave\tA/B/C/D\tsegments.\tFor\tthe\tsecond\tlayer,\twe\thave\tassigned\teach\tof our\tcustomers\tindex\tand\thave\tranked\tour\tcustomers\tinto\tthree\tcategories,\tnamely\thigh,\tmedium\tand\tlow. For\tthe\tsecond\tfactor,\twe\thave\ttwo\tlayers\tfor\ttesting,\tnamely\t20%\toff\tthe\tentire\tpurchase\tand\tup\tto\t30%\toff. Based\ton\tour\tcount\trequest,\twe\twill\tdetermine\tthe\tminimum\tquantities\tper\tcell\tand\ttesting\tperiod\tneeded to\timplement\tand\tmeasure\tresults. Segment\tpromotion\tcode\tassignments In\torder\tto\tuniquely\tidentify\t(and\tmeasure)\tthe\trecords\tby\ttheir\tspecific\tpermutations\tof\tsegment\tand\toffer, we\tneed\tto\tassign\tand\tpopulate\tthe\tfollowing\tproposed\tpromotion\tcodes: A:\tSegment\tA B:\tSegment\tB C:\tSegment\tC D:\tSegment\tD H:\tHigh\tranking\tsegment M:\tMedium\tranking\tSegment L:\tLow\tranking\tSegment CO:\tCurrent\toffer 20:\t20%\toff\tentire\tpurchase 30:\tup\tto\t30% Segment/offer High ranking Medium ranking Low ranking A B C D A B C D A B C D Current offer 20% of entire purchase Up to 30% off Total A(H)CO B(H)CO C(H)CO D(H)CO A(M)CO B(M)CO C(M)CO D(M)CO A(L)CO B(L)CO C(L)CO D(L)CO A(H)20 B(H)20 C(H)20 D(H)20 A(M)20 B(M)20 C(M)20 D(M)20 A(L)20 B(L)20 C(L)20 D(L)20 A(H)30 B(H)30 C(H)30 D(H)30 A(M)30 B(M)30 C(M)30 D(M)30 A(L)30 B(L)30 C(L)30 D(L)30 Performance\tMeasurement We\twill\tmeasure\t(by\tsegment)\tthe\timpact\tof\tthe\toffers\ton\tcertain\tKey\tPerformance\tIndicators,\tand\twe\thave summarized\tthem\tas\tprimary\tKPI\tthat\tneeds\tto\tbe\tprioritized\tand\tsecondary\tKPI. Page\t6 As\tstated\tin\tthe\tback\tground,\twe\tknow\tthat\tthe\tbusiness\tgoal\tis\tto\tgrow\tbusiness\tfor\tApparel\tXYZ.\tTherefore we\thave\tdeveloped\tPrimary\tKPI\tas\tIncrease\tsales,\trevenue\tand\tmargins.\tAlso\tin\torder\tto\tgrow\tbusiness, Apparel\tXYZ\tcan\talso\tacquire\tnew\tcustomers\tthrough\tconversion.\tTherefore\tincreasing\tconversion\trate\tis another\tprimary\tKPI. l Increase\tSale: n Total\torders n Total\tunit n Total\tGross\tRevenue l Conversion: n Click\trate n Click\tthrough\trate n Response\trate Deliverables Under\tthis\tstatement\tof\twork,\tour\tcompany\twill\tprovide\tspecific\tdeliverables\tto\thelp\tyour\tbrand\tunderstand the\timpact\tof\tdifferent\tshipping\toptions\ton\tcustomers. These\tdeliverables\tinclude\ttest\tdevelopment\tand set-up,\tdata\tmanagement,\tresults\tanalysis,\tand\tfinal\trecommendations/next\tsteps. At\tthe\tconclusion\tof\tthe project,\tour\tcompany\twill\tconduct\ta\tformal\tonsite\treview\tof\tfindings,\taccompanied\tby\ta\tPowerPoint presentation\twith\tdetails\taround\teach\tof\tthe\tdeliverables\tdescribed\tabove. Implementation: DESCRIPTION CATEGORY APPROXIMATE TIMELINE Test\tSetup Deliverable 1-2\tweeks Task (TBD) Deliverable 1\tweek - Determine\tsample\tsizes - Finalize\ttest\tparameters\tand\tdesign - Setup\tcoding Data\tcollection Performance\tAnalysis\tand\tPresentation Background X.com marketing director has consulted with a few agencies and discussed potential impact of building a few landing pages to maximize engagement, registrations and net sales on the retail website. You were hired as a consultant to help the director make a decision. The marketing manager decided to test 4 landing pages: A - \"On Sale\" products without an offer B - \"On Sale\" products with a strong offer: 30% off C - Hot New Merchandise: New Arrivals with an offer 20% off D - Hot New Merchandise: New Arrivals without an offer E (Control) - Current frontpage Assumptions: 1) All landing pages include registration bottom 2) All landing pages have a similar creative design 3) Average order revenue = $125 4) Landing page subsequent click rate off the current frontpage = 1.4%, and in external media we don't mention any promotions or discounts. 5) COGS = 30% of Gross Revenue 6) Production cost of 1 landing page = $3000 Definitions: - Engagement is defined as number of pages per visit. - Registrations are defined as number of visitors who registered but not purchased. - Net Sales is total revenue minus ( COGS, total offer costs and promotion costs). Questions/Deliverables: Write a test design including the following sections: - Marketing objectives - The profit drivers: Clearly define what are the KPIs - Detailed test design: what do we test, how do we test? o - - Test administration and implementation: o Determine at 95% confidence internal o What are the minimum samples sizes needed per cell? o Please provide start and end dates for the testing Proforma P&L o - Hypotheses testing: write at least 4 H0/H1 pairs Please develop a Proforma P&L to assess the viability and test priorities Primary and Secondary KPIs