Experiments in Organizations (4) What is the unit of randomization? Is it a cluster randomization? Unit of randomization: All members of the same unit are assigned to the same treatment group (e.g., day, store, customer) . Cluster randomization: Unit of randomization is larger than unit of outcome (e.g., randomize by store and measure purchases by customer) (5) What is the total sample size? What is the total number of clusters (if applicable)? Ex. Randomize by store: Total sample size is 1,000 customers, total number of clusters is 10 stores Does the sample size seem large enough? (6) Are there any concerns with the design and randomization? Selection: The people in the different treatment groups may not be identical at baseline - i.e., they have pre-existing difference that may partially determine which group they end up in and may also affect the outcome (e.g., people who shop at store(s) assigned to treatment are more likely to purchase than people who shop at store(s) assigned to control). Spillovers: The people in different treatment groups might interact or learn about the other groups in ways that affect the impact of treatment Effects of being in an experiment: c.g., novelty, know you're in an experiment or being watched, know your behavior will affect policy, experimenter demand -- participants behave the way they think the experimenter wants them to Results (1) What is the primary outcome? What are the main results, including direction, size and significance (statistical and economic) of the treatment effects? (2) Are there any interesting secondary effects (subgroups, interim, longer term, spillovers)? . Are there any important outcomes missing? (3) What are possible mechanisms driving the results? Do you agree with the authors' interpretation of the results or could something else be going on? Discussion (1) What are the best features of the experiment? What are its shortcomings? Do you think the results would generalize? (2) What should the organization do next? Refine the design? Scale it up? Run it for longer? Roll it out? What are challenges to organization wide implementation? (3) How could you use the results of the experiment? Is there another context where this could be interesting? How would you adapt or improve upon the design