1. Briefly decribe on trap that Denrell suggests managers can fall into from the use of biased information as discussed in the article 2. Denrell suggests that managers in emerging industries may be lee suscptible to selection bias. In one or two sentenses, explain his reasoning.

HBR AT LARGE - Selection Bias and the Perils of Benchmarking read books about the manufacturing strategies of the Japanese innovators of that time. Where the Dangers Lie What kinds of traps do managers fall into when they rely on biased data? Three are likely. Perhaps the most prevalent mistake is to overvalue risky business practices. The problem is easy to see in the exhibit \"The Effects of Bias,\" which illustrates what happens when trends are drawn from incomplete data elds. The graphs plot the relationship be- tween engaging in a risky organizational practice and subsequent corporate per- formance. The rst graph records data from all companies that have ever im- plemented the risky practice, while the second one excludes companies that failed. As you might expect, the perfor- mance of rms that do not employ the practice at all is relatively stable. But the greater the degree to which rms engage in the practice, the wider the gap between the successful and unsuc cessful companies becomes. as perfor- mance either spikes or plummets. 0n average,though, as the trend line shows, engaging in the risky practice somewhat reduces performance. Now suppose that we observed this industry only after many of the worst- performing companies had gone out of business or had been acquired by other rms. In that case, we would have seen the successes but few of the failures asso- ciated with the risky practice. As a result, the observed association between the risky practice and performance would be positive, as the second graph shows the reverse of the true association. Lee Fleming aptly illustrates this dy- namic in his Harvard Business Review article "Perfecting Cross-Pollination" (September 2oo4). Fleming nds that, on average, the value of innovations coming out of diverse, cross-functional teams is lower than the value of inner vations produced by teams of scientists whose backgrounds are similar to one another. But the innovations that the more heterogeneous teams produce tend to be either breakthroughs or dis- mal failures. In fact, the distribution of the innovation values as the diversity of team members increases looks quite similar to what we see in our rst graph. In most instances, however, data on failed projects are not available, at least about failed projects in other compa- nies, so most managers would be able to observe just the distribution pattem we Many industries work the same way. For example, a telephone company or a software rm that had a large market share in 2004 will probably also have a large market share in zoos, owing to customer inertia and switching costs. Thus, even if managers do a poor job in 2005, such a company might still turn in high prots, as managers coast on their past accomplishments or good luck. Focusing on stock market returns in- stead of prots mitigates this problem, since changes in stock prices, in a well- functioning market, do reect changes How many managers study the corporate collapses of the 19805? Yet they still read about the Japanese manufacturing strategies of that time. see in our second graph. As a result,they would overestimate the value of cross- functional teams. Only by collecting data on both successes and failures, as Fleming did, could they spot the risks involved in using cross-functional re- search teams. (Another example of the same dynamic is described in the side- bar, " How Wrong Can You Get?") A second trap for unwary managers arises from the fact that performance often feeds on itself, so that current ac- complishments are unfairly magnied by past achievements. To see how this works, imagine that a company is a run- ner competing against other runners. 1f the runner wins ten independent races. he is probably better than the others, who can learn from him. But suppose instead that the outcome of one race affects subsequent races. That is, if the runner wins by one minute in the rst race, he gets a one-minute head start in the next race, and so on. Clearly, win- ning ten such races is less impressive, since a victory in the rst race gives the runner a higher chance of winning the second, and an even higher chance of winning the third, and so on. jerker Denrell is an assistant professor of organizational behavior at Stanford Gradu ate School of Business in Stanford, Cahirnia. A more-derailed discussion of the concepts in this article can be found in his paper, \"Vicarious Learning, Undersompling of Fail- ure, and the Myths ofManagement,\" Organization Science, Maylune 2003. 116 in performance. But dening success by stock market returns introduces other problems. As Wharton professor Sidney Winter has pointed out, a company's stock price will hold steady when one excellent CEO succeeds another. How- ever. the share price will increase when the company exchanges an inferior CEO for a better. but still substandard, CEO. Maintaining excellence, in other words. might be less well rewarded than be- coming merely mediocre. A third problem with looking only at high performers for clues to high per- formance is the issue of reverse causal- ity. Data may. for instance, reveal a strong association between the strength of a company's culture and its perfor- mance. But does a strongculture lead to high performance or the other way around? The chicken-and-egg problem is especially knotty in this instance since high performance in itself affects cor- porate culture in several ways. To begin with, it's probably easierto build a team- based culture in a healthy rm than in a failing one, where workers are likely to be demoralized and disloyal. High- perforrning companies also can afford to institute programs and practices that low-performing rms cannot. Some of these expensive and time-consuming activities might actually reduce perfor- mance at struggling companies. HARVARD BUSINESS REVIEW Selection Bias and the Perils of Benchmarking - HBR AT LARGE The Effects of Bias What happens when people draw conclusions from incomplete data? Sup- p05e you are investigating the relationship between corporate performance and a particular risky business practice, such as using cross-functional teams. lfyou plotted the results of all companies that engaged in this practice (or any such practice),you would nd that the more widespread the practice, the more volatile company performance, and your graph would look like this: All Companies h Igh Company . zero performance Performance trend line low low Prevalence of risky 7 -) high business practice On average, as the trend line indicates, any such risky practice is correlated somewhat negativeiy with performance. But suppose that you looked only at existing companies and excluded all those that had gone out of business while engaged in this practice.Then your graph would look like this: Only Existing Companies high trend line Company zero performa nce performance low low - Prevalence of risky business practice >high Now the trend line will indicate a positive correlation,which ofcourse is not the case. Thus selection bias will cause you to draw precisely the wrong conclusion. APRIL 2005 What's more, managers' expectations of performance may influence their choice of strategies and thus confound interpretations of the effect those choices have. As William Boulding and Markus Christen observed in \"First-Mover Dis- advantage\" (HBR, October 2001). com- panies that have innovative products and strong distribution capabilities often choose to enter new markets early. Their strong products and capabilities produce high returns. As a result, how- ever, their managers associate early entry with high performance in all cases, even when there is a rst-mover disadvantage. Bias, Bias Everywhere Many of the popular theories on per- formance are riddled with selection bias. One of the most enduring ideas in management, for instance, is the no- tion that successful rms are those that focus most of their resources on one area or technology rather than diver- sifying. Books such as in Search of Ex cellence, Built to Last, and Prot from the Core all recommend that managers \"stick to their knitting\" and \"focus on the core." Typically, the research studies behind these books look only at existing com panies or even more narrowly only at highly successful companies. As a re- sult,their authors overestimate the ben- ets of focus. Consider, for example, Chris look and James Allen's nding in Pror'om the Core that 78% of all high performance rms focused on one set of core activities while only 22% of lower- performance rms did. The study com- prised some 1,854 companies, judging high-performance according to share price returns, sales, and prot ratios, but it included only businesses that survived throughout the study period. It did not, therefore, consider any company that started with a focused strategy but then failed. Including those failures would have changed the picture substantially. Ac- cording to Zook and Allen, 13% of all rms achieved high performance, of which 78% or 188 rms focused on the core. lfin that period just 200 other 117 HBR AT LARGE - Selection Bias and the Perils of Benchmarking companies with focused strategies that had gone out of business had been in- cluded in the sample, then the true re lationship between focus and perfor- mance would be the precise opposite of the one Zook and Allen infer. Another fond notion often lauded by management gurus and the popular press is that CEOs should be bold and take risks. Indeed, many stories in the business press celebrate the intuition of certain great leaders. No less an au- thority than Jack Welch entitled his autobiography Straight from the Cut. Some leaders notably Sony's Akio Morita have gone so far as to eschew market research altogether, believing their instincts are a better guide to mar- ket changes. It's certainly true that companies can be handsomely rewarded when their CEOs take big risks. Suppose you are operating in an industry fashion. say. or consumer electronics where rst movers have an advantage but where there is also considerable uncertainty regarding consumer preferences. To gain rstmover advantage. a company must act quickly. The top-performing companies will be those that. led largely by the instincts of their senior manag ers, are lucky enough to launch products that happen to appeal to customers. But the worst-performing companies will also be those that act on hunches and happen to launch products that don't appeal to customers. Since few people advertise their failures, and many of these unfortunate rms cease to exist. we hear mainly about the 5qu cess of decisions based on gut feelings and little about the countless"visionar- ies"who similarly tried to revolutionize industries but did not. The point here is not that all the pop- ular theories about performance are wrong. I don't know. There may be a genuine link between success and focus. In some industries, the strength of a cul- ture may matter regardless of its nature. And the instincts of some managers may be as sound a basis for strategic de cision making as any amount of analy- sis. But what I do know is that no man- agers should accept a theory about US business unless they can be condent that the theory's advocates are working off an unbiased data set. Fixing the Problem The most obvious step to take to guard against selection bias is to get all the data you can on failure. Within your or- ganizations, you must insist that data on internal failures be systematically collected and analyzed. Such informa- tion can otherwise easily disappear be- cause the people responsible may leave the organization or be unwilling totalk. Looking outside your company, you should extend your benchmarking ex- ercises to include lessthan-successful rms. Industry associations can help you collect data about failures of new prac- tices and concepts. Despite your best efforts, it's unlikely you can ever be completely condent that your data are unbiased. Fortu- nately, you do have some backup, be- cause economists and statisticians have developed a number of tools to correct for selection bias. These tools, however, are grounded in certain assumptions, How Wrong Can You Get? S: which may be more or less realistic, de- pending on the context. Suppose,for example,that we want to estimate the average return on equity of all companies in a given industry, but we have available only the ROE data of sur- viving rms. Since lowprot businesses are more likely to fail,just taking the av- erage ROE of all surviving rms would lead to too high an estimate. But sup pose we assume that ROE is distributed along a standard bell curve and that all businesses with a negative ROE will fail. Then we can use the data we have to estimate the average ROE for all rms because the information on hand is enough to tell us how steep the curve is, how broad. and what the average is. This approach can be used to correct for bias in any situation in which we can apply formal statistical tools. For in- stance, let's say we suspect that in a par- ticular industry, the more training a company's sales staff gets, the higher the average salesperson's performance will be and the more consistent the en- tire sales staff's performance will be. Suppose further that we have detailed During World War II,the statistician Abraham Wald was assessing the vul nerability of airplanes to enemy re. All the available data showed that some parts of planes were hit disproportion- ately more often than other parts. Mill, tary personnel concluded. naturally enoughI that these parts should be rein forced. Wald, however, came to the ot} posite conclusion: The parts hit least often should be protected. His recom- mendation reected his insight into the selection bias inherent in the data. which represented only those planes that returned. Wald reasoned that a plane would be less likely to return if it were hit in a critical area and. conversely, that a plane that did return even when hit had probably not been hit in a critical loca- tion. Thus, he argued, reinforcing those parts of the returned planes that sus- tained many hits would be unlikely to pay off.1 I. The Wald story is one of the most widely cited anecdotes m the statistical community. To nd out more about it, see w Allen WJIIIS' " The Statistical Research Group, \")4: 4945.\";loomatoj them-neuron StoreshcaMssorrarmr-i, June 1930, and M. Manqel and El. Samameqo 'Aoraham wald's work on Aircraft Survivahiliiy.",r:.vurnorojthMmencan Statistical Association. June 1934 HARVARD BUSINESS REVIEW Rotman data on the investments in training made by most firms currently operating 20:20 VISION - WHAT WILL THE in the industry. If we can safely assume [BUSINESS] WORLD LOOK LIKE IN 2020? that performance follows some speci- A Conference for Thought Leaders fied distribution pattern, we can in prin- JUNE 3, 2005 - ROTMAN SCHOOL OF MANAGEMENT, TORONTO ciple use the data we have to obtain an unbiased estimate of how investments CONFERENCE CO-CHAIRS: ANDREW GOWERS, Editor, Financial Times; Author in training actually do influence the av- ROGER MARTIN, Dean, Rotman School & Director, AIC Institute for Corporate Citizenship @ Rotman; Author erage level and variability of sales staff SESSIONS: performance, even if we do not have "The Future and Its Enemies: the Conflict Over Creativity, Enterprise and Progress" data on firms that failed. Essentially, VIRGINIA POSTREL, Author, The Substance of Style; The Future and Its Enemies; what we are doing is inferring the shape Columnist, The New York Times, Forbes of a particular iceberg by observing its "The Rise of China and India: Global Implications" WENDY DOBSON, Director & Professor, Institute for International Business, Rotman School; Author tip and making (we hope) a reasonable "Measuring the Effectiveness of Boards: Shining Light on Unlit Areas of Corporate Endeavour" assumption about the relationship be- TIM ROWLEY, Deloitte & Touche Professor of Strategic Management & Director, tween the tips of icebergs and the rest Clarkson Centre for Business Ethics & Board Effectiveness, Rotman School of them. "Beyond Offshoring: Assessing Your Company's Global Potential" JOANNE OXLEY, Associate Professor of Strategic Managem ent, Rotman School; Author The pioneer of these statistical meth- "The Flight of the Creative Class: The New Global Competition for Talent" ods was James Tobin, winner of the 1981 RICHARD FLORIDA, Author, The Flight of the Creative Class; The Rise of the Creatve Class; Nobel Prize in economics. His work was Hirst Professor of Public Policy, George Mason University later built upon by James Heckman, "Irrational Exuberance Part 2: Risk in the 21st Century" ROBERT SHILLER, Author, Irrational Exuberance; The New Financial Order, who himself received a Nobel Prize in Stanley B. Resor Professor of Economics, Yale University 2000 for his contributions in this area. COST: C$1000 per person; no charge for Rotman School MBA and PhD alumni In recent years, management scholars COMPLETE DETAILS: www.rotman.utoronto.ca/2020 have applied these methods to correct for selection bias in their own research Joseph L. Rotman School of Management University of Toronto and have started to advocate for their use in the broader managerial com- munity. In "Getting the Most out of All Your Customers" (HBR, July-August 2004), for instance, Jacquelyn S. Thomas, Werner Reinartz, and V. Kumar demon- strate how such tools can be used to im- prove the cost-effectiveness of market- ing investments. Cautionary words and counsels of fail- ure, I know, are seldom well received. Managers crave certainties and role models from business literature, and to some extent they have to. They live in a fast-paced world, and they often cannot afford to postpone action until they get Smart managers don't have all the answers- better data. But there really is no excuse for ignoring the glaring traps we've de- they just know where to find them. scribed in these pages. Success may be more inspirational, but the inescapable Case in Point" Leadership challenges from a real world perspective logic of statistics dictates that managers in pursuit of high performance are more The Essential Leader" likely to attain their goal if they give the A suite of interactive online programs to build high performance leaders stories of their competitors' failures as Harvard ManageMentor" full a hearing as they currently do the An online resource with more than 35 essential business topics stories of their successes. Reprint RO504H Call 1-800-795-5200 to receive your HARVARD free brochure, or visit BUSINESS To order, see page 135. SCHOOL Priority Code 9763 www.elearning.hbsp.org. PUBLISHING elearning APRIL 2005