Please read the question

Question: What is the main points discussed and write a 1-2 paragraph summary on what is your opinion. Make sure to include a a brief description of replication and why it is important in research. Also give your opinion about the the idea of stereotype threat and the research behind it. Do you still think some of the findings are valid, why or why not? What, if anything, can we learn from it to influence our work in classrooms with diverse students?

Scenario:

JA: Do you can I ask you do you do you just want to like lay out for us the chronology of your obsession. I think I feel comfortable saying it was. Has it become an obsession or just a just a dalliance? DE: I mean. SW: I mean or maybe you just noticed a crumbling building and ran over to stick your pen... DE: I mean I am, I'm a contrarian. And I'm interested in alternative facts about science let's say. (laughs) JA: Hey, I'm Jad Abumrad. RK: I'm Robert Krulwich. This is Radiolab. JA: And a little while back, our editors Soren Wheeler and I ... we talked to science journalist named Dan Engber who got us kind of tangled up about something that we thought we knew about about the world, about ourselves... RK: Something beautiful as I recall. JA: Yeah, that we talked about at great length on this show. CS: Okay. JA: Hello? CS: Hello? SW: And to set that conversation up we're gonna start with this guy. JA: Hi is this Claude Steele? CS: Yes. SW: Professor Claude Steele. CS: I'm the Lucie Stern's emeritus professor of psychology at Stanford University. CS: Yes, exactly. [laughs] So the the topic is a sort of family birthright. SW: We actually had him on the show number of years ago CS:Some time ago, I can't remember how many years ago. But- SW: But then years and years later, midway through his career as a psychology professor, Claude ran into a demonstration of the power of race that really surprised him. CS: I got a job offer - this is in the 80s at the University of Michigan. And it was part psychology and part to administer a minority student program there. JA: Yeah it's been a - it's been a - CS: Uh- JS: it's been a long time. I'm looking at the sheet here and it says SA: This is Claude Steele in the original program that we ran back in 2009. 2009. CS: And um in the process - I saw data that surprised me. CS: Whoa! JA: That it was JA: What he saw was a troubling trend. Two kids would enter Michigan. One was black, one was white. They'd come in at the exact same level. CS: That blows my mind. CS: Same skills. Same SAT score. JA: Yeah that was a long- JA: So theoretically they should do the same when they get to Michigan. But, without fail or almost without fail, after one semester. CS: I was a lot younger man then. JA: [laughs) i know. CS: The black kid was winding up with lower grades. JA: How much lower? SW: And the reason that we had him on the show back then was because he had done a study in the mid-90s that just completely changed the way that we thought about the power of stereotypes. CS: Pretty pretty um - pretty dramatic. At least two thirds of a letter grade. JA: Meaning if the white kid got an A, the black kid who should be getting an A, too, is instead getting a B CS: Well, my mother was white, my father was African American they - and they were very active in the civil rights movement. So you can imagine race was no distant or passing thing in in our life and family. CS: That's right JA: Or a B+, SA: For Claude, growing up, that was just everyday dinner table talk. JA: That's significant. CS: That's significant. That's significant. TV show: The Bell Curve, intelligence and class structure in American life. JA: And he also by the way saw this performance gap between women and men when it came to math JA: To the same degree? SW: The Bell Curve argued that one explanation for the achievement gap among others was genetic and IQ-based. Now, of course, it's not. But even though there's no scientific backing at all for genetic differences like this. The Bell Curve was still significant just because of the kinds of conversations it was creating at the time and the effect that it had on researchers in this field. CS: The same degree in advanced math courses it was comparable. I learned this is a national phenomenon. If if I was to walk into almost any college class in the United States uh - I'd have a very high probability of finding exactly that CS: Well, The Bell Curve is one point in a long history of that kind of argument ... DE: So I think it's important to put it in context of what was going on at the time... (FADES UNDER) SW: This is Dan Engber again and he says that that gap in achievement between black and white students that Claude had noticed, that was actually a huge topic of conversation at the time. that the difference between groups really is rooted in genetic differences let's just be frank and honest and if you really can't admit that, then you don't have the courage needed to be a real scientist. That's that's a thumbnail way of describing this experience that I've had being confronted with with that notion. SW: And this was obviously disturbing to Claude, well, first on a personal level DE: there's a lot of discussion of what to do about the achievement gap and the familiar argument, is well this has to do with systemic racism and systemic differences in opportunities that play out through an individual lifespan. Now that seems right. It's also daunting because how are we ever going to like cure all of the socioeconomic disparities in this country...? CS: in order to be a scientist are you supposed to actually be open to the possibility that you and your family and your whole race have some genetic limitation? SW: But also because it was so weak scientifically, That's a big problem yeah SW:. But then, in 1994, a different, and in many ways, very dangerous ideas was being tossed into this debate. CS: It's been very difficult, impossible to produce anything like definitive data that the differences in test scores between groups is genetic. Reporter: Charles Murray - coauthor of the book The Bell Curve, SW: When The Bell Curve came out. JA: So Claude Steele thought all right - step one complete, I've got a lab situation that resembles the real world, good. Now the next step is to tweak things a little bit, see if I can mess around with it. JA: Now, normally in these situations - SW: But while the differences between these students that Claude was seeing at Michigan, it also didn't seem to him like it could be explained simply by their backgrounds or their opportunities. CS: Because, you take, say, a black kid and white kid at Michigan. They both have extraordinary scores like, you know, they're in the 98th percentile on their SATS. So, um, the background between the two kids, whatever it is, has not resulted in a difference to that moment in time. So, if going forward and taking a test, black kid gets a lower score or a lower performance in a course of some sort, then something must be happening right there, right there. SW: Something must be happening in the moment. CS: There was something there that people didn't understand and that we certainly didn't understand. CS: The test giver's got a white lab coat on, and he brings in a big stack of cellophane wrapped tests. And he puts a clock on the table - and it's all- it's all - you know it's like - that's - [laughs) that's gonna intimidate almost anybody! JA: Maybe that's what's happening, he thought. What if I took away the clock, took away the coat and most importantly, right before the test, I had the test giver instead of saying the normal - I'm going to give you a test, pre-test thing. Maybe instead, say something like this. CS: Look you may have heard that uh women don't do as well as men on difficult standardized math tests. You may have heard that. But that is not true for this particular test. JA: So he figured he would start with the women in math issue. He brought a bunch of women in and a bunch of men. Sophomores. CS: Brought them into the laboratory one at a time. Gave them a half an hour section of the graduate record exam you take if you're a math major. Very, very difficult math. JA: Ohh. CS: This particular test does not show gender differences, never has never will. JA: Mm. JA: He wondered if maybe saying that simple sentence before giving the test would have an effect. CS: And sure enough, the women who had all the same credentials coming into that situation performed dramatically worse than the men. JA: Worse as in- CS: It'd be a couple hundred points on an SAT test CS: And sure enough, I wouldn't be here if their performance didn't go up to match that of the equally skilled men. JA: That performance gap totally vanished. CS:Sheee look at this thing. So we raced and did it very quickly, the same kind of an experiment with African Americans. JA: Big difference. CS: So this was - this was a big effect. JA: There, the pre-test disclaimer went like this, CS: This is an instrument that we use to study problem solving and it is not diagnostic of individuals intellectual ability. As it is, as an IQ test, it's in fact seen as the gold standard of IQ test because it's quote culture free. JA: In other words, this is not a test of your intelligence. I repeat not an IQ test. ED: There's no math, there's no reading. CS: Because it doesn't involve language. Uh or you could represent the exact same test as a puzzle. CS: So just do the best you can. JA: And with that simple disclaimer at the start- RB: Puzzles. CS: Same kind of an effect. ED: Puzzle. JA: The black students and the white students were now equal. JA: Meaning you can give an IQ test to a bunch of kids and the black students will perform worse. But if you give that same test - lose the word test, lose the word IQ, and just call it a puzzle? CS: Just recently uh Ryan Brown and Eric Day did a even clever-er treatment. They - there is an IQ test, which is nonverbal. RB: The black participants suddenly jump up in their performance. RB: It's called the advanced progressive matrices. ED: Basically we got a reversal. CS: When you represent it as a puzzle, blacks perform as well as whites. CS: It has figures. ED: Very abstract, they got lines crossing. RB: They - they did, yeah. CS: That you have to match and so on. ED: Checks uh- JA: That's all it takes. Just change a few words. JA: Stereotypes are powerful. Okay, that makes sense. That - but in terms of understanding how this works, can you make this tactile for me? Like if the stereotype that's having all these effects is a thing - like a - RB: It's essentially pattern matching. ED: Diamonds with dots in em. It's - ED: Mmhm. RB: Totally visual. JA: Like a little gremlin that bites? Like when in the test taking process, does it actually like do its damage? ED: Yeah it's very - CS: And so they could represent that test - ED: That - that's JA: When does it actually RB: I don't know if you can do this. CS: Oh s****, is what they say about us true? ED: gonna - that's gonna be way open to debate. JA: What does seem to be clear from the data, according to Eric Day and Ryan Brown and Claude Steele, is that the gremlin only seems to appear when the test is sufficiently hard. RB: They don't think you can do it, JA: All the usual stuff. And even if the student doesn't believe it, which is likely RB: See you don't have to believe it, that's the kind of insidious thing CS: If the test is easy, it's important to point out uh these effects don't happen. JA: It's not that the gremlin is not there. here. JA: Just the fact that he has now this extra bit of mental chatter. RB: Well he walks in with you, but he doesn't speak necessarily until things get challenging RB: That little guy whispering. CS: As soon as it - the test gets difficult, RB: That's where the voices kick in, JA: Well it's a distraction. CS: And that makes their performance go down JA: Just a little bit. JA: Which means that for most of the test, everybody's doing about the same. It's only at problem number 17. The one about... cosines and factorials and whatever, where things start to go wrong, and - at least that's the theory. At that problem, the black student starts to stiffen up a little bit. CS: That's right. JA: And Claude Steele's measured this. RB: All of this dialogue is keeping you from being 100 percent focused on the task at hand which is solving these problems. JA: So the real subtle power of a stereotype isn't that it prevents you from doing a thing you want to do. It distracts you for just a beat from doing the thing you want to do and that may be all the difference. SW: So that's how we ended the piece that we did back in 2009. But in the years after Claude did that original study, the effect, which he called "Stereotype Threat," became one of the biggest... Male voice: Stereotype threat- CS: Their blood pressure's elevated, their short term memory is impaired. JA: It's that flicker of frustration through their body that wakes up JA: the gremlin who starts - to whisper in their ear. SW: and most important ideas Female voice: Stereotype threat. SW: in all of social psychology. Male voice: Now some psychologists say, stereotypes can become self- fulfilling prophecies. SW: And Claude Steele... MI: a professor of psychology at the University of Toronto. JA: So when did you begin -- if you could give us sort of the cliff notes version of your history with stereotype threat in particular? MI: I mean I think... I was certainly attracted to that that part of social psychology that dealt with prejudice and discrimination, so I'm Jewish, I went to you know, Jewish day school and high school and kind of perhaps baked into me was a desire to, for social justice, um, and you know seeing the evils of prejudice and how those evils taken to their logical extreme, what could happen. So I was passionate about the topic. I, um ... so it seemed like a very hopeful, sort of explanation that also offered relatively easy solutions to fix. JA: Sounds like you really came into it to with a very social, political sort of bent to it. Male voice: Ladies and gentlemen it's my great pleasure to present, Claude Steele. SW: became a sort of academic rockstar. Male voice: Professor Claude Steele. Female voice: Dr. Claude Steele. SW: Speaking to overflowing audiences at places like Columbia and Cornell. MI: Yes, that's right. I wanted my work to have an impact. I wanted it to, yeah, to change the world. JA: Coming up, after the break, the Michael tries to change the world but the world kind of changes... Male voice: Welcome Dr. Steele. RK: him... Female voice: Dr. Steele. JA: Him, yeah... JA: Jad SW: And this idea of a stereotype threat was shown to be relevant in cases that had to do with age and socioeconomic status. There were studies about women playing chess, men being tested on like social sensitivity. I mean, Claude's work ended up inspiring sort of a whole generation of social psychologists. RK: Robert MI: Yeah, I would say that the original stereotype threat paper by Steele and Aronson blew me away. It just spoke to me and was beautiful and it seemed to offer answers, you know, to questions that troubled me. JA: Radiolab. We're back and we're talking about Claude Steele's seminal research stereotype threat: how the threat of negative stereotypes can impact a person's academic performance. SW: So this is Michael Inzlicht... SW: But along the way he says, he noticed some things that troubled him. SW: Right and uh just before the break we were talking to Michael Inzlicht who, after getting into stereotype threat research, he went onto grad school and really focused in on stereotype threat as his field of choice. SW: And weirdly enough it all started he says, when he moved to Toronto, MI: Yeah, so... MI: I started my - my current job at the University of Toronto in 2005 - DE: How many studies of stereotype threat did you end up doing? MI: I would say in the order of, like, in terms of numbers of studies 20 maybe? 15 to 20? - something like that? SW: And while he was there he was running a lot of his stereotype threat research on women in math you know giving women a test and then doing some intervention to reduce the stereotype and seeing if there would be a difference in their test scores SW: He did a lot of studies on women's performance in math but also just looking at different environments and how they create or encourage these stereotypes. And over the course of his career he ended up editing a book about stereotype threat. MI: And, simply put - SA: For the first time - MI: Oh, well yeah... yeah, yeah... MI: I was not able to consistently show any effect of, of, of stereotype threat. SW: Gave numerous talks on the subject. JA: Hm. MI: All over the world SA: He even signed a brief, uh - kind of explaining stereotype threat SA: In other words, the women who had received this stereotype-threat- reducing intervention performed just about the same as the women who hadn't. MI: to the Supreme Court of the United States - Male reporter: The Supreme Court heard the oral argument in Fisher v. University of Texas at Austin - MI: So I think when I first got here, I failed to replicate some of these effects, and like let me go back to the drawing board. Let me, you know, think about you know-what is my-how is my population different? SA: And his first thought was maybe it has something to do with Toronto. MI: So Toronto uh - incredibly diverse place. SW: As it applied to the question of affirmative action Male reporter: The court will decide whether race used in university and college admission policies is constitutional. SW: like take the freshman pysch classes he was teaching. MI: Um so I - you know put my name to this, SA: But then MI: We're talking about you know a third East Asian, a third South Asian, and only about 15% Caucasian or Eur-European Canadian. And then a little bit of everything else. SA: So he started wondering, SW: ... how just by doing some very standard practices in psychology research... MI: To what extent do our students even ha - are they aware of these stereotypes? About, you know, women and math? Um, and remarkably, when I would ask, you know, our students this MI: ... using these techniques you are taught explicitly in grad school ... SW: you could sometimes end up with these sort of ludicrous conclusion. SA: Like he'd ask, you know, Who here has heard that women tend to be worse at math? JA: So you're saying there were this paper was showing ways in which experimenters could subtly unknowingly juice their results. Is that essentially what it was? MI: I would say no more than like a quarter to a third had you know a strong awareness of this stereotype? MI: Yeah, essentially... SW: So he started running the experiments again but SW: So the paper was pointing out that it becomes sort of standard practice that you know when you were researching for some effect like oh how happy people were or how well they did on a test that you would measure that thing in a couple of different ways. MI: I'd only pick those women and men who actually had awareness but even then. I still then couldn't get the effect. I was like, okay, well - maybe I'm doing something wrong, SW: Like maybe the interventions he was designing weren't reducing the threat or with these students the threat just wasn't that much of a threat in the first place. But by this time Michael had already done a ton of research on stereotype threat and MI: you know, I'm I'm a person who gets bored eh rather quickly um, and I just you know, started, eh - started losing interest. MI: Which to some extent is considered good practice, right? You want to, no one measure captures your construct perfectly. So you should measure that thing in as many ways as possible. But now what if you find that it works," your hypothesis is confirmed with one of those measures but it's not confirmed with three or four of the other ones. SW: And it was not uncommon practice at that point to just report the one place where you got in effect. SA: But then um just a couple years later - MI: You only report the one that worked. MI: Well, so um - things changed (laughs) SA: On October 17, 2011... SW: You could even argue that you were just dialing in exactly where this effect happens. But the paper, this false psychology paper pointed out that if you ignore the places where it didn't work... MI: This one paper was published. SA: It was a paper called False Positive Psychology. MI: That's not really a full picture of what that data actually looked MI: And this paper detailed... like. SW: In part, because a group of scientists started thinking, "Wow, maybe we really need to go back and reinvestigate some of the key findings in our field." SW: The data as a whole, if you looked at all of it, might not actually support your conclusion. MI: I remember reading that. My jaw dropped. I sent this paper i circulated it to the other faculty at my department and all of us are many or many of us saw the importance of this paper. We, we called an emergency meeting for, you know JA: You called an emergency meeting? DE; They call them high powered replications or just kind of doing the same thing but with more people. SW: So they started doing these studies with bigger sample sizes and with strict rules about what data you were looking at. DE: They say ahead of time exactly what they're going to do and how they're going to analyze the data. So you know there's no possibility of monkeying around to get the answer you want. MI: Yeah! I'll never forget it. I mean it was JA: What does an emergency meeting look like? MI: The faculty, and the graduate students just to discuss the contents of the paper and to see what it meant and what the implications were. SA: And as they talked it over, they realized that some ways probably some of their own work had fallen prey to the problems that this paper was pointing out SW: And these attempts to replicate or reproduce, you know, major findings in social psychology and the sort of panic that went along with it seemed to be known as the replication crisis. ARCHIVAL: Replication - it's the cornerstone of science. ARCHIVAL: News articles this week are talking about the reproducibility crisis in science. MI: Yeah, I did see myself in some of this and I thought, you know, "Wow... like, you know, what has been implicated? What papers of mine but papers more generally have been implicated in the field you know writ large?" DE: It's like, oh man - what? You know, what is going on here? SW: Because as they re-examine some of these studies ARCHIVAL: The facial feedback effect SW: And, in fact, meetings like the one that Michael found himself in started happening at universities and conferences all over the world. DE: Yeah, people looked at that paper and everyone thought, "Oh s***". SA: ... studies that you'll find in just about any psych 101 textbook. Some of the replications were failing. ARCHIVAL: The study you'll be taking part in today involves... SW: This is Dan Engber again. DE: You know, this is, this is what, we're... we're all doing this stuff and now we know from this one paper it's very, very easy to turn up spurious findings this way. MI: Yeah it changed, it changed everything. DE: So specifically what got me really into covering the replication crisis was news about ego depletion. SW: So this is a whole literature of studies that were all about how we sort of use up our willpower. DE: This is as rigorous a replication as you can get. And they just found like no effect. SW: Basically no effect. DE: The original study is you go into a lab and you're presented with a dish of delicious fresh baked chocolate chip cookies. I love the method section of that paper they describe baking the cookies in the lab so that the smell will be around the subjects when they come in and they put these cookies out. They say you can't have any cookies. And then they leave you alone with the cookies. JA: But I guess I still don't understand like how is that they're finding nothing now but before they had a study that was then replicated a bunch of times in a bunch of different labs like what - I still don't get what's going on? SW: And what they found was that, you know, if someone has to sit there resisting a cookie with it right in front of them and the smell was wafting up their nose as they sit there if they have to go through that it'll actually be much harder for them to complete certain kinds of logic puzzles. And the argument was: DE: Well the, the- so you have that original cookie study. Yeah I think it's notable that no other studies that I know of used cookies. Now, I found studies where M&Ms were used so that just makes me wonder. I have no idea how that lab did their study but it makes me wonder what would would happen if I ran a lab and I wanted reproduce this cookle finding an extended in a new direction and I kept trying it with cookies and it just never worked. Let me try it with Charleston Chews. Doesn't work. Hmm? DE: you use willpower on you know Task A, then you try to do task B and you just won't have a store of willpower in less later studies found you drank some lemonade in the meantime. SW: Because it's not hard to resist a Charelston Chew. SW: Because sugar, the argument goes, replenishes your willpower. DE: So there were sort of like increasingly bizarre elaborations of this original theory. It ended up working with M&Ms. It ended up working with cookies. It ended up... DE: Yeah, maybe Charleston Chews are not good candy. And I end up like... I do it with M&Ms and it works. Boom! There's my dissertation I publish a paper out of it. So now that's in literature. And so now ego depletion seems like an even stronger more valid thing because it's not just about cookies. Now, it's about cookies or M&Ms. My point is you don't really know how many things were tried in each individual lab. JA: Do you think that might be what's happening in these labs is that there's a lot of trial and error and the error sort of swept aside? And the successes are offered up and then suddenly you have one more success that bolsters the idea. Is that what you think might be happening? SW: Over the years, Dan says this idea that you use up willpower in one place and have less and another just started entering all different corners of our lives. DE: The insight of the original study was replicated again and again and again for decades. DE: I think that's that is the heart of it. SW: But then this group of scientists did this massive effort to replicate the original study, they had over 2000 subjects. They followed these rigorous rules about like what data they were going to look at. SW: But whatever the problems are not with those follow-up studies, the big thing was that scientists were continuing to fail at replicating these big SW: It's the "****" effect (laughing) and, and fairly well known studies like the idea that when you smile it changes your mood or... ARCHIVAL (TED Talk): So, I want to start by offering you a free, no-tech life hack... DE: Yeah, exactly, like I'll show you. CS: Well let me let me say this. Maybe this will help there because this is something, you know, I thought a lot about from the very beginning. SW: So when we talked to Claude Steele about this, he had a couple things to say. First of all: SW: The idea presented in one of the most watched TED talks ever, that the way you sort of hold yourself or stand... ARCHIVAL (TED Talk): So you make yourself big, you stretch out, you take up space, you're basically opening up - it's about opening up. CS: This research has been dramatically well replicated. SW: The stereotype threat effect has been demonstrated, you know, way more times and in way more contexts than really any of those other social pyschology studies. SW: The idea that that could have a measurable impact on your behavior or even your hormone levels. That one also failed to replicate. And that kept happening with study after study after study. And, of course, you know, people would come back and say that the replication effort wasn't done right or you, you didn't really design it well. DE: You're seeing a lot of the researchers who have made their careers studying certain effects. They're just not budging. A few of them are but most of them are not budging. So you just have a split forming. One researcher described it to me as like a civil war within social psychology CS: I don't know if there is another, if there's another phenomenon that has produced so many demonstrations. And, if you can't replicate one of them or six of them, I don't know, I wouldn't... that doesn't surprise me. SW: And he says, you know, you could even see the failures as just information about where the effect really applies and where it doesn't. JA: So is stereotype threat now itself under threat. Is it one of these bodies of research that's now being rethought? CS: So this is, this is science gradually getting sophisticated enough to help apply it in appropriate places. DE: Well, no-one has yet done the big multi-site pre-registered replication that they did for ego depletion, the one that like really woke me up to this? SW: But Dan says there have been sort of smaller scale, sort of here and there, attempts to replicate some of the studies. And some of those have failed. SW: For example, he says he would only expect the effect to appear at times when the person is really invested in what they're trying to do and thus, the negative stereotype really is threatening. On top of that, he says, the kinds of stereotypes that are actually threatening to a given group might change over time. DE:Some studies came out that found that, "You know what? I tried to redo, I tried to do the stereotype threat thing on a big group of students and I found that sometimes the opposite happened. When I tried to induce stereotype threat, the students did better. CS: I'm not sure that the stimuli and the procedures would have the same meaning with today's college students that they had then. You know, social psychology is... the meanings come from the contemporary moment, the state of the culture at that time. CS: That's really where the, you know, the tire meets the road... is can you actually move the educational performance and commitment of real people in real school situations, JA: Yeah, that is interesting actually. Like, if what this research is doing is studying an individual's relationship to like threat... and like, that's gonna be different depending on who you are because you're gonna find different things threatening depending on where you are and even when you are... SW: And Claude actually sent us a list of several dozen studies showing that... CS:... interventions designed to reduce stereotype threat can have dramatic and long-lasting effects on achievement. CS:Yeah, I don't think I know enough about the culture of Black students" today versus "the culture of Black students" 25 years ago but it wouldn't surprise me that there are some real differences. So I don't put as much stock in the exact replication in Experiment A or B as I do in the conceptual replication. DE: Is it possible... I mean, you mentioned there are just hundreds of studies that that kind of circle around the same idea in different ways That is, on its face, very compelling evidence that, you know, this is a robust phenomenon. But, at the same time, you know there's people who gather all the data together and they say look, you know maybe there's some kind of bias that slips in when people are doing this research, that they just keep trying different versions until they get something that looks like a result. Does that seem plausible to you? SW: Now, we should say, there was at least one case... there was a study done in 2006 that a researcher tried to replicate in 2011. DE: It's just much smaller than the original 2006 effect. He did it again with more students this time and he got pretty much nothing. This is the same guy in the same school system trying to do the same study with, you know, hundreds and hundreds of kids and he came up with nothing JA: I'm curious to know - I mean given that replication has become a conversation that you have to unfortunately contend with, I'm just curious if it's changed your opinion of the work. CS: No, I don't think there's anything that could make me go, "Uh oh, this whole thing is not true." SS: I want the truth out there more than I want anything else. CS: Boy, that's a deeply uh I guess cynical? - you know I, I - account of, of a scientific literature this big. It would seem to - that, that that doesn't seem highly probable to me, it doesn't seem highly likely to me. It's clearly real and replicable under these circumstances. Just because they're not everything doesn't mean they're not ...incredibly important to... to the progress of this society. SW: This is Steve Spencer. He was an early collaborator with Claude Steele, especially on those studies involving women and math tests. SW: And, in fact, Claude points out that stereotype threat is pretty unique in the fact that many of the studies are not just in the lab. They've been taken out into the streets with real people. SS: I did recognize in some of the critiques real issues that we need to deal with. SW: But Steve, like Claude, is very confident in, in the results of his studies and the effect of the stereotype threat in general. So much so that... SS: I'm writing right now an article where I'm going to disclose every single study I've ever done, what the results are and put the data up for everybody to look at. I will admit to the questionable research practices that I've done and be as forthcoming and honest with everything in my own lab. In addition to doing that, I've entered into an adversarial collaboration it's called... SW: Which just means he's going to do a big scale reproduction study. MI: To be faced with the probability, the likelihood that all this might have been for naught... or much of it might have been for not. It's you know it's unsettling. It's it's a loss of meaning. MI: You know was I was doing good work was a controlling knowledge. I was a part of the problem was I chasing you know signal or was I chasing noise. I mean, I think the effect is, you know, it might be there. But it might be so small as to not be, not be meaningfully important. JA: Huh, that's interesting. It's hard to know where to stand on this. SW: Well, one thing, that, you know, we should make clear is that stereotypes can be really damaging. I mean, having someone tell you you suck at something when you're under the gun to do it, that's always... that's gonna have an effect. I guess the question is... SS: ... with people who have serious doubts about whether stereotype threat is real. You know, I can't say ahead of time what my reaction will be. I think what I can promise is that I will take the findings very seriously and I will do my utmost not to be defensive about them. DE: Are you nervous about this? SS: No. JA: Yeah. DE: I mean the stakes are pretty high. SS: No no. I mean, you know, what do you mean the stakes are high? I mean I'm a full professor. I have tenure. What are the real stakes for me in this? Not really much. SW: ... how, you know... what effect exactly and when and how and what can you do about it? And those things feel like maybe... I guess it feels like the part of me that wanted this to be a kind of very simple fix that would work everywhere and, sort of, save the world... you know, that part is, you know, is feeling a little bit of a loss--like worrying that his whole thing is shrinking on me a bit JA: Yeah, totally. It does feel like we're all growing up a little bit. You kind of just have to walk away from those big simple promises. SW: Maybe that's exactly what you need in order to be able to find the smaller places where you can have an effect, you know? Right here, right now, with this person trying to do a particular thing. SW: But... MI: Well.... SW: Well, not everyone in the field is these kinds of niggling doubts so well. MI: There are so many pieces of evidence that things are not alright. JA: Producer, editor Soren Wheeler. SW: This again is Michael Inzlicht, the professor from Toronto. RK: This piece was produced by Simon Adler and Amanda Aronczyk. JA: Ok, I guess we should go? RK: Yeah. JA: Thanks for listening. JA: Do you can I ask you do you do you just want to like lay out for us the chronology of your obsession. I think I feel comfortable saying it was. Has it become an obsession or just a just a dalliance? DE: I mean. SW: I mean or maybe you just noticed a crumbling building and ran over to stick your pen... DE: I mean I am, I'm a contrarian. And I'm interested in alternative facts about science let's say. (laughs) JA: Hey, I'm Jad Abumrad. RK: I'm Robert Krulwich. This is Radiolab. JA: And a little while back, our editors Soren Wheeler and I ... we talked to science journalist named Dan Engber who got us kind of tangled up about something that we thought we knew about about the world, about ourselves... RK: Something beautiful as I recall. JA: Yeah, that we talked about at great length on this show. CS: Okay. JA: Hello? CS: Hello? SW: And to set that conversation up we're gonna start with this guy. JA: Hi is this Claude Steele? CS: Yes. SW: Professor Claude Steele. CS: I'm the Lucie Stern's emeritus professor of psychology at Stanford University. CS: Yes, exactly. [laughs] So the the topic is a sort of family birthright. SW: We actually had him on the show number of years ago CS:Some time ago, I can't remember how many years ago. But- SW: But then years and years later, midway through his career as a psychology professor, Claude ran into a demonstration of the power of race that really surprised him. CS: I got a job offer - this is in the 80s at the University of Michigan. And it was part psychology and part to administer a minority student program there. JA: Yeah it's been a - it's been a - CS: Uh- JS: it's been a long time. I'm looking at the sheet here and it says SA: This is Claude Steele in the original program that we ran back in 2009. 2009. CS: And um in the process - I saw data that surprised me. CS: Whoa! JA: That it was JA: What he saw was a troubling trend. Two kids would enter Michigan. One was black, one was white. They'd come in at the exact same level. CS: That blows my mind. CS: Same skills. Same SAT score. JA: Yeah that was a long- JA: So theoretically they should do the same when they get to Michigan. But, without fail or almost without fail, after one semester. CS: I was a lot younger man then. JA: [laughs) i know. CS: The black kid was winding up with lower grades. JA: How much lower? SW: And the reason that we had him on the show back then was because he had done a study in the mid-90s that just completely changed the way that we thought about the power of stereotypes. CS: Pretty pretty um - pretty dramatic. At least two thirds of a letter grade. JA: Meaning if the white kid got an A, the black kid who should be getting an A, too, is instead getting a B CS: Well, my mother was white, my father was African American they - and they were very active in the civil rights movement. So you can imagine race was no distant or passing thing in in our life and family. CS: That's right JA: Or a B+, SA: For Claude, growing up, that was just everyday dinner table talk. JA: That's significant. CS: That's significant. That's significant. TV show: The Bell Curve, intelligence and class structure in American life. JA: And he also by the way saw this performance gap between women and men when it came to math JA: To the same degree? SW: The Bell Curve argued that one explanation for the achievement gap among others was genetic and IQ-based. Now, of course, it's not. But even though there's no scientific backing at all for genetic differences like this. The Bell Curve was still significant just because of the kinds of conversations it was creating at the time and the effect that it had on researchers in this field. CS: The same degree in advanced math courses it was comparable. I learned this is a national phenomenon. If if I was to walk into almost any college class in the United States uh - I'd have a very high probability of finding exactly that CS: Well, The Bell Curve is one point in a long history of that kind of argument ... DE: So I think it's important to put it in context of what was going on at the time... (FADES UNDER) SW: This is Dan Engber again and he says that that gap in achievement between black and white students that Claude had noticed, that was actually a huge topic of conversation at the time. that the difference between groups really is rooted in genetic differences let's just be frank and honest and if you really can't admit that, then you don't have the courage needed to be a real scientist. That's that's a thumbnail way of describing this experience that I've had being confronted with with that notion. SW: And this was obviously disturbing to Claude, well, first on a personal level DE: there's a lot of discussion of what to do about the achievement gap and the familiar argument, is well this has to do with systemic racism and systemic differences in opportunities that play out through an individual lifespan. Now that seems right. It's also daunting because how are we ever going to like cure all of the socioeconomic disparities in this country...? CS: in order to be a scientist are you supposed to actually be open to the possibility that you and your family and your whole race have some genetic limitation? SW: But also because it was so weak scientifically, That's a big problem yeah SW:. But then, in 1994, a different, and in many ways, very dangerous ideas was being tossed into this debate. CS: It's been very difficult, impossible to produce anything like definitive data that the differences in test scores between groups is genetic. Reporter: Charles Murray - coauthor of the book The Bell Curve, SW: When The Bell Curve came out. JA: So Claude Steele thought all right - step one complete, I've got a lab situation that resembles the real world, good. Now the next step is to tweak things a little bit, see if I can mess around with it. JA: Now, normally in these situations - SW: But while the differences between these students that Claude was seeing at Michigan, it also didn't seem to him like it could be explained simply by their backgrounds or their opportunities. CS: Because, you take, say, a black kid and white kid at Michigan. They both have extraordinary scores like, you know, they're in the 98th percentile on their SATS. So, um, the background between the two kids, whatever it is, has not resulted in a difference to that moment in time. So, if going forward and taking a test, black kid gets a lower score or a lower performance in a course of some sort, then something must be happening right there, right there. SW: Something must be happening in the moment. CS: There was something there that people didn't understand and that we certainly didn't understand. CS: The test giver's got a white lab coat on, and he brings in a big stack of cellophane wrapped tests. And he puts a clock on the table - and it's all- it's all - you know it's like - that's - [laughs) that's gonna intimidate almost anybody! JA: Maybe that's what's happening, he thought. What if I took away the clock, took away the coat and most importantly, right before the test, I had the test giver instead of saying the normal - I'm going to give you a test, pre-test thing. Maybe instead, say something like this. CS: Look you may have heard that uh women don't do as well as men on difficult standardized math tests. You may have heard that. But that is not true for this particular test. JA: So he figured he would start with the women in math issue. He brought a bunch of women in and a bunch of men. Sophomores. CS: Brought them into the laboratory one at a time. Gave them a half an hour section of the graduate record exam you take if you're a math major. Very, very difficult math. JA: Ohh. CS: This particular test does not show gender differences, never has never will. JA: Mm. JA: He wondered if maybe saying that simple sentence before giving the test would have an effect. CS: And sure enough, the women who had all the same credentials coming into that situation performed dramatically worse than the men. JA: Worse as in- CS: It'd be a couple hundred points on an SAT test CS: And sure enough, I wouldn't be here if their performance didn't go up to match that of the equally skilled men. JA: That performance gap totally vanished. CS:Sheee look at this thing. So we raced and did it very quickly, the same kind of an experiment with African Americans. JA: Big difference. CS: So this was - this was a big effect. JA: There, the pre-test disclaimer went like this, CS: This is an instrument that we use to study problem solving and it is not diagnostic of individuals intellectual ability. As it is, as an IQ test, it's in fact seen as the gold standard of IQ test because it's quote culture free. JA: In other words, this is not a test of your intelligence. I repeat not an IQ test. ED: There's no math, there's no reading. CS: Because it doesn't involve language. Uh or you could represent the exact same test as a puzzle. CS: So just do the best you can. JA: And with that simple disclaimer at the start- RB: Puzzles. CS: Same kind of an effect. ED: Puzzle. JA: The black students and the white students were now equal. JA: Meaning you can give an IQ test to a bunch of kids and the black students will perform worse. But if you give that same test - lose the word test, lose the word IQ, and just call it a puzzle? CS: Just recently uh Ryan Brown and Eric Day did a even clever-er treatment. They - there is an IQ test, which is nonverbal. RB: The black participants suddenly jump up in their performance. RB: It's called the advanced progressive matrices. ED: Basically we got a reversal. CS: When you represent it as a puzzle, blacks perform as well as whites. CS: It has figures. ED: Very abstract, they got lines crossing. RB: They - they did, yeah. CS: That you have to match and so on. ED: Checks uh- JA: That's all it takes. Just change a few words. JA: Stereotypes are powerful. Okay, that makes sense. That - but in terms of understanding how this works, can you make this tactile for me? Like if the stereotype that's having all these effects is a thing - like a - RB: It's essentially pattern matching. ED: Diamonds with dots in em. It's - ED: Mmhm. RB: Totally visual. JA: Like a little gremlin that bites? Like when in the test taking process, does it actually like do its damage? ED: Yeah it's very - CS: And so they could represent that test - ED: That - that's JA: When does it actually RB: I don't know if you can do this. CS: Oh s****, is what they say about us true? ED: gonna - that's gonna be way open to debate. JA: What does seem to be clear from the data, according to Eric Day and Ryan Brown and Claude Steele, is that the gremlin only seems to appear when the test is sufficiently hard. RB: They don't think you can do it, JA: All the usual stuff. And even if the student doesn't believe it, which is likely RB: See you don't have to believe it, that's the kind of insidious thing CS: If the test is easy, it's important to point out uh these effects don't happen. JA: It's not that the gremlin is not there. here. JA: Just the fact that he has now this extra bit of mental chatter. RB: Well he walks in with you, but he doesn't speak necessarily until things get challenging RB: That little guy whispering. CS: As soon as it - the test gets difficult, RB: That's where the voices kick in, JA: Well it's a distraction. CS: And that makes their performance go down JA: Just a little bit. JA: Which means that for most of the test, everybody's doing about the same. It's only at problem number 17. The one about... cosines and factorials and whatever, where things start to go wrong, and - at least that's the theory. At that problem, the black student starts to stiffen up a little bit. CS: That's right. JA: And Claude Steele's measured this. RB: All of this dialogue is keeping you from being 100 percent focused on the task at hand which is solving these problems. JA: So the real subtle power of a stereotype isn't that it prevents you from doing a thing you want to do. It distracts you for just a beat from doing the thing you want to do and that may be all the difference. SW: So that's how we ended the piece that we did back in 2009. But in the years after Claude did that original study, the effect, which he called "Stereotype Threat," became one of the biggest... Male voice: Stereotype threat- CS: Their blood pressure's elevated, their short term memory is impaired. JA: It's that flicker of frustration through their body that wakes up JA: the gremlin who starts - to whisper in their ear. SW: and most important ideas Female voice: Stereotype threat. SW: in all of social psychology. Male voice: Now some psychologists say, stereotypes can become self- fulfilling prophecies. SW: And Claude Steele... MI: a professor of psychology at the University of Toronto. JA: So when did you begin -- if you could give us sort of the cliff notes version of your history with stereotype threat in particular? MI: I mean I think... I was certainly attracted to that that part of social psychology that dealt with prejudice and discrimination, so I'm Jewish, I went to you know, Jewish day school and high school and kind of perhaps baked into me was a desire to, for social justice, um, and you know seeing the evils of prejudice and how those evils taken to their logical extreme, what could happen. So I was passionate about the topic. I, um ... so it seemed like a very hopeful, sort of explanation that also offered relatively easy solutions to fix. JA: Sounds like you really came into it to with a very social, political sort of bent to it. Male voice: Ladies and gentlemen it's my great pleasure to present, Claude Steele. SW: became a sort of academic rockstar. Male voice: Professor Claude Steele. Female voice: Dr. Claude Steele. SW: Speaking to overflowing audiences at places like Columbia and Cornell. MI: Yes, that's right. I wanted my work to have an impact. I wanted it to, yeah, to change the world. JA: Coming up, after the break, the Michael tries to change the world but the world kind of changes... Male voice: Welcome Dr. Steele. RK: him... Female voice: Dr. Steele. JA: Him, yeah... JA: Jad SW: And this idea of a stereotype threat was shown to be relevant in cases that had to do with age and socioeconomic status. There were studies about women playing chess, men being tested on like social sensitivity. I mean, Claude's work ended up inspiring sort of a whole generation of social psychologists. RK: Robert MI: Yeah, I would say that the original stereotype threat paper by Steele and Aronson blew me away. It just spoke to me and was beautiful and it seemed to offer answers, you know, to questions that troubled me. JA: Radiolab. We're back and we're talking about Claude Steele's seminal research stereotype threat: how the threat of negative stereotypes can impact a person's academic performance. SW: So this is Michael Inzlicht... SW: But along the way he says, he noticed some things that troubled him. SW: Right and uh just before the break we were talking to Michael Inzlicht who, after getting into stereotype threat research, he went onto grad school and really focused in on stereotype threat as his field of choice. SW: And weirdly enough it all started he says, when he moved to Toronto, MI: Yeah, so... MI: I started my - my current job at the University of Toronto in 2005 - DE: How many studies of stereotype threat did you end up doing? MI: I would say in the order of, like, in terms of numbers of studies 20 maybe? 15 to 20? - something like that? SW: And while he was there he was running a lot of his stereotype threat research on women in math you know giving women a test and then doing some intervention to reduce the stereotype and seeing if there would be a difference in their test scores SW: He did a lot of studies on women's performance in math but also just looking at different environments and how they create or encourage these stereotypes. And over the course of his career he ended up editing a book about stereotype threat. MI: And, simply put - SA: For the first time - MI: Oh, well yeah... yeah, yeah... MI: I was not able to consistently show any effect of, of, of stereotype threat. SW: Gave numerous talks on the subject. JA: Hm. MI: All over the world SA: He even signed a brief, uh - kind of explaining stereotype threat SA: In other words, the women who had received this stereotype-threat- reducing intervention performed just about the same as the women who hadn't. MI: to the Supreme Court of the United States - Male reporter: The Supreme Court heard the oral argument in Fisher v. University of Texas at Austin - MI: So I think when I first got here, I failed to replicate some of these effects, and like let me go back to the drawing board. Let me, you know, think about you know-what is my-how is my population different? SA: And his first thought was maybe it has something to do with Toronto. MI: So Toronto uh - incredibly diverse place. SW: As it applied to the question of affirmative action Male reporter: The court will decide whether race used in university and college admission policies is constitutional. SW: like take the freshman pysch classes he was teaching. MI: Um so I - you know put my name to this, SA: But then MI: We're talking about you know a third East Asian, a third South Asian, and only about 15% Caucasian or Eur-European Canadian. And then a little bit of everything else. SA: So he started wondering, SW: ... how just by doing some very standard practices in psychology research... MI: To what extent do our students even ha - are they aware of these stereotypes? About, you know, women and math? Um, and remarkably, when I would ask, you know, our students this MI: ... using these techniques you are taught explicitly in grad school ... SW: you could sometimes end up with these sort of ludicrous conclusion. SA: Like he'd ask, you know, Who here has heard that women tend to be worse at math? JA: So you're saying there were this paper was showing ways in which experimenters could subtly unknowingly juice their results. Is that essentially what it was? MI: I would say no more than like a quarter to a third had you know a strong awareness of this stereotype? MI: Yeah, essentially... SW: So he started running the experiments again but SW: So the paper was pointing out that it becomes sort of standard practice that you know when you were researching for some effect like oh how happy people were or how well they did on a test that you would measure that thing in a couple of different ways. MI: I'd only pick those women and men who actually had awareness but even then. I still then couldn't get the effect. I was like, okay, well - maybe I'm doing something wrong, SW: Like maybe the interventions he was designing weren't reducing the threat or with these students the threat just wasn't that much of a threat in the first place. But by this time Michael had already done a ton of research on stereotype threat and MI: you know, I'm I'm a person who gets bored eh rather quickly um, and I just you know, started, eh - started losing interest. MI: Which to some extent is considered good practice, right? You want to, no one measure captures your construct perfectly. So you should measure that thing in as many ways as possible. But now what if you find that it works," your hypothesis is confirmed with one of those measures but it's not confirmed with three or four of the other ones. SW: And it was not uncommon practice at that point to just report the one place where you got in effect. SA: But then um just a couple years later - MI: You only report the one that worked. MI: Well, so um - things changed (laughs) SA: On October 17, 2011... SW: You could even argue that you were just dialing in exactly where this effect happens. But the paper, this false psychology paper pointed out that if you ignore the places where it didn't work... MI: This one paper was published. SA: It was a paper called False Positive Psychology. MI: That's not really a full picture of what that data actually looked MI: And this paper detailed... like. SW: In part, because a group of scientists started thinking, "Wow, maybe we really need to go back and reinvestigate some of the key findings in our field." SW: The data as a whole, if you looked at all of it, might not actually support your conclusion. MI: I remember reading that. My jaw dropped. I sent this paper i circulated it to the other faculty at my department and all of us are many or many of us saw the importance of this paper. We, we called an emergency meeting for, you know JA: You called an emergency meeting? DE; They call them high powered replications or just kind of doing the same thing but with more people. SW: So they started doing these studies with bigger sample sizes and with strict rules about what data you were looking at. DE: They say ahead of time exactly what they're going to do and how they're going to analyze the data. So you know there's no possibility of monkeying around to get the answer you want. MI: Yeah! I'll never forget it. I mean it was JA: What does an emergency meeting look like? MI: The faculty, and the graduate students just to discuss the contents of the paper and to see what it meant and what the implications were. SA: And as they talked it over, they realized that some ways probably some of their own work had fallen prey to the problems that this paper was pointing out SW: And these attempts to replicate or reproduce, you know, major findings in social psychology and the sort of panic that went along with it seemed to be known as the replication crisis. ARCHIVAL: Replication - it's the cornerstone of science. ARCHIVAL: News articles this week are talking about the reproducibility crisis in science. MI: Yeah, I did see myself in some of this and I thought, you know, "Wow... like, you know, what has been implicated? What papers of mine but papers more generally have been implicated in the field you know writ large?" DE: It's like, oh man - what? You know, what is going on here? SW: Because as they re-examine some of these studies ARCHIVAL: The facial feedback effect SW: And, in fact, meetings like the one that Michael found himself in started happening at universities and conferences all over the world. DE: Yeah, people looked at that paper and everyone thought, "Oh s***". SA: ... studies that you'll find in just about any psych 101 textbook. Some of the replications were failing. ARCHIVAL: The study you'll be taking part in today involves... SW: This is Dan Engber again. DE: You know, this is, this is what, we're... we're all doing this stuff and now we know from this one paper it's very, very easy to turn up spurious findings this way. MI: Yeah it changed, it changed everything. DE: So specifically what got me really into covering the replication crisis was news about ego depletion. SW: So this is a whole literature of studies that were all about how we sort of use up our willpower. DE: This is as rigorous a replication as you can get. And they just found like no effect. SW: Basically no effect. DE: The original study is you go into a lab and you're presented with a dish of delicious fresh baked chocolate chip cookies. I love the method section of that paper they describe baking the cookies in the lab so that the smell will be around the subjects when they come in and they put these cookies out. They say you can't have any cookies. And then they leave you alone with the cookies. JA: But I guess I still don't understand like how is that they're finding nothing now but before they had a study that was then replicated a bunch of times in a bunch of different labs like what - I still don't get what's going on? SW: And what they found was that, you know, if someone has to sit there resisting a cookie with it right in front of them and the smell was wafting up their nose as they sit there if they have to go through that it'll actually be much harder for them to complete certain kinds of logic puzzles. And the argument was: DE: Well the, the- so you have that original cookie study. Yeah I think it's notable that no other studies that I know of used cookies. Now, I found studies where M&Ms were used so that just makes me wonder. I have no idea how that lab did their study but it makes me wonder what would would happen if I ran a lab and I wanted reproduce this cookle finding an extended in a new direction and I kept trying it with cookies and it just never worked. Let me try it with Charleston Chews. Doesn't work. Hmm? DE: you use willpower on you know Task A, then you try to do task B and you just won't have a store of willpower in less later studies found you drank some lemonade in the meantime. SW: Because it's not hard to resist a Charelston Chew. SW: Because sugar, the argument goes, replenishes your willpower. DE: So there were sort of like increasingly bizarre elaborations of this original theory. It ended up working with M&Ms. It ended up working with cookies. It ended up... DE: Yeah, maybe Charleston Chews are not good candy. And I end up like... I do it with M&Ms and it works. Boom! There's my dissertation I publish a paper out of it. So now that's in literature. And so now ego depletion seems like an even stronger more valid thing because it's not just about cookies. Now, it's about cookies or M&Ms. My point is you don't really know how many things were tried in each individual lab. JA: Do you think that might be what's happening in these labs is that there's a lot of trial and error and the error sort of swept aside? And the successes are offered up and then suddenly you have one more success that bolsters the idea. Is that what you think might be happening? SW: Over the years, Dan says this idea that you use up willpower in one place and have less and another just started entering all different corners of our lives.