OpenNashPodcasts

Freakonomics, M.D.

12. Where Do All the Bad Ideas Go?

19 Nov 2021 · 34m · Bapu Jena, Steve Levitt, Emily Oster · 21 quoted facts

▶ Play episodeOriginal source ↗

In brief

Researchers abandon most of their ideas, and the studies that get published are a skewed sample of all the work done. Bapu Jena, Steve Levitt and Emily Oster argue that failure is normal: Levitt fails on about 19 of 20 projects. Journals favor positive findings, so the published record overstates how often hypotheses work. Their practical answer is to test ideas quickly and cheaply and to treat published results with skepticism.

The tension is whether the abandoned ideas deserve a home. Jena worries that wasted effort and publication bias distort science. Levitt and Oster doubt that most failed ideas are worth publishing, because zeros often come from mistakes and redoing work is cheap. They also admit that being slow to quit costs real time. Levitt lost about 18 months on a crime-and-television idea that he believed in too strongly.

Key insights

  1. Most research ideas fail, and that's normal

    Levitt says he fails on roughly 19 of 20 projects, and Jena and Oster describe many dead projects too. Good ideas are rare, so a productive year may yield only three or four from hundreds of hours of thinking.

  2. Quick first looks beat long commitments

    Levitt advocates a "fast first cut", a rough analysis that can often show within a day whether an idea can work. Oster recalls a teacher's rule that many projects deserve two weeks and few deserve six months.

  3. Belief in an idea can waste months

    Levitt's idea that television raised 1960s crime looked great in a first look, then stopped working. He kept going for about 18 months because he believed it, and Levitt says that was at least 16 months too long.

  4. Journals favor positive results, distorting the record

    Ted Sterling found in the late 1950s that 97% of papers in four psychology journals reported positive results. Jena notes that this publication bias persists and may be getting worse, so published findings are a selected sample.

  5. Cheap data does not guarantee creative questions

    Levitt says the cost of empirical research has plunged, from a summer of RA work on census tapes to a day. He and Oster worry that starting from a dataset leads to fewer interesting questions.

Questions

What is the file drawer problem in research?

It is the habit of shelving studies that did not work out or found no effect, so they never get published. Jena says the term was coined in the 1970s by psychologist Robert Rosenthal.

Why do scientific journals prefer positive results?

Journals tend to see studies that confirm a hypothesis as more worth the limited page space than "null" findings, which found no effect. Jena says research shows positive results are more likely to be published, so the published record fills up with them.

Should researchers publish their failed ideas and null results?

Jena would like those ideas to be more visible, but Levitt and Oster are doubtful. Levitt says zeros often come from mistakes and that everyone discards them, so readers should just be skeptical of what does get published.

How do top researchers decide when to give up on an idea?

They try to find out quickly. Levitt says a day is usually enough for a rough first look, and Oster says many projects are worth two weeks but few are worth six months.

Did texting while driving turn out to be dangerous in Levitt's study?

Levitt's study linked insurer and phone-company data and found no effect of texting on bad driving outcomes. He says the insurer only saw hard braking, not crashes, and he thinks people slowed down while texting. The paper was never published, because he expected readers to be angry and not to believe it.

Facts

Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.

The term "file drawer" for unpublished research was coined in the 1970s by psychologist Robert Rosenthal.

“That's a term that was coined in the 1970s by the psychologist Robert Rosenthal.”

0:02:11 · Bapu Jena · File drawer · 1970s

In the late 1950s, researcher Ted Sterling found that 97% of papers in four major psychology journals reported positive results.

“in the late 1950s, a researcher named Ted Sterling found that 97 % of papers in four major psychology journals had positive results.”

0:13:27 · Bapu Jena · Publication bias · 1950s

Jena says his research found that gun injuries fall when the National Rifle Association holds its annual meeting.

“Like, do gun injuries fall when the National Rifle Association holds its annual meeting? It turns out they do.”

0:14:25 · Bapu Jena · Gun injuries

During the COVID-19 pandemic, Emily Oster built a national database on schools and found the risk of infection among students was low.

“during the pandemic, she became the de facto face of school reopenings, building a national database and finding that the risk of COVID-19 infection among students has been low.”

0:07:13 · Bapu Jena · School reopenings · 2020

Steve Levitt has studied the link between legalized abortion and crime, and corruption among sumo wrestlers.

“Steve has studied the link between legalized abortions and crime. He's looked at corruption among sumo wrestlers.”

0:06:41 · Bapu Jena · Levitt research

Steve Levitt and Matt Gentzkow worked on a project linking early television to rising 1960s crime for about 18 months, which Levitt says was at least 16 months too long.

“And we probably worked on it for 18 months, at least 16 months too long.”

0:16:16 · Steve Levitt · TV and crime · 1960s

Levitt's study linking insurance driving data to phone texting data found no impact of texting on bad driving outcomes.

“And in the end, we found that there was no impact of texting on any bad driving outcomes.”

0:21:36 · Steve Levitt · Texting and driving

In Levitt's texting study, the insurer's data recorded hard braking, not actual crashes.

“the insurance company didn't actually know when people crashed. They just knew when they slammed on the brakes.”

0:21:13 · Steve Levitt · Texting and driving

Jena used insurance claims data to test whether heart attacks, anxiety and medication use rose after the Ashley Madison data breach, and found nothing.

“we looked at insurance claims data to see whether things like heart attacks, anxiety, medications, whatever, went up in the days after Ashley Madison breach occurred.”

0:16:35 · Bapu Jena · Ashley Madison

Jena's rough analysis of US data found no evidence that having daughters instead of sons changes a parent's life expectancy.

“whether you have a son or daughter may affect your quality of life and how you die, but it doesn't affect your life expectancy.”

0:17:34 · Bapu Jena · Daughters and longevity

Jena studied whether Veterans Affairs researchers writing about the VA found more positive results, did not find that effect, and published the study.

“We actually wrote that up because I thought this question of ideological bias, no matter what you found, would be interesting. And we got it published”

0:19:36 · Bapu Jena · Ideological bias

Jena replicated a controversial finding about patient outcomes during cardiology meetings in a different clinical scenario and set of meetings, but struggled to publish it.

“this finding that patients do better when cardiologists go to their meetings, I replicated it at different clinical scenario, different set of meetings.”

0:28:23 · Bapu Jena · Replication

Levitt recalls that using census data once required one research assistant roughly a summer to copy it off data tapes, and that it can now be done in a day.

“it used to be if you wanted to use the census for anything, it took one RA, roughly a summer, to spin the data off of actual data tapes to get a data set together.”

0:30:44 · Steve Levitt · Data costs

Emily Oster recalls her grad school teacher David Laibson saying that many projects are worth two weeks but few are worth six months.

“David Leibson told us many projects are worth two weeks. Few projects are worth six months.”

0:26:01 · Emily Oster · Project time

After her freshman year, Emily Oster worked as a research assistant for Harvard Kennedy School economist Chris Avery on education projects, which led her to choose economics.

“the second job was I was a research assistant for Chris Avery, who is an economist at the Kennedy School, working on some projects about education.”

0:06:15 · Emily Oster · Oster's career

Jena applied to a University of Chicago dual-degree program seeking a medical degree and a biology PhD, and was steered toward economics.

“Almost 20 years ago, when I was interviewing for a dual degree program at the University of Chicago, I was trying to get my medical degree and a PhD in biology.”

0:03:55 · Bapu Jena · Jena's career

Levitt took introductory economics at Harvard because it was the biggest course on campus, and decided on economics after a lesson on comparative advantage.

“The introductory economics course was the biggest course on campus, so I had to take it.”

0:04:54 · Steve Levitt · Levitt's career

Levitt says three or four good ideas a year from a 300 to 400 hour investment in ideas would be a great year.

“if I had three or four good ideas a year off of a 300 or 400 hour investment, I'd call that a great year.”

0:11:32 · Steve Levitt · Idea generation

Levitt says he fails on about 19 out of 20 projects.

“I probably fail on 19 out of 20 projects, so there's an endless supply.”

0:15:02 · Steve Levitt · Research failure

The American Journal of Gastroenterology dedicated its November 2016 and May 2020 issues to null findings.

“The American Journal of Gastroenterology dedicated its November 2016 and May 2020 issues to null findings.”

0:33:44 · Bapu Jena · Null results · 2016

In 2019, the Berlin Institute of Health launched a program offering researchers 1,000 euros to publish replication studies or null results.

“In 2019, the Berlin Institute of Health launched a program that offers researchers 1 ,000 euros to publish replication studies or no results.”

0:34:17 · Bapu Jena · Replication incentives · 2019

Chapters

  1. 0:01:07Introducing the file drawer of unpublished ideas

    Jena frames the episode around studies that never get published and asks whether those scrapped ideas should be shared.

    “Because if they did, I wouldn't have written a paper about it.”
  2. 0:05:22How two economists ended up in economics

    Levitt and Oster explain accidental routes into the field, showing how chance shapes careers.

    “That was literally the reason I did economics.”
  3. 0:08:07Where research ideas come from

    The three compare ways of generating ideas, from asking what puzzles you to spending dedicated time on ideas.

    “Think about when you look out at the world, what's puzzling?”
  4. 0:12:54Publication bias and the value of null results

    Jena explains why journals favor positive findings and how this skews the scientific record.

    “journals aren't usually as interested in publishing null results”
  5. 0:16:16Levitt's costly TV-and-crime project

    Levitt describes a promising early result that fell apart, and the 18 months he spent on it anyway.

    “I wasted almost an infinite amount of time relative to any other failed project I've ever done.”
  6. 0:17:03Dead ends: Ashley Madison and daughters

    Jena shares two of his own failed studies, showing how null results look in practice.

    “We tried all sorts of little things to figure out if there's something there, and there wasn't.”
  7. 0:21:36The texting-and-driving study that never ran

    Levitt shows a null result that was stopped by expected publication hurdles and not by bad data.

    “we found that there was no impact of texting on any bad driving outcomes”
  8. 0:25:40The fast first cut and when to quit

    Levitt and Oster set out rules of thumb for how long to spend on an idea before dropping it.

    “I'm a huge believer in what I call a fast first cut.”
  9. 0:27:43Should zeros and replications be published?

    The group debates whether failed ideas deserve a venue, with Levitt and Oster doubting it and Jena stressing replication.

    “we know that everyone throws away their zeros.”
  10. 0:30:44Cheap data and the state of creativity

    Levitt and Oster discuss whether easier access to data has made economics more or less creative.

    “the cost of doing empirical research has plunged.”

Transcript

0:00:00
Speaker 4

Your business relies on technology every day, but how confident are you in the security behind it? As your business evolves, so do the cybersecurity challenges that come with it. That's why BPM's Cybersecurity Risk Assessments help leaders better understand cyber risk, identify vulnerabilities, and develop a practical roadmap for stronger security. Build confidence in your cybersecurity strategy. Learn more at bpm.com slash cyber. ever check out online and think wow that was easy then you might have used link no credit card hunt no address typing no checkout obstacle course link fills in your details protects eligible purchases and makes subscriptions easier to manage it's used by 300 million people which means you might already be one of them next time you're at

0:00:49

checkout look for the green button link the smarter way to pay

0:00:57
Bapu Jena

So let me just ask you, do you think that hospitalizations for hyperglycemia go up among children with diabetes after Halloween? Yes. No, they don't.

0:01:06
Emily Oster

Really?

0:01:07
Bapu Jena

Because if they did, I wouldn't have written a paper about it. That's me talking to two economists whom I mostly admire. Steve Levitt, University of Chicago professor and co-author of Freakonomics, and Emily Oster, Brown University professor and author of parenting books like The Family Firm. You've probably read or heard about lots of their studies, but I asked them to sit down with me because I wanted to know about the studies you haven't heard about. You see, the nature of research is to come up with ideas, to ask questions.

0:01:46

And if you're one of the lucky ones, you get to ask really creative, important questions. and then design studies to answer them. But coming up with a good idea, designing a good study, and coming up with a good answer is a lot like playing the lottery. Sometimes you strike big, as Steve and Emily have done a lot of times, but most of the time you lose. And just like in the lottery, the news doesn't feature the losers.

0:02:11

In fact, most of our ideas end up in what we call the proverbial file drawer. That's a term that was coined in the 1970s by the psychologist Robert Rosenthal. Maybe the idea, the question you wanted to explore, just wasn't that good to begin with. Maybe the idea was a good one, but there was just no good way or good data to study it. Maybe another better question pulled you in a different direction. Or maybe there was an answer to your good question, and that answer was simply no.

0:02:44

There was no impact. There was no change in outcome. There was no significant difference. Now, I'm a big fan of good ideas, but I'm not such a big fan of the file drawer because it's basically where some of my favorite ideas go to die. Ideas that, even if they didn't work out, they might still make you say, hmm, that's kind of interesting, which has led me to wonder, is there something we or I should be doing with all those scrapped ideas?

0:03:11

From the Freakonomics Radio Network, This is Freakonomics, MD. I'm Bapu Jena. I'm a medical doctor and an economist. And this is a show where I dissect fascinating questions at the sweet spot between health and economics. At the heart of each of those questions was a spark, an idea that set researchers on a path of discovery. Today, Emily Oster, Steve Levitt, and I talk about that path and how we approach the often daunting task of asking questions for a living and if there's something that we should be doing with those questions that are lying around in the back of our file drawers.

0:03:49
Steve Levitt

Somebody approached me and they asked me if I could study texting and driving and what the effect was.

0:03:55
Bapu Jena

Yep, believe it or not, that's in a file drawer. Before we get to that, though, I wanted to know a little bit about how Steve and Emily became economists. As for me, I didn't actually plan to become an economist. Almost 20 years ago, when I was interviewing for a dual degree program at the University of Chicago, I was trying to get my medical degree and a PhD in biology. The director of the program asked if I wanted to try for a PhD in economics instead.

0:04:25

I happened to have studied econ in college, so he thought I'd be a good fit for that program. I was impressionable, maybe a little bit crazy. And before I knew it, I was taking graduate courses in economics while going to medical school at the same time. So for me, becoming an economist was all about chance, which is a feature of life that has actually influenced a lot of my work. Chances are, if you're listening to this, though, you know who Steve Levitt is.

0:04:51

He's a world-class economist. But how did he become one?

0:04:54
Steve Levitt

So I didn't give any thought to anything. I was at Harvard, and in my freshman year, I had a rule that I just took the biggest, easiest courses on campus. And I figured I believed in markets even before I was an economist. The introductory economics course was the biggest course on campus, so I had to take it. And I remember I had this interesting moment maybe a month into the course where they were teaching about the concept of comparative advantage.

0:05:22

It's something that was so intrinsic to me that I thought, how can they teach this? And as I walked out of the class, my best friend, who was also in the same class, said, oh my God, that was the most complicated thing I've ever seen. I don't think I could ever learn that. And I thought, whoa, wait a second, maybe I should do economics. I'm good at it. That was literally the reason I did economics. Emily, how about you?

0:05:43
Bapu Jena

Tell us why you got into economics.

0:05:45
Emily Oster

Yeah, so my parents are economists, both of them. And so I know what comparative advantage was at five. But I actually was definitely not planning to be an economist. When I entered college, I was going to do like bench science. And the summer after my freshman year, I had two jobs. One of them was working in a fruit fly lab where I was responsible for dissecting the fruit fly larvae brains. And I don't remember what I did with them after that, but that's what I did.

0:06:15

And then the second job was I was a research assistant for Chris Avery, who is an economist at the Kennedy School, working on some projects about education. And at the end of the summer, I decided to be an economist. How much of that was the love of the stuff I did with Chris and how much of it was the tremendous dislike of the fruit flies? We will never know. But that was the sort of ultimate dividing line.

0:06:39

Steve teaches at the University of Chicago and

0:06:41
Bapu Jena

Emily at Brown University. They've both tackled some of the most creative questions that I have seen. For example, Steve has studied the link between legalized abortions and crime. He's looked at corruption among sumo wrestlers. Emily has studied the impact of cable TV on the status of women in India, and she's helped us understand by looking at people with Huntington's disease, how expectations about our future health impact the educational and health investments we make in ourselves today.

0:07:13

Emily's also written a lot about the economics of parenting, and during the pandemic, she became the de facto face of school reopenings, building a national database and finding that the risk of COVID-19 infection among students has been low. So these are two incredibly creative people, and I wanted to know how they come up with ideas. Was this something they were born with, or did they learn it?

0:07:40
Emily Oster

I think that there's an inherent piece and a taught piece. When you are doing economics or probably any field, the hardest thing in graduate school is the part where you go from, you know, doing problem sets that someone else has set for you to having to look at a blank piece of paper and come up with an idea. Some people are better at that than others, but there are things that you can teach people like think about what you're interested in in the world.

0:08:07

Think about when you look out at the world, what's puzzling? What's exciting? What do you want to understand? And the students I have who I think are going to be the most successful are those who are always asking those questions and not asking the question of, you know, I'm going to read 17 issues of the American Economic Review and figure out what's the kind of niche thing that I could do that would be a small innovation on someone else's thing.

0:08:29

Those are good papers. We all write some of those papers. But I think that the most successful people are those who say, hey, what do I want to know about and how can I be the one to find that out?

0:08:38
Bapu Jena

So, Emily, let me follow up on that. Do you do anything yourself proactive to come up with ideas or do you just kind of live your life and then, you know, you're walking around, you're seeing something that, you know, your kid did and you just come up with an idea?

0:08:49
Emily Oster

I think at this point, most projects kind of come out of some other project. So it's sort of that I'm doing something and then I wonder about something and then it leads to something else. I don't ever do the thing where I sit down and I'm like, OK, now I'm going to come up with that idea because I've tried that and it's really it's deeply unsuccessful. I think actually Steve was the one who explained to me that would be deeply unsuccessful, but I still sometimes then have tried to do it and found he is correct.

0:09:15
Bapu Jena

Steve, before I ask you, let me just tell you, I'm guessing I didn't follow your advice because you know what I do probably three times a week for about half an hour to an hour. Usually when I'm driving to soccer, I'll just jump on the phone with my research assistants and whoever's working with me. And we have these idea sessions and I'll just start. I'll jump on the phone and say, who's got an idea? and invariably most of the ideas that people say and that I say are not going to be great.

0:09:37

But once in a while, there's a nugget there. But I also think that there's a causal effect of just that sort of discipline in that it helps me see things around me in ways that other people might not. And I don't think it's something that's inherent in me because if we can teach doctors how to transplant hearts from one human body to another, it's hard to imagine that we can't teach them how to think creatively. As a practical matter, I bet that you probably read the New York Times and Wall Street Journal and Washington Post.

0:10:06

And I'll read some of those things too. But I also read like People and Yahoo News, not to dismiss the newsworthiness or credibility of Yahoo News. But I read those kinds of things because I feel like they kind of help me get ideas as well. What are your thoughts on all this?

0:10:21
Steve Levitt

So I agree with both of you said 100%. I would add a couple of small things. One is, Bapu, we don't try to teach people to be creative in economics. There's zero time devoted to it. Of course, you can teach people to be creative. When I was a young grad student, I actually spent a lot of time studying creativity and the advertising industry. And in particular, there's a book called A Whack on the Side of the Head.

0:10:45

It's just a book on creativity. It's a fantastic book about ideas. So I think we can teach it. The other thing I think that we don't stress enough to grad students or to anyone interested and creativity is you have to invest time that most ideas are bad. And most of the grad students I know might spend an hour a month or less on ideas. They're always working on something. They're never looking at ideas. But I spent probably an hour a day on ideas.

0:11:13

And it wasn't, like Emily says, it wasn't sitting on the couch because sitting on the couch didn't turn out to be my best way of generating ideas. But with intentionality, like you're talking about, Bapu, I would seek out a conversation with somebody who knew something about fruit flies, because who knows, maybe fruit flies would lead to someplace.

0:11:30
Emily Oster

They don't lead. I assure you.

0:11:32
Steve Levitt

Or I would just go into the stacks in the library and I would go to some section I would never want to go to usually, like, you know, the Middle Ages. And I would just start opening books and looking for stuff. And I never knew where it would lead. And you don't have to have very many good ideas. is if I had three or four good ideas a year off of a 300 or 400 hour investment, I'd call that a great year.

0:11:59
Bapu Jena

But even if you have a good idea, sometimes getting that idea out into the world, which in the academic world often means getting it published in a good journal, doesn't typically happen. And I get it. The ideas may have been bad or they just didn't work out. But what if the seed of an idea was good, but maybe just executed poorly? What if another scientist has a better way of looking at the same thing? Wouldn't it be good if they're aware of these quote-unquote failed attempts?

0:12:28

Some questions are worth answering, no matter what the answer is. But for other questions, scientific journals are often focused on publishing so-called positive findings. It works like this. When you write a research paper, you typically have to provide a hypothesis. What do you think is going to happen when you run an experiment or analyze some data? If you confirm your hypothesis, that's great. That's a positive finding. And research shows you're more likely to get it published.

0:12:54

If you don't, well, that means you had what we call a null finding. You didn't find anything. And journals aren't usually as interested in publishing null results. You didn't prove anything after all. So why dedicate limited page space and resources to a study that didn't find anything? You can see how this creates a problem. Not only may people waste a lot of time looking at ideas that others have already looked at, but if journals favor positive findings, the scientific record becomes increasingly filled with positive findings.

0:13:27

This is a problem that's been known about for a long time. For example, in the late 1950s, a researcher named Ted Sterling found that 97 % of papers in four major psychology journals had positive results. Recent analyses have shown that this publication bias, as Sterling called it, it still exists and could be worsening. So knowing that for some types of ideas, a positive finding is important, how do we think about what makes for a good research idea?

0:13:57

When it comes to good ideas, there are two buckets in my mind. The first bucket is questions that are so interesting and important that no matter what their answer is, positive or not, the answer is going to be important. For instance, do minimum wage laws have an effect on employment? No matter what the answer is, people are probably going to want to hear about it. The second bucket is for questions that really are only interesting if there is a positive relationship.

0:14:25

Like, do gun injuries fall when the National Rifle Association holds its annual meeting? It turns out they do. But people are probably only going to want to know the answer to that question if the answer is an emphatic yes. Mostly out of preference, though, a lot of my research falls into that second category. And those questions tend to be kind of high risk. If I don't find any relationship, there's just no story there. Now, I have my fair share of research questions and projects that have failed for one reason or another, but it turns out that so do Emily and Steve.

0:15:02
Steve Levitt

I probably fail on 19 out of 20 projects, so there's an endless supply. So one that really comes to mind, though, is a particular project that actually relates to Emily because her husband,

0:15:14
Bapu Jena

Jesse Shapiro, Matt Jenskow, had done work on early TV. Jesse Shapiro is an economist at Brown University, and Matt Jenskow is an economist at Stanford. And I had done a lot of work looking

0:15:25
Steve Levitt

at crime and why crime had fallen in the 1990s. And I had spent a lot of time trying to figure out why crime had gone up in the 1960s, but I could never figure it out. And I had this great idea. Maybe it was TV. The advent of TV led to increased crime for a variety of reasons. You might imagine why that might be. And interestingly, we did a fast first cut. By fast first cut,

0:15:50
Bapu Jena

Steve isn't making a surgical reference, thank goodness. He means an initial analysis of some

0:15:55
Steve Levitt

data. It was with Matt Jenskow, and we looked at it for a day, and the results were unbelievable. I mean, they were fantastic. We had the first figure of what was going to be an amazing paper explaining a puzzle no one could ever explain. And then it didn't work. And every month, Matt would say, we should really stop working on this. This is stupid. It's not working. And I'd say, no, no, no. I believe it. I feel this.

0:16:16

It's going to work. And we probably worked on it for 18 months, at least 16 months too long. And it was a great example of, in part because that best first cut worked, and partly because I really actually believed in my heart that this was a true story. I wasted almost an infinite amount of time relative to any other failed project I've ever done.

0:16:35
Bapu Jena

So that was one. I'll just give you two of mine. So you may remember a few years ago, there was this data breach from something called Ashley Madison. This is a site for married people to find affairs. And so I thought, okay, if it's really the case that a lot of men were using Ashley Madison, that data breach should have been extraordinarily stressful. So we looked at insurance claims data to see whether things like heart attacks, anxiety, medications, whatever, went up in the days after Ashley Madison breach occurred.

0:17:03

And then we subsetted it to people who were taking erectile dysfunction drugs. We tried all sorts of little things to figure out if there's something there, and there wasn't. So in case you were wondering, how does this fit with the mRNA technology? It's a similar level of importance. And the other thing came from something I used to see in the hospital a lot. When we'd be taking care of these patients who are older, it'd often be the case that daughters were coming into the hospital to chat with the doctors and other healthcare providers about their parent, as opposed to the sons.

0:17:34

And it made me wonder, do people with daughters live longer than people with sons? And it's hard to do unless you have like Swedish data where you can link these families over time. But we did a rough cut with some data that's available in the U.S. and didn't find any evidence at all. So whether you have a son or daughter may affect your quality of life and how you die, but it doesn't affect your life expectancy.

0:17:57

Those are two examples from my archives. Emily Goat, and then I'm going to go back to you, Steve, for the second one.

0:18:02
Emily Oster

There's a lot of projects that have died. But the thing that comes to mind is at some point in the last several years, I got very interested in scientific publishing and the process of scientific publishing and what people want to publish, how that relates to the sort of prevailing narrative. So if people think something is true, are you more likely to see papers published reinforcing that? It relates to some work that was successful in sort of a related space.

0:18:29
Bapu Jena

What were you trying to figure out? Publication bias or something?

0:18:31
Emily Oster

Yeah, publication bias, but not the kind of publication bias in favor of significant findings, but publication bias in favor of some prevailing. Like if we all think something matters, Do we publish papers that say it matters? Like, if it turns out then there's a big study that says, oh, actually, that was wrong. Then do we publish a bunch of papers saying that it's wrong? You know, is it hard to publish things that other people disagree with?

0:18:53

And so I just kept having my RAs do different things involving scraping information about publications. And we would try something and it wouldn't work. And I'd be like, oh, try this other thing. And they split on for like a year before finally those RAs graduated. And I was like, you know what? I'm not going to put other people on this. Like, I just have to let this go.

0:19:11
Bapu Jena

Let me just piggyback off that. So I did this thing where we looked at people who worked in a Veterans Affairs hospital and often VA researchers, they write about health care issues and quality of care issues in the VA system. And I thought, hmm, I bet that VA researchers, when they write studies about the VA, are going to be finding things that are positive, like the VA does this good, the VA does that good, et cetera.

0:19:36

And we didn't find that. We actually wrote that up because I thought this question of ideological bias, no matter what you found, would be interesting. And we got it published, but it's got me thinking about COVID-19, because if you think about people's preferences, let's say based on masks, could it be the case that if you looked at studies about the effects of mask mandates or various non-pharmaceutical interventions, and you classify them as positive, meaning there's an effect of them on slowing spread versus negative, meaning There's no effect.

0:20:07

And then you look at the authors of those studies and look at whether or not they were wearing masks on social media before the study was published or whatever. Would that be a way to identify a bias that people might have as they're going into research? Because like you said, and Steve said, you have an idea about what's the right answer and you're pushing the data and pushing the analysis to try to get there. And you may not get there.

0:20:27

Steve, you got another example?

0:20:29
Steve Levitt

I do have one. So somebody approached me and they asked me if I could study texting and driving and what the effect was. and I never thought about the problem, and I didn't have any great ideas. So almost on a lark, I called up a friend I had at a big insurer, and I said, hey, if I could link the data you have on driving to data on texting, would you be interested? And he said, sure.

0:20:53

And then I called up a really big telecom company, a friend I had there, and said, hey, if I could link your data to this insurance today, would you be happy to do it? And amazingly, they said yes. And we developed this partnership where we were the trusted third party of these data that they wouldn't give to the other company, but they would give to us. And we went to a lot of trouble to put it together.

0:21:13

And it wasn't perfect, but it was pretty good. Now, part of the problem was the insurance company didn't actually know when people crashed. They just knew when they slammed on the brakes. But it turned out that when people were texting, they just slowed way down. They hardly ever slammed on their brakes because I think they were driving so slow and were so far away from other cars that they probably weren't driving very well and they weren't doing anything that was discernible that was dangerous.

0:21:36

And in the end, we found that there was no impact of texting on any bad driving outcomes. And it could have been a good paper, but I think nobody would have believed it. And the companies we were working with weren't very excited about it. And then my RA went off to grad school and the project never died. It just didn't happen. It was one of those cases where this bias that arises in publication, which we had a perfectly good result.

0:22:03

We had a draft of a paper, but everybody just ran out of steam because we knew it would be such a war to try and get it published because it would go so much against people's priors, but not in a way that will make people excited, just in a way that will make them angry and frustrated. So that's a very different example of a paper failing.

0:22:18
Bapu Jena

And you mean get angry and excited and text about it to their friends while they're dropping?

0:22:21
Steve Levitt

At least it won't hit anybody while they're doing it because they'd be slowed down.

0:22:24
Emily Oster

Exactly. Yeah. Going too slow.

0:22:27
Bapu Jena

Coming up next, when is it time to bail on a project? And while it's been fun hearing about all these ideas, where they came from, what went wrong, we'll try to figure out, is there something we should be doing with them? My conversation with economists Emily Oster and Steve Levitt continues right after this.

0:22:52
Speaker 4

Your business relies on technology every day. But how confident are you in the security behind it? As your business evolves, so do the cybersecurity challenges that come with it. That's why BPM's Cybersecurity Risk Assessments help leaders better understand cyber risk, identify vulnerabilities, and develop a practical roadmap for stronger security. Build confidence in your cybersecurity strategy. Learn more at bpm.com slash cyber. Online checkout shouldn't feel like an obstacle course. card hunting, password fumbling. That's why there's Link.

0:23:26

Link auto-fills your information and protects eligible purchases. Next time you pay online, look for the green button. Link, the smarter way to pay. What do I gift my mom and nieces? Walmart has top brands you know, like Ninja, Disney, let go. Oh, dad loves music. Sonos can bring his songs to life with sound that hits just right. And my coffee-obsessed bestie. DeLonghi Brews Coffee right for morning's extra price. All those great brands at Walmart?

0:23:58

Who knew? We knew, now you do too. Shop gifts though love at Walmart.

0:24:07
Bapu Jena

So far, the economists Steve Levitt and Emily Oster and I have been sharing some of our research questions that failed to really see the light of day. whether it was because the data didn't work out or the research assistants helping us left for bigger and better things, or we just knew a particular paper would never get accepted by an academic journal. We all pretty much have too many of these stories to count, but not all failures amount to months and months of wasted time.

0:24:36

Here's Emily again.

0:24:38
Emily Oster

There are a lot of times where I spent a day or two in some data set, just trying to figure it out. So, you know, there was a period in which we had this really interesting, I think you probably still have it on my computer, like some really interesting data from India where they were doing tests, like, do you know letters and could you read and doing it for kids who were in school or not in school.

0:25:01

It's a data set that I used in various ways, but I remember spending a lot of days just like in that data, sort of poking around and trying to figure out, okay, can I say something about the relationship between knowing a little bit of math and knowing a little bit of reading and how does it evolve differently for different kinds of kids. And it just kind of never went anywhere. It's not a failed project because there wasn't really a project, but it was those moments of I kind of have an idea or an inkling of something.

0:25:23

I'm going to spend a little time looking at it. And then I really am just going to let it go at the end of a couple of days and say, you know what, there's kind of wasn't anything

0:25:30
Bapu Jena

there and it's just going to die. Yeah. Steve, how much time would you spend on an idea before you, let's say, quote unquote, gave up? What's your kind of cutoff? When's your drop dead period?

0:25:40
Steve Levitt

I'm a huge believer in what I call a fast first cut. And I think there's almost always a way to get some idea within a day about whether you have a good chance of succeeding or failing. I think a month is way too long unless you got a good reason for saying there's no way to figure out in a day.

0:26:01
Emily Oster

When I was in grad school, David Leibson told us many projects are worth two weeks. Few projects are worth six months. Maybe I'm a little slower than Steve. But that really stuck with me. And I tell my students that a lot. But I think it's hard. You know, people get like into it. They want to clean the data. It's hard for people to accept. You just got to like do something fast and see if it's worth going forward.

0:26:28
Bapu Jena

Emily and Steve, I'm sure there's plenty of people who've got really interesting ideas that the world won't hear about. What do you think we should do about them? Is there a place where this stuff should go? Is it something that you think academics should value? Because I'd be shocked to think of any academic department that would value faculty writing up a bunch of interesting null findings.

0:26:48
Emily Oster

There's this pre-registration idea in clinical trials or in any kind of randomized experiments, right, where you sort of don't want people to file drawer findings that are in the wrong direction or whatever it is. And so there's sort of that push. But what you're talking about is something different, which is like, I had this idea, it was kind of insane, it turned out to be wrong. Shouldn't people know about that? And I think what I push back is, do they need to know about that?

0:27:11

I don't know.

0:27:12
Steve Levitt

So why are we worried about zeros not being published? One is because it gives people bad incentives to go and try to manufacture positive results where they aren't really there. And that's obviously a terrible incentive. People often say, yeah, and it's good to know the zeros because then other people don't have to go and follow that same path. But I actually think there's nothing less valuable than research or time. and people going and redoing it, it's not the worst thing in the world because a lot of problems where you get zeros are actually maybe because you made a mistake.

0:27:43

I mean, an easy way to get a zero is you messed up the data and then everything kind of leads back to zero. So I wouldn't want to discourage people from retreading the same ground. And I think when you get zeros, it often, it's because you're not thinking straight and it encourages you to think better and to find something that's not a zero in the same area. So I'm less worried about the fact that zeros don't get published, we know that everyone throws away their zeros.

0:28:08

And so you have to look with a lot of skepticism on what does get published. So as long as everyone knows that it's a highly selected set of stuff that gets published, then I think the costs of throwing away these worthless zeros on with Emily is not that high. On the skepticism point, I don't feel

0:28:23
Bapu Jena

like academics rewards replication, right? I mean, there's tons of studies that come out where like extraordinarily controversial, could be very important, would be great to have it replicated. But we don't typically see replications. And the one instance in my own history where I try to replicate a controversial finding, which is this finding that patients do better when cardiologists go to their meetings, I replicated it at different clinical scenario, different set of meetings. And I thought, you know, people would want to know that that strange finding actually held up.

0:28:54

I had like the hardest time getting it published. And the journals would just say, yeah, you know, no, you already published this. It's not that interesting to show it again. I was like, well, that's not the point. I'm not trying to show it again, per se. I'm trying to replicate it. But I feel like the incentives for that kind of activity are very limited.

0:29:09
Emily Oster

I think that's why a lot of certain psychology, social psychology, which had a little bit more of a crisis of faith around this, has, I think, pushed towards the idea that replication is something that happens almost outside of the standard journal publication.

0:29:23
Steve Levitt

What is this outside mechanism?

0:29:24
Emily Oster

So there's this group and they have people replicate studies. They provide a platform for people to surface replications and they actually finance some of them. Gotcha.

0:29:35
Bapu Jena

Just remind me, how do I not get on the hit list? I need to figure that out. You should want to be on the hit list, Bapu.

0:29:40
Emily Oster

Yeah, it sounds like your stuff is all replicatable. It's all perfect.

0:29:49
Bapu Jena

Steve, this question is going to go to you because you have the longest arc of the field. That's another way to say that you're older than Emily and I. One thing I feel like I've seen is that economics has become, in some respects, more creative as it applies to data than it was, let's say, 30 years ago. And I want to reflect on an analogous thing that we've seen in health care, which is that there's been this big push towards quality measurement, which is, say, measuring the quality of doctors or measuring the quality of hospitals.

0:30:17

and that was all enabled by essentially improvements in computing and data. Like all this information was available, so it allowed for this field to develop in a way that it otherwise wouldn't have been able to. And my question for you is, do you think that availability of data has had an impact on the types of questions that people are asking in economics, particularly around how creative they are?

0:30:44
Steve Levitt

So let's just put this in economic terms. the cost of doing empirical research has plunged. I remember when I was first showing up at graduate school, Josh Angrist, who just won the Nobel Prize, he was complaining because it used to be if you wanted to use the census for anything, it took one RA, roughly a summer, to spin the data off of actual data tapes to get a data set together. And Josh was lamenting that, you know, with this new technology, you could do it in a day.

0:31:14

And that was destroying his advantage because he had all sorts of RAs who could do this. And now anyone could do research on it. And he was saying that somewhat tongue in cheek, but it's certainly true that it's become much, much easier. And so it used to be everybody used government data because that was really the only data he had available. But now there's enormous amounts of corporate data. There's data you can scrape. Now someone had the good idea to start using satellite data.

0:31:37

So there's all sorts of technology. Whether that's increased creativity, I haven't thought about that. I mean, certainly we have more options. My basic take on economics is not that it's overwhelmed with creativity. I would actually say my feeling about economics in the recent past is that it's really embraced technicality and sophisticated mathematics over creativity relative to what I remember when I was younger. What do you think, Emily?

0:32:06
Emily Oster

I think now people much more frequently start with the data. Like I found this interesting data set and they're thinking about that as the starting point and then trying to build a question on top of it. And I'm not always sure that that is a way into the most creative ideas. And there's a balance because if you just come up with some question and then there's no way to answer it with data, well, that's just impossible.

0:32:29

But I do think this sort of availability of data is like, you can do everything, it's right there. And yet like, well, what are you gonna do with it?

0:32:35
Steve Levitt

Yeah, Emily, I've never succeeded, although I've tried many times to start with a data set and create an interesting project without an idea. And the other thing that happens to me a lot is people reach out to me and they'll say, I have a data set, an amazing data set, and I'll sit there for an hour and I won't have the slightest idea what to do with it. So I've learned what for me is the right question to ask when I'm faced with the data set.

0:33:03

And the question I ask myself is, what is unique about this data set, this company, this situation that I couldn't do with any other data set? And if I can't come up with something that answers that question, then I know I don't have anywhere to go with that data.

0:33:21
Bapu Jena

Can we just agree that we'll do this once a year so that you can hear any ideas that I have that

0:33:25
Emily Oster

didn't work out in the 12 months? I will be back here in 12 months.

0:33:29
Steve Levitt

You're so productive, Bob, that we should do it every six months.

0:33:31
Emily Oster

I agree.

0:33:32
Steve Levitt

Did you say six minutes or six months? No, months. Every six months. You have enough bad ideas to fill a show every six months.

0:33:37
Bapu Jena

Yeah, that's right. Exactly. All right. All right. Thank you both so much. Thank you, everybody.

0:33:42
Emily Oster

Thanks, everybody.

0:33:44
Bapu Jena

That's it for today's show. Many thanks to Emily Oster and Steve Levitt for taking the time to spill their failed research questions with me. I still found them super interesting and thought-provoking. By the way, there have been some attempts to combat this publication bias issue. A bunch of journals, like the aptly named Journal of Negative Results, exist solely to try to solve this problem. The American Journal of Gastroenterology dedicated its November 2016 and May 2020 issues to null findings.

0:34:17

In 2019, the Berlin Institute of Health launched a program that offers researchers 1 ,000 euros to publish replication studies or no results. I wish I knew about that, otherwise I'd be a rich man. Anyway, I'm ready to do my part. I'd love to hear what you thought about this episode. And if you have had any ideas or research questions that never really made it off the ground, send me an email at bapu at freakonomics.com. That's B-A -P-U at freakonomics.com.

0:34:47

Thanks for listening.

0:35:17
Speaker 5

and Tricia Bobita. Our staff also includes Allison Craiglow, Greg Rippin, Emma Terrell, Jasmine Klinger, Lyric Bowditch, Jacob Clemente, and Stephen Dubner. If you like this show or any other show in the Freakonomics Radio Network, please recommend it to your family and friends. That's the best way to support the podcasts you love. As always, thanks for listening.

0:35:43
Steve Levitt

I think somebody did have a Pokemon Go paper, Bapu. Oh, yeah, there was a Pokemon Go paper. Yeah, yeah. My wife banned me from Pokemon Go when she caught me playing it while I was driving.

0:35:52
Bapu Jena

My wife banned me from Ashley Matt. No, I'm just joking. That's a joke.

0:36:00
Speaker 5

The Freakonomics Radio Network. The hidden side of everything. Stitcher.

0:36:10
Speaker 4

Your business relies on technology every day. But how confident are you in the security behind it? As your business evolves, so do the cybersecurity challenges that come with it. That's why BPM's Cybersecurity Risk Assessments help leaders better understand cyber risk, identify vulnerabilities, and develop a practical roadmap for stronger security. Build confidence in your cybersecurity strategy. Learn more at bpm.com slash cyber. Ever check out online and think, wow, that was easy? Then you might have used Link.

0:36:44

No credit card hunt, no address typing, no checkout obstacle course. Link fills in your details, protects eligible purchases, and makes subscriptions easier to manage. It's used by 300 million people, which means you might already be one of them. Next time you're at checkout, look for the green button. Link, the smarter way to pay.