OpenNashPodcasts

Masters in Business

Daniel Kahneman on Behavioral Economics (Podcast)

14 May 2021 · 50m · Daniel Kahneman, Barry Ritholtz · 26 quoted facts

▶ Play episodeOriginal source ↗

In brief

Daniel Kahneman argues that organizations suffer from two kinds of judgment error: bias, which pushes everyone the same way, and noise, which is unwanted disagreement among people who should agree. Noise is larger than leaders expect, wherever human judgment is involved, from insurance and courts to hiring. It stays hidden, and it can be reduced with structured processes.

The tension is between how reasonable disagreement feels and how costly it is. Experts expected underwriters to differ by about 10% but found 55%, and confidence grows even without feedback. Groups often hide noise by letting people influence each other. Kahneman's remedy is to measure noise with a noise audit, gather independent judgments, average them, and delay intuition.

Key insights

  1. Insurance underwriters disagreed five times more than expected

    Executives expected two underwriters to differ by about 10% on the same case, but the audit found 55%. Kahneman says executives guessed this could cost hundreds of millions or billions of dollars at a large insurer.

  2. Errors cancel only when judging the same thing

    Averaging two ratings of one risk improves accuracy, but overpricing one policy and underpricing another leaves two mistakes. Kahneman says a good average can hide poor individual performance.

  3. Letting people influence each other hides noise

    An admissions committee had officers speak in sequence because open disagreement was uncomfortable. Judgments stopped being independent, so the noise was concealed rather than removed, and a CEO speaking first makes the others' agreement uninformative.

  4. Pattern noise is the biggest source of noise

    Kahneman names three sources: differences in average severity, differences in personal taste (pattern noise), and mood or occasion. Pattern noise is the largest, because each judge reacts to cases in a consistent but personal way.

  5. Decision hygiene delays intuition rather than banning it

    Break a decision into parts, assess each factually, build a profile and only then consult intuition, much like a structured interview. Kahneman says it works against errors without knowing which specific biases are at play.

Questions

What does 'noise' mean in decision-making?

Kahneman defines judgment noise as disagreement between people, or the same person at different times, when their judgments should be identical. Examples include sentencing, pricing insurance or diagnosing illness.

How is noise different from bias?

Bias is the average error, with everyone leaning the same way. Noise is scatter, with some people guessing too high and others too low, and Kahneman says you can have zero bias and still perform poorly.

What is a noise audit and why would a company run one?

A noise audit gives several employees the same realistic cases and compares their answers. Kahneman says he first ran one at an insurer, found 55% disagreement among underwriters, and predicts most organizations will find more noise than they expect.

How can a business reduce noise in hiring?

Kahneman recommends structured interviews that collect information systematically about different traits. Better still, several people form independent impressions, average them and only then discuss, which he says Google does.

Why do experts stay confident when their judgments are inconsistent?

Kahneman says people suffer from naive realism, assuming they see the world correctly, and often get little feedback on whether they were right. Underwriters become faster and more confident because they agree with themselves.

Facts

Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.

Daniel Kahneman shared the 2002 Nobel Memorial Prize in Economic Sciences with Vernon Smith for empirical findings from his work with Amos Tversky.

“He was awarded the 2002 Nobel Memorial Prize in Economic Sciences, which he shared with Vernon Smith for his empirical findings, the work he did with Amos Tversky.”

0:02:25 · Barry Ritholtz · Nobel Prize · 2002

Barry Ritholtz says Kahneman's book Thinking, Fast and Slow was a bestseller of 2011 and won several awards.

“His previous book, Thinking Fast and Slow, was the bestseller of 2011 and won a variety of different awards”

0:02:25 · Barry Ritholtz · Thinking, Fast and Slow · 2011

Kahneman defines judgment noise as variability among judgments of the same object that should be identical.

“we speak about judgment noise, and this is a situation in which judgments should be identical.”

0:03:45 · Daniel Kahneman · Noise definition

Kahneman says the idea for the book came from a noise audit he ran while consulting at an insurance company seven or eight years before the interview.

“So I was consulting in an insurance company seven, eight years ago. and I had the idea of running what today we would call a noise audit”

0:06:25 · Daniel Kahneman · Noise audit

Executives surveyed by Kahneman's team typically said a 10% difference between two people's judgments of a quantity would be reasonable.

“and 10 % seems to be what we expect a reasonable difference to be, which is tolerable when two people make judgments of a quantity.”

0:07:12 · Daniel Kahneman · Expected disagreement

In the insurer's noise audit, the average difference between pairs of underwriters was 55%, more than five times the 10% executives expected.

“Now, the correct answer among the underwriters in that company was 55%, more than five times as much as expected.”

0:07:57 · Daniel Kahneman · Underwriter noise

Executives estimated that a 15% pricing error by underwriters could cost a very large insurer hundreds of millions or billions of dollars.

“but people estimated on that basis that this could be in the hundreds of millions or billions of dollars this is a very large company”

0:10:07 · Daniel Kahneman · Cost of noise

Kahneman says averaging two underwriters' ratings of the same risk is usually more precise than either individual judgment.

“if you have two separate underwriters estimating the same risk and you average their ratings, then the average will be usually more precise than the individual judgments.”

0:12:00 · Daniel Kahneman · Averaging

Kahneman describes an admissions committee that originally collected independent judgments and averaged them, before switching to speaking in sequence.

“The admissions committee used to operate in what we consider the correct manner. And that is, everybody would individually make their judgments, and then they would reveal all judgments together and average them.”

0:15:32 · Daniel Kahneman · Admissions committee

Kahneman defines naive realism as the assumption that one sees the world correctly, which stops people from noticing noise.

“we think we're right. We think we have the right view of situation. We think we understand things correctly. In short, we see the world as the world is. That's naive realism.”

0:19:11 · Daniel Kahneman · Naive realism

Kahneman says bail judges receive one-sided feedback because they learn about crimes by released defendants but not about those held in custody.

“So a bail judge may get feedback on somebody who was released and committed a crime, but a bail judge will never know if somebody who was incarcerated would have committed a crime.”

0:21:18 · Daniel Kahneman · Feedback

Kahneman says well-run airlines and air traffic systems track near misses closely so they can learn without accidents.

“in well-run airlines and air traffic systems, keep track very closely of near misses because those are their opportunities to learn without tragedies.”

0:22:35 · Daniel Kahneman · Near misses

Kahneman says COVID lockdowns led him and co-authors Olivier Sibony and Cass Sunstein to write Noise through daily video calls, which proved more productive than in-person visits.

“Zooming one or two hours a day turned out to be a much better way of writing the book, and this is what happened.”

0:24:22 · Daniel Kahneman · Book writing · 2020

Kahneman says structured interviews, which gather information systematically about different characteristics, hire better than unstructured interviews.

“A much better way of doing it is what is called a structured interview, a structured process, where you accumulate information systematically about different characteristics of the person.”

0:28:13 · Daniel Kahneman · Hiring

Kahneman recommends that several people form independent impressions of a candidate, average them and then discuss the average.

“And better yet is having several people do the hiring, each of them forming an independent impression, and then they average, and then they discuss the average.”

0:28:13 · Daniel Kahneman · Hiring

Kahneman and his co-authors introduce decision hygiene, a set of general procedures to reduce errors without identifying specific biases.

“we introduce a concept that we call decision hygiene. And, you know, the word is not particularly appealing, but it's intended to bring to mind the image of washing your hands.”

0:30:17 · Daniel Kahneman · Decision hygiene

Kahneman says a key principle of decision hygiene is to delay intuition until information has been gathered systematically.

“That is a key principle of decision hygiene is not to avoid intuition altogether, but to delay it.”

0:31:59 · Daniel Kahneman · Intuition

Kahneman cites evidence that judges sentence more severely on hot days and after their football team has lost.

“judges who pass sentences on criminals are more severe on hot days and they They're more severe if their football team lost the game last Sunday.”

0:34:36 · Daniel Kahneman · Occasion noise

The book identifies three sources of noise: differences in average level, pattern noise from personal taste, and occasion noise within one person.

“Those are the three are differences in average level for a judge's severity, differences in taste, what we call pattern noise, and within subject, within person variability.”

0:37:17 · Daniel Kahneman · Sources of noise

Kahneman says the biggest source of noise, which surprised the authors, is pattern noise, where individuals have distinct consistent tastes in cases.

“The biggest source of noise is that people really don't see the world in the same way.”

0:35:43 · Daniel Kahneman · Pattern noise

Kahneman says averaging multiple judgments of a case is the surefire way to eliminate noise, though it does nothing to reduce bias.

“The only way, by the way, to get rid of noise, and that's really quite important, is average judgments.”

0:38:08 · Daniel Kahneman · Averaging

Kahneman says precise rules tend to eliminate noise, while vague standards reduce it only partly.

“And rules, if followed, they're like computations. A computation is a rule, and rules tend to eliminate noise.”

0:40:20 · Daniel Kahneman · Rules and standards

Kahneman recommends that organizations where several people make the same judgment without good feedback run a noise audit by giving them the same problem.

“if you're in an organization where you have multiple people making the same judgment and no very good feedback, conduct what we call a noise audit.”

0:44:04 · Daniel Kahneman · Noise audit

Kahneman says he has returned to studying well-being, a topic he worked on about 20 years earlier.

“Almost by accident, I'm back studying well-being, and I'm involved in several research projects.”

0:44:45 · Daniel Kahneman · Next project

Kahneman says he and Amos Tversky worked closely together for about 12 years, spending many hours a day together.

“amos and i worked very closely together for about 12 years and we spent many hours a day together”

0:48:11 · Daniel Kahneman · Amos Tversky

Kahneman says a new edition of Nudge by Richard Thaler and Cass Sunstein, titled the final edition, was due in August, and that the original appeared around 2008.

“I'm reading the new edition of Nudge, which is coming out, I think, in August. And it's called Nudge, the final edition by Dick Thaler and Cass Sunstein”

0:49:15 · Daniel Kahneman · Nudge · 2008

Chapters

  1. 0:04:32Defining judgment noise

    Kahneman explains noise as unwanted disagreement among people who should agree. This frames the whole book and why it differs from ordinary error.

    “If they don't agree and are expected to agree, we speak about judgment noise.”
  2. 0:07:57The insurance noise audit

    The origin story: underwriters differed by 55% when executives expected 10%. It shows the scale of the problem and its potential cost.

    “Now, the correct answer among the underwriters in that company was 55%”
  3. 0:12:00Why errors do not cancel out

    Averaging helps only when people judge the same item. Separate mistakes across different cases add up rather than offset.

    “But errors when you're responding to different objects do not cancel out.”
  4. 0:16:07The illusion of agreement in committees

    Admissions officers shifted to speaking in sequence, which hid disagreement. This explains why group discussion can mask noise instead of fixing it.

    “when judgments are not independent, they are less effective statistically.”
  5. 0:19:45Naive realism and missing feedback

    People assume others see the world as they do, and professionals rarely learn whether they were right. Confidence grows while accuracy goes unchecked.

    “naive realism prevents us from becoming aware of the amount of noise that there is”
  6. 0:28:13Hiring and structured interviews

    Unstructured interviews are a poor hiring method. Structured, independent assessments perform better, though luck still plays a role.

    “A much better way of doing it is what is called a structured interview”
  7. 0:31:59Decision hygiene

    Kahneman's practical remedy treats options like job candidates and delays intuition until facts are gathered.

    “not to avoid intuition altogether, but to delay it”
  8. 0:35:43The three sources of noise

    Level differences, occasion noise and pattern noise combine to produce the variability seen in judgments, with pattern noise the largest.

    “people really don't see the world in the same way”
  9. 0:38:08Averaging, rules and standards

    Averaging independent judgments reliably removes noise, and precise rules cut it more than vague standards.

    “wherever there is judgment, there is noise and more of it than you think”
  10. 0:48:11Amos Tversky and closing questions

    Kahneman reflects on his collaboration with Tversky and the humor in hunting for mistakes in their own thinking.

    “we were looking for mistakes in our own thinking”

Transcript

0:00:00
Speaker 3

At KKR, we've spent 50 years understanding what matters to our clients and companies. From pioneering private equity to investing across credit, real assets, insurance, and sports, KKR has built the experience to navigate complexity, the judgment to recognize opportunity, and the conviction to create lasting value. Because when people and institutions are counting on us, understanding what matters isn't just important, it's everything. KKR, Understanding what matters. Visit kkr.com slash what matters. Running a business means keeping pace with the latest tech.

0:00:34
Barry Ritholtz

You've put a lot into building your enterprise software stack and scrapping it to integrate AI is out of the question. That's why IBM technology works with all the apps, data, and models you need. It's another way IBM helps you build on the technology you rely on. Let's create smarter business.

0:00:53
Speaker 3

IBM. Visit IBM.com to learn more.

0:01:01
Barry Ritholtz

This is Masters in Business with Barry Ritholtz on Bloomberg Radio. This week on the podcast, what can I say? Another extra special guest, Danny Kahneman, Nobel Prize winner, author of Thinking Fast and Slow. His new book is Noise, A Flaw in Human Judgment. And Danny is just so knowledgeable. Please call me Danny. I feel like I have to call him Professor Kahneman. And he insists. He's 87 years old and incredibly sharp and insightful and just so much wisdom and knowledge.

0:01:41

If you liked Thinking Fast and Slow, which is about judgment error in humans, in individuals, well, noise is about how flaws in judgment within broader institutions come about. And it's a totally different area, and it's absolutely fascinating. I'm a big fan of behavioral finance in general, plus all of Danny's work. Historically, if you are remotely interested in this, then strap yourself in. This is another doozy. With no further ado, my conversation with Danny Kahneman.

0:02:19
Speaker 3

This is Masters in Business with Barry Ritholtz on Bloomberg Radio.

0:02:25
Barry Ritholtz

My extra special guest this week is Danny Kahneman. He was awarded the 2002 Nobel Memorial Prize in Economic Sciences, which he shared with Vernon Smith for his empirical findings, the work he did with Amos Tversky. And what's so fascinating about that Nobel Prize is that Danny is a psychologist. The work they did challenged the prevailing thoughts in economic theory by establishing a basis for common human errors. His previous book, Thinking Fast and Slow, was the bestseller of 2011 and won a variety of different awards, including the National Academy's Communication Award for Best Creative Work.

0:03:11

His latest book is just out, Noise, A Flaw in Human Judgment, which Danny Kahneman wrote with Olivier Sibony and Cass Sunstein. Danny Kahneman, welcome back to Bloomberg.

0:03:26
Daniel Kahneman

I'm delighted to be here.

0:03:28
Barry Ritholtz

You always say, call me Danny, and I always feel awkward, and I feel like I should call you professor, but let me just get that out of it.

0:03:35
Daniel Kahneman

I insist. Call me Danny.

0:03:38
Barry Ritholtz

All right, Danny. So let's start very basically. What is noise? How does it happen? And where does it come from?

0:03:45
Daniel Kahneman

Okay, well, the term noise is an accepted term in statistics. We talk about statistical noise, which is variability, and that's where it comes from. We talk about noise of measurement, which is unreliability in measurement, where measurements that should be identical turn out to vary. So that's the background in the use of the term. As we use it specifically, we speak about judgment noise, and this is a situation in which judgments should be identical. people or the same individual judging the same object at different times or different people, judging the same object.

0:04:32

If they don't agree and are expected to agree, we speak about judgment noise. And in general, people are expected to agree when they're trying to be accurate. So when you have a group of people trying to make their best guess about a quantity, it could be the sentence that somebody should get for a crime. It could be the value of a company. It could be the premium that somebody should be charged, or it could be a diagnosis, a medical diagnosis.

0:05:04

In all these cases, you might have several people looking at the same information, making judgments. If they don't agree, there is noise. And noise is the topic of the book we wrote.

0:05:17
Barry Ritholtz

So it's fascinating how we start to see noisy decision-making come up over and over again in the same fields. And you just mentioned a few, medicine, criminal justice, finance. Are there certain fields that are more susceptible to problems in expert judgment than others? Or is it just that the results of those sort of noisy decisions are so much more significant than other fields?

0:05:47
Daniel Kahneman

Well, we use the word judgment when there is room for reasonable disagreement. That is, you know, we don't use the word judgment for computation. And when computation is appropriate, we wouldn't be talking of noise. We would be talking of people making mistakes. And we talk about noise when it's a matter of judgment. And so the existence of noise by itself is not a surprise. What is a surprise is the amount of noise. There's just a lot more than would be expected.

0:06:25

And here, I think the best way to explain this is to tell you the story of how I started to work on noise and where the whole thing began. So I was consulting in an insurance company seven, eight years ago. and I had the idea of running what today we would call a noise audit that is underwriters to take one example we had several underwriters saw some realistic cases the same cases they were constructed by executives of experts in underwriting so they were completely realistic and you might have 50 underwriters looking at the same premium.

0:07:12

Now, nobody would expect the numbers to be exactly the same, but I asked executives, if you take a pair of underwriters at random, by how much would you expect them to differ in percentages? That is, you take the average of the pair, you take the difference, you divide the difference by the average, what percentage looks reasonable to you? And their answer typically was 10%. And we have, by the way, we have surveyed hundreds of executives since then, and 10 % seems to be what we expect a reasonable difference to be, which is tolerable when two people make judgments of a quantity.

0:07:57

Now, the correct answer among the underwriters in that company was 55%, more than five times as much as expected. That's the phenomenon. So we expect disagreement where judgment is involved. We just don't expect that much disagreement. And this basically was the observation that started us on that path of writing a book, Because it turns out that you find astonishing amount of disagreement when you look for it. And you find it wherever judgment is involved. So engineers who make estimates on the basis of objective data, they don't have a problem of judgment.

0:08:40

But to the extent they do have a problem of judgment, you will expect a lot of noise. So that's the basic finding. And, you know, wherever precision is important, wherever it is important to get to the right number, noise is a source of error. So if you have some people overestimating and others underestimating, they're making errors.

0:09:02
Barry Ritholtz

So let me roll back to that insurance company, which you discuss in the book. And there were two particular areas where there were these broad disagreements. The first was when people were trying to estimate the risk involved with some insurance. And so how you price that very much determines if you're too expensive, meaning you think it's a high risk, you're not going to win the business. And if it's too cheap relative to the risk, well, you'll win the business, but it won't be profitable.

0:09:40

The cost will be higher. And then on the other end, in the appraisal of, hey, what are the damages here, figuring out how much something should be covered by insurance, what the dollar amount is, and the same situation. You can't be too stingy or you lose customers, but you can't be too generous or you give the house away. How significant a financial issue was this for the insurance company?

0:10:07
Daniel Kahneman

Well, you know, it's not easy to estimate that exactly, but I can tell you the question that I asked some executives. I said, suppose there is a correct number, say, for the underwriters, and you have somebody who overestimates the underwriting cost by 15%. How much would you expect that to cost the company? and the same question for underestimating by 15% now in fact 15% on either side is much less than noise than we had discovered but people estimated on that basis that this could be in the hundreds of millions or billions of dollars this is a very large company so So underwriters have a lot of important decisions to make.

0:11:01

Claims adjusters make important decisions which are really consequential for the company. And errors of the magnitude that we observe are costly. The main reason that they may be less costly is that if error is present in all insurance companies, if all insurance companies are noisy, then some of the damage to each individual company will be reduced. But that's the best that we can say.

0:11:29
Barry Ritholtz

Well, one would imagine the insurance company that could reduce noise would find itself at a competitive advantage. There was something you had written that really stood out to me. There's an assumption when you have noisy systems in everything from criminal justice to medicine to insurance that these errors tend to cancel out. But you found out that noisy systems have errors. Not only do they not cancel out, they tend to add up. Explain.

0:12:00
Daniel Kahneman

Well, if you have two separate underwriters estimating the same risk and you average their ratings, then the average will be usually more precise than the individual judgments. because errors in measuring the same object do cancel out. But errors when you're responding to different objects do not cancel out. So if you overprice one policy and you underprice another policy, that doesn't cancel out. You've made two mistakes. And, you know, it's the same thing with two judges. if one defendant is punished too much and another defendant is punished too little.

0:12:49

On average, you know, punishment was right. But who cares about the average? Two mistakes were made. So there is some confusion because people think about canceling out, but that happens when people evaluate or judge or measure the same thing there, and errors do cancel out.

0:13:09
Barry Ritholtz

I recall a book a couple of years ago called The End of Average that looked at that exact issue and said, you know, we tend to look at these averages as if anyone is experiencing an average. But what you're really saying is, hey, if it averages out to be the right answer, it means you have a lot of long answers. That's right.

0:13:32
Daniel Kahneman

Averaging out to the right answer is not a guarantee. And that is a nice example of the phenomenon we're discussing in the book, the neglect of noise. People really tend to focus on bias, which is the average error. But you can have a zero bias and a very poor performance if you have a lot of overestimates and a lot of underestimates.

0:13:57
Barry Ritholtz

estimates. Quite interesting. So one of the things in the book that I was so taken by had to do with the admissions committee for a university, and they used to have all the admission officers do a blind review and get together and try and hash out who they thought would be a good fit for the school and who wouldn't. But it led to a problem, and they started having the first person who reviewed the application put their review number on the corner, like they would actually put their rating on the page and then hand it off to the second person.

0:14:38

And you describe this as the illusion of agreements in organizations. Tell us about that.

0:14:46
Daniel Kahneman

Well, you know, this is an experience that any teacher has had, for example. when you are looking at a test booklet, a student has written several essays. If you score a test booklet, you score the first question, then the second, then the third, then in general, you'll find that your grades do not vary very much. On the other hand, if you read the same test across all students and write the score at the back of the booklet so that you don't know when you read the second question where the first question was, you will often be shocked by the discrepancy between the first and the second.

0:15:32

There is a mechanism by which people, if you gave a good grade the first time, you're going to be inclined to give the benefit of the doubt to the student if there is any ambiguous answer. Exactly the same thing happens in deliberations and in the example that we gave. The admissions committee used to operate in what we consider the correct manner. And that is, everybody would individually make their judgments, and then they would reveal all judgments together and average them.

0:16:07

But they changed the system so that now people spoke in sequence. And the question was asked, why do you do this? This is not optimal. And they say, well, we used to do it the other way. We used to have people prepare their judgments individually, but there was so much disagreement that we stopped. And that's an example where people manage to avoid finding out how much noise there really is. Because when people are allowed to influence each other or influence themselves, in the case of a teacher reading multiple booklets, when judgments are not independent, they are less effective statistically.

0:16:50

You just have less information. Think of the example in which the first person to talk is a CEO, and then everybody agrees. Then the agreement of other people is not informative. In fact, you have one person making the judgment. That's the extreme of abolishing, of eliminating the appearance of noise without eliminating the reality.

0:17:15
Speaker 3

At KKR, we've spent 50 years understanding what matters to our clients and companies, From pioneering private equity to investing across credit, real assets, insurance, and sports, KKR has built the experience to navigate complexity, the judgment to recognize opportunity, and the conviction to create lasting value. Because when people and institutions are counting on us, understanding what matters isn't just important, it's everything. KKR, understanding what matters. Visit KKR.com slash what matters.

0:17:46
Barry Ritholtz

So it sounds like groups and corporations, institutions, schools, they seem to amplify noise. Is that just the nature of bigger numbers of people working together, that they're going to create additional noise?

0:18:03
Daniel Kahneman

No, not necessarily. What happens in a group, if they make their dreadments individually, is not that noise is amplified. The true noise is revealed. So suppose you had underwriters. Suppose you had multiple underwriters judging routinely every risk. Then the optimal procedure would be to have them making independent judgments and only then revealing the two judgments and averaging them. That's clearly the optimal procedure. Sure. are. And the optimal procedure reveals noise and then reduces it by averaging.

0:18:46

But when a single individual makes a judgment, that judgment will be noisy. And when individuals are allowed to influence each other, then it's more like a single judgment than it is like having multiple judgments are the same object.

0:19:03
Barry Ritholtz

So you use the phrase naive realism. What does that mean relative to noise in groups?

0:19:11
Daniel Kahneman

Well, what naive realism means is a statement which most of us are most of the time, that we think we're right. We think we have the right view of situation. We think we understand things correctly. In short, we see the world as the world is. That's naive realism. And if I see the world as it is, and there are friends and colleagues looking at the same world, and I like and respect them, then naturally I assume that they see the world as I do because I see it right.

0:19:45

And if I respect them, they see it right as well. So that's naive realism. And naive realism prevents us from becoming aware of the amount of noise that there is. We just assume noise away. We saw that very nicely among underwriters. When you interview an underwriter, what happens to them? How does an underwriter become expert in the absence of any feedback? Because they don't get any feedback from reality about their underwriting. and what happens is that they become increasingly confident and largely because they agree with themselves so when you agree with yourself a lot and you think you're right and you make judgments with increasing speed and confidence so that makes you think that you're even brighter that's naive realism

0:20:38

allowing massive noise to occur with everybody convinced that they're doing the right thing, but in fact, they may not be doing the right thing because if they were looking at the

0:20:49
Barry Ritholtz

same problem, that would be the fruit. Quite fascinating. So we become familiar with a particular area. That familiarity leads us to think that we're developing an expertise. We tend to make more snap judgments. And without any sort of feedback loop, how can we possibly We know that we're right, and yet that absence of feedback seems to strengthen people's self-confidence. Do I have that right? That's right.

0:21:18
Daniel Kahneman

And think of a number of situations in which exactly this holds. There's a judge who doesn't have feedback as to whether a judgment was correct or not. A bail judge, sometimes there is feedback, but it's asymmetric. So a bail judge may get feedback on somebody who was released and committed a crime, but a bail judge will never know if somebody who was incarcerated would have committed a crime. So feedback is a massive problem, and many professionals act with minimum feedback, and yet they become confident and they feel they're experts.

0:21:55

But in those cases, there is a high risk of noise.

0:21:59
Barry Ritholtz

And a lot of that feedback seems to be only at the extreme. A bridge collapses, a plane crashes, somebody dies, someone out on bail commits a crime. What about all of the, for lack of a better word, near misses where there is a bad judgment, something happens, it's not quite as terrible as an airplane crash, and it's resolved before there's damage, but it's pretty clear the basic judgment was wrong. How does that affect a person's future judgment?

0:22:35
Daniel Kahneman

Well, in situations where there are near misses, there is an opportunity to learn. And in well-run airlines and air traffic systems, keep track very closely of near misses because those are their opportunities to learn without tragedies. But in many situations, you get no feedback at all. I mean, the idea of having sensors in bridges that gives you a sensitive measurement of how much stress there is, that is fairly recent. There used to be very poor feedback on whether a bridge would collapse or not.

0:23:15

And in many situations that professionals make judgment on, there is no feedback at all.

0:23:21
Barry Ritholtz

Quite interesting. So let's talk about this book, which was a collaboration. What was it like working with those two gentlemen versus Thinking Fast and Slow, which I kind of get the sense was you sitting down and putting a lot of your previous work into a context for public consumption.

0:23:47
Daniel Kahneman

Well, writing fast and slow was mostly a very lonely experience. And writing with collaborators was really a pleasure. So it was a relief to be able to count on people, to find mistakes, to correct them. And a lot of the text was actually written by Olivier and by Cass. I had a lot to do with outlining and with critiquing and with rejecting drafts, but I was spared much of the things that I'm most afraid of in writing.

0:24:22

So it was a very good collaboration. And by the way, we benefited a lot from COVID because that forced us into quite an efficient way of collaborating. We used to visit. Olivier would come to New York from Paris, and I would visit Paris for a few days every month. And we had a very good time, but it wasn't productive. Zooming one or two hours a day turned out to be a much better way of writing the book, and this is what happened.

0:24:54
Barry Ritholtz

It sounds like it was just a good excuse to get together in New York and Paris and have a little bit of fun.

0:24:59
Daniel Kahneman

Well, I mean, you know, we didn't think of it as a good excuse, but it turned out that we'd waste a lot of time and a fair amount of money.

0:25:07
Barry Ritholtz

So you mentioned you reviewed a lot of manuscript from Olivier and Cass and rejected stuff. You and Amos very famously would agonize over every sentence in all of your publications. You seem to have spent a lot of time writing meticulously and very thoughtfully. How has that evolved over time? Is this a little easier to sort of be the orchestrator and the editor as opposed to, you know, just agonizingly putting down every single word?

0:25:44
Daniel Kahneman

No, it isn't. I mean, this is part of sort of my intellectual personality or character that I think most clearly when I find flaws in existing texts, and I'm not good at anticipating the flaws. So I see a flaw and I correct it, and then there is new text, and then I discover a new flaw. And I tend to work that way, which is infuriating to my collaborators, and waste a lot of time and effort, but that's the way I am.

0:26:14

On the other hand, I do tend to be very critical, and most of the flaws that I find do exist. So it tends to lead to a good project in a very inefficient way.

0:26:25
Barry Ritholtz

So despite that perfectionism, we all evolve over time. As you were preparing Noise, did you find any of your previous writings or research that you either disagree with or see from a different perspective or light when you were putting this book together?

0:26:45
Daniel Kahneman

No, not really. I mean, in the book, we actually relied on ideas from Thinking Fast and Slow. But the book is really fundamentally different. Thinking Fast and Slow is a book about individuals. And it was a book about the average or a typical individual and how the average or typical mind works. noise is about individual differences. It's about the way that different people think differently. And so this is a really different cut about thinking. It's a different way of looking at thinking.

0:27:22

So we did use some of the material, but noise is not a revision of thinking fast and slow. It is about a truly different topic that we didn't even touch in thinking fast and slow.

0:27:36
Barry Ritholtz

It clearly, it goes in a very different direction and it looks at some systems and some organizations that I don't believe you touched on in Thinking Fast and Slow. So it's kind of interesting because we've already talked about medicine and criminal justice and finance. There was one section I was fascinated by where you discussed hiring and promotions and how, I don't want to use the word random, but how much noise is in that system and how unreliable many organizations' hiring processes are.

0:28:11

Tell us a little bit about that.

0:28:13
Daniel Kahneman

Well, it turns out that people like hiring by interviewing people and forming a general image of the individual that they are thinking of hiring. And it turns out this is not a good way of doing it. A much better way of doing it is what is called a structured interview, a structured process, where you accumulate information systematically about different characteristics of the person. That is less pleasant. It's less enjoyable, but much better. And better yet is having several people do the hiring, each of them forming an independent impression, and then they average, and then they discuss the average.

0:29:05

And this is a procedure, for example, at Google, and it's about state of view. but many places are way short of a state-of-the-art. I should add that state-of-the-art hiring doesn't mean that you're guaranteed a perfect fit. There's so much luck in the world. There's so much uncertainty that the person that you hire may be very good but may run into difficulties with a boss who doesn't like her or something like that.

0:29:39

And by chance alone, you can get a lot of variety. Chance, by the way, is not noise. Chance is something that happens in the real world. Noise is differences among judgments. So hiring is by and large really very poorly done. And it's very poorly done because it doesn't control noise.

0:30:00
Barry Ritholtz

Quite fascinating. So the book goes over how noise affects judgment and how it introduces a variety of errors into our institutional decision-making process. What can we do to improve that process?

0:30:17
Daniel Kahneman

Well, in the book, we introduce a concept that we call decision hygiene. And, you know, the word is not particularly appealing, but it's intended to bring to mind the image of washing your hands. And there is a contrast between debiasing and decision hygiene. Debiasing is like medication or like vaccination. It's specific to a particular disease. When you wash your hands, you don't know which germs you're killing. And if you're successful, you'll never know. So decision hygiene is oriented to improving decision-making and avoiding errors, specifically avoiding noise, but incidentally also avoiding bias, without knowing precisely what biases you're trying to control.

0:31:12

And we have a variety of procedures that we think of as decision hygiene.

0:31:17
Barry Ritholtz

Give us a few examples. What are some of the procedures?

0:31:20
Daniel Kahneman

Well, I'll give you an example that has to do with decision-making. So suppose you are making a decision, and the first step that everyone will tell you is you have to consider your options and have the best possible set of options. But now you come to evaluate options. How do you do that? And here, actually, our advice, we have a slogan. We say options are like candidates. You should think of options in the same way that organizations are advised to operate when they hire candidates.

0:31:59

And we were talking about that earlier. Structure the thinking. Break up each option. Look at the various aspects of it. Assess these aspects in a fact-based way, which is the equivalent of interviewing somebody about different aspects of her character or her experience. and then create a profile of all the information you have about that option and only then invoke intuition. That is a key principle of decision hygiene is not to avoid intuition altogether, but to delay it.

0:32:41

Because intuition is way more effective if it is preceded by a period in which you accumulate information systematically. So that's an example. I have many others.

0:32:53
Speaker 4

With ChatGPT, you can do more than get answers. You can get things done. When you're running a business and looking to expand, you need a clear view of the landscape before making any decisions. ChatGPT can help you prepare. Connect your Google Drive, point to relevant files and reports, and ask Chat to research competitors, industry forecasts, and market signals. It can turn the findings into a strategic deck in your team's template that you can review before presenting to your leadership team.

0:33:22

Growing a business is ambitious, and ChatGPT can help you do it with confidence. Get started at chatgpt.com.

0:33:30
Barry Ritholtz

This is what... And there were quite a few in the book. There were some things that really surprised me about that decision-making process, how people's moods affect their decisions, even the weather affects decision-making. We are essentially different people at different times.

0:33:46
Daniel Kahneman

Oh, yes. That is, there are different sources of noise that we talk about. So there are three of them, actually. But one of them is what we call within-person noise. And that is that the individual is indeed making different judgments depending on circumstances that are irrelevant. So it's true there is evidence that mood really affects the way that people think. people tend to be more creative when they're in a good mood but they tend to be also more gullible and they are more critical when they're in a bad mood so mood affects the way we think and it also affects we're more prone to see good things when we're in a good mood mood is important there is evidence

0:34:36

that judges who pass sentences on criminals are more severe on hot days and they They're more severe if their football team lost the game last Sunday. So there are a lot of irrelevant events or circumstances that influence our judgment. This is one of the three major sources of judgment.

0:34:59
Barry Ritholtz

Let's get to the other two. What are the other two sources?

0:35:02
Daniel Kahneman

Well, one other, which is easy to think about, it's very easy to think about it in terms of judges. Some judges are more severe than others. So their sentences on average are more severe than the sentences of other judges. That's one aspect. And the same is true by the web underwriters. Some underwriters write large premiums on average, and other underwriters write small premiums on average. So there are differences of that kind. But it turns out that the biggest source of noise, And that came as a surprise to us.

0:35:43

The biggest source of noise is that people really don't see the world in the same way. So that different judges have different tastes in crimes and different tastes in criminals. So somebody may be particularly severe about repeat offenders. Somebody else might be extremely lenient, say, about white-collar crime, but really upset by violence. And it turns out that there is, we call that a pattern noise. That is, each judge, each individual has a pattern of judgments, which is different from the pattern of judgments of other people.

0:36:29

And that is the major source of noise. And people are consistent in that way. So, for example, suppose you're a judge and somebody reminds you of your daughter, whether that makes you more lenient or more severe, probably more lenient. Now, on another day, that same person would also remind you of your daughter. So this is not noisy within the individual. This is a characteristic of the individual, but no other judge shares it. And it turns out that this highly case-specific differences in attitudes that are difficult to pin down, they are noise.

0:37:07

Judges have personalities, and judgments differ as much as personalities do.

0:37:13
Barry Ritholtz

And then what is the third source of noise that you identified in the book?

0:37:17
Daniel Kahneman

Those are the three are differences in average level for a judge's severity, differences in taste, what we call pattern noise, and within subject, within person variability. We call that occasion noise because on different occasions you make different judgments. And it's the sum of these three sources of noise that creates, that's the noise that we observe. So in the system, all three operate on any particular judgment.

0:37:50
Barry Ritholtz

So I'm going to ask the question I was thinking about a little differently based on what you just said. What fields seem to manage reducing noise better than others? And are there any fields that are especially susceptible to noise?

0:38:08
Daniel Kahneman

That's a very good question to which I do not have a very good answer. because in our work, we found noise wherever we looked for it. Indeed, our summary conclusion is wherever there is judgment, there is noise and more of it than you think. You know, this has been our conclusion. So we haven't found places that control noise very efficiently. The only way, by the way, to get rid of noise, and that's really quite important, is average judgments.

0:38:44

Take multiple judgments of a case and average them. And this mechanically eliminates noise. If you have enough judgments, the average, it may be biased because averaging does nothing to reduce bias, but it eliminates noise. So that's a surefire way of eliminating noise is averaging multiple cases.

0:39:07
Barry Ritholtz

Very interesting. Let me throw a curveball at you. If you were designing a system to introduce noise to short-circuit human judgment, what would you create to make judgment less effective, noisier?

0:39:25
Daniel Kahneman

I don't think I would do things very differently from the way that they are done in many institutions now. I would let people make individual judgments without feedback. That's all that's needed. Make their individual decisions without feedback, which is a situation that's very common. And that will create a lot of noise eventually. And noise is reduced by feedback. Sometimes it's the feedback of other people. So case conferences can be arranged to some extent control noise. But, you know, you don't have to try very hard to create a lot of noise.

0:40:06

I think the existing organizations do very little to control noise.

0:40:11
Barry Ritholtz

So let's talk a little bit about ways to control noise. And you describe a difference between rules and standards. Tell us about that.

0:40:20
Daniel Kahneman

Well, standards is a way of when you say, for example, that obscenity is something that you recognize. So there is a standard to avoid obscenity, but you do not define it. That's a standard. A rule is more precise than that, and it tells you specifically what you have to do. And rules, if followed, they're like computations. A computation is a rule, and rules tend to eliminate noise. Standards sometimes reduce noise, but standards do not eliminate noise because they're vague.

0:40:58
Barry Ritholtz

So the seven words you can't say on television is a rule, but pornography, I know it when I see it, is a standard? Is that the difference?

0:41:07
Daniel Kahneman

Precisely. Precisely.

0:41:08
Barry Ritholtz

Quite interesting. So given all of the work you've done over the years, all of your research, you seem to have continually identified flaws in cognition, flaws in human judgment. Has this affected the way you view other people? Do you turn around and say, wow, this species is a terrible decision-making apparatus, or is it something less comprehensive than that?

0:41:38
Daniel Kahneman

No, actually for all my career, I've been interested in intuition and intuitive thinking. And I've been interested, and that's a lecture I used to give for many years, intuitions, marvels, and flaws. Because intuition is marvelous, but it's also flawed. And it's true that I have found it more interesting to study the flaws of intuition than its marvels. And there is a lot to do to correct the flaws of intuition. But to say that this has turned me into a pessimist or that I dislike people because their minds are flawed, I think the minds are pretty marvelous.

0:42:20

But they are certainly far from perfect.

0:42:23
Barry Ritholtz

Right. So you're focusing on the small bits that we get wrong. But overall, we manage to navigate through life pretty effectively.

0:42:31
Daniel Kahneman

Well, we certainly manage to navigate through life. And, you know, it would be absurd to focus on the flaws of human beings when you see what they're capable of. On the other hand, if you want to do things better, then you'd better focus on the flaws rather than on what is going well.

0:42:52
Barry Ritholtz

You know, one of the things you said when we spoke last about thinking fast and slow, I asked you about your own investing process. And you said despite knowing everything that you know about human decision making, you still catch yourself making the same sort of mistakes that everybody makes. Is that still the case? Do you still feel that way?

0:43:20
Daniel Kahneman

Oh, yes. I mean, you know, I've been at it for more than 50 years, and I'm really not better than I was. In general, my thinking has been, and it was true when I wrote Thinking Fast and Slow, which was focused on individuals, that the hope of improving thinking is in organizations because organizations think slowly, and they have procedures. and it's by imposing procedures, by adopting procedures, that you can improve things. And in the case of noise, we have a procedure that we recommend to get started, and that's measure noise.

0:44:04

If you're in an organization where you have multiple people making the same judgment and no very good feedback, conduct what we call a noise audit. Give them the same problem and look at their solution. We predict that you'll find more noise than you think you will. That's our prediction. And that's a recommendation to organizations. It's not something that you can recommend to an individual.

0:44:33
Barry Ritholtz

Quite interesting. I have to ask you before we get to our favorite questions, what's the next project? What's the next book look like? What is tickling your curiosity these days?

0:44:45
Daniel Kahneman

Well, actually, I'm back to a topic that I was working on 20 years ago. Almost by accident, I'm back studying well-being, and I'm involved in several research projects. None of them is as big or ambitious as noise was or thinking fast and slow, but all of them are quite interesting, so I'm not bored.

0:45:12
Barry Ritholtz

I can't picture you bored because you always seem to have a lot of different things going on. Let me ask my favorite questions that I ask all of our guests. And let's start with what are you doing to stay entertained during this pandemic lockdown? In addition to working on the book, what are you streaming? What are you watching on Netflix, if anything?

0:45:34
Daniel Kahneman

Oh, I've been watching several series, several very good series. Let's see, the last ones. There is a political series on Netflix, Le Baron Noir, which is a French political thriller. That is very good. There is a Danish political series, Borgen, that is very good. I am now watching Babylon Berlin, about Berlin in the 1920s, which is excellent. And so I do my watching mainly when I exercise, and I exercise a fair amount. But I've seen a lot of series since, well, for the last few years.

0:46:16
Barry Ritholtz

So Balm Noir was the French one. What was the Danish one?

0:46:20
Daniel Kahneman

Bogen. Bogen is bridge. Actually, the Danish one, Bogen is a thriller. It's a Scandinavian thriller. There is a Danish one about a woman prime minister, and it's not Borgen, and I now block on its name, but it will be easy to find. And I really recommend it. It is superb.

0:46:40
Barry Ritholtz

All right, I will check that out. So let's talk about your early mentors. Who helped to shape your career? And I guess we have to include collaborators in that as well.

0:46:51
Daniel Kahneman

Well, I mean, there's been one major influence on my career, and it was Amostrosky. The collaboration with him completely changed my life, and it changed the way I do things, but it gave me a—yeah, it changed my life. And it was the best period of my life, too, professionally.

0:47:16
Speaker 4

With Chachi Bt, you can do more than get answers. You can get things done. When you're running a business and looking to expand, you need a clear view of the landscape before making any decisions. ChatGPT can help you prepare. Connect your Google Drive, point to relevant files and reports, and ask chat to research competitors, industry forecasts, and market signals. It can turn the findings into a strategic deck in your team's template that you can review before presenting to your leadership team.

0:47:45

Growing a business is ambitious, and ChatGPT can help you do it with confidence. Get started at ChatGPT.com.

0:47:53
Barry Ritholtz

The thing I recall from Michael Lewis's undoing project is that people said you guys would lock yourself into an office or a classroom, and all they would hear all day long is peals of laughter coming from within. Is that true? Is that an exaggeration? Or did you guys...

0:48:11
Daniel Kahneman

No, that's really not an exaggeration. amos and i worked very closely together for about 12 years and we spent many hours a day together and he was very funny he had an excellent sense of humor and he loved laughing and in his presence also became funny so we were amusing each other and the field that we studied uh was was one that was conducive to laughter because we were looking for mistakes in our own thinking. And to trap ourselves or to see that you are tempted to make a stupid error, that is quite funny.

0:48:57

And that's the game that we engaged in, in studying judgment and in studying decision-making, was looking for errors in our own thinking. And that was very amusing. I can imagine.

0:49:09
Barry Ritholtz

So let's talk about books. What are some of your all-time favorites, and what are you reading right now?

0:49:15
Daniel Kahneman

Well, I would say my all-time favorites of recent years were Sapiens. I think it's many people's favorite book by Yuval Noah Harari. I read it twice, which is something that I rarely do. And right now, well, I'm reading the new edition of Nudge, which is coming out, I think, in August. And it's called Nudge, the final edition by Dick Thaler and Cass Sunstein, the same Cass Sunstein. And it's quite different from the original Nudge, which appeared, I think, in 2008.

0:49:53

But it had the same characteristic that Nudge had. It's wise and it's funny.

0:49:59
Barry Ritholtz

Right. Dick said it's about two-thirds new, and I think that's August 4th that comes out.

0:50:05
Daniel Kahneman

Yeah, that's right. So I happen to be reading that right now.

0:50:09
Barry Ritholtz

August 3rd, I'm looking at a message from him. He's a very amusing person to begin with, and if you're telling me the book is funny, then I am really looking forward to reading that.

0:50:23
Daniel Kahneman

Oh, the book is, you know, he just, he can't help himself. He's funny all the time. He's my best friend, my best living friend.

0:50:31
Barry Ritholtz

Let me ask you this question. If a recent college graduate asked you for some advice, if he was interested in a career in either psychology or behavioral finance, what sort of advice might you give him?

0:50:47
Daniel Kahneman

Well, you know, I tend to refrain from advice because I don't believe I have a crystal ball into the future. I can tell you what I would have been doing if I was starting today. The fields that are very exciting from my perspective are neuroscience, including neuroeconomics, which is the neuroscience of decision making, and artificial intelligence. I mean, in those two areas right now, there are extraordinary developments, very exciting. And so when you see that, and they're attracting massive talent, both areas.

0:51:28

So you know that for the next decade or so, they'll be cooking. A lot is going to happen. And what happens after that, I have no idea.

0:51:38
Barry Ritholtz

And our final question, what do you know about the world of psychology, behavioral finance, economics today that you wish you knew 50 or so years ago when you were first getting started?

0:51:52
Daniel Kahneman

Oh, well, so much has been learned. I can't say that I wish I had known earlier, but it has been so much fun to find out over the years, both in my work and in the work of others. So I can't think of things that would have made me act differently. But all I can say to you is, oh, yes, things have really changed since I've been in that field.

0:52:22
Barry Ritholtz

Huh, quite fascinating. Thank you, Danny, for being so generous with your time. We have been speaking with Danny Kahneman, whose new book, Noise, A Flaw in Human Judgment, was co-authored with Olivier Sibony and Cass Sunstein. If you enjoy this conversation, check out any of our previous 400 such interviews. You can find those at iTunes, Spotify, Google, Bloomberg.com, wherever you get your podcast each week. We love your comments, feedback, and suggestions. Write to us at mibpodcasts at Bloomberg.net.

0:52:57

You can sign up for my daily reads at Ritholtz.com. Check out my weekly column on Bloomberg.com slash opinion. Follow me on Twitter at Ritholtz. I would be remiss if I did not thank the crack staff that helps put together this conversation each week. Tim Harrow is my audio engineer. Matiko Valbron is my project manager. Michael Boyle is my producer. Michael Batnick is my head of research. I'm Barry Ritholtz. You've been listening to Masters in Business on Bloomberg Radio.

0:53:35
Speaker 3

Buy a digital pack, rip it open, and reveal a real physical card. Football, basketball, even Pokemon. Packs start at just $5. Ship it straight to your door or sell it back instantly at full market value. Max value up to $125 ,000. Download Underdog and get in on the chase. Deposit $5, get five free packs. Real rips, real cards, real cash. Only on Underdog. Unleash your dog. 18 plus terms apply. Top grails are extremely rare. This is Jana Kramer.

0:54:03

Chat GPT for teens is designed to help teens learn, think critically and grow their skills. Chat GPT for teens supports active learning by breaking concepts down rather than giving teens the answer. Study mode helps teens work through problems and interactive quizzes test their knowledge and provide immediate feedback. Safety protections are on by default and parental controls offer families additional options like setting study hours and quiet hours. Visit chatgpt.com slash parents to learn more.