In brief
Daniel Kahneman argues that organizations suffer from two kinds of judgment error: bias, which pushes everyone the same way, and noise, which is unwanted disagreement among people who should agree. Noise is larger than leaders expect, wherever human judgment is involved, from insurance and courts to hiring. It stays hidden, and it can be reduced with structured processes.
The tension is between how reasonable disagreement feels and how costly it is. Experts expected underwriters to differ by about 10% but found 55%, and confidence grows even without feedback. Groups often hide noise by letting people influence each other. Kahneman's remedy is to measure noise with a noise audit, gather independent judgments, average them, and delay intuition.
Facts
Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.
Daniel Kahneman shared the 2002 Nobel Memorial Prize in Economic Sciences with Vernon Smith for empirical findings from his work with Amos Tversky.
“He was awarded the 2002 Nobel Memorial Prize in Economic Sciences, which he shared with Vernon Smith for his empirical findings, the work he did with Amos Tversky.”
0:02:25 · Barry Ritholtz · Nobel Prize · 2002
Barry Ritholtz says Kahneman's book Thinking, Fast and Slow was a bestseller of 2011 and won several awards.
“His previous book, Thinking Fast and Slow, was the bestseller of 2011 and won a variety of different awards”
0:02:25 · Barry Ritholtz · Thinking, Fast and Slow · 2011
Kahneman defines judgment noise as variability among judgments of the same object that should be identical.
“we speak about judgment noise, and this is a situation in which judgments should be identical.”
0:03:45 · Daniel Kahneman · Noise definition
Kahneman says the idea for the book came from a noise audit he ran while consulting at an insurance company seven or eight years before the interview.
“So I was consulting in an insurance company seven, eight years ago. and I had the idea of running what today we would call a noise audit”
0:06:25 · Daniel Kahneman · Noise audit
Executives surveyed by Kahneman's team typically said a 10% difference between two people's judgments of a quantity would be reasonable.
“and 10 % seems to be what we expect a reasonable difference to be, which is tolerable when two people make judgments of a quantity.”
0:07:12 · Daniel Kahneman · Expected disagreement
In the insurer's noise audit, the average difference between pairs of underwriters was 55%, more than five times the 10% executives expected.
“Now, the correct answer among the underwriters in that company was 55%, more than five times as much as expected.”
0:07:57 · Daniel Kahneman · Underwriter noise
Executives estimated that a 15% pricing error by underwriters could cost a very large insurer hundreds of millions or billions of dollars.
“but people estimated on that basis that this could be in the hundreds of millions or billions of dollars this is a very large company”
0:10:07 · Daniel Kahneman · Cost of noise
Kahneman says averaging two underwriters' ratings of the same risk is usually more precise than either individual judgment.
“if you have two separate underwriters estimating the same risk and you average their ratings, then the average will be usually more precise than the individual judgments.”
0:12:00 · Daniel Kahneman · Averaging
Kahneman describes an admissions committee that originally collected independent judgments and averaged them, before switching to speaking in sequence.
“The admissions committee used to operate in what we consider the correct manner. And that is, everybody would individually make their judgments, and then they would reveal all judgments together and average them.”
0:15:32 · Daniel Kahneman · Admissions committee
Kahneman defines naive realism as the assumption that one sees the world correctly, which stops people from noticing noise.
“we think we're right. We think we have the right view of situation. We think we understand things correctly. In short, we see the world as the world is. That's naive realism.”
0:19:11 · Daniel Kahneman · Naive realism
Kahneman says bail judges receive one-sided feedback because they learn about crimes by released defendants but not about those held in custody.
“So a bail judge may get feedback on somebody who was released and committed a crime, but a bail judge will never know if somebody who was incarcerated would have committed a crime.”
0:21:18 · Daniel Kahneman · Feedback
Kahneman says well-run airlines and air traffic systems track near misses closely so they can learn without accidents.
“in well-run airlines and air traffic systems, keep track very closely of near misses because those are their opportunities to learn without tragedies.”
0:22:35 · Daniel Kahneman · Near misses
Kahneman says COVID lockdowns led him and co-authors Olivier Sibony and Cass Sunstein to write Noise through daily video calls, which proved more productive than in-person visits.
“Zooming one or two hours a day turned out to be a much better way of writing the book, and this is what happened.”
0:24:22 · Daniel Kahneman · Book writing · 2020
Kahneman says structured interviews, which gather information systematically about different characteristics, hire better than unstructured interviews.
“A much better way of doing it is what is called a structured interview, a structured process, where you accumulate information systematically about different characteristics of the person.”
0:28:13 · Daniel Kahneman · Hiring
Kahneman recommends that several people form independent impressions of a candidate, average them and then discuss the average.
“And better yet is having several people do the hiring, each of them forming an independent impression, and then they average, and then they discuss the average.”
0:28:13 · Daniel Kahneman · Hiring
Kahneman and his co-authors introduce decision hygiene, a set of general procedures to reduce errors without identifying specific biases.
“we introduce a concept that we call decision hygiene. And, you know, the word is not particularly appealing, but it's intended to bring to mind the image of washing your hands.”
0:30:17 · Daniel Kahneman · Decision hygiene
Kahneman says a key principle of decision hygiene is to delay intuition until information has been gathered systematically.
“That is a key principle of decision hygiene is not to avoid intuition altogether, but to delay it.”
0:31:59 · Daniel Kahneman · Intuition
Kahneman cites evidence that judges sentence more severely on hot days and after their football team has lost.
“judges who pass sentences on criminals are more severe on hot days and they They're more severe if their football team lost the game last Sunday.”
0:34:36 · Daniel Kahneman · Occasion noise
The book identifies three sources of noise: differences in average level, pattern noise from personal taste, and occasion noise within one person.
“Those are the three are differences in average level for a judge's severity, differences in taste, what we call pattern noise, and within subject, within person variability.”
0:37:17 · Daniel Kahneman · Sources of noise
Kahneman says the biggest source of noise, which surprised the authors, is pattern noise, where individuals have distinct consistent tastes in cases.
“The biggest source of noise is that people really don't see the world in the same way.”
0:35:43 · Daniel Kahneman · Pattern noise
Kahneman says averaging multiple judgments of a case is the surefire way to eliminate noise, though it does nothing to reduce bias.
“The only way, by the way, to get rid of noise, and that's really quite important, is average judgments.”
0:38:08 · Daniel Kahneman · Averaging
Kahneman says precise rules tend to eliminate noise, while vague standards reduce it only partly.
“And rules, if followed, they're like computations. A computation is a rule, and rules tend to eliminate noise.”
0:40:20 · Daniel Kahneman · Rules and standards
Kahneman recommends that organizations where several people make the same judgment without good feedback run a noise audit by giving them the same problem.
“if you're in an organization where you have multiple people making the same judgment and no very good feedback, conduct what we call a noise audit.”
0:44:04 · Daniel Kahneman · Noise audit
Kahneman says he has returned to studying well-being, a topic he worked on about 20 years earlier.
“Almost by accident, I'm back studying well-being, and I'm involved in several research projects.”
0:44:45 · Daniel Kahneman · Next project
Kahneman says he and Amos Tversky worked closely together for about 12 years, spending many hours a day together.
“amos and i worked very closely together for about 12 years and we spent many hours a day together”
0:48:11 · Daniel Kahneman · Amos Tversky
Kahneman says a new edition of Nudge by Richard Thaler and Cass Sunstein, titled the final edition, was due in August, and that the original appeared around 2008.
“I'm reading the new edition of Nudge, which is coming out, I think, in August. And it's called Nudge, the final edition by Dick Thaler and Cass Sunstein”
0:49:15 · Daniel Kahneman · Nudge · 2008