Topic
Roxana Daneshjou
Roxana Daneshjou is a Stanford dermatologist who runs an AI for healthcare lab, and in podcast appearances on NEJM AI Grand Rounds and the Microsoft Research Podcast she discussed bias and fairness in medical AI, especially in dermatology. She says she sees patients half a day a week and spends the rest of her time on the lab.
She told the Microsoft Research Podcast that her team's benchmark found AI models performed significantly worse on brown and black skin, and that fine-tuning on diverse skin tones eliminated the bias. She also said GPT-3.5 and GPT-4 suggested using race to calculate kidney function, which she calls untrue. A 2024 study by her team found 65 percent of surveyed dermatologists reported using large language models.
25 facts from 2 episodes, in date order. These are what speakers said on the shows, each linked to the moment it was said. They aren't independently verified.
2016
Daneshjou's team drew some of its test questions from a 2016 PNAS paper on false race-based beliefs held by medical trainees.
“there's this 2016 PNAS paper that looks at some of the incorrect race-based misconceptions that medical trainees have.”
Roxana Daneshjou · Interrogating AI Fairness and Bias in Dermatology and Beyond with Dr. Roxana Daneshjou, 0:35:33
2024
In a 2024 study, Roxana Daneshjou's team found 65 percent of surveyed dermatologists reported using large language models in their practices.
“65 % of dermatologists reported using large language models in their practices”
Roxana Daneshjou · The AI Revolution in Medicine, Revisited: Laws, norms, and ethics for AI in health, 1:13:04
Undated
Roxana Daneshjou attended the Texas Academy of Mathematics and Science, a boarding high school where students live on a college campus.
“I went to a high school that was the Texas Academy of Mathematics and Science.”
Roxana Daneshjou · Interrogating AI Fairness and Bias in Dermatology and Beyond with Dr. Roxana Daneshjou, 0:04:29
Undated
Roxana Daneshjou became interested in computational genomics in a bioengineering seminar taught by Russ Altman at Stanford, and later did a PhD with him.
“I was taking a bioengineering seminar with Dr. Russ Altman, and he talked about using computational methods to study human genomics.”
Roxana Daneshjou · Interrogating AI Fairness and Bias in Dermatology and Beyond with Dr. Roxana Daneshjou, 0:05:44
Undated
Daneshjou sees dermatology patients half a day a week and spends the rest of her time running her AI for healthcare lab at Stanford.
“So I practice half a day a week seeing patients. And then I run my AI for healthcare lab the rest of the time”
Roxana Daneshjou · Interrogating AI Fairness and Bias in Dermatology and Beyond with Dr. Roxana Daneshjou, 0:09:14
Undated
Daneshjou's team reviewed the dermatology AI literature and found that almost none of the papers reported which skin tones were used for training or testing.
“found that almost none of those papers actually even reported what skin tones were used in training or testing.”
Roxana Daneshjou · Interrogating AI Fairness and Bias in Dermatology and Beyond with Dr. Roxana Daneshjou, 0:10:54
Undated
The Fitzpatrick scale has six categories, which Daneshjou says does not capture the full diversity of human skin tone.
“And there's six categories in it, which is obviously not inclusive of the full diversity of human skin tone.”
Roxana Daneshjou · Interrogating AI Fairness and Bias in Dermatology and Beyond with Dr. Roxana Daneshjou, 0:12:42
Undated
In work with Matt Groh, Daneshjou found that dermatologists labeling Fitzpatrick skin type from images usually differed from each other by no more than one point.
“most of the time, if you have dermatologist labeling images, they're within one point, either below or above each other.”
Roxana Daneshjou · Interrogating AI Fairness and Bias in Dermatology and Beyond with Dr. Roxana Daneshjou, 0:14:54
Undated
Daneshjou's benchmark images were labeled using pathology reports, reviewed by a dermatopathologist and a dermatologist, rather than by visual impression alone.
“We were not just looking at the image and saying, oh, we think it's this. We actually had the pathology reports.”
Roxana Daneshjou · Interrogating AI Fairness and Bias in Dermatology and Beyond with Dr. Roxana Daneshjou, 0:16:35
Undated
Daneshjou's skin-tone dataset was designed as a benchmark for testing algorithms and is too small to train a deep learning model.
“It is not a large enough data set to train an entire deep learning model on, but it is one that you can use to benchmark against.”
Roxana Daneshjou · Interrogating AI Fairness and Bias in Dermatology and Beyond with Dr. Roxana Daneshjou, 0:17:05
Undated
Daneshjou's team matched Fitzpatrick 1-2 and 5-6 patients by sex, age, photograph date and diagnostic category to allow a head-to-head comparison.
“we tried to match the patients by sex, age, the time that the photograph was taken”
Roxana Daneshjou · Interrogating AI Fairness and Bias in Dermatology and Beyond with Dr. Roxana Daneshjou, 0:18:11
Undated
Because commercial developers would not give access, the team tested open-source dermatology models that were not necessarily intended for clinical use.
“And we ended up testing models that were open source. So these are not necessarily models that are going to be used clinically”
Roxana Daneshjou · Interrogating AI Fairness and Bias in Dermatology and Beyond with Dr. Roxana Daneshjou, 0:20:33
Undated
Daneshjou described a dermatoscope as a magnifying device with a special light that can attach to an iPhone, and joked that it costs about $900.
“So a dermatoscope is like basically a fancy magnifying glass that costs $900 for no reason.”
Roxana Daneshjou · Interrogating AI Fairness and Bias in Dermatology and Beyond with Dr. Roxana Daneshjou, 0:28:28
Undated
Daneshjou's team simply asked chatbots how to calculate eGFR, a kidney function measure, without supplying patient details.
“We actually made it very simple. We just said, how do you calculate EGFR? We just asked it.”
Roxana Daneshjou · Interrogating AI Fairness and Bias in Dermatology and Beyond with Dr. Roxana Daneshjou, 0:37:19
Undated
Some language models justified using the race-based eGFR equation by citing a difference in muscle mass between races, which Daneshjou calls a debunked racist trope.
“there's difference in muscle mass between races, which is a completely debunked racist trope.”
Roxana Daneshjou · Interrogating AI Fairness and Bias in Dermatology and Beyond with Dr. Roxana Daneshjou, 0:38:35
Undated
Models gave race-neutral answers on lung capacity unless a patient's race was provided, at which point they began citing debunked racial differences.
“But if you ask it about calculating lung capacity and give a patient race, then it starts talking about debunked differences.”
Roxana Daneshjou · Interrogating AI Fairness and Bias in Dermatology and Beyond with Dr. Roxana Daneshjou, 0:40:00
Undated
Daneshjou argued that medical diagnosis is less clear-cut than people assume, citing a case where unclear biopsy results required follow-up notes to assign a label.
“I actually think that diagnosis is a lot harder than people realize”
Roxana Daneshjou · Interrogating AI Fairness and Bias in Dermatology and Beyond with Dr. Roxana Daneshjou, 0:54:48
Undated
Daneshjou works with a human scribe in clinic and says it lets her avoid the computer entirely during patient visits.
“I actually work with a human scribe. And when I started working with a human scribe, it was totally changed my life because I'm not at the computer at all now with my patients”
Roxana Daneshjou · Interrogating AI Fairness and Bias in Dermatology and Beyond with Dr. Roxana Daneshjou, 0:57:04
Undated
Roxana Daneshjou says the majority of large genetic studies were conducted on people of European ancestry, leaving many global populations excluded.
“the majority of these genetic studies were on European ancestry individuals”
Roxana Daneshjou · The AI Revolution in Medicine, Revisited: Laws, norms, and ethics for AI in health, 1:03:53
Undated
A Science Advances paper from Roxana Daneshjou's team built a diverse skin-disease benchmark and found AI models performed significantly worse on brown and black skin.
“and one of our landmark papers, which was in Science Advances, showed we created a diverse data set, our own diverse benchmark of skin disease and showed that the models performed significantly worse on brown and black skin”
Roxana Daneshjou · The AI Revolution in Medicine, Revisited: Laws, norms, and ethics for AI in health, 1:05:48
Undated
Daneshjou's team found that fine-tuning dermatology AI models on diverse skin tones eliminated the performance bias.
“because when we fine-tuned on diverse skin tones, you could make that bias go away.”
Roxana Daneshjou · The AI Revolution in Medicine, Revisited: Laws, norms, and ethics for AI in health, 1:05:48
Undated
A Proceedings of the National Academy of Sciences paper on untrue race-based beliefs among medical trainees inspired Daneshjou's team to test whether language models repeat them.
“there was a paper in the Proceedings of the National Academy of Sciences, which did not look at technology at all, but actually looked at the race-based biases of medical trainees”
Roxana Daneshjou · The AI Revolution in Medicine, Revisited: Laws, norms, and ethics for AI in health, 1:09:12
Undated
Daneshjou says GPT-3.5 and GPT-4 responses suggested using race in calculating kidney function, a claim she calls untrue because race is a social construct.
“they said that we should use race in calculating kidney function because there were differences in muscle mass between the races”
Roxana Daneshjou · The AI Revolution in Medicine, Revisited: Laws, norms, and ethics for AI in health, 1:10:57
Undated
Roxana Daneshjou says ChatGPT flagged an incorrect medication dose in her daughter's after-visit summary after her daughter broke an arm.
“I gave it the whole after visit summary and I said, are there any mistakes here? And it clued in that the dose of the medication was wrong.”
Roxana Daneshjou · The AI Revolution in Medicine, Revisited: Laws, norms, and ethics for AI in health, 1:19:21
Undated
In a dermatology study with Matt Groh, AI assistance increased diagnostic accuracy for both groups, but the performance gap between skin tones also grew.
“there was actually situations where like the accuracy increased um in both groups but the gap actually also increased”
Roxana Daneshjou · The AI Revolution in Medicine, Revisited: Laws, norms, and ethics for AI in health, 1:23:04