Roxana Daneshjou attended the Texas Academy of Mathematics and Science, a boarding high school where students live on a college campus.
“I went to a high school that was the Texas Academy of Mathematics and Science.”
Person
Roxana Daneshjou is a Stanford dermatologist who runs an AI for healthcare lab, and the podcasts here cover bias and fairness in medical AI, especially for skin conditions and race-based medicine. She appeared on NEJM AI Grand Rounds and the Microsoft Research Podcast.
On the Microsoft Research Podcast, Daneshjou said her team's Science Advances paper found AI models performed significantly worse on brown and black skin, and that fine-tuning on diverse skin tones eliminated the bias. She also said GPT-3.5 and GPT-4 suggested using race in kidney function calculations, which she calls untrue. On NEJM AI Grand Rounds, she said almost no dermatology AI papers reported which skin tones were used.
2 episodes · 32 quoted claims
Roxana Daneshjou attended the Texas Academy of Mathematics and Science, a boarding high school where students live on a college campus.
“I went to a high school that was the Texas Academy of Mathematics and Science.”
Roxana Daneshjou became interested in computational genomics in a bioengineering seminar taught by Russ Altman at Stanford, and later did a PhD with him.
“I was taking a bioengineering seminar with Dr. Russ Altman, and he talked about using computational methods to study human genomics.”
Daneshjou sees dermatology patients half a day a week and spends the rest of her time running her AI for healthcare lab at Stanford.
“So I practice half a day a week seeing patients. And then I run my AI for healthcare lab the rest of the time”
Phytophotodermatitis, nicknamed margarita dermatitis, typically appears after skin touches certain plant products, such as lime juice, and is then exposed to sunlight.
“And it turns out that this diagnosis is usually after handling certain plant-based products and then being exposed to the sun.”
Daneshjou's team reviewed the dermatology AI literature and found that almost none of the papers reported which skin tones were used for training or testing.
“found that almost none of those papers actually even reported what skin tones were used in training or testing.”
The Fitzpatrick skin tone scale was originally developed to gauge how easily someone burns, so dermatologists could dose photochemotherapy.
“It was originally developed to help dermatologists look at how easily someone might burn, sunburn, in order to help people dose photochemotherapy.”
The original version of the Fitzpatrick scale excluded brown and black skin tones, which were added later.
“when the scale was first developed, it actually excluded brown and black skin tones. That was added later.”
The Fitzpatrick scale has six categories, which Daneshjou says does not capture the full diversity of human skin tone.
“And there's six categories in it, which is obviously not inclusive of the full diversity of human skin tone.”
In work with Matt Groh, Daneshjou found that dermatologists labeling Fitzpatrick skin type from images usually differed from each other by no more than one point.
“most of the time, if you have dermatologist labeling images, they're within one point, either below or above each other.”
Daneshjou's benchmark images were labeled using pathology reports, reviewed by a dermatopathologist and a dermatologist, rather than by visual impression alone.
“We were not just looking at the image and saying, oh, we think it's this. We actually had the pathology reports.”
Daneshjou's skin-tone dataset was designed as a benchmark for testing algorithms and is too small to train a deep learning model.
“It is not a large enough data set to train an entire deep learning model on, but it is one that you can use to benchmark against.”
Daneshjou's team matched Fitzpatrick 1-2 and 5-6 patients by sex, age, photograph date and diagnostic category to allow a head-to-head comparison.
“we tried to match the patients by sex, age, the time that the photograph was taken”
Because commercial developers would not give access, the team tested open-source dermatology models that were not necessarily intended for clinical use.
“And we ended up testing models that were open source. So these are not necessarily models that are going to be used clinically”
Daneshjou described a dermatoscope as a magnifying device with a special light that can attach to an iPhone, and joked that it costs about $900.
“So a dermatoscope is like basically a fancy magnifying glass that costs $900 for no reason.”
The International Skin Imaging Collaboration, one of the largest public dermatology AI image datasets, consists largely of dermoscopy images.
“the International Skin Imaging Collaboration, is largely dermoscopy images.”
Daneshjou and Andy Beam first heard about ChatGPT from one of Beam's students while walking back from a NeurIPS party in New Orleans.
“We were walking back from one of the hosted company parties that they have at NeurIPS in New Orleans. And we were walking back and your student said, you need to look at this.”
Daneshjou's team drew some of its test questions from a 2016 PNAS paper on false race-based beliefs held by medical trainees.
“there's this 2016 PNAS paper that looks at some of the incorrect race-based misconceptions that medical trainees have.”
The medical field stopped using race in kidney function calculations before ChatGPT models were released, yet the chatbots still gave the race-based equation.
“that's why, for example, we don't use race in kidney function calculation. And in fact, that change was made before, for example, the chat GPT models came out.”
Daneshjou's team simply asked chatbots how to calculate eGFR, a kidney function measure, without supplying patient details.
“We actually made it very simple. We just said, how do you calculate EGFR? We just asked it.”
Some language models justified using the race-based eGFR equation by citing a difference in muscle mass between races, which Daneshjou calls a debunked racist trope.
“there's difference in muscle mass between races, which is a completely debunked racist trope.”
Models gave race-neutral answers on lung capacity unless a patient's race was provided, at which point they began citing debunked racial differences.
“But if you ask it about calculating lung capacity and give a patient race, then it starts talking about debunked differences.”
Daneshjou argued that medical diagnosis is less clear-cut than people assume, citing a case where unclear biopsy results required follow-up notes to assign a label.
“I actually think that diagnosis is a lot harder than people realize”
Daneshjou works with a human scribe in clinic and says it lets her avoid the computer entirely during patient visits.
“I actually work with a human scribe. And when I started working with a human scribe, it was totally changed my life because I'm not at the computer at all now with my patients”
Roxana Daneshjou says the majority of large genetic studies were conducted on people of European ancestry, leaving many global populations excluded.
“the majority of these genetic studies were on European ancestry individuals”
A Science Advances paper from Roxana Daneshjou's team built a diverse skin-disease benchmark and found AI models performed significantly worse on brown and black skin.
“and one of our landmark papers, which was in Science Advances, showed we created a diverse data set, our own diverse benchmark of skin disease and showed that the models performed significantly worse on brown and black skin”
Daneshjou's team found that fine-tuning dermatology AI models on diverse skin tones eliminated the performance bias.
“because when we fine-tuned on diverse skin tones, you could make that bias go away.”
A Proceedings of the National Academy of Sciences paper on untrue race-based beliefs among medical trainees inspired Daneshjou's team to test whether language models repeat them.
“there was a paper in the Proceedings of the National Academy of Sciences, which did not look at technology at all, but actually looked at the race-based biases of medical trainees”
Daneshjou says GPT-3.5 and GPT-4 responses suggested using race in calculating kidney function, a claim she calls untrue because race is a social construct.
“they said that we should use race in calculating kidney function because there were differences in muscle mass between the races”
In a 2024 study, Roxana Daneshjou's team found 65 percent of surveyed dermatologists reported using large language models in their practices.
“65 % of dermatologists reported using large language models in their practices”
Stanford held a red teaming event, where participants deliberately stress-test AI, bringing together 80 physicians, computer scientists and engineers to probe language models with realistic medical tasks.
“we held this red teaming event at stanford where we brought together 80 physicians computer scientists engineers”
Roxana Daneshjou says ChatGPT flagged an incorrect medication dose in her daughter's after-visit summary after her daughter broke an arm.
“I gave it the whole after visit summary and I said, are there any mistakes here? And it clued in that the dose of the medication was wrong.”
In a dermatology study with Matt Groh, AI assistance increased diagnostic accuracy for both groups, but the performance gap between skin tones also grew.
“there was actually situations where like the accuracy increased um in both groups but the gap actually also increased”