# Interrogating AI Fairness and Bias in Dermatology and Beyond with Dr. Roxana Daneshjou

- **Show:** NEJM AI Grand Rounds
- **Published:** 17 Jan 2024
- **Source:** https://ai-podcast.nejm.org/e/14-interrogating-ai-fairness-and-bias-in-dermatology-and-beyond-with-dr-roxana-daneshjou/
- **Transcript:** https://podcasts.opennash.com/shows/nejm-ai-grand-rounds/interrogating-ai-fairness-and-bias-in-dermatology-and-beyond-with-dr-roxana-dane-bc40d5/
- **Provider:** OpenNash Podcasts (machine-generated)

## In brief

Dr. Roxana Daneshjou argues that AI in medicine inherits the biases of the human data and systems it learns from, so fairness must be built in from the start. Her dermatology work shows image models doing worse on brown and black skin. Her language-model work shows chatbots repeating debunked race-based medicine. She says technology alone cannot fix health disparities.

The tension runs between AI's promise and the slow, messy reality of medicine. Daneshjou is hopeful about scribes, decision support and a new generation that cares about equity. She is cautious about diagnosis, which she says is less clear-cut than people assume, and about rushing language models into care. Fixing human training and access to care has to happen alongside better algorithms.

## Key insights

1. **Dermatology AI performs worse on darker skin** Daneshjou's team tested three open-source models on a benchmark matched across skin tones. All dropped in accuracy on new data, and they did notably worse on Fitzpatrick 5 and 6 (brown and black skin) than on 1 and 2.
2. **Skin-tone scales in AI are imperfect borrowed tools** The Fitzpatrick scale was built to predict sunburn and dose light therapy, and it originally left out brown and black skin. Machine learning adopted it to label skin color in images, with only six categories.
3. **Chatbots repeat debunked race-based medical equations** When asked how to calculate eGFR, models cited the race-free kidney equation but still gave the old race-based one. Some justified it with a false muscle-mass trope.
4. **Fixing AI bias requires fixing human medicine** Daneshjou says models mirror biased human data and training, such as textbooks that underrepresent dark skin. A perfectly fair algorithm would not repair access gaps or training gaps in the health system.
5. **Near-term AI gains are likely administrative** She expects scribes and note-writing tools to help soon, provided they protect privacy, show where each statement came from and avoid hallucination. She is wary of AI for actual medical care and says diagnosis is harder than people realize.

## Questions this episode answers

### Is AI worse at diagnosing skin conditions on dark skin?

In Daneshjou's benchmark, three open-source dermatology models did worse on brown and black skin (Fitzpatrick 5 and 6) than on white skin (Fitzpatrick 1 and 2). She notes that humans are also trained mostly on lighter skin images.

### Why do AI chatbots repeat race-based medicine?

Daneshjou says large language models learn from human text, so they pick up human biases. In her study, models knew the race-free kidney equation yet still gave the old race-based one when asked how to calculate eGFR.

### What is the Fitzpatrick skin tone scale and why is it a problem?

It is a six-category scale originally built to judge how easily skin burns, to help dose light therapy. Daneshjou says it was repurposed to label skin color in images and is not inclusive of the full range of human skin tones.

### Is AI already used in dermatology clinics?

Not in the US for image diagnosis. Daneshjou says no image-based algorithm is FDA-approved, though some are approved elsewhere, and she guesses a dermatoscope add-on may arrive in 5 to 10 years.

### Will AI make doctors more empathetic with patients?

Daneshjou thinks it could if it works as a digital scribe, so doctors can make eye contact instead of typing. It must protect privacy, let doctors verify what was said, and avoid hallucinations.

## Facts

- **Roxana Daneshjou** [0:04:29]: "I went to a high school that was the Texas Academy of Mathematics and Science."
- **Roxana Daneshjou** [0:05:44]: "I was taking a bioengineering seminar with Dr. Russ Altman, and he talked about using computational methods to study human genomics."
- **Roxana Daneshjou** [0:09:14]: "So I practice half a day a week seeing patients. And then I run my AI for healthcare lab the rest of the time"
- **Roxana Daneshjou** [0:08:00]: "And it turns out that this diagnosis is usually after handling certain plant-based products and then being exposed to the sun."
- **Roxana Daneshjou** [0:10:54]: "found that almost none of those papers actually even reported what skin tones were used in training or testing."
- **Roxana Daneshjou** [0:12:42]: "It was originally developed to help dermatologists look at how easily someone might burn, sunburn, in order to help people dose photochemotherapy."
- **Roxana Daneshjou** [0:12:42]: "when the scale was first developed, it actually excluded brown and black skin tones. That was added later."
- **Roxana Daneshjou** [0:12:42]: "And there's six categories in it, which is obviously not inclusive of the full diversity of human skin tone."
- **Roxana Daneshjou** [0:14:54]: "most of the time, if you have dermatologist labeling images, they're within one point, either below or above each other."
- **Roxana Daneshjou** [0:16:35]: "We were not just looking at the image and saying, oh, we think it's this. We actually had the pathology reports."
- **Roxana Daneshjou** [0:17:05]: "It is not a large enough data set to train an entire deep learning model on, but it is one that you can use to benchmark against."
- **Roxana Daneshjou** [0:18:11]: "we tried to match the patients by sex, age, the time that the photograph was taken"
- **Roxana Daneshjou** [0:20:33]: "And we ended up testing models that were open source. So these are not necessarily models that are going to be used clinically"
- **Roxana Daneshjou** [0:28:28]: "So a dermatoscope is like basically a fancy magnifying glass that costs $900 for no reason."
- **Roxana Daneshjou** [0:29:03]: "the International Skin Imaging Collaboration, is largely dermoscopy images."
- **Roxana Daneshjou** [0:32:31]: "We were walking back from one of the hosted company parties that they have at NeurIPS in New Orleans. And we were walking back and your student said, you need to look at this."
- **Roxana Daneshjou** [0:35:33]: "there's this 2016 PNAS paper that looks at some of the incorrect race-based misconceptions that medical trainees have."
- **Roxana Daneshjou** [0:36:07]: "that's why, for example, we don't use race in kidney function calculation. And in fact, that change was made before, for example, the chat GPT models came out."
- **Roxana Daneshjou** [0:37:19]: "We actually made it very simple. We just said, how do you calculate EGFR? We just asked it."
- **Roxana Daneshjou** [0:38:35]: "there's difference in muscle mass between races, which is a completely debunked racist trope."
- **Roxana Daneshjou** [0:40:00]: "But if you ask it about calculating lung capacity and give a patient race, then it starts talking about debunked differences."
- **Roxana Daneshjou** [0:54:48]: "I actually think that diagnosis is a lot harder than people realize"
- **Roxana Daneshjou** [0:57:04]: "I actually work with a human scribe. And when I started working with a human scribe, it was totally changed my life because I'm not at the computer at all now with my patients"

## Transcript

**[0:00:03] Roxana Daneshjou:** I mean, it's not like the models magically learned bias by themselves. These models learn from human data. And so the biases in human data get picked up and regurgitated with large language models. People will say like, well, humans are biased and therefore the models are going to be biased or that's okay. And I actually don't think that's okay. I think if we're going to build these systems and put them into the health care system, we need to be building systems that are fair and are not going to worsen health disparities or continue to perpetuate them as they exist now.

**[0:00:40] Roxana Daneshjou:** And as I mentioned before, even if you built the most perfect model, like having these systemic issues on the human side, you have to fix that too. Technology is not going to save everything.

**[0:00:58] Arjun Manrai:** Welcome to another episode of NEJM AI Grand Rounds. I'm Raj Manrai, and I'm here with my co-host, Andy Beam. And today we bring you our conversation with Roxana Daneshjou. Roxana is an assistant professor of biomedical data science at Stanford University, and she's also a practicing dermatologist. She's been working in AI for dermatology and for medicine more broadly for several years. Andy, I think this marks the beginning of season two, And Roxana is a wonderful guest to kick off our new season.

**[0:01:28] Arjun Manrai:** So to me, I think Roxana is a clinician scientist who truly and impressively draws and mixes together both her clinical and her technical skills to do unique work at the intersection of AI and dermatology. She also cares a lot about algorithmic bias and healthcare disparities. We really got into that in the conversation. I thought she had a lot of insights to share there about both the potential for AI to exacerbate disparities, but also how it might be used as a tool for good.

**[0:01:57] Arjun Manrai:** You really feel, I think, the urgency of these problems when speaking with Roxana. And I learned a lot from our

**[0:02:03] Andrew Beam:** conversation with her. Yeah, Raj, I agree. It was a great pleasure to talk to Roxana. I first became aware of Roxana, you know, like I do of a lot of scientists through Twitter, where she's prolific. The first time I met her was at NeurIPS, and I was very pleasantly surprised to find that she's as nice in person as she is on Twitter. I agree that her work on algorithmic bias is timely and urgent. And when I think of Roxana, I really think of her as an exemplar for future leaders who are clinicians who are really pushing the boundaries for AI and medicine.

**[0:02:37] Andrew Beam:** I often get approached by junior clinicians saying, how can I mold myself to have a career and really make an impact at this interface of AI and medicine. And almost always, I point to Roxana as the template to do that. She's clearly a rising star in the area. And I think she is uniquely good at combining her deep clinical expertise to inform her research directions in AI. And so for a lot of reasons, it was really great to have her on the podcast.

**[0:03:06] Arjun Manrai:** The NEJM AI Grand Rounds podcast is brought to you by Microsoft, Viz.ai, Lyric, and Elevance

**[0:03:14] Andrew Beam:** Health. We thank them for their support. And now we bring you our conversation with Roxana Daneshjou. Welcome to AI Grand Rounds, Roxana. We're really excited to have you here today.

**[0:03:28] Roxana Daneshjou:** Thanks for having me.

**[0:03:30] Arjun Manrai:** Roxana, this is a question that we always like to get started with. Could you tell us about the training procedure for your own neural network? How did you get interested in artificial intelligence? And what data and experiences led you to where you are today?

**[0:03:44] Roxana Daneshjou:** Well, I guess how far back do we want to go?

**[0:03:47] Andrew Beam:** I mean, as far back as we like to go back to when the neural net was initialized.

**[0:03:54] Roxana Daneshjou:** The pre-training that happened. Basically, I grew up to two Iranian immigrant parents who really liked to incorporate science very early on. I have memories of doing these long road trips with my dad and mom. My dad would always play these astrophysics recordings. And so I had to, you know, from a young age, just develop this curiosity for how things work and how to build things. And I actually, of my own volition, wanted to go to essentially nerd school.

**[0:04:29] Roxana Daneshjou:** I went to a high school that was the Texas Academy of Mathematics and Science. And it's just as nerdy as it sounds. High school students get to live on a college campus and take college classes together. And it's a boarding school. And that was actually when I first got into research because I joined a neuroscience research lab with Dr. Janin Fuchs and was doing a lot of mouse work at that time. And then I went to Rice and I studied bioengineering because I was very interested in understanding how the human body works, but also how to build things that could improve the human health condition.

**[0:05:12] Roxana Daneshjou:** And I think that that was a really formative experience for me because as a bioengineer, we talked a lot about design thinking. We talked a lot about identifying problems and then developing solutions and testing. And I think that formed a real foundation. And to be honest, in college, I waffled a lot about whether I wanted to get an MD or a PhD. I waffled back and forth many times. I ended up actually applying to go to medical school and get an MD.

**[0:05:44] Roxana Daneshjou:** And I landed at Stanford, which has a heavy research focus. And I took, I was taking a bioengineering seminar with Dr. Russ Altman, and he talked about using computational methods to study human genomics. And that really sparked my interest. I was a first year medical student. I emailed him, I met with him. I started doing research with him. And then at some point, I realized two years into that, and then after having done a one-year research fellowship, that I wanted to do a PhD.

**[0:06:18] Roxana Daneshjou:** So I actually didn't go into medical school as an MD-PhD. I was an MD-only who suddenly had struggled with that decision before coming and had just picked one and then decided to add a PhD in the middle of medical school. So I did that with Russ and then I completed my medical school. And right about that time, there was all the interesting work coming out around computer vision and I had become clinically interested in dermatology.

**[0:06:48] Roxana Daneshjou:** And so to me, it seemed like a real opportunity. AI seemed like a real opportunity to solve problems in dermatology, which is a very visual field and all of healthcare in general. So I ended up doing a residency in dermatology and then a postdoc with Dr. James Zhu in healthcare AI. So I was in training for a long time, as you can tell from that story.

**[0:07:13] Andrew Beam:** Can I just follow up? What was it about dermatology that sort of drew you to that?

**[0:07:18] Roxana Daneshjou:** Well, I think there was a couple. So on clinical rotations, I really enjoyed basically every rotation I went through. The one thing that's really nice about dermatology is that it's a very visual field. You can walk into the room and see what's going on a person's skin and put the pattern together with some history. Sometimes you don't even need the history. Sometimes you walk into the room and you can just tell exactly what's going on. And actually one of my favorite diagnosis is this like phytophotodermatitis, which is they get geometric brown or red patches on their hands and they've suddenly appeared.

**[0:08:00] Roxana Daneshjou:** And it turns out that this diagnosis is usually after handling certain plant-based products and then being exposed to the sun. And so you start to ask them questions. So lime juice is one of the things that can do this. So you ask them questions like, oh, so this past weekend, did you hang out with your friends? It's actually called margarita dermatitis. Did you handle limes because you were making margaritas or, you know, having a barbecue or doing something else?

**[0:08:30] Roxana Daneshjou:** And their eyes will just widen because that's exactly what had happened. And so there are other diagnoses in dermatology like that as well. Like when you see the pattern, it corresponds to some exposure. So it's pretty fun to like be able to make diagnoses like this. I think that's part of it. I think also I build, I have really great longitudinal relationships with my patients on an outpatient basis. So I see them regularly. I know their stories.

**[0:08:59] Roxana Daneshjou:** I know all about their families and what their travels are. And so I enjoyed also like the aspect of being able to build these really long-term relationships with the patients that

**[0:09:08] Andrew Beam:** I take care of. That's a great overview of your training. What are you up to now?

**[0:09:14] Roxana Daneshjou:** Yeah, so I am an assistant professor of biomedical data science at Stanford, and also joint with dermatology. So I practice half a day a week seeing patients. And then I run my AI for healthcare lab the rest of the time, which is essentially like the physician scientist dream is to be able to still have your hand in clinical work, because I think one, it's very gratifying to work with patients. And two, it gives you a window into exactly what's happening into the healthcare system.

**[0:09:48] Roxana Daneshjou:** And the rest of the time, I get to work with my amazing, the postdocs, the graduate students and medical students who are in the lab.

**[0:09:56] Andrew Beam:** Awesome. I think that is a good transition to talking about some of your research. So the first paper that I want to talk about is called Disparities in Dermatology AI Performance on a Diverse Curated Clinical Image Data Set. So one, could you give us the setup for this paper? And then I think we'd like to dive into some of the details.

**[0:10:13] Roxana Daneshjou:** The setup for this paper is that I'm interested in AI fairness and bias. And Dr. Adey Adamson, who is a friend and colleague, had written a perspective piece in JAMA Dermatology, essentially saying early on that he was very concerned about whether or not data sets that were being used to train dermatology AI models were representative of the full skin tone spectrum. And I think that his concerns were really appropriate because as many other dermatologists had pointed out before, like Dr.

**[0:10:54] Roxana Daneshjou:** Jenna Lester, the education in dermatology, like the textbooks, the training in and of itself is not very representative of diverse skin tones. So you're looking at the training being like that for humans, and then you start to worry about the training for models having the same problem. And before we actually wrote this paper, we wrote a different paper where we actually went through and looked at the literature that had been published in the dermatology AI space and found that almost none of those papers actually even reported what skin tones were used in training or testing.

**[0:11:34] Roxana Daneshjou:** And that for the ones that did report that, they basically either significantly underrepresented or excluded brown and black skin tones. And so the impetus for this paper was we wanted to create a data set, sort of like a benchmark, so that we could test how algorithms perform across diverse skin tones.

**[0:12:01] Andrew Beam:** That's great. Could you actually, for the non-dermatologists who are listening, how do you measure skin tone? There's a scale. Could you walk us through actually how you assess the variety in skin tone versus just like dark versus light?

**[0:12:12] Roxana Daneshjou:** Right. So there is a scale. It's actually not a great scale. And there has been a lot of discussion and efforts on developing new scales because the current scale is not very inclusive. However, much of the machine learning world has been using that imperfect scale. And that's actually what we do in our paper as well. But I just wanted to put that out there. I think that scale is problematic. It's called the Fitzpatrick skin tone scale.

**[0:12:42] Roxana Daneshjou:** It was originally developed to help dermatologists look at how easily someone might burn, sunburn, in order to help people dose photochemotherapy. So it was originally about how easily you tanned, how easily you burned. And then it got co-opted into being used to assign color from an image. So that's not how it's meant to be used. And also when the scale was first developed, it actually excluded brown and black skin tones. That was added later. And there's six categories in it, which is obviously not inclusive of the full diversity of human skin tone.

**[0:13:22] Roxana Daneshjou:** But that imperfect scale is what we used when we were trying to collect our images. And we tried to actually assign the skin tone based on what was recorded by the clinician seen in person because in photography, like the lighting can impact what the skin looks like. So that's a scale. And so we particularly the light skin tones are Fitzpatrick one and two, three and four sort of in the middle, and then five and six usually represents brown and black skin.

**[0:13:52] Roxana Daneshjou:** So we in our paper are looking at performance of Fitzpatrick skin tones one and two versus five and six, which I'll just call like white versus brown and black skin.

**[0:14:05] Arjun Manrai:** Go ahead, Rush. Roxana, how are dermatologists trained to rate Fitzpatrick skin types? Are they trained to rate these skin types?

**[0:14:14] Roxana Daneshjou:** So it's definitely discussed clinically, like when you write your medical note. And the reason that people care, again, is to try to make a decision on how easily someone might hyperpigment, meaning that how easily their skin might turn brown from inflammation, or how easily they might burn with certain treatments that involve light. So it is something that we actually go through in training. And actually in work with Matt Groh, we looked at the variability of labeling skin images, which is different, obviously, than the in-person labeling or asking people about how easily they burn.

**[0:14:54] Roxana Daneshjou:** But for a machine learning purpose, we wanted to understand how much variability there was between people. And it turns out there is some variability, but most of the time, if you have dermatologist labeling images, they're within one point, either below or above each other. Got it. And just before we move on real quick,

**[0:15:15] Andrew Beam:** just because I find this interesting, how does that impact treatment recommendation or a patient's clinical course, what Fitzpatrick bucket they're put into? The real way to get clinically speaking,

**[0:15:26] Roxana Daneshjou:** it shouldn't just be what you think they're Fitzpatrick, but you should be asking them questions about how photosensitive they are, how easily they burn. If they have a history of having their skin sort of change color after inflammation, because you can't always know that without actually asking. Someone could have brown skin and actually be very photosensitive because of a medication that they're on. So it's important not to make assumptions without actually asking your patients about their experiences clinically.

**[0:16:01] Roxana Daneshjou:** Right. You could be a three or four, but if you're

**[0:16:03] Andrew Beam:** on a lot of antibiotics or something, then you could be photosensitive or something like that. Yeah. Okay. So I think that was very helpful background. So if I understand correctly, you saw that there was this unmet need in the literature that most of the data sets were not diverse. We've seen this in other areas too, like GWAS, where a lot of the studies were done in white Europeans. You kind of had a sense of that this was happening for AI for dermatology.

**[0:16:24] Andrew Beam:** So you actually collected a big data set that had a huge representation of different types of skin tones, right? Could you tell us how you curated that data? Yeah. So we curated and de-identified

**[0:16:35] Roxana Daneshjou:** data from Stanford. It was an effort of many researchers to make sure that we were comparing the, we were using the pathology reports to make the diagnosis. We were not just looking at the image and saying, oh, we think it's this. We actually had the pathology reports. We wanted the labels to be as clean as possible. We had a dermatopathologist sitting with a dermatologist reviewing what's in the report, what it looks like clinically to assign each of those labels.

**[0:17:05] Roxana Daneshjou:** We were looking at the assigned Fitzpatrick skin tone label in the chart and then having two dermatologists confirm that they thought that that label was correct. So for us, this was meant to be a benchmark. It is not a large enough data set to train an entire deep learning model on, but it is one that you can use to benchmark against.

**[0:17:29] Andrew Beam:** How many images did you end up with?

**[0:17:32] Roxana Daneshjou:** I believe it is 656.

**[0:17:36] Andrew Beam:** Okay, nice. I mean, it sounds like a lot of manually intensive labor, given all of the eyes that sort of looked at each image.

**[0:17:44] Roxana Daneshjou:** Yes, it was. And I know that that is not tenable when you're talking about wanting something like 10 ,000, you know, 100 ,000 images for training.

**[0:17:56] Andrew Beam:** Yeah, everything doesn't have to be ImageNet. I think that having very good, high quality benchmark data sets that have a good amount of TLC, which it sounds like this data set definitely does, are super important things to have to understand how these models work.

**[0:18:11] Roxana Daneshjou:** That's how we felt too. And I think that, you know, for us, the other thing that we did was that for Fitzpatrick one and two or the white skin tones and Fitzpatrick five and six, the brown and black skin tones, we tried to match the patients by sex, age, the time that the photograph was taken, meaning like to try to match the quote camera technology and sort of the diagnostic category so that we could do sort of a head-to-head comparison of those two groups.

**[0:18:44] Roxana Daneshjou:** And then we even looked at, for example, we had look at photo quality between the two groups. We had dermatologists label photo quality to make sure that the photo quality was similar between the two groups so that when people are running algorithms against looking at performance in Fitzpatrick 1 and 2 and 5 and 6, we could minimize as many confounders as possible in that comparison.

**[0:19:08] Andrew Beam:** It's almost like you have a counterfactual lesion for each person in the data set. Like the only thing that has been changed has been the skin tone.

**[0:19:14] Roxana Daneshjou:** So it's not quite perfect. I wish it were because, you know, in some cases we had to match categories as like non-melanoma skin cancers had to be a bucket in and of itself because it becomes difficult to match lesions perfectly. But in general, like every benign lesion has a benign lesion that's in a similar diagnostic space.

**[0:19:41] Andrew Beam:** Yeah, makes sense. So what did you find? How do the current dermatology classifiers do?

**[0:19:45] Roxana Daneshjou:** Yeah, so I mean, we, one thing is, I'm all about open science. I think that's why we need open science, because there's a lot of commercial algorithms or algorithms that have been published about that we couldn't get access to. We asked, we went and asked around, hey, you've published on this algorithm, you're making claims that this might be used commercially someday or touch patients. can we test it on the benchmark? And usually the answer is no.

**[0:20:14] Roxana Daneshjou:** So we ended up -

**[0:20:15] Arjun Manrai:** And Roxana, when it's access to the algorithm, you don't even necessarily need the weights, right? No, no, no. The model itself. You just need some type of API or ability to run an image through it. And even access to that was difficult for some of these commercial models that we use.

**[0:20:33] Roxana Daneshjou:** Yes, that is correct. Thank you for clarifying. Yeah, we're not asking for access to the weights. were just asking for access to the interface at the time or an API or something. And we ended up testing models that were open source. So these are not necessarily models that are going to be used clinically, but there were ones that we could access to and had been previously published on and had really good performance. And we looked at three different models.

**[0:21:04] Roxana Daneshjou:** And the interesting thing, I mean, this is not a surprise to anyone who's kind of been following this space, is that all three models had performance drop-offs in general when they tested on a new data set. And we know this because sometimes these models overfit to features in the data set that they were trained on. And when you introduce some kind of new external data set to them, they'll have some performance drop-off. It could be because of differences in the camera technology used to acquire the images, differences in lighting.

**[0:21:36] Roxana Daneshjou:** Dermatology is kind of hard because we don't have standards for how we take our images. So that was the first thing we noticed. But the part that was like most concerning to us was that there were significant differences in how the algorithms performed on Fitzpatrick 1 and 2, the white skin tones versus Fitzpatrick 5 and 6, the brown and black skin tones.

**[0:22:00] Andrew Beam:** That sadly is not surprising. Can I ask a philosophical question? Do humans, are they worse at diagnosing dark skin tones also? Has anyone looked at that?

**[0:22:11] Roxana Daneshjou:** And that's an excellent question. So I mentioned that a lot of the education materials for humans has underrepresented brown and black skin tones. And this has been something many other dermatologists in there's a skin of color society that focuses on making sure that we have equitable care across skin tones. There are many dermatologists who have brought this up over and over again about representation and education and training of dermatologists. And in terms, you know, there have been survey studies that have shown that dermatology residents, some portion of them don't feel as comfortable making diagnosis across diverse skin tones.

**[0:22:57] Roxana Daneshjou:** There is an amazing TED Talk by Dr. Jenna Lester exactly on this topic. in terms of has anyone sort of systematically looked at this, stay tuned. We have a paper coming out where we did this with just images, which is obviously different than clinical care, but we wanted to look at it in the sort of like the teledermatology where you just have some image and not that much history. So there are differences that we saw in that upcoming paper.

**[0:23:31] Andrew Beam:** Because I ask because I'm wondering like how we fix this. And if you trained the model on a more representative data set, but the labels are coming from this very error prone process on dark skin, would that fix it? Or is there something more structural that needs to happen? You did this very thoughtful curation, where you sat down and intentionally looked at the darker skin tones. And I guess like, what does the path forward look like?

**[0:23:54] Roxana Daneshjou:** So I feel confident about our labels because we looked at things that were biopsies and we had path reports on them. Now, of course, there's variability in pathology that exists. But one thing we actually did was have dermatologists label the images and we saw differences in labeling between the two skin tone groups. Of course, that is not representative of what happens in clinical care, that's just giving people an image and asking them to label a disease, which is they don't have the opportunity to ask history or do their clinical exam on the lesion.

**[0:24:35] Roxana Daneshjou:** But from a labeling standpoint, we think that actually, if you are just relying on dermatologists for labeling and you don't have ground truth, that's a loaded word. If you don't have a histopathology label, you have more noise if you have the dermatologist labeling the data. To go back what you're saying, like, how do we fix this, right? Because I'm diving into the machine learning side of it. But I think that there's the AI realm, but we need systemic change in medicine too.

**[0:25:10] Roxana Daneshjou:** Like this is not an AI only problem. AI is reflecting the biases that exist in the human realm. And my feeling has always been that you cannot rely on AI to fix problems that you have to actually tackle in the human realm, which is we need to make a concerted effort to improve the training of dermatologists so that dermatologists do a better job. We need to improve access to care. A lot of the health disparities that exist in dermatology are not AI related issues, but they spill into AI.

**[0:25:51] Roxana Daneshjou:** And I don't think that even if you created the most perfect, fair AI algorithm, that it's going to necessarily be a band-aid for the problems that exist systemically within the medical system. At the risk of ruining what would be a perfect

**[0:26:06] Andrew Beam:** transition to the next set of questions that we want to ask you, I just want to ask one quick follow-up. Just since you're an AI researcher and a practicing dermatologist, what's your sense of the penetration of this technology in dermatology? So I've seen in radiology, it just all of a sudden has happened where there are AI systems in reading rooms and triaging chest x-ray reads. What's your sense of how much penetration there's been for AI in dermatology?

**[0:26:31] Andrew Beam:** So it's been interesting. In the

**[0:26:32] Roxana Daneshjou:** US, image-based, there's other forms, image-based AI, there's been no FDA-approved algorithms as of us talking. There are people running trials to try to get FDA approval. So, and there've also been, now, finally, there've been several, for a while, there were no even prospective clinical trials of, say, like a dermatologist using AI to see if it improved, like, their sensitivity or specificity for finding skin cancers. So now there's been some prospective trials. So we in dermatology are definitely, I like to joke that we're a decade behind radiology.

**[0:27:13] Roxana Daneshjou:** I don't know if it's truly a decade. In other countries, there have been things that have received, for example, CE mark and have been used in clinics. They're also direct to consumer apps, which are a little concerning because as I mentioned, Some of them make diagnostic claims without actually having FDA approval or published trials. So we actually, one of the things that we've done is we've looked at some of the consumer apps that are available in app stores in the US.

**[0:27:44] Roxana Daneshjou:** And I mean, a majority of this, like, we don't know anything about how they actually perform. There's like no published material. But in terms of penetrations, like, is there AI in my clinic? There's not AI in my clinic yet.

**[0:27:58] Arjun Manrai:** how long do you think it is going to be until there's ai in my clinic in your clinic roxana

**[0:28:03] Roxana Daneshjou:** yeah you know maybe five to ten years the first place that i my this is my guess well i guess what are we calling ai in my clinic because if we're talking about like large language models in the emr system people are already trying to test that out now so maybe before like you know So if we're talking about dermatology AI and like image-based -

**[0:28:26] Arjun Manrai:** Image-based dermatology AI.

**[0:28:28] Roxana Daneshjou:** My guess is that the first thing that'll come out, and this is just me guessing, is something with like a dermatoscope, which is the - So a dermatoscope is like basically a fancy magnifying glass that costs $900 for no reason. It's like this device that you put on the lesion, it magnifies it, it shines a special light on it. It's nothing more than that, but it's a little bit more standardized because you put the magnifying glass and everything in that field of view can be captured in an image.

**[0:29:03] Roxana Daneshjou:** And one of the largest public data sets of dermatology AI images for training models, the International Skin Imaging Collaboration, is largely dermoscopy images. And so that space has moved a lot faster than the clinical image-based models. So my guess is the first thing is that there'll be some AI companion to the dermatoscope that comes out. And there've been some clinical trials in that space. So we'll see. To be determined.

**[0:29:32] Arjun Manrai:** Does the dermatoscope plug directly into the EMR or how does it interface? Or does it have its own output and then you have to enter something?

**[0:29:40] Roxana Daneshjou:** No, right now it's low tech. It's a magnifying glass. It has an attachment. You can attach an iPhone to it so you can take a picture. So my guess is that basically based on some of the trials, there'll be some readout. Yeah, you'll actually have to use like your iPhone to take the picture with the hardware, right? The hardware is just going to magnify the image. You'll use your iPhone to take the picture and you'll have something that runs and gives you some readout.

**[0:30:09] Roxana Daneshjou:** That's my guess.

**[0:30:10] Arjun Manrai:** Got it. Very interesting to see. And thank you for entertaining the question because it's a little unfair question too. I could be completely wrong. Yeah. We appreciate the speculation. And I think you're the top person to do it. So I think this is a good transition point, actually. So you mentioned large language models. We want to stay on the topic of algorithmic bias connecting to your work in dermatology. But now switching over to LLM's large language models.

**[0:30:43] Arjun Manrai:** You published a paper pretty recently. I think it's titled Large Language Models Propagate Race-Based Medicine. And as I understand it, it also hits another theme that you referenced a few moments ago, which is AI picking up bias that is in society, that is in the way we practice medicine, and potentially amplifying it and propagating existing ways of practicing medicine, but not necessarily updating as these models have changed, as race has been removed from some of these equations, not staying up to date with current best clinical practice and recommendations from clinical societies.

**[0:31:24] Arjun Manrai:** So could you maybe just set the stage for us by telling us about what motivated this paper and what you did for the study and maybe briefly what you found?

**[0:31:37] Roxana Daneshjou:** Yeah. So I am a dermatologist, but as I like to say, I like to do research throughout all of healthcare AI, having done like a year of internal medicine training, having done rotations as a medical student. I'm not just focused only on things that impact dermatology. And of course, when I remember I was at NeurIPS, as many of us were, and Andrew, I think actually one of your students was the first person to say, hey, did you guys know that there was an update to chat GPT

**[0:32:11] Andrew Beam:** had just been announced? Yeah. I remember that we were at like a cocktail party or something. Yeah. My student was like, there's this thing called chat GPT. I think we should probably pay

**[0:32:19] Roxana Daneshjou:** attention to it. We're like, man. Yeah. Yeah. No, because this is, this is, this is a year ago now,

**[0:32:26] Arjun Manrai:** right? November, 2020. It feels like 10 years ago at this point. It does feel like 10 years ago.

**[0:32:31] Roxana Daneshjou:** Yeah. That's how I feel about having started my faculty job. It's been like a month and I'm like, that month feels like a decade. But yes, actually, it's hilarious that we're talking now because I remember the stage. We were walking back from one of the hosted company parties that they have at NeurIPS in New Orleans. And we were walking back and your student said, you need to look at this. And we were both like, oh, we've played with GPT-3 and like, eh.

**[0:33:01] Roxana Daneshjou:** And then actually, I remember I went back to my hotel room. And instead of getting ready to go to bed, I logged on, made an account and started talking to it. And then I said, whoa, this is very different than GPT-3. And I started asking it a lot of medical questions, which I had done with GPT-3 as well. And now GPT 3.5 was just like answering it in ways that shocked me. And of course, I think for anyone who's ever interacted with these systems, like for the first time, there's that moment of being very impressed.

**[0:33:39] Roxana Daneshjou:** And then you start to dig in and try to think about, okay, where the problems are. And I think that over the last year, a lot of people, and of course, GPT-4 has come out now and people have seen where some of the issues are. And so one thought I had, the reason we did this paper is because medicine is very slow to adopt things. And I had been working on image-based AI. We had even done a clinical trial of one of our models.

**[0:34:07] Roxana Daneshjou:** And that model was not even a diagnostic model. And it was just very slow adoption. And then all of a sudden, it felt like overnight, there were hospital systems saying, hey, we're piloting these large language models into our medical systems. And that was a little surprise. I don't know if you guys have been surprised and shocked, but I've been kind of.

**[0:34:26] Andrew Beam:** Well, thankfully, the hospitals we're affiliated with move even slower than the ones that you're affiliated with. So we haven't had to address that yet.

**[0:34:34] Roxana Daneshjou:** So I said, whoa, like people are actually talking about integrating this into the medical system. So and we haven't had.

**[0:34:47] Arjun Manrai:** I have so many. I have so many questions, but keep going. Keep going. Because I want your perspective also on that at Stanford too, but because it probably is faster than, as Andy is saying, it's probably faster than the way things are moving here. But I think, yes, there is a extreme amount of energy to get these into practice.

**[0:35:05] Roxana Daneshjou:** And I will say that a lot of this stuff is still in the pilot study phase, but it's not just Stanford. It's many academic institutions. I just had never seen something go from it just came out to now we're doing clinical pilot studies in such a short period of time. And so my research group, we've thought about AI fairness and bias for a while, mostly in computer vision. And so then we started thinking about, like, how could this show up with large language models?

**[0:35:33] Roxana Daneshjou:** And so one thought we had is that we knew that physicians and patients were likely asking these models questions, clinical questions. And so we pulled, so there's this 2016 PNAS paper that looks at some of the incorrect race-based misconceptions that medical trainees have. So we pulled some questions from that paper. And then we also had a group of experts who are all authors on the paper sit together and think like what other questions we might ask.

**[0:36:07] Roxana Daneshjou:** And so there's been a lot of discussion around the use of race in medical algorithms because race is not biological. It's a social construct. And so studies have shown that there can be disparate outcomes from using race in algorithms. And so that's why, for example, we don't use race in kidney function calculation. And in fact, that change was made before, for example, the chat GPT models came out. So if you ask chat GPT about it, it knows about that change.

**[0:36:43] Roxana Daneshjou:** And yet, if you ask it, how do you calculate EGFR?

**[0:36:48] Arjun Manrai:** EGFR. So it references the new National Kidney Foundation, American Cite of Nephrology, recommended race-free equation, but then recommends using the race-based previous equation.

**[0:37:02] Roxana Daneshjou:** So yeah. I mean, I'm just saying like, if you ask it, do you know about this information?

**[0:37:06] Arjun Manrai:** Okay. So it's aware of it. It's aware of it. It's in its artificial brain somewhere. Yes. But when you ask it, this is my creatinine, this is my patient, or this is my age, what is my EGFR, it will use the race stratified or the race-based.

**[0:37:19] Roxana Daneshjou:** We actually made it very simple. We just said, how do you calculate EGFR? We just asked it. I see. So if you ask it, if it knows about the newer way of doing things, it knows. So that's just to show that it's not, that it doesn't know. It does know. But if you ask it, how do you calculate EGFR? It will give you, and actually, you know, all the models we tested had problems. So it's, I'm not just picking on chat GPT.

**[0:37:46] Arjun Manrai:** What other models? Did you test the other?

**[0:37:48] Roxana Daneshjou:** Like Bard.

**[0:37:49] Arjun Manrai:** Proprietary models, yeah.

**[0:37:50] Roxana Daneshjou:** Bard, Claude, 3.5 and 4 for GPT.

**[0:37:54] Arjun Manrai:** Were the models meaningfully different from one another?

**[0:37:56] Roxana Daneshjou:** You know, the way that the answers were written were certainly different. We have sort of a heat map in our paper that shows like which models were better than the other. In general, many of them failed similarly on certain questions. And many of them passed, for example, most of them, except for one, passed the question on the genetic basis of race, which, again, race doesn't have a genetic basis. It's a social construct. So it's just interesting to see what they failed at and what they – I'm sure there's that reinforcement learning with human feedback for some of the questions that we asked is my guess.

**[0:38:35] Roxana Daneshjou:** But actually, one thing that was really interesting to me that I wanted to point out is some of the models. Not only did they give the incorrect race-based equation, they actually gave an argument for why you should use that equation using like a false racist trope, which is that there's difference in muscle mass between races, which is a completely debunked racist trope. And so that was even more interesting in a bad way. Not surprising, because that stuff is out in the world, but deeply concerning, because if somebody comes in with those biases,

**[0:39:14] Andrew Beam:** it just gets confirmed, right? I'm curious, did the sort of rationalization come unprompted? It's like, hey, trust me, I'm not racist. Like, here's the reasoning behind

**[0:39:25] Roxana Daneshjou:** why I'm doing this. It was unprompted. I'm saying that this appeared simply in response to just asking like, how do you calculate EGFR? Yeah, there's a like, me thinks the lady doth protest too much component to that, it seems like. Right. I mean, we also asked like, how do you calculate EGFR and gave the race of the patient as well, but even if you don't give a race of patient. So it's interesting because when you talk about lung capacity, if you just ask about calculating lung capacity, it doesn't give a race-based answer.

**[0:40:00] Roxana Daneshjou:** But if you ask it about calculating lung capacity and give a patient race, then it starts talking about debunked differences. It starts claiming that there are differences in lung capacity between races that are not

**[0:40:15] Arjun Manrai:** supposed to be used that don't exist. So, you know, Roxana, similar to Andy's question about dermatology and how dermatologists fare in rating, basically treating patients with different skin tones. Do you think that the models here, so the large language models, do you think they're also reflecting the societal bias, the way that race has been and currently still remains in many instances embedded into physiological equations and is used as sort of an inductive bias right into your model for physiology for many but i think reducing applications in medicine do you think these models are essentially reflecting that i guess is the first question and then my second question is how do you think large language models the ones that you tested

**[0:41:07] Arjun Manrai:** compare to practicing physicians in terms of bringing bias into the clinical encounter

**[0:41:15] Roxana Daneshjou:** So I think for the first question, the answer is yes, because obviously the large language models are first trained on large amounts of text data that reflects human thinking. I mean, it's not like the models magically learned bias by themselves. These models learn from human data. And so the biases in human data get picked up and regurgitated with large language models. I think your second question is a really difficult one. I'm not sure how to answer.

**[0:41:49] Roxana Daneshjou:** People will say humans are biased and therefore the models are going to be biased or that's okay. And I actually don't think that's okay. I think if we're going to build these systems and put them into the healthcare system, we need to be building systems that are fair and are not going to worsen health disparities or continue to perpetuate them as they exist now. And as I mentioned before, even if you built the most perfect model, having these systemic issues on the human side, you have to fix that too.

**[0:42:20] Roxana Daneshjou:** Technology is not going to save everything.

**[0:42:24] Arjun Manrai:** Yeah, no, so I agree. And I think the time to do that is now, before they are widespread. But I think understanding human bias is maybe a path to thinking about designing a robust human AI collaboration. And so we have all these nice examples of GPT passing this or that, right? And we're almost desensitized to these papers now, right? Because it's been done so much. But we have very few studies of how humans and AI operate together.

**[0:42:54] Arjun Manrai:** And I think we're going to have more of those. But just as in the context of accuracy, I think thinking about bias, if we understand human bias and we understand the way the algorithm is biased, I wonder if there is a path forward that you see where we could design large language models to actually inject some nuance into the way that humans are reasoning. Because humans have a ton of bias too, right? And it's not to say that that excuses algorithmic bias, but it is to say that we should be aware of the status quo and design systems and improve systemic education, all that.

**[0:43:27] Arjun Manrai:** But I think design systems that hopefully allow that collaboration of the human and the AI together to reduce bias. I agree with, I mean,

**[0:43:34] Roxana Daneshjou:** you could imagine that you could have the system designed to always give the correct type of algorithm that's like the most up-to-date and explain to the human doctor why. Like, this is why.

**[0:43:50] Arjun Manrai:** Yes. Like the kidney function example, that what that would have looked like would be in 2021, the National Kidney Foundation, American Cite Nephrology, recommended a race-free creatinine based equation, here it is. And then if they wanted context, they could have context on the previous race based equations, and further information, right? That's not an explanation

**[0:44:12] Roxana Daneshjou:** why, right? Because when those decisions were made, there were explanations of why, like, how the prior equation could cause disparate outcomes, why some of the assumptions that were made were harmful. You could build that in. I see what you're saying. Like it could be actually a tool for when you do the human AI collaboration, the AI helps sort of the human understand like

**[0:44:37] Arjun Manrai:** the state of the art now. Yeah. Great. All right. I think that was Andy. Is this a good time? I think it's a good time for the lightning round. Are you ready? Oh boy. Roxana, we have a lightning

**[0:44:50] Roxana Daneshjou:** round. I am aware. I've listened to your podcast. Okay. So you don't even need the setup so we can

**[0:44:55] Andrew Beam:** just hop right into it. So I think that this is a good first lightning round question just because of how nicely it dovetails. Will LLMs be net positive for medicine over the next five years?

**[0:45:07] Roxana Daneshjou:** I don't have a great answer. I don't. Five years. I'm not allowed to say I don't know. These have to be short answers. Is it going to be net positive five years? I'm a little concerned because we really need to do more research and have frameworks for assessing when they're working, when they're not working and how biased they are. I think in the long term, they could be net positive, but I don't know if five years is long enough.

**[0:45:36] Roxana Daneshjou:** So maybe I'll just say perhaps net

**[0:45:39] Andrew Beam:** neutral because they won't do much of anything over the next five years. Is that I wouldn't say

**[0:45:43] Roxana Daneshjou:** They weren't going to do anything. I just, yeah, I'm terrible at predicting the future. Now, if there are several companies working on it, helping it write medical notes, and if they get that to work well, that would really make my life better. Right. So there's some administrative

**[0:45:59] Andrew Beam:** things like that that are probably not harmless, but the stakes are lower that seem likely to come

**[0:46:05] Roxana Daneshjou:** over the next five years. And so if those come in the next, I think they'll cause improvements and quality of life on administrative stuff. I'm not sure in terms of medical care. I'm still wary about them being ready for actual medical care.

**[0:46:19] Arjun Manrai:** Got it. Roxana, if you weren't in medicine, what job would you be doing?

**[0:46:23] Roxana Daneshjou:** So I'm going to say like not medicine, STEM or science related at all, because my answer to this is that I would be a national park ranger and I would be the park ranger that gets to hike all day and check on everyone. Because I love, I just love being in nature. I love hiking. And it just seems like it would seem like a cool job. You get to meet people from all over the world and hike all day.

**[0:46:45] Arjun Manrai:** Follow on question. Top one or two national parks.

**[0:46:49] Roxana Daneshjou:** So, I mean, I live in California. So, you know, I've been to Yosemite.

**[0:46:54] Arjun Manrai:** You have an embarrassment of riches.

**[0:46:56] Roxana Daneshjou:** Yeah, I've kind of lost. I've kind of lost count number time. Pacific Symposium on Biocomputing is in Big Island. So Volcanoes National Park is another one I've been to many, many times, which is an amazing park because it changes every time we go and there's lava and you know I know you guys have your competing conference um but we have volcanoes and that conference was

**[0:47:21] Andrew Beam:** started completely out of jealousy for PSB just to be clear um so I like that because I learned two things one is that you like the outdoors and be a park ranger and two I think we need to switch up our lightning round questions because that one seemed prepared it seemed like you had thought

**[0:47:34] Roxana Daneshjou:** about that before. Oh, no. Actually, it's because it feels like every icebreaker that I do always has that question. So literally, we just had a retreat where that was the icebreaker question.

**[0:47:46] Andrew Beam:** Got it. Okay. So the next one, what is the best thing that you have read or watched in the last

**[0:47:51] Roxana Daneshjou:** year? The best thing? The AI Grand Rounds podcast. Actually, the podcast is really good. It's been

**[0:47:59] Arjun Manrai:** No, no, no. Just get something other than the podcast. Thank you, though. Your check's in the mail.

**[0:48:06] Roxana Daneshjou:** No, no, no. The podcast is quite excellent. You know, I'm just trying to think because it's been hard with small kids to like really watch much TV.

**[0:48:19] Andrew Beam:** What's your favorite episode of Bluey?

**[0:48:22] Roxana Daneshjou:** My daughter loves number blocks. so it's been a lot of i've been listening to um i've been listening to a lot of audiobooks that's how i get through i'm just trying to see sorry give me a second i know that it's supposed to be

**[0:48:36] Andrew Beam:** fast that one well it doesn't have to be fast because we can make it seem fast in the post oh

**[0:48:41] Roxana Daneshjou:** yeah that's true i'm like trying to even i like literally can't even remember i've listened to some audiobooks but i'm just trying to remember um what i've even like listened to recently

**[0:48:54] Arjun Manrai:** this is very embarrassing all the like net netflix or like hbo nothing

**[0:49:03] Roxana Daneshjou:** i mean i nothing uh oh you know what i i did enjoy i did enjoy i just recently listened to the audiobook range which i liked because similar to in that book i have spent a lot of time in training and exploring different things. And that's kind of helped me put stuff together. And so that book talks about essentially how people have come at problems in new ways by bringing in like past experiences from different fields.

**[0:49:34] Roxana Daneshjou:** Nice. All right. So I think I know the

**[0:49:37] Arjun Manrai:** answer to this question, but I'm very, very curious if I'm right here. Will AI and medicine be driven more by computer scientists or by clinicians? I think it has to be driven by

**[0:49:49] Roxana Daneshjou:** clinicians in the sense, I actually think it really has to be driven by teams, not one or the other, and interdisciplinary people who understand both sides. Because domains, I think domain-specific expertise is so important. I talk to computer scientists all the time who are going after a problem that's not even a real problem in medicine. And, you know, I'm like, don't spend all your time and effort on this. Or they make some sort of assumption about the data in medicine that's not true.

**[0:50:21] Roxana Daneshjou:** And so then they built this whole model that's built on some assumption about what the data is like that ends up being untrue. So I do think they need domain expertise help. And obviously clinicians who don't have AI experience think that AI can do things that it cannot do. or think don't realize how the models are trained and don't realize what pitfalls or biases exist in the model. So really, I think it has to be interdisciplinary people or teams.

**[0:50:51] Andrew Beam:** I think that that's been a recurrent theme on the podcast. So I think there's not much to object with there. Right. So if you could have dinner with one person alive or dead, who would it be?

**[0:51:01] Roxana Daneshjou:** So you can see I really did not prepare for these questions. that's the point we like your candy i know um who would i have dinner with oh my god oh murray curry could i mean as a as a female scientist like oh yeah yeah you know i'd love to i mean she's clearly brilliant and would just love to talk to her and see how her brain works

**[0:51:31] Arjun Manrai:** Yeah, that's a good choice. Awesome answer. All right, our last lightning round question. Do you think that things created by AI can be considered art?

**[0:51:41] Roxana Daneshjou:** That is an excellent question. I think that things created by AI usually are not in isolation from human. So I love art. And I think that art is about expression of emotion and feeling and experience. and it makes you feel something. And from what I've seen from AI-generated art, there's usually a human still behind it, right? A human trying to express, using AI to express. The prompter.

**[0:52:18] Andrew Beam:** Can I change the setup then slightly? Let's say that we hook ChatGPT up to Dolly, which you can already, and say, make art.

**[0:52:27] Roxana Daneshjou:** Okay, so that doesn't feel like art to me because again, I feel like art is about a human conveying their experience in some way. So I think humans and AI working together can make art because there's some emotion or experience that's being conveyed that can then be taken in by the observer, but maybe not like just randomly generated images from.

**[0:52:53] Andrew Beam:** If I could put on my like elbow patches for just a second, if a human looking at the thing created by the AI has some type of feelings or, you know, beauty being in the eye of the beholder.

**[0:53:07] Roxana Daneshjou:** Right. I guess in that case, I think this is a very nuanced, complicated question that you're trying to make me put my foot down. I don't want to get canceled on Twitter.

**[0:53:17] Arjun Manrai:** We'll leave it at that. All right, Roxana, you survived the lightning round. That was excellent. We just have a few concluding big picture questions for you. The first is what areas of medicine do you think will be the most resistant to change from AI?

**[0:53:34] Roxana Daneshjou:** Resistant to change? I mean, I think some of it is appropriate resistant to change. I think the human element is just so important in medical care. As someone who's received medical care, as someone who's had family, who's received bad news, Like there's something so important about, you know, the empathy of another human being in the room, conveying that information, helping answer your question. So, for example, we should not be using AI to deliver bad news. And I think that actually people keep trying to use AI to build diagnostic algorithms, particularly with imaging data.

**[0:54:14] Roxana Daneshjou:** But I think the actual diagnostic process that happens in medicine is quite nuanced, more than people realize. And this is an example that I actually recently gave, just a very easy, like not even a complicated case. So we're looking at a lesion. We're trying to label some dermatology data, right? We have images of the lesions. We have the biopsy results. Turns out the biopsy results are not very definitive. And then we're looking at follow-up notes, like literally this dermatopathologist and I are trying to like label this image data.

**[0:54:48] Roxana Daneshjou:** And we're reading the follow-up notes to see what treatments were tried, like what the dermatologist thought in order to come up with a diagnostic label. Because I think people think that diagnosis is always black and white. So I actually think that diagnosis is a lot harder than people realize. And I think diagnostic tasks, there might be some resistance there compared to using AI to help with administrative, triage, decision support. But like straight up diagnosis is actually a lot harder than here's the diagnosis.

**[0:55:25] Roxana Daneshjou:** In some cases, some things are straightforward, but many things are not.

**[0:55:29] Arjun Manrai:** So for the first example that you gave, which is appropriately resistant, and I like the way you phrased that, appropriately resistant to change, things like delivering bad news, right, or counseling a patient that you don't want to come from a computer, you want this to come from another human. A lot of AI leaders now, medical AI leaders are arguing that AI might actually enable doctors to have more time with their patients because they absorb some of that administrative burden or because they're allowing the doctor to make eye contact with the patient and not just be entering things into the computer.

**[0:56:03] Arjun Manrai:** I'm just curious your personal stance. Are you optimistic about, let's say, digital scribes or AI agents that are listening to the encounter that allow you to then make eye contact, focus on your patient, have time to be there?

**[0:56:16] Roxana Daneshjou:** I would love a digital scribe. I would say two things about it. One, need to ensure patient privacy on any company that's listening into the encounter and building such a model. Patient privacy is key. to there needs to be ways to verify like in case you don't hey i don't remember actually the patient saying that like being able to look in you need to know provenance exactly and there are companies who are doing it in exactly that way so i think if you have those two sort of pillars and also like the third making sure there's not any kind of hallucination that happens I think an AI scribe, if you can meet those criteria, would absolutely allow more empathetic care,

**[0:57:04] Roxana Daneshjou:** more eye contact. I actually work with a human scribe. And when I started working with a human scribe, it was totally changed my life because I'm not at the computer at all now with my patients, which is what I prefer. I don't want to be looking at the computer. But when you have so many encounters back to back. If you don't write notes down, you're not going to remember exactly what happened, which is important. And so it makes such a huge difference to have somebody, whether that's human or AI, kind of document that encounter so that you can just focus in.

**[0:57:38] Roxana Daneshjou:** Because the thing is, is that when you're with patients, it's really important to look at their face to make eye contact, but also to sort of read what's happening. how is the person reacting to the information they're sharing? That's an important part of the art of medicine. Do they seem anxious? Do they seem concerned? Do they look like they might cry? Like you need to know that information. You can't do that if you're staring at a computer.

**[0:58:04] Andrew Beam:** Great. So I like to ask a completely different kind of question. So in addition to being AI nerds, the three of us also have something else in common. We're all junior faculty who have small kids. So I have a four-year-old. We have another one on the way. I started my faculty job in the same month as our daughter was born. So that was a crazy time in my life. So I was wondering if you'd be willing to share what your experience of going on this sort of academic ride has been like, while also balancing the needs of small kids and like what strategies do you have?

**[0:58:35] Andrew Beam:** What has worked well? What hasn't worked well?

**[0:58:38] Roxana Daneshjou:** Does the word chaos feel relevant?

**[0:58:41] Andrew Beam:** It feels exhaustive. Super relevant. It feels relevant and exhaustively descriptive.

**[0:58:47] Roxana Daneshjou:** Yes. I see the little trampoline you have behind you. Yeah. So I have, she just turned five and I actually also have another one on the way. So I'm 33 weeks pregnant. Congratulations. And yeah, starting a faculty job while pregnant has been a interesting experience. But as I tell, I try to tell the trainings, I'm like, there's really no good time. Like also you cannot time it. Sometimes things don't work out the way you expect it to.

**[0:59:18] Roxana Daneshjou:** You just have to do what's best for your family and not try to time things because unfortunately it doesn't work out like that. I think that for me, like my family is very important to me. And so I prioritize. I mean, one thing that's nice about our job is flexibility because I'm a clinician half day a week. And I cannot cancel clinic last minute. But if I needed to work on some writing and something happens, like I can write later that day or later in the evening when my daughter's gone to sleep.

**[0:59:55] Roxana Daneshjou:** So I do appreciate some of the flexibility afforded by an academic research career. I think I try to talk about this because the training is so long, especially for MD PhDs, which is again why I say like you just have to do what's best for your family. We actually don't live near any family, which makes it extra difficult. But we do have family that has come in and supported as needed. And I think that's really huge.

**[1:00:25] Roxana Daneshjou:** I think it's really hard. I think institutions need to do a much better job of supporting young trainees and faculty that have kids because whether that's like affordable child care, because childcare is so expensive or just having like easy backup options.

**[1:00:44] Andrew Beam:** I don't know, how do you guys feel? No, that all resonates with me. I love that you also share that with your trainees because I try and do the same thing. Like for instance, Friday is Veterans Day and so that means daycare is closed. And so that means we usually have our lab meetings on Friday. And so it means that there's no lab meetings. You know, we're recording this podcast interview later than we intended to because my daughter got a fever and couldn't go to daycare.

**[1:01:10] Andrew Beam:** So we had to reschedule. And thank you for understanding that. But I think that I try and be transparent like that with my trainees too, because then they know sort of what I'm up to. But I think the surprise for me was how little insight I had into what being a working parent would be before I became one myself. And so I think that there's a big information asymmetry and you don't really know what it's going to be like until you're there.

**[1:01:33] Andrew Beam:** So at least I'm trying to hand it down to the next generation so that they can know what the other side's going to look like.

**[1:01:39] Roxana Daneshjou:** I also think it's very helpful when you have colleagues who are working parents who then understand that meetings at a certain time, which corresponds exactly to the preschool pickup or drop-off, don't work so well. Or completely, like when you emailed me that your child had a fever, my immediate reaction was like, yeah, I've totally been there. I'm not frazzled at all by that. In fact, you had my deep empathy. also because many times they come home and then they feel better and they're running around the house and then they can't go back for 24 hours, but they're not actually doing well.

**[1:02:15] Andrew Beam:** I always say that they often have a virus, but they're not sick. Yeah, exactly. Ineligible for daycare, but not really sick.

**[1:02:27] Arjun Manrai:** All right, Roxana, this is our last question. What are you most optimistic about for the future

**[1:02:33] Roxana Daneshjou:** of AI in medicine? What I'm most optimistic about is the future, like the people who are in training now, the medical students, the graduate students, the postdocs. I have had a chance to work with many of them and to give lectures. And a lot of people are caring about fairness and equity in AI. And I think that in order for this to go correctly, it can't be like, oh yeah, there's a subset of researchers who care about fairness and equity and AI in medicine, right?

**[1:03:06] Roxana Daneshjou:** And everyone else just does whatever. It actually has to be, like, this has to be, like, baked into the system. It has to be, everybody has to be an AI researcher who cares about fairness and equity, not just one portion. And as I've given talks, I have found that I think, like, the next generation of researchers who are coming down the pipeline think this is deeply important. Care about this. Bring it up. Ask me about it.

**[1:03:35] Roxana Daneshjou:** Even before I bring up the topic, they're asking me about it. And that, I think, gives me optimism for the future because these are going to be sort of our colleagues and who are going to be hopefully helping build better systems and helping implement them. That's great. And just to connect that to

**[1:03:55] Arjun Manrai:** your large language models paper on race-based medicine, I think it was medical students across the country who were a big part of pushing for change in the way that these equations are incorporating race over the past few years too. So I think you're totally spot on. And this has been really, really amazing. Thank you so much for joining us for AI Grand Rounds.

**[1:04:17] Roxana Daneshjou:** Thank you so much for having me. This was fun. Thank you.
