# The AI Revolution in Medicine, Revisited: Laws, norms, and ethics for AI in health

- **Show:** Microsoft Research Podcast
- **Published:** 1 May 2025
- **Source:** https://www.microsoft.com/en-us/research/podcast/laws-norms-and-ethics-for-ai-in-health/
- **Transcript:** https://podcasts.opennash.com/shows/microsoft-research-podcast/the-ai-revolution-in-medicine-revisited-laws-norms-and-ethics-for-ai-in-health-9313df/
- **Provider:** OpenNash Podcasts (machine-generated)

## In brief

Health AI is arriving faster than the rules for it, so the answer is neither a ban nor a free-for-all. Laura Adams, Vardit Ravitsky and Roxana Daneshjou argue for shared principles, new ways of regulating, honest disclosure to patients, and routine testing for bias. They treat AI as a tool to deploy responsibly now, because its benefits are real and so are its risks.

The central tension is speed against safety. Adams fears old-style regulation will stifle innovation without preventing harm. Ravitsky asks whether AI is held to a higher standard than error-prone human doctors, while still insisting patients be told when AI is used. Daneshjou shows bias comes from data and from humans, so fixing the model alone is not enough.

## Key insights

1. **Shared principles beat a patchwork of AI guidelines** Adams reviewed at least 60 sets of AI principles and found gaps such as post-launch monitoring and environmental impact. She wants "governance interoperability" so organizations are not stuck with conflicting rules, as happened with state privacy laws and health data exchange.
2. **Regulation must cover AI after deployment** Adams says checking devices as safe before release is not enough. Hospitals also need ways to confirm AI works as intended at the point of care, and she suggests regulatory sandboxes and treating AI somewhat like a licensed provider.
3. **Patients preferred AI messages until told it was AI** In Ravitsky's survey, AI-written replies were longer and seemed more empathetic, and patients rated them higher. Satisfaction dropped once AI authorship was disclosed, so undisclosed AI would win on satisfaction, which she argues is the wrong measure given trust and the right to know.
4. **Training data, not the algorithm, drove skin-tone bias** Daneshjou's team built a diverse skin-disease benchmark and found models did worse on brown and black skin. Fine-tuning on diverse skin tones made the gap go away.
5. **A fair AI can still widen human bias gaps** In a study with Matt Groh, AI assistance raised accuracy in both groups, but the gap grew because physicians trusted the AI more on white skin. Daneshjou concludes the combined human-plus-AI system must be tested, not just the model.

## Questions this episode answers

### What is the National Academy of Medicine's AI Code of Conduct?

It is a set of principles and six short commitments, led by Laura Adams, meant as a touchstone for organizations writing their own AI rules. It was drafted last year and put out for public comment, and a revised version is coming.

### Should patients be told when AI writes their doctor's messages or takes notes?

Vardit Ravitsky says yes, at least for now. In her survey patients liked AI replies until they learned who wrote them, and she argues informed consent and long-term trust require disclosure while AI use is not yet the norm.

### Do AI chatbots show bias in medical answers?

Yes. Roxana Daneshjou's team tested several models, including GPT-3.5, GPT-4 and Claude, and all gave some race-based medical answers. One example was advising race be used to calculate kidney function, which she calls a false trope.

### Will AI replace doctors like dermatologists or radiologists?

Daneshjou doesn't think it is close for dermatology. She says the job involves skin exams, biopsies, judging intermediate cases and choosing treatments, and AI works best on well-defined tasks, which much of medicine is not.

### Why is regulating generative AI in healthcare so hard?

Laura Adams says generative AI is "wickedly challenging" to regulate and that old methods haven't even kept up with earlier AI. She worries they would stifle innovation without stopping harm, so she favors new approaches like regulatory sandboxes (safe testing zones with regulator oversight).

## Facts

- **Peter Lee** [0:00:38]: "Shortly after OpenAI's GPT-4 was publicly released, Kerry Goldberg, Dr. Zak Kohani, and I published The AI Revolution in Medicine"
- **Peter Lee** [0:08:12]: "So, November 2022, CHAT GPT. Later in the spring of 2023, GPT-4."
- **Laura Adams** [0:04:59]: "National Academy was formed more than 50 years ago. It was formed by the federal government, but it is not the federal government."
- **Laura Adams** [0:04:59]: "some probably know of the National Academy of Medicine when it was Institute of Medicine and produced such publications as to Air is Human and Crossing the Quality Chasm"
- **Laura Adams** [0:06:10]: "especially the six commitments that were based on the idea of complex adaptive systems, simple rules"
- **Laura Adams** [0:18:09]: "We looked at least at 60 different sets of these principles, guidelines, and frameworks."
- **Laura Adams** [0:18:45]: "Also, the impact on the environment was a significant gap as well."
- **Laura Adams** [0:19:15]: "we opened it up for public comment once we did the draft last year"
- **Peter Lee** [0:16:59]: "Also, our listeners might want to know that you led something called the Rhode Island Quality Institute, which was really, I think, arguably the most important early kind of examples"
- **Laura Adams** [0:26:15]: "by Bakul Patel and David Blumenthal, who posited, should we be regulating generative AI as we do a licensed and qualified provider?"
- **Laura Adams** [0:26:51]: "I chair the Global Opportunities Group for a regulatory and innovation AI sandbox in the UK."
- **Vardit Ravitsky** [0:32:26]: "We've been around for more than 55 years, and we're considered one of the organisations that actually founded the field known today as bioethics"
- **Vardit Ravitsky** [0:48:45]: "We surveyed hundreds of respondents. And basically what we discovered is that there was a statistically significant difference in their level of satisfaction"
- **Vardit Ravitsky** [0:49:34]: "it turns out that they preferred the messages written by AI. They were longer, more detailed, even conveyed more empathy."
- **Vardit Ravitsky** [0:49:34]: "But then when we disclosed who wrote the message, they were less satisfied when they were told it was AI."
- **Roxana Daneshjou** [1:03:53]: "the majority of these genetic studies were on European ancestry individuals"
- **Roxana Daneshjou** [1:05:48]: "and one of our landmark papers, which was in Science Advances, showed we created a diverse data set, our own diverse benchmark of skin disease and showed that the models performed significantly worse on brown and black skin"
- **Roxana Daneshjou** [1:05:48]: "because when we fine-tuned on diverse skin tones, you could make that bias go away."
- **Roxana Daneshjou** [1:09:12]: "there was a paper in the Proceedings of the National Academy of Sciences, which did not look at technology at all, but actually looked at the race-based biases of medical trainees"
- **Roxana Daneshjou** [1:10:57]: "they said that we should use race in calculating kidney function because there were differences in muscle mass between the races"
- **Roxana Daneshjou** [1:13:04]: "65 % of dermatologists reported using large language models in their practices"
- **Roxana Daneshjou** [1:14:00]: "we held this red teaming event at stanford where we brought together 80 physicians computer scientists engineers"
- **Roxana Daneshjou** [1:19:21]: "I gave it the whole after visit summary and I said, are there any mistakes here? And it clued in that the dose of the medication was wrong."
- **Roxana Daneshjou** [1:23:04]: "there was actually situations where like the accuracy increased um in both groups but the gap actually also increased"
- **Peter Lee** [1:24:48]: "Jeff Hinton, Turing winner and one of the grandfathers of the AI revolution, famously had predicted many years ago that by 2018 or something like that, we wouldn't need human radiologists because of AI."

## Transcript

**[0:00:02] Peter Lee:** This is the moment for broad, thoughtful consideration of how to ensure maximal safety and also maximum access. Like any medical tool, AI needs those guardrails to keep patients as safe as possible. But it's a tricky balance. Those safety measures must not mean that the great advantages that we document in this book end up unavailable to many who could benefit from them. One of the most exciting aspects of this moment is that the new AI could accelerate healthcare in a direction that is better for patients, all patients, and providers as well, if they have access.

**[0:00:38] Peter Lee:** This is the AI Revolution in Medicine Revisited. I'm your host, Peter Lee. Shortly after OpenAI's GPT-4 was publicly released, Kerry Goldberg, Dr. Zak Kohani, and I published The AI Revolution in Medicine to help educate the world of healthcare and medical research about the transformative impact this new generative AI technology could have. But because we wrote the book when GPT-4 was still a secret, it, we had to speculate. Now, two years later, what did we get right and what did we get wrong?

**[0:01:20] Peter Lee:** In this series, we'll talk to clinicians, patients, hospital administrators, and others to understand the reality of AI in the field and where we go from here. The passage I read at the top there is from Chapter 9, Safety First. One needs only to look at examples such as laws mandating seatbelts in cars and, more recently, internet regulation to know that policy and oversight are often playing catch-up with emerging technologies. When we were writing our book, Carrie, Zach, and I didn't claim that putting frameworks in place to allow for innovation and adoption while prioritizing inclusiveness and protecting patients from hallucination and other harms would be easy.

**[0:02:06] Peter Lee:** In fact, in our writing, we posed more questions than answers in the hopes of highlighting the complexities at hand and supporting constructive discussion and action in the space. In this episode, I'm pleased to welcome three experts who have been thinking deeply about these matters, Laura Adams, Vardit Ravitsky, and Dr. Roxana Daneshjou. Laura is an expert in AI, digital health, and human-centered care. As a senior advisor at the National Academy of Medicine, or NAM. She guides strategy for the Academy's science and technology portfolio and leads the Artificial Intelligence Code of Conduct National Initiative.

**[0:02:46] Peter Lee:** Vardit is President and CEO of the Hastings Center for Bioethics, a bioethics and health policy institute. She leads research projects funded by the National Institutes of Health, is a member of the committee developing the National Academy of Medicine's AI Code of Conduct and is a senior lecturer at Harvard Medical School. Roxana is a board-certified dermatologist and an assistant professor of both dermatology and biomedical data science at Stanford University. Roxana is among the world's thought leaders in AI, healthcare, and medicine, thanks in part to groundbreaking work on AI biases and trustworthiness.

**[0:03:27] Peter Lee:** One of the good fortunes I've had in my career is the chance to work with both Laura and Vardit, mainly through our joint work with the National Academy of Medicine. They're both incredibly thoughtful and decisive leaders working very hard to help the world of healthcare and healthcare regulators come to grips with generative AI. And over the past few years, I've become an avid reader of all of Roxana's research papers. Her work is highly technical, super influential, but also informative in a way that spans computer science, medicine, bioethics, and law.

**[0:03:59] Peter Lee:** These three leaders, one from the medical establishment, one from the bioethics field, and the third from clinical research, provide insights into three incredibly important dimensions of the issues surrounding regulations, norms, and ethics of AI in medicine. Here's my interview with Laura Adams. Laura, I'm just incredibly honored and excited that you're joining us here today. So welcome.

**[0:04:29] Laura Adams:** Thank you, Peter. My pleasure. Excited to be here.

**[0:04:32] Peter Lee:** So Laura, I've been working with you at the NEM for a while, and you are a strategic advisor at the NEM. But I think a lot of our listeners might not know too much about the National Academy of Medicine and then within the National Academy of Medicine what the strategic advisor does. So why don't we start there? How would you explain to a person's mother or father what the National Academy of Medicine is?

**[0:04:59] Laura Adams:** Sure. National Academy was formed more than 50 years ago. It was formed by the federal government, but it is not the federal government. It was formed as an independent body to advise the nation and the federal government on issues of science and primarily technology-related issues as well. So with that 50 years, some probably know of the National Academy of Medicine when it was Institute of Medicine and produced such publications as to Air is Human and Crossing the Quality Chasm, both of which were seminal publications that I think had a dramatic impact on quality safety and how we saw our healthcare system and what we saw in terms of its potential.

**[0:05:38] Peter Lee:** So now, for your role with NAM, what does the senior advisor do? What do you do?

**[0:05:43] Laura Adams:** What I do there is, in the course of leading the AI Code of Conduct project, my role there was in framing the vision for that project. really understanding what do we want it to do? What impact did we want it to make? So, for example, some thought that it might be that we wanted everyone to use our code of conduct. And my advice on that was let's use this as a touchstone. We want people to think about their own codes of conduct for their use of AI.

**[0:06:10] Laura Adams:** That's a valuable exercise to decide what you value, what your aspirations are. I also then do a lot of the field alignment around that work. So, I probably did 50 talks last year, conference presentations, webinars, different things where the code of conduct was presented so that the awareness could be raised around it so people could see the practicality of using that tool, especially the six commitments that were based on the idea of complex adaptive systems, simple rules, where we could recall those in the heat of decision-making around AI, in the heat of application, or even in the planning and strategic thinking around it.

**[0:06:50] Peter Lee:** All right, we're going to want to really break into a lot of details here. But I would just like to rewind the clock a little bit and talk about your first encounters with AI. And there's sort of, I guess, two eras. There's the era of AI and machine learning before ChatGPT, before the generative AI era, and then afterwards. Before the era of generative AI, what was your relationship with the idea of artificial intelligence? Was it a big part of your role and something you thought about, or was it just one of many technologies that you considered?

**[0:07:31] Laura Adams:** It was one of many. Watching it help us evolve from predictive analytics to predictive AI, which, of course, I was fascinated by the fact that it could use structured and unstructured data, that it could learn from its own processes. These things were really quite remarkable, but my sense about it was that it was one of many. We were looking at telemedicine, we were looking at a variety of other things, particularly wearables and things that were affecting and empowering patients to take better care of themselves and have more agency around their own care.

**[0:08:05] Laura Adams:** So, I saw it as one of many, and then the world changed in 2022, changed dramatically.

**[0:08:12] Peter Lee:** Right. Okay. So, November 2022, CHAT GPT. Later in the spring of 2023, GPT-4. And so, you know, what were your first encounters and what were you feeling? What were you experiencing?

**[0:08:30] Laura Adams:** At the time, I was curious and I thought, I think I'm seeing four things here that make this way different. And one was, and it proved to be true over time, the speed with which this evolved. And I was watching it evolve very, very quickly and thinking, this is almost, this is kind of mind-blowing how fast this is getting better. And then this idea that, you know, we could scale this. As we were watching the early work with ambient listening, I was working with a group of physicians that were lamenting the cost and the unavailability of scribes.

**[0:09:08] Laura Adams:** They wanted to use scribes. And I'm thinking, we don't have to incur the cost of that. We don't have to struggle with the unavailability of that type of someone in the workforce. And then I started watching the ubiquity. And I thought, oh my gosh, this is unlike any other technology that we've seen. Because with electronic health records for example it's had its place but it was over here we had another digital technology maybe telehealth over here this was one and i thought there will be no aspect of health care that will be left untouched by ai that blew my mind and then i think the last thing was the democratization and i realized wow anyone with um a smartphone has

**[0:09:51] Laura Adams:** access to the most powerful large language models in the world. And I thought, this, to me, is a revolution in cheap expertise. Those were the things that really began to stun me, and I just knew that we were in a way different

**[0:10:09] Peter Lee:** era. It's interesting that you first talked about ambient listening. Why was that of particular

**[0:10:17] Laura Adams:** initial interest to you specifically? It was because one of the things that we were putting together in our code of conduct, which began pre-generative AI, was the idea that we wanted to renew the moral well-being and the sense of shared purpose to the healthcare workforce. That's one of the six principles. And I knew that the cognitive burden was becoming unbearable. When we came out of COVID, it was such a huge wake-up call to understand exactly what was going on at that point of care and how challenging it had become because information overload is astonishing in and of itself.

**[0:10:55] Laura Adams:** And that idea that we have so much in the way of documentation that needed to be done and how much of a clinician's time was taken up doing that rather than doing the thing that they went into the profession to do. And that was interact with people, that was to heal, that was to develop human connection that had a healing effect. And they just, so much of the time was taken away from that activity. I also looked at it and because I studied diffusion of innovations theory and understand what causes something to move rapidly across the social system and get adopted, it has to have a clear relative advantage.

**[0:11:32] Laura Adams:** It has to be compatible with the way that processes work. So I didn't see that this was going to be a hugely disruptive activity to workflow, which is a challenge of most digital tools, is that they're designed without that sense of how does this impact the workflow. And then I just thought that it was going to be a front runner in adoption, and it might then start to create that tsunami, that wave of interest

**[0:11:58] Peter Lee:** in this. And I don't think I was wrong. I have to ask you, because I've been asking every guest, there must have been moments early on in the encounter with generative AI where you felt doubt or skepticism. Is that true? Or did you immediately think, wow, this is something very

**[0:12:15] Laura Adams:** important. No, I did feel doubt and skepticism. My understanding tells me of it and told me of it in the very beginning that this is trained on the internet with all of its flaws. When we think about AI, we think about it being very futuristic, but it's trained on data from the past. I'm well aware of how flawed that data, how biased that data is. Mostly men, mostly white men, when we think about it during a certain age grouping.

**[0:12:42] Laura Adams:** So, I knew that we had inherent massive flaws in the training data, and that concerned me. I saw other things about it that also concerned me. I saw that it's difficulty in beginning to use it and govern it effectively. You really do have to put a good governance system in if you're going to put this into a care delivery system. And I began to worry about widening a digital divide that already was a chasm. And that was between those well-resourced, usually urban hospitals and health systems that are serving the well-resourced and the inner city hospital in Chicago or the rural hospital in Nebraska or the Mississippi Community Health Center.

**[0:13:27] Peter Lee:** Yes. So I think this skepticism about technology, new technologies in healthcare is very well-earned. So you've zeroed in on this issue of technology where oftentimes we hope it'll reduce or eliminate biases, but actually seems to oftentimes have the opposite effect. And maybe this is a good segue then into this really super important national effort that you're leading on the AI code of conduct. because in a way, I think those failures have passed. And even just the idea, the promise that technology should make a doctor or a nurse's job easier, not harder, even that oftentimes seems not to have panned out in the way that we hope.

**[0:14:17] Peter Lee:** And then there's, of course, the well-known issue of hallucinations or of mistakes being made. How did those things inform this effort around a code of conduct? And why a code of conduct?

**[0:14:34] Laura Adams:** Those things weighed heavily on me as the initiative leader, because I had been deeply involved in the spread of electronic health records. Not really knowing and understanding that electronic health records were going to have the effect that they had on the providers that used them. Looking back now, I think that there could have been design changes, but we probably didn't have as much involvement of providers in the design. In some cases, we did. We just didn't understand what it would take to work it into their workflows.

**[0:15:08] Laura Adams:** So I wanted to be sure that the code of conduct took into consideration and made explicit some of the things that I believe would have helped us had we had those guardrails or those guidelines explicit for us. And those are things like our first one is to protect and advance human health and connection. We also wanted to see things about openly sharing and monitoring because we know that for this particular technology, it's emergent. We're going to have to do a much better job at understanding whether what we're doing works and works in the real world.

**[0:15:41] Laura Adams:** So the reason for a code of conduct was we knew that the good news when the here comes AI and it's barreling toward us, the good news was that everybody was putting together guidelines, frameworks, and principle sets. The bad news was, same, that everybody was putting together their own guideline principle and framework set. And I thought back to how much I struggled when I worked in the world of health information exchange and built a statewide health information exchange and then turned to try to exchange that data across the nation and realized that we had a patchwork of privacy laws and regulations across the state.

**[0:16:21] Laura Adams:** it was extremely costly to try to move data. And I thought we actually need, in addition to data intraoperability, we need governance intraoperability, where we can begin to agree on a core set of principles that will more easily allow us to move ahead and achieve some of the potential and the vision that we have for AI if we are not working with a patchwork of different guidelines, principles, and frameworks. So that was the impetus behind it.

**[0:16:51] Laura Adams:** Of course, we again want it to be used as a touchstone, not everybody wholesale adopt what we've said. We want people to

**[0:16:59] Peter Lee:** think about this and think deeply about it. Yeah, Laura, I always am impressed with just how humble you are. You were indeed one of the prime instigators of the digitization of health records leading to electronic health record systems. And I don't think you need to feel bad about that. That was a tremendous advance. I mean, moving a fifth of the U.S. economy to be digital, I think, is significant. Also, our listeners might want to know that you led something called the Rhode Island Quality Institute, which was really, I think, arguably the most important early kind of examples that set a pattern for how and why health data might actually lead very directly to improvements in human health at a

**[0:17:51] Peter Lee:** statewide level or at a population level. And so I think your struggles and frustrations on how to expand that nationwide, I think, are really, really informative. So let's get into what these principles are, you know, what's in the code of conduct.

**[0:18:09] Laura Adams:** Yeah. The six simple rules were derived out of a larger set of principles that we pulled together. And the origin of all of this was we did a fairly extensive landscape review. We looked at least at 60 different sets of these principles, guidelines, and frameworks. We looked for areas of real convergence. We looked for areas where there was inconsistencies. And we looked for out and out gaps. The out-and-out gaps that we saw at the time were things like a dearth of references to advancing human health as the priority.

**[0:18:45] Laura Adams:** Also, monitoring post-implementation. So, at the time, we were watching these evolve, and we thought these are very significant gaps. Also, the impact on the environment was a significant gap as well. And so, when we pulled that together, we developed a set of principles and crosswalked those with learning health system principles. And then once we got that, we again wanted to distill that down into a set of commitments, which we knew that people could find accessible.

**[0:19:15] Laura Adams:** And we published that draft set of principles last year. And we have a new publication that will be coming out in the coming months that will be the revised set of principles and code commitments that we got because we took this out publicly. So we opened it up for public comment once we did the draft last year. Again, many of those times that I spoke about this, almost all of those times came with an invitation for feedback.

**[0:19:43] Laura Adams:** And conversations that we had with people shaped it. And it is in no way, shape, or form a final code of conduct, the set of principles and commitments, because we see this as dynamic. But what we also knew about this was that we wanted to build this with a super solid foundation, a set of immutables, the things that don't change at some vicissitudes or the whims of this or the whims of that. We wanted those things that were absolutely foundational.

**[0:20:11] Peter Lee:** Yeah. So we'll provide a link to the documents that describe the current state of this, but can we give an example of one or two of these principles and one or two of the commitments?

**[0:20:23] Laura Adams:** I've mentioned the Protect and Advance Human Health and Connection as the primary aim. We also want to ensure that the equitable distribution of risks and benefits and that equitable distribution of risks and benefits is something that I was referring to earlier about when I see well-resourced organizations. And one that's particularly important to me is engaging people as partners with agency at every stage of the AI lifecycle. That one matters because this one talks about and speaks to the idea that we want to begin bringing in those that are affected by AI, those on whom AI is used, into the early development and conceptualization of what we want this new tool, this new application to do.

**[0:21:10] Laura Adams:** So that includes the providers that use it, the patients. And we find that when we include them, the ethicists that come along with that, we develop much better applications, much more targeted applications that do what we intend them to do in a more precise way. The other thing about that engaging with agency, by agency, we mean that person, that participant can affect the decisions and they can affect the outcome. So it isn't that there's sort of a token person coming into the table and will allow you to tell your story or so, but this is an active participant.

**[0:21:46] Laura Adams:** We practiced what we preached when we developed the code of conduct, and we brought patient advocates in to work with us on the development of this, work with us on our applications, that first layer down of what the applications would look like, which is coming out in this new paper. We really wanted that component of this because I'm also seeing that patients are definitely not passive users of this. And they're having an agency moment, let's say, with generative AI because they've discovered a new capacity to gain information, to in many ways claim some autonomy in all of this.

**[0:22:26] Laura Adams:** And I think that there is a disruption underway right now, a big one that has been in the works for many years, but it feels to me like AI may be the tipping point for that disruption of the delivery system as we know it.

**[0:22:38] Peter Lee:** Right. I think it just exudes sort of inclusivity and thoughtfulness in the whole process. During this process, were there surprises, things that you didn't expect, things about AI technology itself that surprised you?

**[0:22:57] Laura Adams:** The surprises that came out of this process for me, one of them was I surprised myself. We were working on the commentary paper, and Stephen Lin from Stanford had strong input into that paper. And when we looked at what we thought were missing, he said, let's make sure we have the environmental impact. And I said, oh, really, Stephen, we really want to think about things that are more directly aligned with health, which I couldn't believe came out of my own mouth.

**[0:23:29] Laura Adams:** And Stephen, without saying, do you hear yourself? I mean, I think he could have said that, but he was more diplomatic than that. And he persisted a bit longer and said, I think it's actually the greatest threat to human health. And I said, of course, you're right. But that was surprising and embarrassing for me, but it was eye-opening in that even when I thought that I had understood the gaps and using this as a touchstone.

**[0:23:56] Laura Adams:** So the learning that took place and how rapidly that learning was happening among people involved in this. The other thing that was surprising for me was the degree at which patients became vastly facile with using it to the extent that it helped them begin to, again, build their own capacity. The hashtag patients use AI from Dave DeBronckart. Watch that one. this is more revolutionary than we think. And so I watched that, the swell of that happening, and it sort of shocked me because I was envisioning this as, again, a tool for use

**[0:24:39] Peter Lee:** in the clinical setting. Yes. Okay, so we're running now towards the end of our time together, and I always like to end our conversations with a more provocative topic. And I thought for you, I'd like to use the very difficult word regulation. And when I think about the book that Kerry, Zach, and I wrote, we have a chapter on regulation, but honestly, we didn't have ideas. We couldn't understand how this would be regulated, and so we just defaulted to publishing a conversation about regulation with GPT-4.

**[0:25:21] Peter Lee:** And in a way, I think, I don't know that I or my co-authors were satisfied with that. In your mind, where do we stand two years later now when we think about the need or not to regulate AI, particularly in its application to healthcare, and where has the thinking evolved to? There are two big differences

**[0:25:44] Laura Adams:** that I see in that time that has elapsed. And the first one is we have understood the insufficiency of simply making sure that AI-enabled devices are safe prior to going out into implementation settings. We recognize now that there's got to be this whole other aspect of regulation and assurance that these things are functioning as intended and we have the capacity to do that in the point of care type of setting. So that's been one of the major ones.

**[0:26:15] Laura Adams:** The other thing is how wickedly challenging it is to regulate generative AI. I think one of the most provocative and exciting articles that I saw written recently was by Bakul Patel and David Blumenthal, who posited, should we be regulating generative AI as we do a licensed and qualified provider? Should it be treated in the sense that it's got to have a certain amount of training and a foundation that's got to pass certain tests? It has to demonstrate that it's improving and keeping up with current literature.

**[0:26:51] Laura Adams:** Does it be responsible for mistakes that it makes in some way, shape, or form? Does it have to report its performance? And I'm thinking, what a provocative idea, but it's worth considering. I chair the Global Opportunities Group for a regulatory and innovation AI sandbox in the UK. and we're hard at work thinking about how do you regulate something as unfamiliar and untamed, really, as generative AI. So, I'd like to see us think more about this idea of sandboxes, more this idea of should we be just completely rethinking the way that we regulate.

**[0:27:30] Laura Adams:** To me, that's where the new ideas will come because the danger, of course, in regulating in the old way, First of all, we haven't kept up over time, even with predictive AI, even with pregenerative AI, we haven't kept up. And what worries me about continuing on in that same vein is that we will stifle innovation and we won't protect from potential harms. Nobody wants an AI Chernobyl. Nobody. But I worry that if we use those old tools on the new applications, that we will not only not regulate, then we'll stifle innovation.

**[0:28:08] Laura Adams:** And when I see all of the promise coming out of this for things that we thought were unimaginable, then that would be a tragedy.

**[0:28:19] Peter Lee:** You know, I think the other reflection I've had on this is the consumer aspect of it, because I think a lot of our current regulatory frameworks are geared towards experts using the technology. So when you have a medical device, you know you have a trained, board-certified doctor or licensed nurse using the technology. but when you're putting things in the hands of a consumer, I think somehow the the surface area of risk seems wider to me and so I think that's another thing that somehow our current regulatory concepts aren't really ready for.

**[0:29:03] Peter Lee:** I would agree with that and the

**[0:29:06] Laura Adams:** I think a few things to consider vis-a -vis that is that this revolution of patients using it is unstoppable, so it will happen. But we're considering a project here at the National Academy about patients using AI and thinking about, let's explore all the different facets of that. Let's understand what does safe usage look like? What might we do to help this this new development enhance the patient-provider relationship and not erode it as we saw.

**[0:29:37] Laura Adams:** Don't confuse your Google search with my medical degree type of approach. Thinking about how does it change the identity of the provider? What can we do to safely build a container in which patients can use this without giving them the sense that it's being taken away or that because I just don't, I don't see that happening. I don't think they're going to let it happen. That to me feels extremely important for us to explore all the dimensions of that.

**[0:30:04] Laura Adams:** And that is one project that I hope to be following on to the AI Code of Conduct and applying the Code of Conduct principles with that project.

**[0:30:13] Peter Lee:** Well, Laura, thank you again for joining us. And thank you even more for your tremendous national, even international leadership on really helping mobilize the greatest institutions in a diverse way to fully confront the realities of AI in healthcare. I think it's tremendously important work. Peter, thank you for having me.

**[0:30:38] Laura Adams:** This has been an absolute pleasure.

**[0:30:50] Peter Lee:** I've had the opportunity to watch Laura in action as she leads a national effort to define an AI code of conduct. And our conversation today has only heightened my admiration for her as a national leader. What impresses me is Laura's recognition that technology adoption in healthcare has had a checkered history, and furthermore, oftentimes not accommodated the huge diversity of stakeholders that are affected equally. The concept of an AI code of conduct seems straightforward in some ways, but you can tell that every word in the emerging code has been chosen carefully.

**[0:31:27] Peter Lee:** And Laura's tireless engagement, traveling to virtually every corner of the United States, as well as to several other countries, shows real dedication. And now, here's my conversation with Vardit Ravitsky. Vardit, thank you so much for joining.

**[0:31:51] Vardit Ravitsky:** It's a real pleasure. I'm honored that you invited me.

**[0:31:55] Peter Lee:** You know, we've been lucky. We've had a few chances to interact and work together within the National Academy of Medicine and so on. But I think for many of the normal subscribers to the Microsoft Research Podcast, they might not know what the Hastings Center for Bioethics is and then what you, as the leader of the Hastings Centre, do every day. So I'd like to start there, first off, with what is the Hastings Centre?

**[0:32:26] Vardit Ravitsky:** Mostly we're a research centre. We've been around for more than 55 years, and we're considered one of the organisations that actually founded the field known today as bioethics, which is the field that explores the policy implications, the ethical social issues in biomedicine. So we look at how biotechnology is rolled out. We look at issues of equity, of access to care. We look at issues at the end of life, the beginning of life, how our natural environment impacts our health.

**[0:33:00] Vardit Ravitsky:** Any aspect of the delivery of healthcare, the design of the healthcare system, and biomedical research leading to all this. Any aspect that has an ethical implication is something that we're happy to explore. We try to have broad conversations with many, many stakeholders, people from different disciplines, in order to come up with guidelines and recommendations that would actually help patients, families, communities. We also have an editorial department. We publish academic journals, we publish a blog, and we do a lot of public engagement activities, webinars, in-person events.

**[0:33:38] Vardit Ravitsky:** So, you know, we just try to promote the thinking of the public and of experts on the ethical aspects of health and healthcare. One thing I've been impressed with, with your work

**[0:33:48] Peter Lee:** and the work of the Hastings Center is it really confronts big questions, but also gets into a lot of practical detail. And so we'll get there. But before that, just a little bit about you. The way I like to ask this question is, how do you explain to your parents what you do every day?

**[0:34:09] Vardit Ravitsky:** Funny that you brought my parents into this, Peter, because I come from a family of philosophers. Everybody in my family is in humanities, in academia. When I was 18, I thought that that was the only profession and that I absolutely had to become a philosopher or else what else can you do with your life? I think being a bioethicist is really about on one hand keeping an eye constantly on the science as it evolves. When a new scientific development occurs, you have to understand what's happening so that you can translate that outside of science.

**[0:34:48] Vardit Ravitsky:** So if we can now make a gamete from a skin cell so that babies will be created differently? You have to understand how that's done, what that means, and how to talk about it. The second eye, you keep on the ethics literature. What ethical frameworks, theories, principles have we developed over the last decades that are now relevant to this technology? So you're really a bridge between science,

**[0:35:18] Peter Lee:** biomedicine on one hand and humanities on the other hand. Okay, so let's shift to AI. And here, I'd like to start with a kind of an origin story, because I'm assuming before generative AI and ChatGPT became widely known and available, you must have had some contact with ideas in data science, in machine learning, you know, and in the concept of AI before ChatGPT. Is that true? And, you know, what were some of those early encounters like for you?

**[0:35:58] Vardit Ravitsky:** The earlier issues that I heard people talk about in the field were really around diagnostics and reading images and, ooh, it looks like machines could perform better than radiologists. And, oh, what if women prefer that their mammographies be read by these algorithms? And does that threaten us clinicians because it sort of highlights our limitations and weaknesses as, you know, the weakness of the human eye and the human brain? So there were early concerns about will this outperform the human and potentially take away our jobs?

**[0:36:43] Vardit Ravitsky:** Will it impact our authority with patients? What about de-skilling clinicians or radiologists or any type of diagnostician, losing the ability, some abilities that they've had historically because machines take over? So those were the early day reflections. And interestingly, some of them remain even now with generative AI. All those issues of the standing of a clinician and what sets us apart and will a machine ever be able to perform completely autonomously? And what about empathy and what about relationships?

**[0:37:19] Vardit Ravitsky:** Much of that translated later on into the more advanced technology.

**[0:37:24] Peter Lee:** I find it interesting that you use words like our and we to implicitly refer to humans, homo sapiens, to human beings. And so do you see a fundamental distinction, a hard distinction that separates humans from machines? or, you know, how, if there are replacements of some human capabilities or some things that human beings do by machines, how do you think about that?

**[0:37:57] Vardit Ravitsky:** Oh, you're really pushing hard on the philosopher in me here. I've read books and heard lectures by those who think that the line is blurred, and I don't buy that. I think there's a clear line between human and machine. I think the issue of AGI, of artificial general intelligence, and will that amount to consciousness, again, it's such a profound, deep philosophical challenge that I think it would take a lot of conceptual work to get there. So how do we define consciousness?

**[0:38:36] Vardit Ravitsky:** How do we define morality? The way it stands now, I look into the future without being a technologist, without being an AI developer, and I think, maybe I hope, that the line will remain clear that there's something about humanity that is irreplaceable. but I'm also remembering that Immanuel Kant the famous German philosopher when he talked about what it means to be a part of the moral universe what it means to be a moral agent he talked about rationality and the ability to implement what he called the categorical imperative and he said that would apply to any creature not just humans and that's so interesting it's always fascinated me that so many centuries ago, he said such a progressive thing.

**[0:39:32] Peter Lee:** That's amazing.

**[0:39:33] Vardit Ravitsky:** It is amazing because I often, as an ethicist, I don't just ask myself what makes us human. I ask myself what makes us worthy of moral respect? What makes us holders of rights? What gives us special status in the universe that other creatures don't have? And I know this has been challenged by people like Peter Singer who say some animals should have the same respect. And what about fetuses and what about people in a coma? I know the landscape is very fraught.

**[0:40:08] Vardit Ravitsky:** But the notion of what makes humans deserving of special moral treatment to me is the core question of ethics. And if we think that it's some capacities that give us this respect, that make us hold that status, then maybe it goes beyond humans. So it doesn't mean that the machine is human, but maybe at a certain point these machines will deserve a certain type of moral respect that it's hard for us right now to think of a machine as deserving that respect.

**[0:40:45] Vardit Ravitsky:** that I can see. But completely collapsing the distinction between human and machine?

**[0:40:51] Peter Lee:** I don't think so, and I hope not. Yeah. Well, in a way, I think it's easier to entertain this type of conversation post-ChatGPT. And so now, what was your first personal encounter with what we now call generative AI? And what went through your mind as you had that first encounter? No one's ever asked me this before, Peter.

**[0:41:22] Vardit Ravitsky:** It almost feels exposing to share your first encounter. So I just logged on and I asked a simple question, but it was an ethical question. I framed an ethical dilemma because I thought, if I ask it to plan a trip, like all my friends already did, it's less interesting to me. And within seconds, a pretty thoughtful, surprisingly nuanced analysis was kind of trickling down my screen. And I was shocked. I was really taken aback. I was almost sad because I think my whole life I was hoping that only humans can generate this kind of thinking in using moral and ethical terms.

**[0:42:14] Vardit Ravitsky:** And then I started tweaking my question and I asked for specific philosophical approaches to this. And it just kept surprising me and how well it performed. So I literally had to catch my breath and, you know, sit down and go, okay, this is a new world, something very important and potentially scary is happening here. How is this going to impact my teaching? How is this going to impact my writing? How is this going to impact health?

**[0:42:44] Vardit Ravitsky:** Like, it was really a moment of shock.

**[0:42:47] Peter Lee:** I think the first time I had the privilege of meeting you, I heard you speak and share some of your initial framing of how, you know, how to think about the potential ethical implications of AI and the human impacts of AI in the future. keeping in mind that people listening to this podcast will tend to be technologists and computer scientists, as well as some medical educators and practicing clinicians. What would you like them to know or understand most about your thoughts?

**[0:43:26] Vardit Ravitsky:** I think from early on, Peter, I've been an advocate in favor of bioethics as a field, positioning itself to be a facilitator of implementing AI. I think on one hand, if we remain the naysayers as we have been regarding other technologies, we will become irrelevant because it's happening. It's happening fast. We have to keep our eye on the ball and not ask, should we do it, but rather ask, how should we do it? And one of the reasons that bioethics is going to be such a critical player is that the stakes are so high.

**[0:44:04] Vardit Ravitsky:** The risk of making a mistake in diagnostics is literally life and death. The risk of breaches of privacy that would lead to patients losing trust and refusing to use these tools. The risk of clinicians feeling overwhelmed and replaceable. The risks are just too high, and therefore creating guardrails, creating frameworks with principles that sensitize us to the ethical aspects, that is critically important for AI and health to succeed. And I'm saying it as someone who wants it very badly to succeed.

**[0:44:43] Peter Lee:** You are actually seeing a lot of healthcare organizations adopting and deploying AI. Has any aspect of that been surprising to you? Have you expected it to be happening faster or slower or unfolding in a different way?

**[0:44:59] Vardit Ravitsky:** One thing that surprises me is how it seems to be isolated. Different systems, different institutions making their own decisions about what to acquire and how to implement. I'm not seeing consistency. and I'm not even seeing anybody at a higher level collecting all the information about who's buying and implementing what, under what types of principles and what are their outcomes? What are they seeing? It seems to be just siloed and happening everywhere and I wish we collected all this data.

**[0:45:35] Vardit Ravitsky:** Even about how the decision is made at the executive level to buy a certain tool to implement it, where, why, by whom. So that's one thing that surprised me. The speed is not surprising me because it really solves problems that healthcare systems have been struggling with. What seems to be one of the more popular uses, and again, you know this better than I do, is the help with scribes with taking notes, ambient recording. This seems to be really desired because of burnout that clinicians face around this whole issue of note-taking.

**[0:46:19] Vardit Ravitsky:** And it's also seen as a way to allow clinicians to do more human interaction, you know, look at the patient, talk to the patient, listen, rather than focus on the screen. We've all sat across the desk with a doctor that never looks at us because they only look at the screen. So there's a real problem here and there's a real solution and therefore it's hitting the ground quickly. But what's surprising to me is how many places don't think that it's their responsibility to inform patients that this is happening.

**[0:46:52] Vardit Ravitsky:** So some places do, some places don't. And to me, this is a fundamental ethical issue of patient autonomy and empowerment. And it's also pragmatically the fear of a crisis of trust. people don't like being recorded without their consent surprise surprise yeah people worry about such a recording of a very private conversation that they consider to be confidential such a recorded recording ending up in the wrong hands or being shared externally or going to a commercial entity people care patients care so what is our ethical responsibility to tell them and what is the institutional responsibility to implement these wonderful tools.

**[0:47:38] Vardit Ravitsky:** I'm not against them. I'm totally in favor. Implement these great tools in a way that respects longstanding ethical principles of informed consent, transparency, accountability for, you know, change in practice. And, you know, bottom line, patient's right to know what's happening in their care.

**[0:48:00] Peter Lee:** You actually recently had a paper in a medical journal that touched on an aspect of this, which I think was not with scribes, but with notes that doctors would send to patients. And in fact, in previous episodes of this podcast, we actually talked to both the technology developers of that type of feature as well as doctors who were using that feature. And in fact, even in those previous conversations, there was the question, well, what does the patient need to know about how this note was put together?

**[0:48:39] Peter Lee:** So you and your co-authors had a very interesting recent paper about this.

**[0:48:45] Vardit Ravitsky:** The trigger for the paper was that patients seem to really like being able to send electronic messages to clinicians. We email and text all day long, why not in health, right? People are used to communicating in that way. It's efficient. It's fast. So we asked ourselves, wait, what if an AI tool writes the response? Because again, this is a huge burden on clinicians and it's a real issue of burnout. We surveyed hundreds of respondents. And basically what we discovered is that there was a statistically significant difference in their level of satisfaction when they got an answer from a human clinician, when they got an answer, again, electronic message from AI.

**[0:49:34] Vardit Ravitsky:** And it turns out that they preferred the messages written by AI. They were longer, more detailed, even conveyed more empathy. You know, AI has all the time in the world to write you a text. It's not rushing to the next one. But then when we disclosed who wrote the message, they were less satisfied when they were told it was AI. So the ethical question that that raises is the following. If your only metric is patient satisfaction, the solution is to respond using AI but not tell them that.

**[0:50:16] Vardit Ravitsky:** now when we compared uh telling them the that it was ai or a human or not telling them anything their satisfaction remained high which means that if they were not told anything they probably assumed that it was a human clinician writing because their satisfaction for human clinician or no disclosure was the same so basically if we say nothing and just send back an ai generated response, they will be more satisfied because the response is nicer, but they won't be put off by the fact that it was written by I and therefore, hey presto, optimal satisfaction.

**[0:50:56] Vardit Ravitsky:** But we challenge that and we say, it's not just about satisfaction. It's about long-term trust. It's about your right to know. It's about empowering you to make decisions about how you want to communicate. So we push back against this notion that we're just there to optimize patient satisfaction and we bring in broader ethical considerations that say, no, patients need to know, if it's not the norm yet to get your message from AI, they should know that this is happening.

**[0:51:26] Vardit Ravitsky:** And I think, Peter, that maybe we're in a transition period. It could be that in two years, maybe less than that, most of our communication will come back from AI and we will just take it for granted that that's the case. And at that point, maybe disclosure is not necessary because many, many surveys will show us that patients assume that and therefore they are informed. But at this point in time, when it's transition and it's not the norm yet, I firmly think that ethics requires that we inform patients.

**[0:52:02] Peter Lee:** Let me push on this a little bit, because I think this final point that you just made is, I think, is so interesting. Does it matter what kind of information is coming from human or AI? Is there a time when patients will have different expectations for different types of information from their doctors?

**[0:52:22] Vardit Ravitsky:** I think, Peter, that you're asking the right question because it's more nuanced. And these are the kinds of empirical questions that we will be exploring in the coming months and years. Our recent paper showed that there was no difference regarding the content. If the message was about what we call the serious matter or a less serious matter, the preferences were the same. But we didn't go deep enough into that. That would require a different type of design of study.

**[0:52:58] Vardit Ravitsky:** And you just said, you know, there are different types of information. We need to categorize them. What types of information and what degree of impact on your life? Is it a life and death piece of information? Is it a quality of life piece of information? How central is it to your care and to your thinking? So all of that would have to be mapped out so that we can design these studies. But, you know, you pushed back in that way and I want to push back in a different direction that to me is more fundamental and philosophical.

**[0:53:32] Vardit Ravitsky:** How much do we know now? You know, I keep saying, oh, patients deserve a chance for informed consent and they need to be empowered to make decisions. And if they don't want that tool used in their care, then they should be able to say, no, really? Is that the world we live in now? Do I have access to the black box that is my doctor's brain? Do I know how they performed on this procedure in the last year?

**[0:54:01] Vardit Ravitsky:** Do I know whether they're tired? Do I know if they're up to speed on the literature with this? we already deal with black boxes except they're not AI and I think the evidence emerges that AI outperforms the humans in so many of these tasks so my pushback is are we seeing AI exceptionalism in the sense that if it's AI panic we have to inform everybody about everything and we have to give them choices and they have to be able to reject that tool and the other tool versus, you know, the rate of human error in medicine is awful.

**[0:54:41] Vardit Ravitsky:** People don't know the numbers. The annual deaths attributed to medical error is outrageous. So why are we so focused on informed consent and empowerment regarding implementation of AI and less in other contexts? Is it just because it's new? Is it because it is some sort of existential threat, not just a matter of risk? I don't know the answer, but I don't want us to suffer from AI exceptionalism, and I don't want us to hold AI to such a high standard that we won't be able to benefit from it, whereas, again, we're dealing with black boxes already in medicine.

**[0:55:23] Vardit Ravitsky:** Just to stay on this topic, though, one more

**[0:55:26] Peter Lee:** question, which is maybe almost silly in how hypothetical it is. If instead of email, it were a Teams call or a Zoom call, doctor to patient, except that the doctor is not the real doctor, but it's a perfect replica of the doctor designed to basically fool the patient that this is the real human being and having that interaction. Does that change the bioethical considerations

**[0:55:57] Vardit Ravitsky:** at all? I think it does, because it's always a question of, are we really misleading? Now, if you get a text message in an environment that people know AI is already integrated to some extent, maybe not your grandmother, but the younger generation is aware of this implementation, then maybe you can say it was implied I didn't mean to mislead the patient if the patient thinks they're talking to a clinician and they're seeing like what if it's not you now Peter what if I'm talking to an avatar or some representation of you would I feel that I was misled in recording this podcast yeah I would because you really gave me good reasons to assume that it was you So it's

**[0:56:47] Vardit Ravitsky:** something deeper about trust, I think. And it touches on the notion of empathy. A lot of literature is being developed now on the issue of what will remain the purview of the human clinician. What are humans good for when AI is so successful and especially in medicine? And if we see that the text messages are read as conveying more empathy and more care and more attention, And if we then move to a visual representation, facial expressions that convey more empathy, we really need to take a hard look at what we mean by care.

**[0:57:28] Vardit Ravitsky:** What about then the robots, right, that we can touch, that can hug us? I think we're really pushing the frontier of what we mean by human interaction, human connectedness, care and empathy. this will be a lot of material for philosophers to ask themselves the fundamental question you asked me at first what does it mean to be human but this time what does it mean to be two humans together and to have a connection and if we can really be replaced in the sense that patients will feel more satisfied more heard more cared for do we have ethical grounds for resisting that And if so, why?

**[0:58:11] Vardit Ravitsky:** You're really going deep here into the conceptual questions, but bioethics is already looking at that.

**[0:58:18] Peter Lee:** Vardit, it's just always amazing to talk to you. The incredible span of what you think about from those fundamental philosophical questions all the way to the actual nitty-gritty, like, you know, what parts of an email from a doctor to a patient should be marked as AI. I think that span is just incredible and incredibly needed and useful today. So,

**[0:58:43] Vardit Ravitsky:** thank you for all that you do. Thank you so much for inviting me.

**[0:58:56] Peter Lee:** The field of bioethics, and this is my take, is all about the adoption of disruptive new technologies into biomedical research and healthcare. And Vardit is able to explain this with such clarity. I think one of the reasons that AI has been challenging for many people is that its use spans the gamut from the nuts and bolts of how and when to disclose to patients that AI is being used to craft an email, all the way to what does it mean to be a human being caring for another human.

**[0:59:27] Peter Lee:** What I learned from the conversation with Vardit is that bioethicists are confronting head-on the issue of AI in medicine, and not with an eye towards restricting it, but recognizing that the technology is real, it's arrived, and needs to be harnessed now for maximum benefit. And so now, here's my conversation with Dr. Roxana Daneshjou. Roxana, I'm just thrilled to have this chance to chat with you.

**[0:59:55] Roxana Daneshjou:** Thank you so much for having me on today. I'm looking forward to our conversation.

**[1:00:00] Peter Lee:** In Microsoft research, of course, we think about healthcare and biomedicine a lot. But I think there's less of an understanding from our audience what people actually do in their day-to-day work. And, of course, you have such a broad background, both on the science side with a PhD and on the medical side. So what's your typical work week like?

**[1:00:28] Roxana Daneshjou:** I spent basically 90 % of my time working on running my research lab and doing research on how AI interacts with medicine, how we can implement it to fix the pain points in medicine, and how we can do that in a fair and responsible way. and 10 % of my time I am in clinic. So I am a practicing dermatologist at Stanford and I see patients half a day a week.

**[1:01:01] Peter Lee:** And your background, it's very interesting. There's always been these MD-PhDs in the world, but somehow, especially right now, what's happening in AI, people like you have become suddenly extremely important. because it suddenly has become so important to be able to combine these two things. Did you have any inkling about that when you started, let's say, on your PhD work?

**[1:01:30] Roxana Daneshjou:** So I would say that during my, because I was in training for so long, during my PhD was in computational genomics. I still have a significant interest in precision medicine, And I think AI is going to be central to that. But I think the reason I became interested in AI initially is that I was thinking about how we associate genetic factors with patient phenotypes, patient phenotypes being how does the disease present? What does the disease look like?

**[1:02:10] Roxana Daneshjou:** and I thought you know AI might be a good way to standardize phenotypes from images of say skin disease because I was interested in dermatology at that time and you know the the part about phenotyping disease was a huge bottleneck because you know you would have humans sort of doing the phenotyping. And so in my head, when I was getting into the space, I was thinking, I'll bring together, you know, in computer vision, and genetics to try to, you know, make new discoveries about how genetics impacts human disease.

**[1:02:53] Roxana Daneshjou:** And then when I actually started my postdoc to learn computer vision, I went down this very huge rabbit hole, which I am still, I guess, falling down where I realized, you know, about biases in computer vision and how much work needed to be done for generalizability. And then after that, large language models came out and like everybody else became incredibly interested in how this could help in healthcare. And now also vision language models and multimodal models.

**[1:03:27] Roxana Daneshjou:** So, you know, we're just tumbling down the rabbit hole.

**[1:03:30] Peter Lee:** Indeed, I think you really made a name for yourself by looking at the issues of biases, for example, in training data sets. And that was well before large language models were a thing. Maybe our audience would like to hear a little bit more about that earlier work.

**[1:03:53] Roxana Daneshjou:** So as I mentioned, my PhD was in computational genetics. In genetics, what has happened during the genetic revolution is these large-scale studies to discover how genetic variation impacts human disease and human response to medication. So that's what pharmacogenomics is, is human response to medications. And as I got, you know, entrenched in that world, I came to realize that I wasn't really represented in the data. And it was because the majority of these genetic studies were on European ancestry individuals, you weren't represented either.

**[1:04:40] Roxana Daneshjou:** Many diverse global populations were completely excluded from these studies. And genetic variation is quite diverse across the globe. And so you're leaving out a large portion of genetic variation from these research studies. Now, things have improved. It still needs work in genetics, but definitely there has been many amazing researchers, you know, sounding the alarm in that space. And so during my PhD, I actually focused on doing studies of pharmacogenomics in non-European ancestry populations. So when I came to computer vision, and in particular, dermatology, where there were a lot of papers being published about how AI models perform at diagnosing skin disease, and several papers essentially saying, oh, it's equivalent to a dermatologist.

**[1:05:48] Roxana Daneshjou:** Of course, that's not completely true because it's a very sort of contrived setting of diagnosis. But my first inkling was, well, are these models going to be able to perform well across skin tones? and one of our landmark papers, which was in Science Advances, showed we created a diverse data set, our own diverse benchmark of skin disease and showed that the models performed significantly worse on brown and black skin. And I think the key here is we also showed that it was an addressable problem because when we fine-tuned on diverse skin tones, you could make that bias go away.

**[1:06:38] Roxana Daneshjou:** So it was really, in this case, about what data was going into the training of these computer vision models.

**[1:06:45] Peter Lee:** Yeah, and I think if you're listening to this conversation, if you haven't read that paper, I think it's really must-reading. It was not only, Roxana, it wasn't only just a landmark scientifically and medically, But it also sort of crossed the chasm and really became a subject of public discourse and debate as well. And I think you really changed the public discourse around AI. So now I want to get into generative AI. I always like to ask, what was your first encounter with generative AI personally?

**[1:07:25] Peter Lee:** And what went through your head? What was that experience like for you?

**[1:07:30] Roxana Daneshjou:** Yeah, I mean, I actually tell this story a lot because I think it's a fun story. So I would say that I had played with, you know, GPT-3 prior and wasn't particularly, you know, impressed by how it was doing. And I was at NeurIPS in New Orleans and I was we were walking back from a dinner. I was with Andrew Beam from Harvard. I was with his group. And we were just sort of, you know, walking along, enjoying the sights of New Orleans, chatting.

**[1:08:05] Roxana Daneshjou:** And one of his students said, hey, OpenAI just released this thing called ChatGPT.

**[1:08:14] Peter Lee:** So this would be New Orleans in December 2, right? Yes. Okay.

**[1:08:21] Roxana Daneshjou:** So I went back to my hotel room. I was very tired, but I, you know, went to the website to see, okay, like, what is this thing? And I started to ask medical questions, and I started all of a sudden thinking, uh-oh, we have made a leap here. Something has changed.

**[1:08:47] Peter Lee:** So it must have been very intense for you from then because months later you had another incredibly impactful or landmark paper basically looking at biases, race-based medicine in large language models. So can you say more about that?

**[1:09:12] Roxana Daneshjou:** Yes, I work with a very diverse team. And we have thought about bias in medicine, not just with technology, but also with humans. Humans have biases too. And there's this whole debate around, is the technology going to be more biased than the humans? How do we do that? But at the same time, like the technology actually encodes the biases of humans. And there was a paper in the Proceedings of the National Academy of Sciences, which did not look at technology at all, but actually looked at the race-based biases of medical trainees that were untrue and harmful in that they perpetuated racist beliefs.

**[1:09:59] Roxana Daneshjou:** And so we thought, if medical trainees and humans have these biases, why don't we see if the models carry them forward? And we added a few more questions that we sort of brainstormed as a team. And we started asking the models those questions.

**[1:10:18] Peter Lee:** And by this time, it was GPT-4?

**[1:10:22] Roxana Daneshjou:** We did include GPT-4 because GPT-4 came out as well. And we also included other models as well, such as Claude, because we wanted to look across the board. And what we found is that all of the models had instances of perpetuating race-based medicine. And actually, the GPT models had one of the most, I think, one of the most egregious responses. And again, this is 3.5 and 4. We haven't, you know, fully checked to see what things look like, because there have been newer models.

**[1:10:57] Roxana Daneshjou:** in that they said that we should use race in calculating kidney function because there were differences in muscle mass between the races. And this is sort of a racist trope in medicine that is not true because race is not based on biology. It's a social construct. So yeah, that was that study. And that one did spur a lot of public conversation.

**[1:11:26] Peter Lee:** Yeah, your work there even had the issue of bias overtake hallucination, you know, as really the most central and most difficult issue. So how do you think about bias in LLMs? And does that, in your mind, disqualify the use of large language models from particular uses in medicine?

**[1:11:51] Roxana Daneshjou:** Yeah, I think that the hallucinations are an issue too. And in some senses, they might even go with one another, right? Like if it's hallucinating information, that's not true, but also like biased. So I think these are issues that we have to address with the use of LLMs in healthcare. But at the same time, things are moving very fast in this space. I mean, we have a secure instance of several large language models within our healthcare system at Stanford so that you could actually put secure patient information in there.

**[1:12:28] Roxana Daneshjou:** So while I acknowledge that bias and hallucinations are a huge issue, I also acknowledge that the healthcare system is quite broken and needs to be improved, needs to be streamlined, physicians are burned out, patients are not getting access to care in the appropriate ways. And I have a really great story about that, which I can share with you later. So in 2024, we did a study asking dermatologists, are they using large language models in their clinical practice?

**[1:13:04] Roxana Daneshjou:** And I think this percentage has probably gone up since then. 65 % of dermatologists reported using large language models in their practices on tasks such as, you know, writing insurance authorization letters, you know, writing information sheets for the patient, even sort of using them to educate themselves, which makes me a little nervous. because in my mind, the best use of large language models right now are cases where you can verify facts easily. So for example, I did show and teach my nurse how to use our secure large language model in our healthcare system to write rebuttal letters to the insurance.

**[1:14:00] Roxana Daneshjou:** I told her, hey, you put in these bullet points that you want to make and you ask it to write the letter and you can verify that the letter contains the facts which you want and which are true yes and um we have also done a lot of work to try to stress test models because we want them to be better and so we held this red teaming event at stanford where we brought together 80 physicians computer scientists engineers and had them write scenarios and real questions that they might ask on a day-to-day or tasks that they might actually ask ai to do and then we had them grade the performance um and we did this with the

**[1:14:49] Roxana Daneshjou:** gpt models at the time we were doing it with gpt 3.54 and four with internet. But before the paper came out, we then ran the data set on newer models. And we made the data set public because I'm a big believer in public data. So we made the data set public so that others could use this data set. And we labeled what the issues were in the responses, whether it was bias, hallucination, like a privacy issue, those sort of things.

**[1:15:19] Peter Lee:** If I think about the hits or misses in our book, you know, we actually wrote a little bit, not too much, about noticing biases. I think we underestimated the magnitude of the issue in our book. And another element that we wrote about in our book is that we noticed that the language model, if presented with some biased decision-making, more often than not was able to spot that the decision making was possibly being influenced by

**[1:15:52] Roxana Daneshjou:** some bias what do you make of that so funny enough i think we had a we had a before i i moved from twitter to blue sky but we we had a had a little back and forth on twitter about this which actually inspired us to look into this as a research, and we have a preprint up on it, of actually using other large language models to identify bias, and then to write critiques that the original model can incorporate and improve its answer upon.

**[1:16:29] Roxana Daneshjou:** I mean, we're moving towards this sort of agentic systems framework rather than a singular large language model. And people, of course, are talking about also retrieval augmented generation where you sort of have this corpus of, you know, text that you trust and find trustworthy and have that incorporated into the response of the model. And so you could build systems essentially where you do have other models saying, hey, specifically look for bias and then it will sort of focus on that on that task and you can even you know give it examples of what bias is within context learning now so I do think that we are going to be improving in this space and actually um my team is most

**[1:17:21] Roxana Daneshjou:** uh recently we're working on building patient-facing chatbots. That's where my patient story comes in. But we're building patient-facing chatbots. And we're using prompt engineering tools. We're using automated eval tools. We're building all of these things to try to make it more accurate and less biased. So it's not just like one LLM spitting out an answer. It's a whole system. All right. So let's get to your patient-facing story? Oh, of course. Over the summer, my six-year-old fell off the monkey bars and broke her arm.

**[1:18:01] Roxana Daneshjou:** And I picked her up from school. She's crying so badly. And I just look at her and I know that we're in trouble. And I said, okay, you know, we're going straight to the emergency room. And we went straight to the emergency room. She's crying the whole time. I'm almost crying because it's just like she doesn't even want to go into the hospital and so then my husband shows up and we also had the baby and the baby wasn't allowed in the emergency room so I had to step out and I'm thanks to the Cures Act I'm getting like all the information you know as it's happening like I'm getting the x-ray results and I'm looking at it and

**[1:18:45] Roxana Daneshjou:** I can tell there's a fracture but I can't you know, tell like how bad it is. Like, is this something that's going to need surgery? And I'm desperately texting like all the orthopedic folks I know, the pediatricians I know, hey, what does this mean? Like getting real time information. And later in the process, there was a mistake in her after visit summary about how much Tylenol she could take. But I, as a physician knew that this dose was a mistake.

**[1:19:21] Roxana Daneshjou:** I actually asked chat GPT. I gave it the whole after visit summary and I said, are there any mistakes here? And it clued in that the dose of the medication was wrong. So again, I as a physician with all these resources have difficulty kind of navigating the healthcare system, understanding what's going on in x-rays results that are showing up you know on my phone um can personally identify medication dose mistakes but you know most people probably couldn't and and it could be very i actually you know emailed the team and let them know to give feedback but so i we have a we have a health care system that is broken in so many ways and it's so difficult

**[1:20:12] Roxana Daneshjou:** to navigate so I get it and so that's been a you know a big impetus for me to work in this space and try to

**[1:20:20] Peter Lee:** make things better. That's an incredible story it's also validating because you know one of the examples in our book was the use of an LLM to spot a medication error that you know a doctor or nurse what might make. Right. You know, interestingly, we're finding no sort of formalized use of AI right now in the field. But anecdotes like this are everywhere. So it's very interesting. All right. So we're starting to run short on time. So I want to ask you a few quick questions and a couple of them might be a little provocative.

**[1:21:01] Roxana Daneshjou:** Oh, boy. Well, I don't run away from controversy.

**[1:21:08] Peter Lee:** So, of course, with that story you just told, I can see that you use AI yourself. When you are actually in clinic, when you are being a dermatologist and seeing patients, are you using generative AI?

**[1:21:26] Roxana Daneshjou:** So I do not use it in clinic except for the situation of the insurance authorization letters. and even um i was offered you know sort of an ai based scribe which many people are using there have been some studies that show that they can make mistakes um i'm still i have a i have a human scribe to me writing the notes is actually part of the thinking process so i when i write my notes at the end of the day there have been times that i've all of a sudden had an epiphany particularly on a complex case but I have used it to write you know sort of these insurance authorization letters I've also used it in grant writing so

**[1:22:10] Roxana Daneshjou:** as a scientist I have used it a lot

**[1:22:12] Peter Lee:** more right so I don't know of anyone who has a more nuanced and deeper understanding of the issues of biases in AI in medicine than you. Do you think this biases can be repaired in AI? And if not, what are the implications?

**[1:22:32] Roxana Daneshjou:** So I think there are several things here, and I just want to be thoughtful about it. One, I think the bias in the technology comes from bias in the system and bias in medicine, which very much exists and is incredibly problematic. And so I always tell people, like, it doesn't matter if you have the most perfect, fair AI. If you have a biased human, and you add those together, because you're going to have this human AI interaction, you're still going to have a problem.

**[1:23:04] Roxana Daneshjou:** And there is a paper that I'm on with Dr. Matt Groh, which looked at looking at dermatology diagnosis across skin tones and then with like AI assistance. And we found there's a bias gap, you know, with even physicians. So it's not just an AI problem. Humans have the problem, too. and we also looked at when you have the human ai system how that impacts the gap because you want to see the gap close and it was kind of a mixed result in the sense that there was actually situations where like the accuracy increased um in both groups but the gap actually also increased because they were actually even though they knew it was a fair ai for some reason they

**[1:23:53] Roxana Daneshjou:** were relying upon the AI more often when, or they were trusting it more often on diagnoses on white skin. Maybe they'd read my papers, who knows? Even though we had told them, it was a fair model. So I think for me, the important thing is understanding how the AI model works with the physician at the task. And what I would like to see is it improve the overall bias and disparities with that unit. And at the same time, I tell human physicians, we have to work on ourselves.

**[1:24:33] Roxana Daneshjou:** We have to work on our system, our medical system that has systemic issues of access to care or how patients get treated based on what they might look like or other parts of their background.

**[1:24:48] Peter Lee:** All right, final question. And so, you know, we started off with your stories about imaging in dermatology. And of course, Jeff Hinton, Turing winner and one of the grandfathers of the AI revolution, famously had predicted many years ago that by 2018 or something like that, we wouldn't need human radiologists because of AI. That hasn't come to pass. but since you work in a field that also is very dependent on imaging technologies do you see a future when radiologists or for that matter dermatologists might be largely replaced by

**[1:25:30] Roxana Daneshjou:** machines i think that's a complex question let's say you have the most perfect ai systems i think there's still a lot of nuance um and how these you know things get done i i'm not a radiologist So I don't want to speak to what happens in radiology. But in dermatology, it ends up being quite complex, the process. I don't just look at lesions and make diagnoses. I do skin exams to first identify the lesions of concern.

**[1:26:02] Roxana Daneshjou:** So maybe if we had total body photography that could help catch which lesions would be of concern, which people have worked on, that would be step one. And then the second thing is, you know, it's, I have to do the biopsy. So, you know, is the robots not going to be doing the biopsy? And then the pathology for skin cancer is sometimes very clear, but there's also like intermediate type lesions where we have to make a decision, bringing all that information together.

**[1:26:35] Roxana Daneshjou:** For rashes, it can be quite complex. And then we have to kind of think about what other tests we're going to order, what therapeutics we might try first. that sort of stuff. So, you know, there is a thought that you might have AI that could reason through all of those steps, maybe, but I just don't feel like we're anywhere close to that at all. I think the other thing is, AI does a lot better on sort of, you know, tasks that are, are well defined.

**[1:27:05] Roxana Daneshjou:** And a lot of things in medicine, like it would be hard to train the model on because it's not well defined, that even human physicians would disagree on the next best step.

**[1:27:16] Peter Lee:** Well, Roxanne, for whatever it's worth, I can't even begin to imagine anything replacing you. I think your work has been just so, I think, used the word, and I agree with it, landmark and multiple times. So thank you for all that you're doing. And thank you so much for joining

**[1:27:33] Roxana Daneshjou:** this podcast. Thanks for having me. This was very fun.

**[1:27:39] Peter Lee:** The issue of bias in AI has been the subject of truly landmark work by Roxana and her collaborators, and this includes biases in large language models. This was something that in our writing of the book, Carrie, Zach, and I recognized and wrote about, but in fairness, I don't think Carrie, Zach, or I really understood the full implications of it, and this is where Roxana's work has been so illuminating and important. Roxana's practical prescriptions around run teaming have proven to be important in practice and equally important were Roxana's insights into how AI might always be guilty of the same biases, not only of individuals, but also of whole healthcare organizations.

**[1:28:25] Peter Lee:** But at the same time, AI might also be a potentially powerful tool to detect and help mitigate against such biases. When I think about the book that Kerry, Zach, and I wrote, I think when we talked about laws, norms, ethics, regulations, it's the place that we struggled the most. And in fact, we actually relied on a conversation with GPT-4 in order to tease out some of the core issues. Well, moving on from that conversation with an AI to a conversation with three deep experts who have dedicated their careers to making sure that we can harness all of the goodness while mitigating against the risks of AI.

**[1:29:10] Peter Lee:** It's been both fulfilling, very interesting and a great learning experience. I'd like to say thank you again to Laura, Vardit, and Roxana for sharing their stories and insights. And to our listeners, thank you for joining us. We have some really great conversations planned for the coming episodes, including an examination on the broader economic impact of AI in health and a discussion on AI drug discovery. We hope you'll continue to tune in. Until next time. Thank you.
