OpenNashPodcasts

NEJM AI Grand Rounds

Beyond the Hype: Dr. Xiao Liu on Evaluating Medical AI

16 Sep 2026 · 43m · Xiao Liu, Arjun Manrai, Andrew Beam · 23 quoted facts

▶ Play episodeOriginal source ↗

In brief

Dr. Xiao Liu argues that medical AI earns trust through transparent, rigorous evidence rather than headline performance claims. Reporting guidelines such as CONSORT-AI and SPIRIT-AI do not dictate how to run a study. They make researchers show what they did, so editors, reviewers and readers can judge the work on equal terms. She also argues that evaluation must continue after deployment.

The central tension is between rigor and openness to discovery. Liu spent years cleaning up a hype-filled literature, yet she worries that rigid, slow evaluation habits could block genuinely new medical insight. She imagines AI revealing new biomarkers and disease subtypes while doctors become companions through illness. She admits she has no clear answer for how to balance caution with speed.

Key insights

  1. Reporting guidelines demand transparency, not good methodology

    Liu says checklists like CONSORT only require authors to disclose what they did, whether the method was good or bad. This puts authors, editors and reviewers on equal footing and often nudges researchers toward better methods because they know they will be exposed.

  2. Early medical AI literature was mostly noise

    Her systematic review found fewer than 5% of deep-learning imaging papers met robust standards such as external validation and meaningful comparison with clinicians. Among those, AI performance was at best equivalent to radiologists, so the problem was unreliable evidence, not bad results.

  3. Good consensus guidelines take months of volunteer effort

    Delphi consensus projects involve multiple voting rounds and an in-person meeting, and Liu says doing them properly needs at least six months. Rushing to three months is possible, but the quality of feedback from volunteers suffers.

  4. Industry work taught her implementation realities

    Liu joined Apple partly because advising regulators felt disingenuous without product experience. She learned how a company ships products, including validating Apple Watch sensing algorithms, before moving to Microsoft AI to work on language models for medicine.

  5. Overcaution could block new medical discoveries

    Liu fears that sticking rigidly to slow, familiar evaluation methods could waste a moment when AI may reveal new disease subtypes and biomarkers. She gives white dot syndromes in the eye as diseases grouped by appearance that may have different mechanisms.

Questions

What is a medical reporting guideline, and does it tell researchers how to run their study?

It is a checklist of things a research paper must state, such as how patients were randomized. Liu stresses that it is about transparency, not methodology: it does not say how to do the study, only that you must clearly report what you did.

What are CONSORT-AI and SPIRIT-AI?

They are AI-specific add-ons to existing guidelines for reporting clinical trials (CONSORT) and trial protocols (SPIRIT). Liu says they add items like how data was handled and what the AI's inputs and outputs were.

How good was early AI research in medical imaging, according to Liu?

Liu says her systematic review found fewer than 5% of papers used robust methods like external validation and fair comparison with clinicians. Even those showed AI performing at best about as well as radiologists.

Will AI replace doctors, according to Liu?

She does not predict replacement. She thinks the job won't look dramatically different in five years, but in ten or more years she hopes doctors will be 'companions of experience' who translate knowledge into human terms while AI supplies much of the medical knowledge.

What is Liu most worried about for medical AI?

She worries that the field will be too risk averse and cling to slow, familiar evaluation methods, missing new discoveries. She says she is not asking for reckless risk-taking and admits she lacks a good answer for how to strike the balance.

Facts

Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.

Xiao Liu began a PhD before ophthalmology training because she did not secure a place in ophthalmology on her first attempt.

“my first foray into I wouldn't even say AI necessarily but just research was when I decided to do a PhD before my ophthalmology training and that was actually because I didn't get in the first time.”

0:03:26 · Xiao Liu · Career path

Pearse Keane, an ophthalmologist at Moorfields Eye Hospital, had started a collaboration with DeepMind to develop a retinal diagnosis model.

“one of my co-supervisors, someone called Pierce Keane, who is an ophthalmologist at Moorfields Eye Hospital in London, and he had just initiated a collaboration with DeepMind to develop a retinal diagnosis model.”

0:04:35 · Xiao Liu · DeepMind collaboration · 2017

John Deeks, a biostatistics professor at the University of Birmingham, did key work on diagnostic test evaluation and shaped Liu's methodological foundations.

“The first one is a professor of biostatistics called John Deeks, who many listeners may know, who did a lot of some of the key work in diagnostic test evaluation.”

0:05:32 · Xiao Liu · Diagnostic test evaluation

Livia Faes, a Swiss ophthalmologist, was co-first author with Xiao Liu on a systematic review of deep learning.

“The second person is someone called Livia Fais, who is a Swiss ophthalmologist and also my co-first author on my first more well-known publications, which was a systematic review on deep learning.”

0:06:01 · Xiao Liu · Systematic review

Lauren Oakden-Rayner is an Australian radiologist who wrote an influential blog on AI that shaped Liu's thinking.

“Lauren Oakden Rayner, an Australian radiologist who used to publish this fantastic blog on AI.”

0:06:39 · Xiao Liu · AI blog

Liu worked on a medical algorithmic audit framework setting out steps for auditing an algorithm after deployment.

“did a piece of work around the medical algorithmic audit which was this theoretical framework which was around these are the steps that you can take to audit an algorithm”

0:09:43 · Xiao Liu · Algorithmic audit

Liu's systematic review comparing deep learning with healthcare professionals in medical imaging was published in The Lancet Digital Health in 2019.

“One of them was a Lancet Digital Health 2019 paper that, if I'm getting this right, you were working on during graduate school and published.”

0:10:46 · Arjun Manrai · Systematic review · 2019

Fewer than 5% of deep-learning imaging papers in Liu's review met the robust methodological standard of other areas of medicine.

“Less than 5 % of the papers actually really met that very standard bar in other areas of medicine.”

0:14:10 · Xiao Liu · Evidence quality · 2019

Among the methodologically robust studies in Liu's review, AI models performed at best equivalently to radiologists.

“And we found that the performance of those models within that cohort was at best equivalent to radiologists.”

0:14:10 · Xiao Liu · AI vs radiologists · 2019

A reporting-guideline consensus meeting involving Xiao Liu and Andy Beam took place in Birmingham, UK in January 2020.

“So there was this meeting in Birmingham, UK that I believe you hosted. I at least remember you leading it. And the reason, again, like I remember it was in January 2020.”

0:15:04 · Andrew Beam · Consensus meeting · 2020

CONSORT covers randomized trials, PRISMA systematic reviews, STARD diagnostic test evaluation and TRIPOD prediction or prognostic models.

“probably one of the best known ones is consort for randomized trials there's also for example prisma for systematic reviews starred for diagnostic test evaluation and tripod for prediction prognostic models”

0:17:35 · Xiao Liu · Reporting guidelines

CONSORT-AI extends the CONSORT trial reporting guideline for AI interventions, and SPIRIT-AI was developed alongside it for clinical trial protocols.

“consort ai was the ai extension of the consort guidelines and we also developed spirit ai at the same time which is for clinical trial protocols”

0:17:35 · Xiao Liu · CONSORT-AI

The AI reporting guidelines were developed through a Delphi study, a multi-stage vote on items drawn from literature review and expert opinion.

“It was a Delphi study, which is essentially a large vote. We propose a set of guidelines that are developed through review of the literature and through expert opinion.”

0:18:31 · Xiao Liu · Delphi method

The CONSORT guidelines are led by David Moher, who is based in Canada.

“the consort guidelines is led by David Moer, who is Canada based.”

0:24:56 · Xiao Liu · CONSORT leadership

Liu largely ran the CONSORT-AI and SPIRIT-AI effort by herself in 2019.

“when we did consort and spirit ai it was pretty much just me as a one man band one woman band back in 2019”

0:22:48 · Xiao Liu · CONSORT-AI development · 2019

The STANDING Together recommendations were developed by a team of four including Joe Alderman, Elinor Laws and Joanne Palmer, working for about a year.

“by the time we did standing together we had a really amazing team so we had joe alderman and eleanor laws and joe palmer and there were four of us”

0:22:48 · Xiao Liu · STANDING Together · 2022

Andy Beam suggested Douglas Altman, a prolific medical statistician involved in many guidelines, may explain why UK researchers lead so many reporting guidelines.

“One reason might be the legacy of Douglas Altman. He was one of the world's most prolific medical statisticians and was involved in a lot of the guidelines.”

0:24:05 · Andrew Beam · Guideline history

At Apple, Liu worked as a health scientist designing studies to validate algorithms on wearables such as the Apple Watch.

“I joined Apple as a health scientist. My main role was designing studies to validate algorithms on wearables.”

0:26:40 · Xiao Liu · Apple role

Apple released a hypertension notification feature for the Apple Watch, which Liu helped validate, in the year of the recording.

“One of which I could speak about now because it was released earlier this year, the hypertension notification.”

0:27:20 · Xiao Liu · Apple Watch · 2025

At Microsoft AI, Liu worked on a sequential diagnosis paper published in the year of the recording.

“One of the main projects I was working on was this sequential diagnosis paper, which was published earlier this year.”

0:28:25 · Xiao Liu · Microsoft AI research · 2025

Liu's first job as a doctor was in acute psychiatry, on a rapid response team assessing people in acute distress.

“my first job was in acute psychiatry. You probably didn't want a long answer for this.”

0:30:14 · Xiao Liu · Early career

Liu predicts computer scientists and engineers will drive medical AI in the short term, with clinicians as gatekeepers in the long term.

“I think the short-term computer scientists, engineers, in the long-term, clinicians.”

0:32:55 · Xiao Liu · Who drives AI

Liu says white dot syndromes group eye diseases by appearance, and they may be driven by different mechanisms.

“we have tons of diseases that are like loosely characterized as quote white dot syndromes just because they happen to produce white dots at the back of the eye”

0:41:19 · Xiao Liu · White dot syndromes

Chapters

  1. 0:00:03Reporting guidelines as transparency tools

    Liu explains that reporting guidelines make research transparent rather than prescribe methods, which levels the field for reviewers and editors.

    “They're not methodological guidelines. They're transparency guidelines. So tell us what you did.”
  2. 0:03:26From ophthalmology to AI evaluation

    Liu describes how a failed residency application led to a PhD, exposure to DeepMind's eye project, and three mentors who shaped her evaluation thinking.

    “I am a medic by training and towards the end of my medical school I was pretty determined”
  3. 0:10:24Auditing algorithms and assigning responsibility

    Her work with regulators led to a medical algorithmic audit framework meant to clarify who is accountable for monitoring AI after deployment.

    “everyone's just pointing at the other guy being like, is there responsibility?”
  4. 0:14:10Sorting hype from evidence in the imaging literature

    Her systematic review applied established diagnostic-test standards and found very few AI studies were methodologically robust.

    “Less than 5 % of the papers actually really met that very standard bar in other areas of medicine.”
  5. 0:22:48Building CONSORT-AI and SPIRIT-AI through Delphi consensus

    The hosts and Liu describe how these guidelines were created by voting rounds and in-person debate, and how much time that takes.

    “To do these consensus projects properly you really do need at least six months.”
  6. 0:26:40Moving into Apple and Microsoft AI

    Liu explains why she sought product experience in industry and what she works on now in medical language models.

    “I decided that I needed to do this at a company with a product.”
  7. 0:34:35Lightning round on explainability and the doctor's future

    Liu says AI need not be explainable, and that doctors may become companions through illness as knowledge becomes widely available.

    “I think the emphasis will be much more on being companions of experience.”
  8. 0:42:09Medical education and AI's discovery potential

    Liu predicts less raw knowledge memorization in training, then voices optimism about new disease categories and pessimism about excess caution.

    “I worry about the lost opportunity if we are too risk averse”

Transcript

0:00:03
Xiao Liu

It could be something as simple as state the intervention that you're testing in the abstract so that when it's on a bibliographic database it's easily searchable. Or it could be something fairly methodological like in randomized control trials, talk about how the mechanism for randomization and describe that to a certain level of detail. What reporting guidelines don't do is tell you how you should do your study. So they're not methodological guidelines. They're transparency guidelines. So tell us what you did.

0:00:33

Even if it was good methodology, bad methodology, just make it really transparent and clear. And what it enables is it supplies enough information that brings the authors, the editors of a journal, and the peer reviewers on a level playing field. So there's no kind of smoke and mirrors and you can't hide behind, you know, vague detail. And it just enables better critical appraisal overall for everyone.

0:01:01
Arjun Manrai

Hi, and welcome to another episode of NEJM AI Grand Rounds. I'm Raj Manrai. I'm here with my good friend and co-host, Andy Beam. And today, we're really excited to bring you our conversation with Dr. Xiao Liu. Xiao is a clinical scientist, a health research lead at Microsoft AI. She's also an associate professor in AI and digital health at the University of Birmingham in the United Kingdom. And she's also an editor, part of the editorial board of NEJM AI.

0:01:29

Andy, this was a really amazing conversation. I think Xiao has done more in the last five or 10 years than pretty much anyone I know in AI and medicine. She's had a few different careers, and I think she's also juggling them all simultaneously and really doing amazing academic work and also work with Apple and Microsoft and others in industry. So I had a lot of fun just picking her brain and learning about how she approaches problems and how thoughtful she is about constructing solutions in medical AI.

0:01:59
Andrew Beam

Yeah, I agree. There's probably few people on the planet who have done more to improve the evidence base for medical AI than Xiao and her collaborators. You and I have both been involved in several of these efforts to set guidelines for reporting and for standards. A lot of the things that we touch on this episode is how it was really the Wild West five, 10 years ago when it comes to the evidence for medical AI. And again, Xiao has been one of the key players in cleaning that up and turning medical AI into something much more like a real science.

0:02:26

I likewise really, really enjoyed the conversation. I also heard things in this conversation I hadn't heard before. Some of her motivations for going to industry to have more applied product understanding, to understand how the technology is impacting patients. Again, I think it just speaks to the breadth of contributions that she's had in the field and just a really, really fun conversation. the NEJM AI Grand Rounds podcast is brought to you by Microsoft Viz.ai Lyric and Elevent Health we thank them for their support and with that we bring you our conversation with Xiao Liu on AI Grand Rounds Xiao thanks for joining us we're excited to have you on the podcast today

0:03:10
Xiao Liu

Thank you so much for having me.

0:03:13
Arjun Manrai

Xiao, it's great to have you here. This is a question that we always get started with. Could you tell us about the training procedure for your own neural network? How did you get interested in artificial intelligence? And what data and experiences led you to where you are today?

0:03:26
Xiao Liu

Yeah, of course. I would love to say that it's one coherent story, which when you read it backwards, it always seems coherent. But I think along the way, it's always a series of just following my own curiosity and nose and then ending up where I ended up. my original neural network which I suppose has stuck with me is I am a medic by training and towards the end of my medical school I was pretty determined to become an ophthalmologist ophthalmic surgery was my passion my first foray into I wouldn't even say AI necessarily but just research was when I decided to do a PhD before my ophthalmology training and that was actually because I didn't get in the first time.

0:04:07

So in the UK, as with many other places in the world, ophthalmology is a highly competitive specialty. And the first time I tried, I didn't get a place. So I was like, oh, okay, what am I going to do now? And I had the opportunity to do a PhD in a project that was to do with quantifying inflammation in the eye non-invasively. And there's a bit of computer image analysis stuff going on in there. I was not computational or technical at all.

0:04:35

I hadn't even done research before, but I thought, okay, this is pretty cool. And then as with many PhDs, some of the most interesting parts end up being the side quests that you do in the evenings and weekends. So when I was doing this PhD, one of my co-supervisors, someone called Pierce Keane, who is an ophthalmologist at Moorfields Eye Hospital in London, and he had just initiated a collaboration with DeepMind to develop a retinal diagnosis model.

0:05:04

and through working with him I got to learn a bit more about AI this is around 2017 and what it could do and how you go about developing algorithms and so that was what sparked my interest in the first place but in terms of the training of my neural network I think I was fortunate enough to have met a couple of quite significant people that shaped how I think about things and I'm going to mention three people on this podcast.

0:05:32

So the first one is a professor of biostatistics called John Deeks, who many listeners may know, who did a lot of some of the key work in diagnostic test evaluation. And that's where I got my methodological foundations built in my neural network. I really learned how you could properly evaluate diagnostic tests from him. And I was lucky enough that he was right on my doorstep in the sense that he was based at University of Birmingham where I was studying.

0:06:01

The second person is someone called Livia Fais, who is a Swiss ophthalmologist and also my co-first author on my first more well-known publications, which was a systematic review on deep learning. She taught me how to appraise some of these papers that were being published at the time. We worked through tons of publications together, scrutinizing every part, trying to apply reporting guidelines to them, and trying to bridge this well-established methodology in test evaluation against this emerging field of AI and machine learning.

0:06:39

So she was really my partner in crime in that piece of work and helped shape a lot of my thinking. And then the third person, who I'm now fortunate enough to be quite good friends with, is someone called Lauren Oakden Rayner, an Australian radiologist who used to publish this fantastic blog on AI. For several years, she did this. And I stalked her like nothing else. And she may not be aware of this, but I read every single thing she put on that blog.

0:07:10

I used to stalk her on Twitter and try and engage in conversations with her. And now many, many years later, that's paid off. We're pretty good friends and we've collaborated a whole bunch.

0:07:19
Andrew Beam

I'll just hop in here and recommend that blog. It's incredible. I also was a huge fan of it and read it constantly. I think it's still up and there's just so many amazing nuggets in there that only a radiologist could do. And so again, highly recommended and likewise a huge fan.

0:07:34
Xiao Liu

Yeah, I completely agree. It really taught me how to think critically about what was important in medical tasks. And it helped me sift out the noise that was happening in the field and be very targeted around how to formulate a problem that could be solved by machine learning. But yes, that blog is amazing and it's timeless. So still worth reading. Amazing, Xiao. And then maybe just

0:08:00
Arjun Manrai

continue the story on. So you finished your graduate degrees and then what was the next step? How did you think about the next step and where did you go from there? So I took this big foray

0:08:09
Xiao Liu

into AI evaluation in the second year of my PhD. And then in the third year, reluctantly went back to finish my PhD work, which then suddenly seemed a lot less shiny, but needed to be done. And I was very insistent that that was a coherent piece of work by itself. So I didn't include any of the AI stuff in the thesis. But then it was pretty obvious to me then that this was the direction I needed to take my work after the PhD.

0:08:34

I also at this point, applied again for ophthalmology and got where I wanted to be in the country and took on my residency job or in the UK, it's called specialty registrar so started my ophthalmic training whilst balancing around two days a week in research so then at this point I was still doing a lot of work around methodology around reporting and because the team that I was working with we were publishing a lot in this area we started getting a lot of outreach from regulators and from health policy makers to help them think about how they wanted to set the evidence bar for AI as they were thinking about rolling this out.

0:09:17

So certainly in the UK, we're thinking about national rollouts, where to put investment within the NHS. So I did a lot of work working with government agencies at that time, which was very interesting and very fun. And it felt very impactful as well. And I guess my interest around the area evolved a bit as well. So I'd spent a bunch of time working on how do you design a study well and how do you evaluate meaningfully?

0:09:43

At the same time, hospitals were beginning to take up technologies, procure them and roll them out in pilots or in real deployment. And I started thinking a bit more about implementation. How do we monitor? How do we audit for errors? As we know with AI, often you learn more after it's in real use than in the pre-deployment phase. So I started thinking more about what did hospitals need to be able to do that task properly and how do we do that in an accountable and transparent way so did a piece of work around the medical algorithmic audit which was this theoretical framework which was around these are the steps that you can take to audit an algorithm but at

0:10:24

the back of my mind something that i wanted to achieve with it was trying to allocate responsibility clearly across vendors hospitals and regulators because that's conversation sometimes got very murky and And you get the situation where everyone's just pointing at the other guy being like, is there responsibility? Is there responsibility? And then no one's doing monitoring properly.

0:10:46
Arjun Manrai

I think that's a great point. So we want to actually just dig into those papers. And I think you mentioned both of them. One of them was a Lancet Digital Health 2019 paper that, if I'm getting this right, you were working on during graduate school and published. And this is titled, A Comparison of Deep Learning Performance Against Healthcare Professionals in Detecting Diseases for Medical Imaging, a Systematic Review and Meta-Analysis. So I think the title is very informative about what the paper is going to be about.

0:11:10

But maybe you can tell us about the sort of backstory for that. And I think you started to a little bit. But how did you select that topic? How did you select that project? And what might we not know about how that got kicked off?

0:11:22
Xiao Liu

Yeah, right. So I think maybe to paint a picture of how the academic or publication landscape was at the time, I think every week we were seeing new papers being published in top tier journals. with claims that were like, AI is just as good as radiologists or AI beats radiologists at pneumonia detection. And there were so many of them. And we were speaking to editors and editors were also feeling like we don't have a big enough pool of experience.

0:11:56

And they were conflicting,

0:11:57
Arjun Manrai

right? Some of them were conflicting with each other too. So there's a lot of confusion at the

0:12:00
Xiao Liu

time. Yeah, there was a ton of confusion and a lot of hype, a lot of hype, a lot of confusion. And you were hearing different things. So the editors were saying, you know, we don't have a big enough pool of peer reviewers who know this area well enough to help us tease out the good from the bad. There were a ton of methodologists and statisticians being like, we have established methodology for this. And these papers are just ignoring it.

0:12:24

And they're inventing new terminology of concepts that have been there for many, many years.

0:12:30
Andrew Beam

This just takes me back to Twitter during that time. The number of ML versus stats debates slash beefs, there was one of those every day. It was like Frank Harrow, Gary Collins, some new ML paper would come out and they'd be like, guys, there's a field called statistics that has thought about this for over 100 years. Maybe you should look into it. That really takes

0:12:50
Xiao Liu

me back. Yeah. Yeah. And I think there were a few fairly iconic tables or slides at the time with the terms that basically mean the same thing, like F1 and then sensitivity and specificity.

0:13:02
Arjun Manrai

Like the dictionaries translating between the fields, right?

0:13:05
Xiao Liu

Yes, exactly. Exactly. It became a bit of a meme. So I guess the TLDR of it was just tons of material being published. Nobody knew what any of it meant or what we should do with it in clinical practice. And I guess also in those years, we weren't seeing these seemingly incredibly impressive tools being implemented in a way that it brought any benefit to patients. And we didn't know which ones were good and which ones were bad.

0:13:31

So the motivation behind that paper was to try and apply the same level of rigor that we have been for many years to diagnostic evaluation studies, to prediction model studies, and assess the level to which they were reporting the methodology and the results thoroughly. and then really only pull out the ones that were following robust methodologies. So external validations, like for like comparisons, meaningful comparator, like comparing to human clinicians. And we reduced the number really significantly.

0:14:10

Less than 5 % of the papers actually really met that very standard bar in other areas of medicine. And we found that the performance of those models within that cohort was at best equivalent to radiologists. So it wasn't a bad news headline. It was just that there's a ton of noise in the published literature. So the natural next step was thinking, okay, well, what can we do about this? And that's how I ended up working on

0:14:36
Arjun Manrai

reporting guidelines next. Andy, maybe that's a good transition for what you want to dig into.

0:14:40
Andrew Beam

too. Yeah, Xiao, you and I were talking about this before the episode started, but I have this vivid memory of the first time that we met. I think actually through some of these Twitter debates, got pulled into some of the reporting guidelines work. I think I was like representative of the bad side or the dark side of ML, but I also have a degree in statistics. And so I was also sympathetic to... So you're the bridge.

0:15:04

You're the bridge. I was the bridge. And so I feel like I had been balanced enough in some of these crazy Twitter debates that some of the folks were like, Okay, let's bring Andy in here to make sure that some of the ML side is represented. So there was this meeting in Birmingham, UK that I believe you hosted. I at least remember you leading it. And the reason, again, like I remember it was in January 2020.

0:15:23

It was the last trip I took before COVID and didn't leave my basement for the two years after that meeting. I mean, I always look back on that trip fondly because it was one of my last trips before COVID. I also just remember it being an incredibly vibrant intellectual environment. A lot of the things that we were debating about how do you, what's the right way to report AI models for when they're being used as interventions for clinical trials?

0:15:45

What's the same? What's different? And I also realized that debates in real life are significantly better than debates on Twitter. Twitter is great for starting debates, but not actually having them. And a lot of the people who I'd seen involved in this community, once we got everyone in the same room, we all actually didn't disagree that much. So anyway, I just wanted to call that out. It was, I think, an incredible meeting. And I just remember you being one of the leading lights there.

0:16:10

So how about you walk us through what are reporting guidelines and what are spirit in consort and why are they important for AI?

0:16:17
Xiao Liu

Yeah. And we were so glad to have you there, Andy. So thank you for coming out. I'm so glad Birmingham was the trip of the beginning of 2020 for you. Yeah. So reporting guidelines are a fairly well-established practice in medical scientific publishing. There are a checklist of items that should be reported or included in your paper. So it could be something as simple as state the intervention that you're testing in the abstract so that, you know, when it's on a bibliographic database, it's easily searchable.

0:16:51

Or it could be something fairly methodological, like in randomized control trials, talk about how the mechanism for randomization and describe that to a certain level of detail. what reporting guidelines don't do is tell you how you should do your study so they're not methodological guidelines they're transparency guidelines so tell us what you did even if it was good methodology bad method just make it really transparent and clear and what it enables is it supplies enough information that brings the authors the editors of a journal and the peer reviewers on a level playing field so there's no kind of smoke and mirrors and you can't hide behind you know vague detail and it just enables better critical appraisal overall for everyone

0:17:35

so that's what a reporting guideline is there are a number of these for different study designs that have different reporting requirements so probably one of the best known ones is consort for randomized trials there's also for example prisma for systematic reviews starred for diagnostic test evaluation and tripod for prediction prognostic models and as new study design variants have emerged over the years or new interventions have emerged academic groups have developed extensions to these guidelines to address those specific niches and so consort ai was the ai extension of the consort guidelines and we also developed spirit ai at the same time which is for clinical trial protocols and they were essentially additions to the core consort reporting guideline, which just addressed

0:18:31

a few AI specific things like the way that data was handled, describing input and outputs, things like that. You also asked about the process for how these guidelines are developed. So it was a Delphi study, which is essentially a large vote. We propose a set of guidelines that are developed through review of the literature and through expert opinion. And then we put it out to a multi-stakeholder vote on which ones are important, which ones are not.

0:19:00

And we do that in multiple stages. And it ends with an in-person discussion, which as you mentioned, has a lot of debate. And we often learn from each other a lot during that process as well. We end up excluding some items where we can't get consensus and we include the ones that we do.

0:19:19
Andrew Beam

Yeah, and I think it's fair to say that the guidelines have been extremely influential. I was just trying to do a quick calculation on the side here. I think combined, they've been cited over 4 ,000 times since they were published in 2024. So clearly they've seen a lot of uptake. And I think for me, it was a big learning experience because when I was going through the Delphi process and coming to the meeting, I did think more of it being about like recommendations for trial design.

0:19:44

But like, as you like very nicely articulated, it's really just making sure that people know what you did so that the evidence can be evaluated critically and transparently. I think I also have this vivid memory of it just being, what's the nice way to say this, a dumpster fire in terms of the evidence for medical AI. And I just think it was something that was really needed. I also think going back to how lively the debates were, one of my most cited papers came out of that meeting with Lauren on explainable AI.

0:20:14

Because I remember one of the Delphi questions was like, should there be an explainability requirement? or actually how would you report any explainability assessment you did of the AI during the clinical trial? And I remember she and I got up on a little mini soapbox and said, no, like, well, you shouldn't do that at all. And here's why. And I think the editor, I think Rupa from Lancet Digital Health was there and asked us to like write that up.

0:20:36

And I'm sure that that also spawned lots of other projects like that too. So I love the idea of bringing together so many different people and the Delphi process, again, helped orient everyone coming into the meeting. I mean, I found that I really enjoyed it, not just because it was my last trip before COVID.

0:20:52
Xiao Liu

Yeah, no, that's great to hear. And just to add one thing, although the guidelines are not supposed to specify good methodology, often by shining the light on the fact that, hey, you're going to have to report this, that often serves as a soft encouragement for authors to go like, oh, oh, this is, I'm going to be pretty exposed here. And I think it does have this kind of secondary effect where I'm sure if I don't know how it would be measured, but methodological rigor does go up just as a result of needing to be more transparent.

0:21:26
Andrew Beam

Yeah. Sunlight's the best disinfectant or whatever that saying is.

0:21:30
Arjun Manrai

I was going to make like a Heisenberg uncertainty principle. Like you can't measure it without perturbing it in a good way. So I also had the privilege of coming to one of these that you organized, I think the standing together one. What year was this? 2022? 2023? I think 2022. Me too. Yeah. And I was amazed, like this was just one of the best journal clubs that I'd ever been a part of where people are really kind of ripping into language and scrutinizing very, very, very precisely exactly what the implications are of individual words and intent and meaning and what we could be missing and, you know, not including.

0:22:12

And so this was a lot of fun. And And what I also, I think, came to appreciate, like I knew this was probably a fair amount of work for the organizing team, but I had no idea how much work this actually took to just get off the ground, right? To even sort of formulate the pre-reading material or the things for folks to react to, let alone then shepherding the process and getting the final reporting guidelines put together.

0:22:35

And so can you maybe give us a sense? Like I think we, Andy and I probably got to appreciate this by coming to these meetings, but getting a consort AI launched, right, or a spirit AI launched and then published is a tremendous amount of work, right?

0:22:48
Xiao Liu

Yes. And actually, so the academic group that I belong to in Birmingham, we do a lot of these consensus projects. And so over the years, we've built up experience and knowledge of what it takes. when we did consort and spirit ai it was pretty much just me as a one man band one woman band back in 2019 but by the time we did standing together we had a really amazing team so we had joe alderman and eleanor laws and joe palmer and there were four of us kind of working on this together for about a year i think to do these consensus projects properly you really do need at least six months.

0:23:32

I've done expedited ones before where we try to do it in three months. And the reason for that is there's a ton of people giving their free time to this. When you are trying to get consensus amongst people working in the field, we're asking people to review a bunch of items, give considered comments back. And obviously the quality varies depending on how much time people have. And you have to do multiple rounds of this. So to get that engagement, you do need time and you also need reach and you also need the correct ways of communication and so

0:24:05
Andrew Beam

on. I wanted to ask like maybe a slightly different cultural question because it struck me that most of the guidelines are written and initiated and organized by UK-based researchers and like is there something that like Americans are lax in this or is there any historical reason why that's true. One reason might be the legacy of Douglas Altman. He was one of the world's most prolific medical statisticians and was involved in a lot of the guidelines.

0:24:33

And again, in the sort of like ML's lost history, there was a tweet from the University of Montreal about Yoshua Bengio being the first researcher to cross a million citations. But Doug Altman had been there six months to a year before that. So again, like, I think that that history, if it doesn't repeat, tends to rhyme with itself. Do you have any thoughts on that? Or is there not really anything there?

0:24:56
Xiao Liu

Yeah, it's interesting. I mean, I don't know. I do think some groups build up the expertise and how to, I mean, the process is quite similar when you, the methodology, the Delphi methodology is similar. So once you've built up that skill, you can apply it to different areas. And maybe there's a bit of compounding type effect there if you're a group that has done it once. I would say the The consort guidelines is led by David Moer, who is Canada based.

0:25:27

And for us, we certainly in terms of when we try to put together the stakeholder group, the consensus group, we really tried to be as international as possible, not least because some of the leading work was happening in North America. And we brought Lauren over from Australia as well. it's always tricky trying to trying to get because you are somewhat dependent on your network.

0:25:54
Arjun Manrai

Yeah, totally. So Xiao, beyond, you know, all your amazing academic work that we've been talking about, you've had, I feel like a few different careers that I think you're managing to juggle simultaneously now. And so including, I think you're still at the University of Birmingham, you still have an affiliation there. But you're also you spent some time at Apple, and then now you're at Microsoft. And so maybe, and we'll definitely have you and your colleagues back on, but maybe you can just tell us a little bit about some of the work that you've been doing at those companies.

0:26:26

And I think it broadly, as I understand it, relates to medical AI evaluation and deployment, but maybe just tell us a little bit about that and what you're hoping to achieve through those roles that might be harder to achieve through just an academic role.

0:26:40
Xiao Liu

Yes, definitely. About three, four years ago, when I was still juggling academic AI work and clinical ophthalmology, I decided I needed to go full in. And I decided that I needed to do this at a company with a product. The decision was partly driven because I was doing more and more advisory work with regulators, and I was starting to feel disingenuous, having never been on the other side. So that was when I joined Apple. And that was a fantastic place to get that experience because I joined Apple as a health scientist.

0:27:20

My main role was designing studies to validate algorithms on wearables. So I was mainly working on sensing algorithms for the Apple Watch. One of which I could speak about now because it was released earlier this year, the hypertension notification. It was great because the portfolio included things that were very early stage and novel, as well as things that were quite late in the pipeline and nearly getting regulatory approval. The studies were very different, diverse, and for someone with a methodological background was really fun for me.

0:27:51

But also Apple had this amazing machinery that was a well-oiled machinery for shipping products. And that's the experience that I really wanted to get. I was there for just over a year, then I decided that, no, I was going to start a lab at the University of Birmingham. So I went to full-time academia. That was one of my alternative careers. But shortly after I learned about the team at Microsoft AI that was just being set up working on Frontier LLM capabilities for medicine, and I thought, okay, this is very cool.

0:28:25

It was an opportunity that I couldn't really turn down. Here, the focus is very much on generative AI and language models. And even in the year and a half that I've been here, my role, which is primarily research focused, has covered a few different things. One of the main projects I was working on was this sequential diagnosis paper, which was published earlier this year. And I would say the research is broadly with a goal to develop advanced medical intelligence, whatever that means.

0:29:02

And there are many of us trying to define that and trying to define things like medical superintelligence, but really try and push the frontier in medical reasoning as we can today.

0:29:15
Andrew Beam

Awesome. Raj, should we switch to the lightning round if Xiao's ready for it? Are you ready for the lightning round? Yeah. So we like to take a little pause in the middle of the show, do a lightning round, mix things up, and then we'll go into some big picture questions at the end. How does that sound? Sounds good. Cool. So we'll throw some questions at you, keep your responses short, but we may prompt you for additional context depending on your answer.

0:29:48

Are you ready?

0:29:48
Xiao Liu

Okay, I'm ready.

0:29:49
Andrew Beam

Cool. I love this question. It's the one that we have increasingly been starting the lightning round with because we get such a variety of answers. But what was your first job?

0:29:59
Xiao Liu

My first job. This is going to be so boring because my first job was being a doctor.

0:30:07
Andrew Beam

Well, I think that that's still interesting, though, because clearly you've been driven to be a clinician from a very early time.

0:30:14
Xiao Liu

But actually, no, I can give a bit more detail. So my first job was in acute psychiatry. You probably didn't want a long answer for this. So you feel free to cut this out afterwards.

0:30:23
Andrew Beam

Please, please. Yeah.

0:30:25
Xiao Liu

But I was in this rapid response team in the acute psychiatric unit, which means you're basically the small SWAT team in the hospital going to do rapid assessments of people in acute psychiatric distress in the emergency departmental acute medical ward. It was very cool. So you saw a lot of interesting pathology, including catatonia, which is pretty rare as a part of schizophrenia.

0:30:52
Arjun Manrai

All right, Xiao, here's our second question. I think we've touched upon this actually already in a few of the conversations. Should medical AI be explainable?

0:31:03
Xiao Liu

Oh, that is a good lightning question that is not answerable in lightning time.

0:31:09
Arjun Manrai

Maybe it's actually a bad lightning question then.

0:31:12
Xiao Liu

Should it be? No. I think I'm on the no side of that debate. I don't think it has to be. I think we should ask why we want AI or anything to be explainable. And I think if we drill down far enough, ultimately, we just want stuff to have beneficial effect.

0:31:33
Arjun Manrai

Yeah. Xiao, I think you helped spawn that essay that Andy and Co wrote. So I think her answer is consistent, right? with uh with a totally consistent yeah yeah yeah it's full circle there we go i think i wish i

0:31:49
Xiao Liu

wrote every paper zied obermeyer wrote okay cool yeah i think i think science 2019 the knee pain yeah yeah the the recent um the ecg one oh yeah the mi prediction one just every single one for me has just it's like a album of greatest hits i love it he does write like a like very high

0:32:12
Andrew Beam

taste papers i will say like they're usually like very high taste very thoughtful and very clever

0:32:17
Arjun Manrai

i think that's his general philosophy yeah it's a good philosophy yeah and when you meet him in

0:32:22
Andrew Beam

person you're like oh of course it's consistent yeah talking to you i get the experience of

0:32:28
Arjun Manrai

reading your papers totally consistent it's sort of a rare and cool experience that way like you're You're exactly who I would imagine writing these awesome papers.

0:32:36
Andrew Beam

Yes. Yeah. Cool. I'm curious your thoughts on this next one. It's one that we ask a lot of folks. Will medical AI be driven more by computer scientists or clinicians?

0:32:48
Xiao Liu

What do we mean by driven?

0:32:52
Andrew Beam

Changed? Maybe I'll just say changed.

0:32:55
Xiao Liu

Oh, okay. I see. Oof. Okay. I think the short-term computer scientists, engineers, in the long-term, clinicians. So I think what happens is the computer scientists and the people who are handling the data, choosing the metrics, choosing the benchmarks, get to decide in the short-term which direction progress happens. but if we're talking about if we're talking about the end point being being used in actual patients i think clinicians will have much more of a gatekeeping function more distal in that process

0:33:38
Arjun Manrai

got it makes sense all right xiao here's our last lightning round question will a doctor's job look dramatically different five years from now because of ai

0:33:50
Xiao Liu

five years okay five years i don't think it will look dramatically different but 10 years and beyond probably yes and i hope so should i expand i was i was actually gonna say in what way will it look different 10 years from now yeah so i think there's how i think it might go and then there's how I would like it to go. Maybe it's the same thing. If medical intelligence and if models continue to advance in the way that they are, I think the medical profession doesn't need to be about accumulating knowledge or even medical reasoning the way that we're taught in med school.

0:34:35

I think the emphasis will be much more on being companions of experience. So the experience of deteriorating health and having unexpected health events happen to you, the experience of having fluctuations in your well-being just because of everyday life, the experience of having a life-changing diagnosis or having treatment failure and complications and hopefully recovery, but maybe not. And I think there's something deeply interpersonal that we will continue to want as part of that journey, which someone that can leverage the knowledge, but not necessarily embody it, but translate it into human terms.

0:35:20

And that's how I hope the profession will become.

0:35:24
Andrew Beam

I think that's an incredible, well, congrats on the lightning round. But I think that's a great segue into the final series of questions that we'd like to ask from a big picture perspective. and Raj, if you don't mind, I might take the first one here because I'd like to follow up on that. So in that picture though, do you need a medical degree to essentially be the healthcare companion or the someone who's on that journey with the patient?

0:35:46

If so, like how does medical education evolve over the next decade and what is the doctor's like core set of competencies that

0:35:54
Xiao Liu

they would need to have going forward? Yeah, I don't know what the training will look like but what I imagine is it will look quite different to how we're trained right now like I don't imagine we have in medical school we have preclinical years and then clinical years or at least in the UK it's like that so at first you're just accumulating theoretical knowledge and then in the UK in the third year you get dropped onto a ward and you have to try and apply this to real life so what I imagine is that that preclinical phase where you're just accumulating raw knowledge maybe will become less important because knowledge will be at our fingertips in other ways.

0:36:35

But the practice of applying that to people is something that we'll have to spend most of the time training for. And that does require training as well. Some of us are naturally better at it. Some of us are not, but it is a trainable skill too. And I think that will become the bulk of the training. You will need some knowledge. I think we need to be fluid in the way we apply that knowledge. So that some of that needs to be easily recalled in real time, but maybe it doesn't

0:37:03
Arjun Manrai

need to be as deep. So I think you're saying something that's so interesting, right? And I think what's striking to me is that I think a lot of people will agree with that. But then if you ask them to sort of drill down the specifics, right, you'll get maybe very different answers is my guess as to what is sort of required as that base of knowledge. Because I think if you're saying that the knowledge can be less deep, but we have a way of sort of using it and accessing it, then sort of the skill changes to being able to find what you need to find and then apply it to your patient.

0:37:34

And I think at the end of the day, that's what is the most important thing for a physician anyway, right? That is what they do now, I think that most doctors would say. But the question I think for medical education, which I think is fascinating, we're not going to resolve it right now is what is the sort of bare requirements or what will the requirements look like for an internal schema within their mind to be able to even know what to look for or to ask what they need to ask so that they can find information that's available to them digitally that they can then apply to their patient?

0:38:05

I don't know if the Krebs cycle is going anywhere, right? But what is that sort of scaffold? Because I think if I ask some of my medical educator friends and colleagues and other professors, I think they might argue that we're doing that, right? And And so is it like, are we going to sort of just continue to pare that down into sort of that scaffold that is essential to then be able to search? Or are we going to more fundamentally reshape what medical education is?

0:38:32

Like, for example, I think a lot of medical schools are considering how AI is being used in clinical practice, of course, by young and old physicians alike, but also how their students are using AI to learn and to study and to prep and things like that. And so there's a lot of committees on integrating AI into medical education, which are meeting and discussing, right? But like, this sort of whether AI gets integrated into the curriculum in a much more formal, systematic way, how to use AI as a doctor, right, when treating patients, like, do you think it'll reshape more like that?

0:39:06

Or will it just sort of be a kind of pared down version of medical school as it is right now?

0:39:12
Xiao Liu

yeah i mean i'm not i'm not sure i'm not sure exactly how it will play out i mean i'm i think the optimistic side of me feels like that task of knowing what to ask and you know that scaffolding of knowing what information to elicit doesn't necessarily need to be driven by human reasoning but we need to find a way that it can easily be delivered in real time and in life in a way that's

0:39:42
Arjun Manrai

yeah conversational and natural you see ai interacting more with patients directly right in our in our near-term future yeah i think so yeah yeah and doing things like eliciting histories and you know taking doing physicals and uh asking questions and other information gathering that that a physician is currently doing. Right. Yeah. I don't see what I don't see why not. Yeah, I agree. Yeah. Okay. So we have one last sort of big picture question for you.

0:40:10

And this is sort of a, it's a kind of a two part question. And I think, again, I think you've touched on this, but maybe just thinking to narrow this down and make this a little more precise, like the next five years, what are you most pessimistic about? And what are you most optimistic about and you can choose which order to answer those in so totally up to you that way

0:40:32
Xiao Liu

yeah i think i'm i'm more optimistic than i'm pessimistic so maybe i'll answer it that way around um i'm i think i'm very optimistic about um i'm really optimistic about the things that we might discover in the coming years so i think the things that give me the most excitement is the idea of aggregating all of the knowledge that we have in the medical world and understanding better what biomarkers can tell us, understanding better clustering of phenotypes of disease, understanding if we have misunderstandings in our current medical knowledge and how we conceptualize different disease models, different treatment responses, and so on.

0:41:19

and I always you know if we think about the way diagnoses like diabetes has segregated into different subtypes over the years in my area in ophthalmology inflammatory eye disease which I did my PhD on we have tons of diseases that are like loosely characterized as quote white dot syndromes just because they happen to produce white dots at the back of the eye that probably are driven by different mechanisms. I'm excited about understanding those types of things better.

0:41:53

And I hope that one of the ways that we might see the emergence of new intelligence is actually the deconstruction of some of the concepts that we have in medicine today.

0:42:04
Arjun Manrai

It's such a good answer that I almost don't want to hear the pessimistic one.

0:42:09
Xiao Liu

Well, I mean, yeah, that was the basic answer. The pessimistic one is actually that we're resistant to that, because it's unfamiliar. So I worry about the lost opportunity if we are too risk averse in a time where new discoveries might be emerging. And we rigidly stick to established guidelines or methods of evaluation that take many, many years and are just slow. And that's not to say I want us to be gung-ho and take irresponsible risk and be fully experimental.

0:42:46

but I and I don't even have a good answer to how we calibrate this but the part that I'm pessimistic about is that I think as a field we tend to be cautious by nature for good reason but at this point in time the way that we do that shouldn't look the same as it has over the last decade or two for example where there's not been so much technological change.

0:43:12
Arjun Manrai

All right. Xiao, that was an amazing conversation. Thanks so much for coming on AI Grand Rounds. Thanks for coming on.

0:43:18
Xiao Liu

Thank you so much for having me.

0:43:23
Andrew Beam

This copyrighted podcast from the Massachusetts Medical Society may not be reproduced, distributed, or used for commercial purposes without prior written permission of the Massachusetts Medical Society. For information on reusing NEJM Group podcasts, please visit the permissions and licensing page at the NAGM website.