Person
Xiao Liu
Xiao Liu is a doctor and researcher who works on how to test and report on medical artificial intelligence, and this page collects what she said on the NEJM AI Grand Rounds podcast. The episode covered her path from psychiatry and ophthalmology to roles at Apple and Microsoft AI.
Liu said fewer than 5% of deep-learning imaging papers in her 2019 review met the robust methodological standard of other areas of medicine. She also described leading the CONSORT-AI and SPIRIT-AI reporting guidelines and working on an audit framework for algorithms after deployment. Each fact below links to the moment it was said.
1 episodes · 20 quoted claims
Xiao Liu began a PhD before ophthalmology training because she did not secure a place in ophthalmology on her first attempt.
“my first foray into I wouldn't even say AI necessarily but just research was when I decided to do a PhD before my ophthalmology training and that was actually because I didn't get in the first time.”
NEJM AI Grand Rounds · 0:03:26
Pearse Keane, an ophthalmologist at Moorfields Eye Hospital, had started a collaboration with DeepMind to develop a retinal diagnosis model.
“one of my co-supervisors, someone called Pierce Keane, who is an ophthalmologist at Moorfields Eye Hospital in London, and he had just initiated a collaboration with DeepMind to develop a retinal diagnosis model.”
NEJM AI Grand Rounds · 0:04:35
John Deeks, a biostatistics professor at the University of Birmingham, did key work on diagnostic test evaluation and shaped Liu's methodological foundations.
“The first one is a professor of biostatistics called John Deeks, who many listeners may know, who did a lot of some of the key work in diagnostic test evaluation.”
NEJM AI Grand Rounds · 0:05:32
Livia Faes, a Swiss ophthalmologist, was co-first author with Xiao Liu on a systematic review of deep learning.
“The second person is someone called Livia Fais, who is a Swiss ophthalmologist and also my co-first author on my first more well-known publications, which was a systematic review on deep learning.”
NEJM AI Grand Rounds · 0:06:01
Lauren Oakden-Rayner is an Australian radiologist who wrote an influential blog on AI that shaped Liu's thinking.
“Lauren Oakden Rayner, an Australian radiologist who used to publish this fantastic blog on AI.”
NEJM AI Grand Rounds · 0:06:39
Liu worked on a medical algorithmic audit framework setting out steps for auditing an algorithm after deployment.
“did a piece of work around the medical algorithmic audit which was this theoretical framework which was around these are the steps that you can take to audit an algorithm”
NEJM AI Grand Rounds · 0:09:43
Fewer than 5% of deep-learning imaging papers in Liu's review met the robust methodological standard of other areas of medicine.
“Less than 5 % of the papers actually really met that very standard bar in other areas of medicine.”
NEJM AI Grand Rounds · 0:14:10
Among the methodologically robust studies in Liu's review, AI models performed at best equivalently to radiologists.
“And we found that the performance of those models within that cohort was at best equivalent to radiologists.”
NEJM AI Grand Rounds · 0:14:10
CONSORT covers randomized trials, PRISMA systematic reviews, STARD diagnostic test evaluation and TRIPOD prediction or prognostic models.
“probably one of the best known ones is consort for randomized trials there's also for example prisma for systematic reviews starred for diagnostic test evaluation and tripod for prediction prognostic models”
NEJM AI Grand Rounds · 0:17:35
CONSORT-AI extends the CONSORT trial reporting guideline for AI interventions, and SPIRIT-AI was developed alongside it for clinical trial protocols.
“consort ai was the ai extension of the consort guidelines and we also developed spirit ai at the same time which is for clinical trial protocols”
NEJM AI Grand Rounds · 0:17:35
The AI reporting guidelines were developed through a Delphi study, a multi-stage vote on items drawn from literature review and expert opinion.
“It was a Delphi study, which is essentially a large vote. We propose a set of guidelines that are developed through review of the literature and through expert opinion.”
NEJM AI Grand Rounds · 0:18:31
The CONSORT guidelines are led by David Moher, who is based in Canada.
“the consort guidelines is led by David Moer, who is Canada based.”
NEJM AI Grand Rounds · 0:24:56
Liu largely ran the CONSORT-AI and SPIRIT-AI effort by herself in 2019.
“when we did consort and spirit ai it was pretty much just me as a one man band one woman band back in 2019”
NEJM AI Grand Rounds · 0:22:48
The STANDING Together recommendations were developed by a team of four including Joe Alderman, Elinor Laws and Joanne Palmer, working for about a year.
“by the time we did standing together we had a really amazing team so we had joe alderman and eleanor laws and joe palmer and there were four of us”
NEJM AI Grand Rounds · 0:22:48
At Apple, Liu worked as a health scientist designing studies to validate algorithms on wearables such as the Apple Watch.
“I joined Apple as a health scientist. My main role was designing studies to validate algorithms on wearables.”
NEJM AI Grand Rounds · 0:26:40
Apple released a hypertension notification feature for the Apple Watch, which Liu helped validate, in the year of the recording.
“One of which I could speak about now because it was released earlier this year, the hypertension notification.”
NEJM AI Grand Rounds · 0:27:20
At Microsoft AI, Liu worked on a sequential diagnosis paper published in the year of the recording.
“One of the main projects I was working on was this sequential diagnosis paper, which was published earlier this year.”
NEJM AI Grand Rounds · 0:28:25
Liu's first job as a doctor was in acute psychiatry, on a rapid response team assessing people in acute distress.
“my first job was in acute psychiatry. You probably didn't want a long answer for this.”
NEJM AI Grand Rounds · 0:30:14
Liu predicts computer scientists and engineers will drive medical AI in the short term, with clinicians as gatekeepers in the long term.
“I think the short-term computer scientists, engineers, in the long-term, clinicians.”
NEJM AI Grand Rounds · 0:32:55
Liu says white dot syndromes group eye diseases by appearance, and they may be driven by different mechanisms.
“we have tons of diseases that are like loosely characterized as quote white dot syndromes just because they happen to produce white dots at the back of the eye”
NEJM AI Grand Rounds · 0:41:19