In brief
Peter Lee, head of Microsoft Research, argues that GPT-4, the model behind the newest version of ChatGPT, is already capable enough to be used by doctors, nurses and patients every day. Because it will be in everyone's pocket whether or not the medical world is ready, he says clinicians, regulators and the public must urgently debate guidelines and work out how to validate it for clinical use.
The tension runs between astonishment and unease. Lee describes a model that passes medical exams, drafts paperwork, checks its own work and even offers emotional support, yet makes confident mistakes and is not fully understood by its creators. He is uncomfortable that doctors value its human-feeling help, and he admits he does not know how education or empathy will change. His remedy is open, informed debate.
Questions
Can GPT-4 really pass the medical licensing exam?
Peter Lee says GPT-4 got all the US medical licensing exam questions right, while the earlier ChatGPT model (GPT-3.5) reportedly scored about 61%. He adds that it can explain its reasoning, and where it was marked wrong, doctors sometimes disagreed about the correct answer.
Is ChatGPT just a fancy autocomplete?
Andy Beam explains that GPT stands for generative pre-trained transformer and that the model is trained to predict the next word. Lee says that description fits in practice but that something deeper is going on, such as the model solving problems beyond what its training examples covered.
Why is Microsoft so involved in OpenAI?
Lee says it began with Microsoft providing the very large computing infrastructure OpenAI needed, then grew into a formal business partnership with shared research. Microsoft depends on OpenAI for foundation language models, and OpenAI depends on Microsoft's computing power.
What are the risks of using GPT-4 in healthcare?
Lee warns that GPT-4 makes mistakes and hallucinates, meaning it states wrong things in an incredibly convincing way. He says guidelines, regulation and ways to validate it for clinical use are still unsettled, though patients and clinicians are already using it.
Will AI take over doctors' paperwork?
Lee says there is a reasonable chance that in five years the main thing needed will be a recording of the doctor-patient encounter. He says the model can already draft summaries and fill out forms such as a Medicare prior authorization.
Facts
Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.
Peter Lee says his parents immigrated from Korea, earned PhDs, and became a physics professor and a chemistry professor.
“My parents immigrated from Korea and got their PhDs, and my dad became a physics professor, and my mom became a chemistry professor.”
0:04:21 · Peter Lee · Peter Lee background
Peter Lee became a professor at Carnegie Mellon University, working on theoretical computer science.
“I ended up getting my professorship at Carnegie Mellon University doing that kind of theoretical computer science work.”
0:05:04 · Peter Lee · Academic career
After Barack Obama's 2008 election, Peter Lee was pulled into government service at DARPA.
“And then after Barack Obama was elected in 2008, I was Shanghai to serve in the government at DARPA.”
0:05:04 · Peter Lee · DARPA service · 2008
DARPA staged the Network Challenge on the 40th anniversary of the ARPANET, placing 10 red weather balloons across the continental United States.
“we staged a national contest on the 40th anniversary of the ARPANET. We took 10 big red weather balloons across the continental U.S.”
0:07:54 · Peter Lee · DARPA Network Challenge · 2010
The DARPA Network Challenge offered a $40,000 prize to whoever found the precise locations of the 10 balloons.
“said, whoever can find the precise locations of these 10 balloons would win a $40 ,000 prize.”
0:08:28 · Peter Lee · DARPA Network Challenge · 2010
A team from the MIT Media Lab led by Sandy Pentland won the DARPA Network Challenge by mobilizing social media.
“a team from the MIT Media Lab under Sandy Pentland that came up with a beautiful way to mobilize social media”
0:08:28 · Peter Lee · DARPA Network Challenge · 2010
Peter Lee says the Network Challenge became the foundation of a program called Nexus 7, which was deployed in Afghanistan.
“That project, as silly as it was, ended up forming the foundation of a program called Nexus 7 that ended up getting deployed in Afghanistan during that war”
0:09:10 · Peter Lee · Nexus 7
Peter Lee says a Microsoft research project becomes an official incubation when its expense grows large enough to bother the chief financial officer, after which it must be shut down or turned into a real business.
“at some point it gets large enough that the company, and in particular our chief financial officer, starts getting annoyed at the expense. And at that point, we officially designate the effort as an incubation.”
0:12:04 · Peter Lee · Incubations
In 2016, Microsoft chief technology officer Harry Shum told Peter Lee he would be reassigned to work on healthcare.
“in 2016, I received a note from my boss, who at that time was Harry Shum, our chief technology officer at the time, saying that I was going to be reassigned to work in healthcare.”
0:13:18 · Peter Lee · Microsoft healthcare · 2016
Peter Lee says he agreed to lead Microsoft's healthcare work after being cornered by CEO Satya Nadella at a political fundraiser in 2016.
“Satya Nadella was there. And so he cornered me, and I just was not, I didn't have enough spine to stand up to him.”
0:14:32 · Peter Lee · Microsoft healthcare · 2016
Peter Lee says Microsoft partnered with Google, Amazon, IBM, Salesforce and others to advance healthcare technology.
“we actually even partnered with Google, with Amazon, with IBM, with Salesforce and others, in order to try to move the technology forward”
0:17:52 · Peter Lee · Healthcare strategy
Microsoft Research recently established an AI for Science laboratory under Chris Bishop.
“we've just recently established a new laboratory under the leadership of Chris Bishop called AI for Science, Microsoft Research AI for Science.”
0:27:32 · Peter Lee · AI for Science
Microsoft Research's AI for Science lab aims to develop machine learning foundations that could accelerate molecular dynamics simulation by about ten million times.
“to develop the machine learning foundations to, say, achieve 10 million fold acceleration in molecular dynamics simulation.”
0:27:32 · Peter Lee · AI for Science
Peter Lee says the OpenAI and Microsoft relationship began with Microsoft providing computing infrastructure for OpenAI.
“started several years ago a relationship that I think at the beginning was really focused just on Microsoft providing compute infrastructure for OpenAI”
0:29:36 · Peter Lee · OpenAI partnership
Peter Lee says Microsoft offers OpenAI's GPT models through its cloud with assurance that they meet compliance requirements such as HIPAA for medical use.
“we can assure our corporate customers that it meets compliance requirements like HIPAA for use in medical settings.”
0:32:07 · Peter Lee · HIPAA compliance
Peter Lee says ChatGPT using the GPT-3.5 model was reported to score 61% on the US Medical Licensing Exam.
“People have tested the CHAT-GPT with the 3.5 model on the U.S. Medical Licensing Exam and found, I think the report was they achieved a 61 % score”
0:38:46 · Peter Lee · Medical exam performance
Microsoft Research's Sebastian Bubeck and colleagues wrote a paper training a neural network to solve systems of linear equations, which Peter Lee uses to illustrate emergent abilities.
“A colleague of mine at Microsoft Research, Sebastian Bubeck, and his colleagues of a paper where they train a neural net to solve systems of linear equations.”
0:42:39 · Peter Lee · Emergent capability
Peter Lee says GPT-4 was trained only on text yet can generate drawing code for a picture of a unicorn.
“GPT-4 has been trained only on text. And so you can have a conversation where you ask it, do you know what a unicorn is?”
0:46:24 · Peter Lee · GPT-4 generalization
Peter Lee says Microsoft's medical experiments use one GPT-4 to propose a differential diagnosis and a second GPT-4 to check it for errors.
“And so a trick that we've been using now in medical scenarios is to have one GPT-4, let's say proposed differential diagnosis, and then have a second GPT-4 read over the first GPT-4's differential diagnosis and check it for errors”
0:49:29 · Peter Lee · AI verification
Peter Lee describes a doctor who used GPT-4 to prepare for a hard conversation with a pancreatic cancer patient, after which GPT-4 asked how the doctor was holding up.
“after he said thank you to GPT-4, GPT-4 said, and what about you? How are you holding up? Are you getting all the help that you need?”
0:53:23 · Peter Lee · Clinical empathy
Peter Lee served on the founding board of directors of the Kaiser Permanente School of Medicine.
“One of the privileges I've had is being on the founding board of directors for the Kaiser Permanente School of Medicine.”
1:06:21 · Peter Lee · Medical education
Peter Lee says Zak Kohane leads a new department at Harvard Medical School devoted to teaching data and machine learning in medicine.
“Zach Kohani, who created it and is the head of a whole new department at Harvard Medical School, which is devoted to that”
1:07:35 · Peter Lee · Medical education
Peter Lee says the scale hypothesis, that bigger models trained on more data keep getting more capable, has so far held up with no sign of slowing.
“So far, the scale hypothesis has been holding up. And so far, it shows no signs of letting up.”
1:02:28 · Peter Lee · Scale hypothesis