# Upgrading Your Health: Navigating AI's Future In Healthcare with John Halamka of Mayo Clinic Platform

- **Show:** Gradient Dissent
- **Published:** 29 Feb 2024
- **Source:** https://wandb.ai/site/resources/podcast
- **Transcript:** https://podcasts.opennash.com/shows/gradient-dissent/upgrading-your-health-navigating-ais-future-in-healthcare-with-john-halamka-of-m-bbec8a/
- **Provider:** OpenNash Podcasts (machine-generated)

## In brief

John Halamka, president of Mayo Clinic Platform, argues that AI in medicine should be built like good engineering: modular, validated and trusted before it is used on patients. Predictive models in imaging and cardiology are already catching disease earlier than humans. He holds that generative AI is not yet safe for diagnosis or treatment, and that AI should augment clinicians rather than replace them.

The central tension is speed versus safety. Mayo has built a privacy-protected data platform that has produced 180 models, yet generative AI has no agreed way to certify consistent quality. Halamka's answer is curated data, guardrails, risk-based use cases and emerging regulation. Meanwhile, shrinking doctor supply and global access gaps create pressure to deploy AI widely and soon.

## Key insights

1. **Radiology leads AI adoption because humans disagree** Halamka says radiologists reading X-rays are not very consistent with each other, so algorithms add reliability. Tools that flag head bleeds the moment a scan is rendered can cut waits of an hour or two.
2. **Mayo shares data without giving it away** Mayo de-identified 150 years of records on 10 million patients and put them in a cloud container. Outside innovators build models there but cannot buy or copy the data, which Halamka credits for 180 models.
3. **Generative AI is used only for low-risk tasks** Mayo uses it for plain-English explanations, administrative simplification, documentation and chart summaries. It is not used in production for diagnosis or treatment because its quality varies from moment to moment.
4. **Training data bias can quietly skew medical AI** Halamka notes clinical-trial participants tend to be better educated, richer and urban, so models trained on them may be unfair. Mayo is partnering with Mercy and international systems to add geographic and demographic variety.
5. **Doctor shortages make AI close to unavoidable** Halamka says birth rates of 1.3 to 1.4 in countries he visited, plus elders living into their 80s and 90s, mean too few doctors will graduate. He sees AI as essential to care, while keeping humans for listening and judgment.

## Questions this episode answers

### What is the Mayo Clinic Platform and what does it do?

It is a Mayo Clinic initiative, started in 2019 under CEO Gianrico Ferruccia and led by Halamka from 2020, that lets companies and researchers build AI models on de-identified Mayo data. The data stays in a controlled cloud container and is never sold or taken out.

### Can AI really detect diseases before doctors can?

Halamka says Mayo has an algorithm that can spot pancreatic cancer on a CT scan about two years before a human can see it, often at stage zero instead of stage four. Mayo also built a model that reads a single-lead Apple Watch ECG to estimate how well the heart is pumping.

### Is it safe to use ChatGPT for medical questions?

Halamka says there are no objective quality measures yet, so risk matters. Asking a chatbot to summarize what trusted sources like Mayo, Stanford or PubMed say is reasonable, but relying on its general internet training is worse, since that data includes claims like ivermectin curing COVID.

### Will AI replace doctors?

Halamka says no for now: he expects AI to be 'augmented intelligence' that supports clinicians. Patients also want respect, listening and help with care planning from a human, and he expects AI to ease burnout, for example by cutting a radiation-planning task from 16 hours to 45 minutes.

### How does Mayo protect patient privacy when training AI?

It uses what Halamka calls 'data behind glass': the data is fully de-identified, kept in an audited cloud container, limited in what can go in and out, and blocked from being linked with outside datasets. He admits that with enough data and computing power re-identification could become possible, so policy controls are needed too.

## Facts

- **John Halamka** [0:01:00]: "he took a Singer sewing machine motor, a simple standards-based modular component at the time, and some mercury switches."
- **John Halamka** [0:01:39]: "And he called the company International Time Recording and decided that it was such a successful idea that he would expand into other devices. And he renamed the company International Business Machines."
- **John Halamka** [0:07:46]: "So in 1983, Stanford ran what I think may have been their first entrepreneurial conference where it was, you know, there are students that start companies."
- **John Halamka** [0:06:08]: "So Terman was the provost of Stanford University, and he had two students in the 1930s, one named Hewlett, the other named Packard."
- **John Halamka** [0:06:50]: "he founded the Stanford Research Park and is widely acknowledged as being one of those seminal figures that brought Silicon Valley to be."
- **John Halamka** [0:13:18]: "Mycine was the first AI system running in the Stanford University Medical Exchange or Sumax that would provide doctors with expert antibiotic prescribing advice."
- **John Halamka** [0:18:21]: "So in 2019, Gianrico Ferruccia, the CEO of Mayo Clinic, had a notion."
- **John Halamka** [0:19:09]: "I joined Mayo Clinic in January of 2020 as president of Mayo Clinic Platform."
- **John Halamka** [0:19:52]: "So we took the entire Mayo corpus, 150 years of data on 10 million patients, de-identified it, which in itself is tricky."
- **John Halamka** [0:20:50]: "About 80 % of the records, just the nature of the way records are recorded, is text."
- **John Halamka** [0:23:57]: "So we got 125 ,000 Apple watch submitted telemetry lead one ECGs from patients, voluntarily all concepted."
- **John Halamka** [0:23:57]: "And then we were able to, in a clinical trial, show that this algorithm has an area under the curve of 0.92, right?"
- **John Halamka** [0:24:38]: "and FDA has now given it full software as a medical device approval to be used as part of diagnosis for patients."
- **John Halamka** [0:16:14]: "So example, Mayo has an algorithm that can diagnose pancreatic cancer two years before a human can see it."
- **John Halamka** [0:14:59]: "radiology is by far the leading adopter of AI. And why? Well, inter-rater reliability of humans looking at x-rays isn't so great."
- **John Halamka** [0:42:25]: "They have taken their 15 million patients, de-identified them, created a cloud container for the purpose of AI training and validation."
- **John Halamka** [0:31:39]: "So we will, in 2024, using de-identified Mayo data, train a novel large language model ourselves with the notion that maybe if it's trained on just really highly curated medical data, it will have less hallucination."
- **Lukas Biewald** [0:36:56]: "there is this data set called like the truthfulness data set where it only gets 25 % accuracy"
- **John Halamka** [0:50:49]: "We got 355 applications. Of those, we downsized to 42. Ultimately, we chose eight general areas."
- **John Halamka** [0:53:52]: "It can take 16 hours for a physicist to develop a model for a complex head and neck tumor. We have developed a model that does it in 45 minutes."
- **John Halamka** [0:48:00]: "that in medicine, it takes about 20 years on average for an innovation to disseminate widely across the specialty."
- **John Halamka** [0:59:00]: "you restrict the inputs and the outputs, and you prohibit linkage with other external data sets."
- **John Halamka** [1:00:00]: "So HIPAA was written in 1996. It doesn't mention the genome."
- **John Halamka** [1:01:04]: "I'm the second human sequenced in the Personal Genome Project."
- **John Halamka** [0:43:34]: "My wife was diagnosed with stage 3A breast cancer in December of 2011, and she's totally fine today."

## Transcript

**[0:00:01] Lukas Biewald:** You're listening to Gradient Dissent, a show about making machine learning work in the real world, and I'm your host, Lucas Biewald. This is an interview with John Halamka, who is president of the Mayo Clinic. He's an expert on the intersection of machine learning and AI, which anyone who listens to this podcast knows is something I'm super interested in. So already a fantastic interview I'm looking forward to, but I discovered throughout the interview that he actually got his start back in 1981, creating a startup that made tax software for IBM mainframes and built it in the basement of a Silicon Valley legend before it was cool.

**[0:00:40] Lukas Biewald:** And he has a clock from the 1700s in his background. And he's seen the complete arc of AI and medical systems. So this is just a fascinating interview. I hope you enjoy it as much as I enjoyed doing it. So John, I've been told to ask, what is the deal with the clocks

**[0:01:00] John Halamka:** in your room here? So I have long believed as an engineer that anything we create should be modular, upgradable, using standard components, but assembling them in novel ways. And so these clocks remind me of that. The clock on the left in 1905, an engineer in Armonk, New York, had this concept of saying, let's take common things like clocks that we have to wind. Can we reduce human burden by automating them? And he took a Singer sewing machine motor, a simple standards-based modular component at the time, and some mercury switches.

**[0:01:39] John Halamka:** So yes, liquid mercury. And when the weights go halfway down, the clock triggers the motor and winds the weights. So it has been self-winding since 1905. And he called the company International Time Recording and decided that it was such a successful idea that he would expand into other devices. And he renamed the company International Business Machines. And so that's Thomas J. Watson's prototype for his startup company, IBL. The clock on the other side is exactly the opposite.

**[0:02:15] John Halamka:** created by a silversmith, not a single standard component, completely not upgradable. Anytime you need to deal with some kind of fine tuning, it requires tools of the 1700s. So that's Paul Revere's clock. Wow. Perfect example of modular, upgradable, stands the test of time, low maintenance, low cost, and exactly the opposite. it. I try to apply that in everything that I do in creating software and healthcare solutions.

**[0:02:50] Lukas Biewald:** And I do notice that the clock on the left, or I think it's on the right for me with this recording software, that clock seems to actually have kind of the, probably the right time, Eastern time. And is the one on the left actually even running? I can't.

**[0:03:04] John Halamka:** Yeah, they should both be at the right time. So, but you know, there is a little variation of a minute or two here, but... Got it. Awesome.

**[0:03:13] Lukas Biewald:** Well, how long ago did you get these clocks?

**[0:03:16] John Halamka:** So my wife used to run an antique store in Waltham, Massachusetts in the 90s. And so this international time recording company was found in an attic and brought to her in the mid-90s. The Paul Revere clock was actually in 11 different pieces in a garage in Acton, Massachusetts. And I had to rebuild it, truly, using tools from the 1700s.

**[0:03:43] Lukas Biewald:** Wow, and you were actually able to rebuild it? You did that.

**[0:03:46] John Halamka:** I did, yeah. And so if you think about it, in the 1700s, clocks and locks were actually the same tool set. And so I happened to find an old locksmith tool set, and that was sufficient to deal with the cantankerous clock.

**[0:04:05] Lukas Biewald:** And what kind of tools come up? I sort of imagine like an anvil and a hammer or something. What is that tool that you need?

**[0:04:12] John Halamka:** Well, exactly as you say, there are a lot of pins in these things. And so you need the ability to use some very fine hammered pin removal and pin insertion tools.

**[0:04:30] Lukas Biewald:** Interesting. Well, so I have another kind of, at this point, maybe historical question for you. You're obviously a very successful person in your field, but I saw in some of the research that I think you might have actually started a startup or worked at a startup back in the early 80s on the Stanford campus. I was really curious to hear how that happened and what that was like and how that experience might have shaped you.

**[0:04:58] John Halamka:** Sure. So when I arrived in Palo Alto in 1979, it was the Topps 20 operating system, and you had mainframes and you had minis. And at that time, not a single student had a desktop device. And so I built one. That was, of course, the Altair era, the Popular Electronics 1978 company, the 8080, the Z80, and that sort of thing. So I wire wrapped and then had to write the BIOS for this thing and was the first Stanford student to have a desktop.

**[0:05:36] John Halamka:** With that, I thought, I wonder what you could do with this. And I went to the Stanford Law Library and looked at the U.S. tax code and said, it is not that difficult to take the tax laws of the United States and codify them. Back then, this was sort of pre-PC, pre-Apple. So it was Kaypro, Osborne, CPM, that era. And I wrote a predecessor to TurboTax when I was an 18-year-old and then sold that out of my dorm room at Stanford.

**[0:06:08] John Halamka:** and the odd thing that happened my wife to be and i um were walking by the stanford student union truster and we saw this three by five index card wanted young couple to care for aging professor and we're on free room and board which sounded good so we interviewed and the professor was a guy named Frederick Emmons Terman. Wow. You might need to describe who Terman is for some of the others. So Terman was the provost of Stanford University, and he had two students in the 1930s, one named Hewlett, the other named Packard.

**[0:06:50] John Halamka:** And he suggested that maybe Stanford as an academic institution could take some of its unused land and start to incubate companies and academia and industry could be two sides of the same coin. And he founded the Stanford Research Park and is widely acknowledged as being one of those seminal figures that brought Silicon Valley to be. And so we got this position with him and I was his caregiver during the last three years of his life. So the end of that story of writing tax software was that I did it in Dr.

**[0:07:26] John Halamka:** Terman's basement. So here I am in the home of the founder of Silicon Valley, writing tech software on home-built computers in the early 80s.

**[0:07:36] Lukas Biewald:** Well, so at that time in the early 80s, was it already kind of a stereotype to be doing a software startup in a basement? Or was that a really unusual thing?

**[0:07:46] John Halamka:** Oh, that was a very unusual thing. So in 1983, Stanford ran what I think may have been their first entrepreneurial conference where it was, you know, there are students that start companies. Isn't that odd? And so I remember I was one speaker and the other speaker, I think, with me was a guy named Michael Dell. And, you know, I went off to medicine. He went off and did some computer stuff. So it was considered very atypical.

**[0:08:20] John Halamka:** And then when I went to medical school, it was unheard of for a doctor to do something entrepreneurial or a doctor to do engineering. So the early 80s, it was an innocent time.

**[0:08:35] Lukas Biewald:** And it seems like you really love engineering and computers. What inspired you to go to medical school?

**[0:08:43] John Halamka:** this is going to be a slightly peculiar story but i promise it will bear fruit all right so uh although i was born in des moines iowa uh i moved to southern california in the early 60s and in the early 60s the defense contractors raytheon hughes aircraft aerojet general were the kind of Silicon Valley of California, right? I mean, Silicon Valley wasn't named just 71, 72. And so as a young child, I was a free-range kid.

**[0:09:18] John Halamka:** So you wouldn't know what that is, but the idea was mom said, go ride your bike, see you in 12 hours, have fun. So I rode my bike to the dumpsters of all the defense contractors and AT &T. None of the dumpsters were locked. and I've pulled out manuals, discarded components, integrated circuits that had failed testing, and taught myself analog and digital logic and then early microprocessor technology. Then I decided as a high school student to apply for a fellowship at a hospital.

**[0:09:53] John Halamka:** And this was Harbor UCLA Medical Center in Southern California. And the project I was assigned was, if you could imagine very small signals in the brain, the visual and audio signals that travel over the nervous system could be captured if you signal averaged and removed all the ambient noise from all the other cells around them. How could you gather evoked potentials from humans, do signal averaging and fast Fourier transformation? And I said, well, I have 2K of RAM and a knowledge of assembly language.

**[0:10:34] John Halamka:** I bet I can create embedded components to do that. And so I did in the 1970s build this. And so I thought this notion of being able to digitize human potential and analyze and diagnose based on these digital signals is something that is a very natural evolution of computing. I'd like to devote my life to the intersection of medical care and engineering. And the funny end of that story is I did residency in emergency medicine at Harbor UCLA in the 90s.

**[0:11:11] John Halamka:** The device that I built in the 70s was still in production.

**[0:11:17] Lukas Biewald:** Well, so you built a device in the 70s as a teenager. that then was put into production?

**[0:11:23] John Halamka:** Well, it was a simpler time. So you have to think about, you know, in the 70s, building experimental devices that were not invasive and unlikely to cause harm and being used in this kind of research capacity didn't require a lot of FDA scrutiny. I mean, today, of course, everything is scrutinized. But yeah, so back in the 70s, it worked. It did the job, right? You came with running code and if it worked, people built on it.

**[0:11:53] John Halamka:** That's just the way the 70s, the 80s used to be.

**[0:11:56] Lukas Biewald:** Wow, that sounds like a different time, I guess. What is the sort of arc of the intersection of machine learning and medicine look like? I mean, it seems like you've seen the complete span of it. I sort of remember when I was at Stanford in the late 90s and early aughts, there was a sense that in some cases, ML systems would diagnose better, but doctors were reluctant to use them. That was kind of my sense of it.

**[0:12:30] Lukas Biewald:** But I was in a computer science lab. I never got the doctor's perspective at that point. How's that come?

**[0:12:35] John Halamka:** When you were at Stanford, did you use Sumax and have a chance to test mycin?

**[0:12:41] Lukas Biewald:** No. I actually don't even know what those are.

**[0:12:43] John Halamka:** Oh, okay. So Ted Shortliff at Stanford was really one of the inventors, the pioneers of AI, classic AI, right? So this is 1980s. And the idea back then was, you know, Lucas, you have an infection. You know, it's probably strep. What antibiotic should I use? Is penicillin a good one? Oh, you're pen allergic. Oh, what should I use instead? All of these decision trees were actually pretty complicated and mere humans wouldn't always make the right decisions.

**[0:13:18] John Halamka:** So Mycine was the first AI system running in the Stanford University Medical Exchange or Sumax that would provide doctors with expert antibiotic prescribing advice. and it was, of course, rules-based. And that was sort of the 80s. Everyone was writing forward chain, back chaining rules, these kinds of things. And so it was considered a decision support system and a kind of advisor, and I think very credible at the time. It wasn't competitive to anybody's decision-making.

**[0:13:52] John Halamka:** A challenge that we have with AI today is it lacks credibility at times because it doesn't have transparency. How was it created? How do I know for the person in front of me now that it's going to be fair, appropriate, valid, effective, and safe? Does it have consistency or reliability? And so I think we're at this journey right now where through a variety of policies and evolving best practices, we're starting to get AI credibility that will give clinicians the faith that using this thing will do no harm.

**[0:14:36] John Halamka:** So the journey has really been that, of getting us to that point where adoption is more likely because there's a sense of, this is going to help me in my role, not hurt me. Are there areas where it's adopted today? I have

**[0:14:52] Lukas Biewald:** a sense that maybe radiology is starting to move over to automated systems, maybe some other

**[0:14:59] John Halamka:** applications? Sure. So let me start. So Mayo has 180 models that it has created. But if you were to look just generally in the marketplace, radiology is by far the leading adopter of AI. And why? Well, inter-rater reliability of humans looking at x-rays isn't so great. And so this idea that, oh, maybe you could get not only more reliability, but early detection of disease and radiology. And there's a couple of use cases that are kind of interesting.

**[0:15:36] John Halamka:** What if you had a head bonk, and technical term, and then you went in to get a head CT because you had this bad headache. How long would it take a radiologist to review that head CT, even if it was stat? Could be an hour, could be two. What if you have bleeding in your head? Well, the algorithms today that some of them are embedded in the scanners themselves or just riding on top of the PAC software, look at the slice, the millisecond it's rendered and say, oh, there's a bleed there, raise the alarm bells.

**[0:16:14] John Halamka:** So you can understand how safety is going to be enhanced by pre-reading just in time, looking for head bleeds or pulmonary embolism, you know, problem with the lungs or air in the body where it shouldn't be, all that. You also can get sometimes earlier detection using very subtle changes that human eyes can't see. So example, Mayo has an algorithm that can diagnose pancreatic cancer two years before a human can see it. Because there's very subtle changes in the vasculature.

**[0:16:51] Lukas Biewald:** Does it need a picture of your pancreas? Yeah.

**[0:16:55] John Halamka:** So in other words, if you get a CT of your abdomen, it can often detect pancreatic cancer at stage zero. Whereas humans are typically seeing it at stage four when it's too late. Okay. So it's earlier detection, more consistency, better workflows. That's getting quite a lot of adoption. But also cardiology is starting to see a lot with prediction of heart disease or early interpretation of cardiac dysfunction. Mayo has a bunch of algorithms that do that. Neurology, early detection of stroke.

**[0:17:31] John Halamka:** Behavioral health, that's an interesting one. Can I detect depression or anxiety from signals like, how did you use your phone today? Did you pick it up five times or 50? Did you like a few things or not? Can I build a model that helps us understand behavioral change? And that works pretty well. But every specialty, it seems, is developing its own model. but your point well taken, the adoption of those models, their application in clinical care is somewhat limited today, but radiology and cardiology are certainly deployed.

**[0:18:07] Lukas Biewald:** Well, so you've mentioned the Mayo Clinic a few times where you're the president, and I have a sense that that's a big deal, but maybe you could tell me what the Mayo Clinic is trying to do and how it kind of fits into our medical system and how you've ended up with 180

**[0:18:21] John Halamka:** models there. Sure. So in 2019, Gianrico Ferruccia, the CEO of Mayo Clinic, had a notion. Academic healthcare doesn't always collaborate with industry with great agility. It takes a lot of legal agreements. It takes a lot of IT resources and almost every interaction is ad hoc. It's as if you've never done it before. So his question is, could we create a platform, right? I mean, of course, you live in a world of platforms, and you know what a platform is, but an ecosystem of technology and processes and policies such that every participant can benefit from every other participant in a repeatable way.

**[0:19:09] John Halamka:** So he wrote a job description analogous to this. wanted physician engineer with 40 years of digital transformation experience, public policy and economics experience, international travel, writing and speaking to change healthcare globally. Apply with it. I joined Mayo Clinic in January of 2020 as president of Mayo Clinic Platform. And immediately the first question, if you're going to be agile, is how do you, in a privacy-protected, ethical way, make large amounts of multimodal data available to innovators of all kinds?

**[0:19:52] John Halamka:** So we took the entire Mayo corpus, 150 years of data on 10 million patients, de-identified it, which in itself is tricky. You have to do such things as look at proper nouns, job roles, geographies, and familial relationships, look at dates and shift them, these kinds of things. We placed that in a cloud container and then built subtenancies where Mayo, with its 78 ,000 employees, could run low-code or no-code tools against it, but also innovators.

**[0:20:27] John Halamka:** Everything from five-person startups to very mature companies could come in, not buy the data, not exfiltrate the data, but build models on the data. And that acceleration is why we have

**[0:20:41] Lukas Biewald:** 180 models today. And so what does this data look like? This is health records of various people?

**[0:20:50] John Halamka:** Well, so when I say multimodal, let me explain. When you think of healthcare data, some of it's very structured. Your problem list, your medication list, your allergy list, or your laboratories. So part of the data is that very predictable form of words and numbers. About 80 % of the records, just the nature of the way records are recorded, is text. It's histories and physicals, operative notes, diagnostic reports. Those are the tricky ones to de-identify because you can imagine if the notes had no name and no phone number and no medical record number that said this very famous AI expert who hosts podcasts.

**[0:21:37] John Halamka:** I mean, maybe it would be re-identifiable. So we have to change that. But you also have images, right? CT, MRI, PET, echo, and omics, genomic data. Could be of your genome from your body or it could be from your tumor or some such thing. Digital pathology. You could have pieces of your tissue that are digitized. Or then things like your expozo. I know that's a weird English word because it's not real. But expozo, what do you eat?

**[0:22:10] John Halamka:** Where do you live? Are you near a Superfund site? What air do you breathe? Do you exercise? So that kind of stuff. So all of that is the multimodal day. I see.

**[0:22:23] Lukas Biewald:** And you've let people come in and build models on top of it, and you've ended up with 180 models so far? Correct.

**[0:22:32] John Halamka:** I mean, can you just - Some of those are validated, some taken through FDA, yeah.

**[0:22:37] Lukas Biewald:** Taken through FDA to do what then? To make a new kind of treatment or -

**[0:22:43] John Halamka:** Right. Well, so what FDA has this notion of software as a medical device, and their regulatory framework is to say, oh, here's a model. I mean, let's actually regulate it just like we would a pacemaker. That if this model is going to be used for diagnosis or for treatment, that it's going to be deemed safe. And very often these things will even go through clinical trials. But let me give you a case example. So Mayo had this notion.

**[0:23:14] John Halamka:** Could we analyze your heart function? And by heart function, I mean, how is your heart pumping, the ejection fraction? By looking at your lead one Apple Watch data and build a model, because the human can't actually look at your lead one Apple Watch data and determine how your heart is pumping. But oddly enough, a model can't. And what is lead one Apple Watch data? Oh, sorry. The way when a physician gathers your heart tracing. Typically they use something called a 12 lead electrocardiogram, but Apple watches only record one lead.

**[0:23:57] John Halamka:** I see. And so could we take that Apple watch data from a single lead on your watch and be able to analyze your heart functionality? So we got 125 ,000 Apple watch submitted telemetry lead one ECGs from patients, voluntarily all concepted. And then we did echo cardiograms. We did a gold standard analysis of their heart function. And then we were able to, in a clinical trial, show that this algorithm has an area under the curve of 0.92, right?

**[0:24:38] John Halamka:** That is, its positive predictive value is really pretty good. And so the end result, and we took that through FDA, and FDA has now given it full software as a medical device approval to be used as part of diagnosis for patients. Wow. What else have you uncovered? What else do these models do? Right. And so, example that I have a heart condition. Mine is called a supraventricular tachycardia. My heart rate is normally 50, but sometimes it goes to 170.

**[0:25:13] John Halamka:** It's irritating, not life-threatening. So I was concerned, well, maybe I have a cardiomyopathy, some sort of problem with infection in my heart, or a valve problem, or maybe I have pulmonary hypertension. All these diagnoses would require a lot of invasive testing. but Mayo's developed algorithms for each of them. So I, in my home, simply wore a device with a 5G transmitter that took my heart telemetry, sent it to Mayo where all these algorithms were able to analyze it and say, ah, you have a less than 1 % chance of any of these bad things.

**[0:25:56] John Halamka:** You have a conduction problem. It's like a loose wire in your heart. Take 25 cents a day of a calcium channel blocker called diltiazam, and you'll be fine. Well, I took the med, I'm fine. I'm cured. So I just give you this notion in the field of cardiology, a lot of different analytics for a lot of different diseases, analyzing did you have them, will you have them. In the field of neurology, you and I are having a wonderful conversation, and hopefully my thought process sounds clear to you, and I have a certain cadence, a certain amplitude in the way I speak, could you diagnose neuromuscular disease like Alzheimer's or a Lou Gehrig's disease, amyotrophic lateral sclerosis, or Parkinsonism from voice?

**[0:26:48] John Halamka:** Well, interesting question. So Mayo has been working on various kinds of models of doing disease prediction probability based on not what you say, but how you say it. And in cancer, could you predict cancer before it happens based on a variety of historical medical data and genomic data? So these are all examples of predictive kinds of algorithms. Well, it's kind of interesting that the stuff that you bring up here,

**[0:27:15] Lukas Biewald:** I feel like it sounds like it's pulling in sort of passively collected data sources that I wouldn't normally associate with a health record. Are you trying to attach like recordings of people in your database along with their Apple Watch and, you know, maybe like what they're liking on Facebook for the depression stuff that you talked about? Like, how do you actually pull in all this

**[0:27:38] John Halamka:** extra data? Well, sure. So as you say, these algorithms are multimodal in the sense that much of it is based on the medical data that is already recorded about you. But sometimes it has to be supplemented with novel signals. Here's the technical way we did it. There is a standard in medicine, this FHIR standard. It's a standard API using JavaScript object notation. And Mayo actually built into its Mayo app the capacity to receive patient transmitted data from their homes.

**[0:28:16] John Halamka:** And so it goes from you to your phone and from the phone into the electronic health record, but you control it, right? This is all done by the patient, initiated and consented by the patient. So we built that facility.

**[0:28:33] Lukas Biewald:** Cool. And are you trying to then actually commercialize these techniques somehow? Or how did all these things, they sound incredibly powerful and cool, but I haven't experienced them myself in my interactions with the healthcare system. Like, where does it go? Like, do you expect that doctors might prescribe me an Apple Watch in the future to collect data on me or something like that?

**[0:28:57] John Halamka:** Well, or the notion that you would be using a device of your choice and that device of your choice that produced a signal could be uploaded to an algorithm. So Mayo has used these algorithms internally, but has also spun out, for example, in cardiology, a company. And that company now makes those algorithms available to the world. And so Mayo is very creative in that it has incredible partnerships and collaborations with existing companies, sometimes creates novel companies for co-development.

**[0:29:36] John Halamka:** I see.

**[0:29:36] Lukas Biewald:** Right. And how has this explosion in LLMs, do you think, affected this stuff? When I look at these interesting disparate data sources that could be huge, it does seem like these algorithms might be deep learning, but are these algorithms using things like LAMA or GPT or anything like that? Does that affect the kind of results that you can get?

**[0:30:03] John Halamka:** So let's talk through that because, as you pointed out, deep learning, predictive type models, we understand how to measure their accuracy, their consistency, their reliability against ground truth. Now, of course, there is data shift and you have to often maybe reevaluate them, but you still know how to do it. With generative AI, it's really early days. And I think we've seen, and I'm sure you have as many anecdotal examples as I do, of generative AI can have a brilliant output.

**[0:30:41] John Halamka:** And five minutes later, a not so brilliant output. So how do you even certify a product as being appropriate for healthcare if every five minutes its performance characteristics are different? So Mayo's approach to this, there's several. Retrieval augmented generation actually works pretty well. That is, I might take a knowledge graph or a predictive model and it has an output. and I say, generative AI, now explain that, put in words, that actually is pretty consistent. So we have looked at all the ways you might use generative AI and things like simple English explanations given a data set created by a human, administrative simplification, helping with clinical documentation, summarizing a chart.

**[0:31:39] John Halamka:** Those are kinds of things that today we are doing. What we aren't doing yet in production is any kind of diagnosis or treatment based on a generative AI product. We think there's just more work to do, and Mayo is doing several things in that regard. So we will, in 2024, using de-identified Mayo data, train a novel large language model ourselves with the notion that maybe if it's trained on just really highly curated medical data, it will have less hallucination.

**[0:32:16] John Halamka:** We don't know, right? I mean, we're going to give that a try. And we'll do in specialty areas several experiments to say, oh, in this, again, early, earliest of days, multimodal generative AI. Could you take an image of a chest X-ray or an EKG and interpret it? Will that work or not? I don't know. So assume this, that Mayo would never deploy in production at this moment in history, a generative AI system, unless we were completely convinced about its safety, its reliability and consistency were sufficient.

**[0:32:58] Lukas Biewald:** But you did use, I think, you know, GPT-4 recently to edit a manuscript and even acknowledge it, which I thought was super cool. Like, did you feel like the quality was enough for your own publication to kind of put your name behind something? How did you think about that?

**[0:33:15] John Halamka:** Sure. So everything that we do in medicine has to be through the lens of risk. Right. So let me just give you the silliest example. I say, Lucas, I think that you should eat more vegetables. okay, if you do and I'm wrong, what's the risk? What's the harm? Like de minimis, right? So if I say, oh, well, GPT-4 does a pretty good job of creating readable English or whatever language. Saying here's a paper I wrote and I'm a human.

**[0:33:49] John Halamka:** You know, I have my own writing style and sometimes, oh, too many clauses, a little too verbose. Can you clean it up and make it readable? did a great job, right? And so that's one of those very low risk, pretty consistent and reliable ways to use GPT-4 as an editor.

**[0:34:11] Lukas Biewald:** Although I have to say, when we met, I remember I talked to you about how, you know, I'm a nervous parent with two little kids. And, you know, I used to go on Reddit to try to figure out if I should call the doctor or not. And I've kind of evolved to a little bit of Reddit research and a little bit of GPT research, which I think you told me was a bad idea. And I came home and I remember telling my wife about that.

**[0:34:34] Lukas Biewald:** And I think we kind of considered your point of view and unfortunately maybe rejected it. I think we still use GPT. And I find that using it alone seems like it would be unsafe, but our kind of alternatives in these scenarios are sort of like take our child to the hospital and kind of wait in line for a long time, use like a Teladoc service, which has been wrong sometimes in the past, or do Reddit or GPT.

**[0:35:00] Lukas Biewald:** So I do kind of feel like it's hard to not use it as part of your arsenal if you don't have a medical background. I have a feeling I'm not the only one that uses it in that way until there's something more reliable out there. So I'm curious how you think about that. I mean, there must be an overall trend in health of people doing more online research before they go to the doctor where certainly the quality can vary.

**[0:35:27] John Halamka:** So here is, again, you have to be, it's very early days. There aren't objective measures for quality or accuracy. But as you look at risk, here is one that would be, you'd think, sort of reasonable. Maybe there's some trusted sources out there, Mayo Clinic or Stanford or UCSF or the National Library of Medicine. If you could say, okay, generative AI product, whatever it is, could you summarize for me in this particular disease state or for these signs and symptoms what Mayo, Stanford, UCSF, and PubMed said?

**[0:36:07] John Halamka:** Okay, well, probably it's going to do a reasonable job of that. Where I find it not so wonderful is when you're relying on just its training set. Right. Because, of course, these products are mostly trained on the contents of the public internet, which also suggests that ivermectin cures COVID.

**[0:36:30] Lukas Biewald:** Right. Well, I actually saw, you wrote an article on, I think it was preparing for the world of generative AI. And you make this kind of offhanded comment that GPT is right only 25 % of the time. And I was like, how could that possibly be true? I got to follow this citation, right? And so, you know, I kind of followed through the tree of links because 25 % truthful just feels way too low for me.

**[0:36:56] Lukas Biewald:** And I think you were sort of right in the sense that there is this data set called like the truthfulness data set where it only gets 25 % accuracy, but it's actually the data set is designed to be kind of wrong facts and mostly medical facts that people think, like wrong things that people think. Like you get arthritis from cracking your knuckles and a whole bunch of things. I think I might get less than 25 % on that data set if I didn't go and fact check it.

**[0:37:26] Lukas Biewald:** So it does sort of seem like, yeah, I guess it's a, it seems like it's going to be particularly having a hard time in cases where the general consensus is different than expert opinion.

**[0:37:39] John Halamka:** Right. Right. And this is why, again, we think 2024 might be a turning point because then we'll have more purpose-built generative AI using highly curated data sets. But there's still going to be a problem. And as you look at some of the generative AI that's trained on the literature, randomized clinical trials, who enrolls in randomized clinical trials? those who have higher education, higher incomes, urban areas, typically. So you do worry that even if you use, depends on the data set you're going to use, if it's clinical and heterogeneous and includes urban and rural, rich and poor, educated and not, you may be okay.

**[0:38:27] John Halamka:** Just clinical trial data may very be biased in a way that is not going to be fair to many users.

**[0:38:36] Lukas Biewald:** Well, I would imagine the Apple Watch data might have a similar funny bias, right? I mean, Apple Watches are not cheap, and it's probably health-conscious people that purchase them. So wouldn't that kind of come up in sort of any of these data sets that you collect?

**[0:38:51] John Halamka:** And the way we originally developed that model was we actually took millions of ECGs taken from all over the United States. And these are ones that were taken in doctors' offices or hospitals or that kind of thing. So it was kind of all comers. So in this sense, it wasn't that we developed the model on only people who had Apple Watches. Because that, you're correct, would be a probably very biased sample.

**[0:39:20] Lukas Biewald:** Do you feel like for some of these applications, you might need to actually make a explainability quality tradeoff? When I look at an image and I think of a self-driving car, it's hard to point to, okay, that's the pixel that makes me not want to turn left here. That's the pixel that makes me think that there's a pedestrian there. And it does seem like with these bigger models and more disparate data sources, it might just feel like the totality of evidence here tells me that you might be sick and need to go to a doctor.

**[0:39:54] Lukas Biewald:** Do you ever feel like you have that tradeoff? Like I could imagine that, you know, for like simple regressions where it's just like, okay, like people that, you know, get this test tend to have this problem. That might not be realistic in the world that you're creating with these huge data sets.

**[0:40:08] John Halamka:** Right. So you asked this question about explainability or causality. Very, very often you can't just take a model and say, oh, look at those weights. I now know what it's doing. Right. Because there are just too many inputs and too much mathematical complexity. So better what you say is this is what the inputs are. This is what the outputs are. It doesn't seem reasonable that that set of input data could result in a reasonable decision. And so it isn't exact explainability or causality.

**[0:40:44] John Halamka:** It's just sort of reasonableness because you're exactly right. I mean, as a human, I believe that I can keep five conversations going simultaneously, but I can't do six. Well, as we know, any of these models, I mean, they could have thousands of simultaneous conversations, not an issue whatsoever. And so we say, hmm, it may very well be that we are taking in so many inputs to get to an answer. and it isn't one signal from one input, it's a combination of all of them and we'll go for reasonableness over explainability.

**[0:41:27] Lukas Biewald:** When you think about making an LLM that's for Mayo Clinic or only trained on really high-quality data sources, do you worry that there might not be enough high-quality data sources? I feel like one of the things that make these models work is just a massive amount of data. How do you approach that?

**[0:41:53] John Halamka:** Sure. So the way Mayo has thought about this is we started with our 10 million patients. And 10 million, well, certainly it has depth, lots of different kinds of data types. And 10 million, that's a reasonable breadth. But as we've been talking about, it may not have heterogeneity. It may not be large enough because of this whole issue of geography or the lifestyle you have. So we have a partnership with Mercy Health. And Mercy is Missouri, Oklahoma, and Arkansas.

**[0:42:25] John Halamka:** They have taken their 15 million patients, de-identified them, created a cloud container for the purpose of AI training and validation. Then we cite Albert Einstein, Brazil, UHN, Canada, Sheba, Israel. We're currently working with Singapore, Republic of Korea, Denmark. And there's a list of 21 countries that in 2024, we hope, will bring up to hundreds of millions of de-identified birth-to-death records to these cloud containers controlled by the owners of the data, not centralized or moved outside of a country.

**[0:43:09] John Halamka:** that starts to get to this point of having a data set that's got the breadth and the heterogeneity.

**[0:43:17] Lukas Biewald:** And I guess, what do you hope to see happens with that? Besides like sort of giving people answers on what conditions they might have, like what other like applications are you looking into or think we might see the next couple of years?

**[0:43:34] John Halamka:** So, so many possibilities. And so let me give you a couple of personal examples. My wife was diagnosed with stage 3A breast cancer in December of 2011, and she's totally fine today. Mayo has an algorithm that can predict breast cancer before a patient has breast cancer. It's not perfectly correct, right? So I asked my wife, if you were able to have a model with 84 inputs of phenotype and genotype and lifestyle and all the rest, and it said with reasonable certainty five years from now, you could develop breast cancer.

**[0:44:16] John Halamka:** But if you take this pill today, that pill will have side effects, a chemical menopause. You'll have fatigue and muscle cramps, but you won't develop cancer, would you? and she said well my values and this has to be very personal are I would be willing to understand that an imperfect algorithm with sort of reasonable amounts of medication that had reasonable tolerable side effects I'd do it and avoid the cancer altogether so that is this notion of being able to do early detection or prediction and give folks a sense of if you change your lifestyle change your medication, did a different diagnostic test, you might avoid or diminish your impact of disease.

**[0:45:05] John Halamka:** That would be interesting. I travel a lot, and I'm in a lot of places in the world where there are no doctors. And so, you know, here we are debating what is the area under the curve? Is it 0.8, 0.9? Well, what is the person in sub-Saharan Africa who has no capacity to travel to a clinician going to do? I mean, having a dock in their pocket, so to speak, an Android phone accessible model with reasonable performance is going to democratize access to specialty knowledge.

**[0:45:43] John Halamka:** And so one hopes that we can spread this knowledge on a global basis, impacting all those who have this capacity to connect and could be 4 billion or more people.

**[0:46:01] Lukas Biewald:** And I mean, you've written about AI replacing doctors. Doesn't that seem feasible? Like maybe it's sort of like field by field, but it doesn't seem like AI is going to sort of stop at useful, right? It'll kind of continue to improve and improve.

**[0:46:17] John Halamka:** Well, again, I think we have to be careful about that statement, which is why do we go to clinicians? A whole variety of reasons, but sometimes it's for respect and listening. Sometimes it's anxiety reduction, or sometimes it's my preferences and care planning. So I think it's very true that as we look at some aspects of AI, early detection in radiology or prediction in cardiology, those things are really good and will augment human decision-making. But there's something about a human-to-human interaction that, at least in many cases, I think is a positive experience for both parties.

**[0:47:02] John Halamka:** So this is why you've heard me say, I think AI will be augmented intelligence in many clinical cases and not replacing a clinic. That makes sense.

**[0:47:16] Lukas Biewald:** you know one thing i'm curious about from from where i sit not not an expert on on medical applications to any degree but sort of watching the machine learning use cases on our platform which is pretty broad-based is i have a feeling the majority maybe even the vast majority of medical applications on weights and biases is more focused on drug discovery like for example i think most of the big pharma companies are pretty heavy users of of weights and biases, but it's only a few places like Mayo Clinic that actually use this.

**[0:47:49] Lukas Biewald:** Is that a function of kind of where the money is in healthcare, or do you think it's because the drug discovery applications kind of work better or somehow like out ahead of all these amazing applications that

**[0:48:00] John Halamka:** you're talking about? Yeah. And again, I think you've seen the dissemination of innovation data that in medicine, it takes about 20 years on average for an innovation to disseminate widely across the specialty. So I think what we're just saying is that the future is already here. It's just unevenly distributed. And that means there are a few provider organizations, Duke, Stanford, Mayo, that are early adopters of these innovations and willing to take a lot of time to look at safety internally.

**[0:48:39] John Halamka:** Pharma happens to just have the staffing, the expertise, and the culture. So they're just a little bit ahead on that dissemination of

**[0:48:48] Lukas Biewald:** innovation. And so do you think that if I go to the Mayo Clinic, I would get much better outcomes because these techniques would be applied to me today? It's early. And so how about this,

**[0:49:04] John Halamka:** that in the field of radiology, I think that this is just not just Mayo, but I think we were just seeing in the literature that there is more timely and accurate diagnosis. Fractures that are missed are discovered early. Bleeding is detected before there are complications. So I think there's enough to conclude that yes, your care is better with AI over reading radiology than just humans reading it alone. But the integration of AI into workflow, into every average patient's experience is still very early in the cycle.

**[0:49:47] John Halamka:** 2024, we'll see more, but it's still going to be a couple of years

**[0:49:51] Lukas Biewald:** before it becomes a standard of care. And how do you think about what ML projects you bring in house and what you sort of like disseminate to the world? Like I know you have a partnership with Google and it sounds like you're kind of creating a platform where people can go in and do machine learning, but presumably you're doing some of it yourself. Like how do you balance

**[0:50:11] John Halamka:** what's internal and what's external? Sure. So Mayo has 7 ,000 specialists and probably the best way to create a good project is to have a passionate leader of that project who has extraordinary expertise in a field. So where there is that clinical expert who is also trained in ML or has a lab that can do ML, that's tended to be where we're seeing our earliest adoption. But again, I looked at 2024. Our intent is to democratize that.

**[0:50:49] John Halamka:** Here's what we recently did. We believe that generative AI has potential. So we asked the organization through a request for application process, give us your best use case. Detail how it would work. We got 355 applications. Of those, we downsized to 42. Ultimately, we chose eight general areas. And so now we are going to bring expertise, bring engineering and compute capacity to those clinicians in those eight areas. And these may be folks that had not done ML in the past on their own.

**[0:51:29] John Halamka:** So we're starting that cycle of making it such that the average expert clinician, not engineer, can participate.

**[0:51:37] Lukas Biewald:** Oh, that's very cool. And so they kind of come up with a hypothesis of something that they think might matter, some data set and some outcome. Is that how it works? Exactly. Wow. Interesting. and it would kind of come from their experience or somehow some knowledge of how the body works?

**[0:51:55] John Halamka:** Right. So if their belief is that they have seen enough patients of this type with these kinds of characteristics, could you potentially develop a generative AI model that will take into account all the patient history, read the chart, and then suggest a path forward for treatment? Let me give you an example area. We're doing this in rheumatoid arthritis. The challenge with rheumatoid arthritis, there are many different flavors, so to speak, of rheumatoid arthritis. Many different treatments.

**[0:52:31] John Halamka:** Each human is a unique individual. So this is the question we're going to ask. General AI, go read the chart. And then based on our training, can you recommend a path of treatment? and again we're going to try this it's going to be done in a very controlled guidelines and guardrails kind of environment

**[0:52:55] Lukas Biewald:** wow that's super cool um and and then like as that as that starts to work i guess the the health the health care practitioner becomes like less i i guess like kind of taught move to some of the other roles that you mentioned versus figuring out what's going on like he would still go into a doctor and then get this? Or could you do it from your home?

**[0:53:16] John Halamka:** Well, so again, I think it's first, the clinician has to make sure that the data that's going into it is reasonable and credible and complete, and that the recommendations seem within the realm of the reasonable. So for now, it is, again, augmenting human capacity. But let me give you an example that will probably sound very reasonable. When a cancer patient needs radiation therapy, a physicist does immense amounts of math to direct radiation to the tumor, but not the surrounding nerves, arteries, and veins.

**[0:53:52] John Halamka:** It can take 16 hours for a physicist to develop a model for a complex head and neck tumor. We have developed a model that does it in 45 minutes. And so you say, oh, well, here's a model that is doing math and physics. That sounds like a pretty good use of a model so that now the clinician is looking at the output saying, is that reasonable? Is that going to be safe? I might want to tune it a little bit.

**[0:54:24] John Halamka:** And the end result is the radiation oncologists and physicists can see more patients in more geographies than ever before. And I did receive a note which I think does reflect one interesting challenge in medicine today. Clinicians are very burned out, right? They have a huge administrative burden. COVID was hard for everybody. This radiation oncologist said, I'm having dinner with my family for the first time in four years. So think about that. That is, if you can see more patients in more geographies have an over read with quality and safety math and physics coming together with human compassion that's a pretty good combination you know i mean i i hear that a

**[0:55:12] Lukas Biewald:** lot kind of in every field and i think you know as someone who feels pretty financially secure i you know i totally relate to it but you can kind of connect the dots to like a further point down the road where we might not need so many doctors, right? Does everyone feel this positive about what you're doing or do you feel like there's pushback from doctors to adopt these kinds of

**[0:55:35] John Halamka:** automated approaches? So in the last 10 days, I've been in seven countries. In every one of those countries, the birth rate is 1.4 or 1.3, but yet the elders are living to their 80s and 90s. So the problem we have is a supply to ban mismatch. We're not going to graduate enough doctors to serve the elders that need their care. So I will tell you, I think AI is going to be an essential part of being able to actually care for those who need it, given fewer doctors graduating from medical school.

**[0:56:13] Lukas Biewald:** How do you feel about regulation? Do you feel like it's helping you or is it causing you problems? Like it must be really different to operate, you know, in a place that's heavily regulated versus a place with much lighter regulation, I would imagine.

**[0:56:27] John Halamka:** Well, but so I've always said regulation, for example, in the area of AI, isn't restricting innovation. It's telling us how we should innovate. What are the quality measures that we should follow? What are the guidelines and guardrails? So the wonderful thing about this journey over the last year, as you look at the president's executive order and some regulation that has just come out from the Office of the National Coordinator, all developed in public-private partnerships in an open, transparent way, which means that those who are experts have great input into how they should, in fact, be evaluated and measured.

**[0:57:10] John Halamka:** So I feel like the regulation in the world of AI is on the right track.

**[0:57:16] Lukas Biewald:** Although, don't you feel like it's still pretty vague? I mean, you know, we looked at the recent Biden regulations, but it didn't seem like there was enough specific there for me to even have an opinion on if they were good or bad.

**[0:57:31] John Halamka:** So remember, the executive order is a what, not a how. So what it says, for example, in health is that HHS has 180 days to establish a public-private partnership that is going to then promulgate best practices and guidelines and these kinds of things. So all of that is laid out as a process. And I look at the Coalition for Health AI and this work that we've done over the last two years now with 1 ,200 organizations, and you'll see a lot of really solid detail.

**[0:58:04] John Halamka:** if you've got a coalition for healthai.org.

**[0:58:09] Lukas Biewald:** Do you worry at all about privacy implications of being able to discover things through passive signals? Like kind of what you said with the Apple Watch, if you could discover that I have like a heart condition that maybe I don't even know about, or if kind of the medical record becomes more and more parts of your life. Do you see that as becoming an issue down the road, like more of an issue?

**[0:58:32] John Halamka:** Sure. So the way that Mayo protects privacy is something we call data behind glass. And here are some of the notions. First, the data has to be fully de-identified. And Eric Schmidt would tell me there's no such thing as fully de-identified data. But let's just say de-identification means a very low risk of re-identification at this time in history with existing technology. And then you place it in a cloud container that you control.

**[0:59:00] John Halamka:** You audit. you restrict the inputs and the outputs, and you prohibit linkage with other external data sets. And so if you have some of those controls in place, you've done a pretty good job to make the risk of re-ID or privacy breach extremely low. But to your point, I think we've seen this just over the last 20 years, that if you have enough data and enough databases and enough compute, you could probably re-ID. So that's why we've had to put these policy controls around the technology to minimize the risk.

**[0:59:40] Lukas Biewald:** But it's a funny idea, maybe even what, you know, kind of consists of a patient record when you talk about what you're doing, right? Like if my voice can kind of tell you medical things about me, are recordings of me now like sort of HIPAA, like under HIPAA privacy laws?

**[1:00:00] John Halamka:** And of course, HIPAA itself will have to evolve. Right. So HIPAA was written in 1996. It doesn't mention the genome. Right. It was 1996. 96. So I think, you know, you always need to put the needs of the patient first, respect patient preferences, and recognize that at the moment, we can't re-ID you based on that lead one ECDG from your Apple Watch. But, you know, maybe five years from now, they'll say, oh, that lead one ECG, that's like a fingerprint or retinal scan.

**[1:00:38] John Halamka:** Could be. And we'll have to evolve consent and privacy around whatever emerging technology breaks.

**[1:00:45] Lukas Biewald:** Yeah, it's actually kind of striking. I think of all the people working in the medical field on this podcast, you've gone the longest without mentioning genetic information. Is that something that you're a little bit more skeptical about than these other sort of passively collected data sources or less of an emphasis? Or is that just random based on my flow of questions?

**[1:01:04] John Halamka:** Well, so let me give you some detail that will help you with that. So I'm the second human sequenced in the Personal Genome Project. So if you go to personalgenomes.org, patient two, that's me. You can download my full sequence. So for the last 15 years, my genomic data has been public. And I have, on that website, you'll see the 1 ,500 diseases I carry. Now, the challenge is, interpreting the genome is still a very imperfect science.

**[1:01:37] John Halamka:** You know, if you've got 1 ,500 diagnoses all listed as low probability, low impact, What do you do with that? So I think, one, it's early days for turning the genome into care plans. But two, the definition of a de-identified genome is also something that is not quite yet state of the art. So, okay, if you take a subset of my genome, biomarkers, a few mutations, oh, that's probably DID. My 3 billion base pairs, certainly not DID.

**[1:02:14] John Halamka:** Totally. So purely the only reason I hadn't mentioned it is because it is a work in process. And we are actually loading a lot of what I'll call DNA fragments, single nucleotide polymorphisms and mutations now into the database. It just hasn't been a common thing to do in prior years.

**[1:02:40] Lukas Biewald:** Cool. Well, John, let me end with this. It's been really fun talking to you. I'm sure that there's a lot of students that watch this podcast that are kind of in machine learning and interested in medical applications. Do you have any advice on kind of what areas of research you would look into if you were starting grad school or just coming out of Stanford now, 40 years later?

**[1:03:02] John Halamka:** Sure. So there is no question that a partnership between a young mind with engineering background who has the basic principles, knowing ML is going to evolve very rapidly, but the principles behind ML, working closely with a clinician in workflow can do magical things. So it's really interesting when I talk to my clinicians, I say, what are your business problems? You know, what are the use cases? What are the things that are challenging you the most? That then becomes the inspiration for students to do projects.

**[1:03:46] John Halamka:** And the combination of energy, data, and clinical expertise will generate new solutions. And Mayo would be thrilled to work with your students in that regard. Awesome. Well, thanks so much, John. It's great to talk to you. Very cool. Well, and thanks. Happy to do this again, because things are moving so fast a year from now. Who knows what the future will bring?

**[1:04:09] Lukas Biewald:** Thanks so much for listening to this episode of Gradient Dissent. Please stay tuned for future episodes.
