OpenNashPodcasts

NEJM AI Grand Rounds

Epic’s Approach to AI with Seth Hain

18 Feb 2026 · 53m · Seth Hain, Andrew Beam, Arjun Manrai · 24 quoted facts

▶ Play episodeOriginal source ↗

In brief

Seth Hain, head of R&D at Epic, argues that AI in healthcare works best as shared infrastructure. Epic trained a model on structured medical events, such as diagnoses, orders and medications, rather than on text. Performance improves predictably as data and model size grow, much like language models. Careful, staged rollout inside clinicians' existing workflow is how he expects trust to be earned.

The central tension is between ambition and caution. The model could replace many one-off prediction tools and give doctors a view of possible futures for a patient. But it is not yet deployed, has not been compared with Epic's existing sepsis model, has not been tested across hospital types, and cannot yet tell a doctor what to do. Epic's answer is to release it to researchers first and test it silently.

Key insights

  1. Medical events can be modeled like language

    Epic's model reads a patient's history as a sequence of diagnoses, orders, medications and elapsed time, and predicts the next event. It starts with no built-in knowledge of what hypertension or Lipitor is.

  2. Bigger models and more data keep improving results

    Epic found scaling laws, meaning predictable gains as data and parameters grow, in structured medical data. Hain says that if this had not held, he would be much less bullish.

  3. One model can replace many single-purpose predictors

    Hospitals have traditionally built a separate model for each target, such as diabetes complications or length of stay. The generative model produces 20 to 40 simulated patient timelines, so odds for many outcomes come from one system.

  4. The model is not yet proven against existing tools

    Epic has not compared Curiosity with its sepsis model, since it is not deployed anywhere. Hain says it would first be run silently on local patients to establish a baseline.

  5. Trust in generative AI needs new baselines

    Tasks like chart summaries had no human-written version to compare against. Epic worked with researchers, including at the University of Wisconsin, on LLM-as-judge evaluation in an open-source AI Trust and Assurance Suite.

Questions

What is Epic's Curiosity model and what does it do?

It is a model trained on patients' structured medical records that predicts what is likely to happen next in a patient's care. Seth Hain says it can forecast diagnoses, orders, complications and hospital stays by generating many possible timelines.

How big is the data behind it, and where does it come from?

Cosmos, Epic's de-identified pool of records from participating health systems, holds over 300 million unique patients. The largest model so far trained on 118 million patients and over 115 billion medical events.

Can doctors use it today to decide treatments?

No. Hain says the first phase is giving researchers access, and he concedes that recommending interventions raises hard questions about confounding, where other factors distort cause and effect. Andy Beam raised this concern.

Will it work equally well at every hospital?

That is not known yet. Hain says localizing the model to different institutions is a question of deep interest that Epic has not fully researched, though it can adjust results using a hospital's own practices.

Is patient data in Cosmos used for other purposes?

Hain says there is no secondary data use of Cosmos. It is governed by an elected committee of health system professionals and is meant only for research and care at the point of care.

Facts

Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.

Epic's Cosmos dataset contains more than 300 million unique patients, with records stitched together across participating health systems.

“Cosmos these days has over 300 million unique patients in it.”

0:12:09 · Seth Hain · Cosmos dataset

More than 130 studies in over 80 journals have been published using the Cosmos dataset.

“there's been over 130 studies published in over 80 journals built off of that data set.”

0:13:19 · Seth Hain · Cosmos research

Cosmos is governed by a council whose members are elected from participating health systems.

“This effort is governed by a variety of health systems, and individuals from the health systems that are participating are elected to a board that create the Governing Council for Cosmos.”

0:12:09 · Seth Hain · Cosmos governance

Epic's structured medical event model has about a billion parameters and was pre-trained on over 115 billion discrete medical events.

“That training cycle ended up ultimately with a billion-parameter model that used over 115 billion discrete medical events in its pre-training process.”

0:18:18 · Seth Hain · Model training

Epic built 78 evaluations to test its structured medical event model on tasks including diagnosis, prognosis and operational metrics.

“We ended up building out 78 different evaluations for understanding how a structured medical model could impact everything from diagnosis, say, in the near term as you're going in for an office visit,”

0:18:18 · Seth Hain · Model evaluation

The largest Curiosity model to date was trained on 118 million patients.

“The largest model to date we've trained on Curiosity has 118 million patients in it.”

0:25:21 · Seth Hain · Model training

Curiosity was formerly known as Comet in some contexts.

“the current version of Curiosity, or what was formerly known as Comet in some contexts,”

0:25:50 · Seth Hain · Product naming

Epic's first Curiosity model was trained only on adults over 18, with pediatric data planned for later training runs.

“The population we trained the first model on was over 18 years old.”

0:33:11 · Seth Hain · Training population

Curiosity generates a patient's predicted future repeatedly, producing 20 to 40 possible timelines to estimate outcome odds.

“you actually generate that multiple times. The result of this is that you have 20, 30, 40 potential timelines that this patient can go down,”

0:25:50 · Seth Hain · Model method

Cosmos is an Epic community effort to bring real-world evidence into the point of care during patient visits.

“Cosmos is an effort across the EPIC community to bring real-world evidence into the point of care”

0:11:37 · Seth Hain · Cosmos dataset

The Cosmos effort has been underway for almost a decade.

“This is an effort that has been underway for almost a decade, at least at this point,”

0:12:09 · Seth Hain · Cosmos history

Epic has not compared Curiosity against the Epic Sepsis Model, since Curiosity has not been deployed in production.

“we have not done it yet.”

0:35:43 · Seth Hain · Sepsis model

Epic works with researchers, including some at the University of Wisconsin, on evaluation techniques such as LLMs as judges, part of an open-source AI Trust and Assurance Suite.

“which is an open source tool for this.”

0:39:38 · Seth Hain · AI evaluation

Over 440 health systems use Epic's generative AI in production.

“We have over 440 health systems using generated AI in production at this point.”

0:47:16 · Seth Hain · Adoption

Epic began rapidly rolling out generative AI in late 2022, starting with outpatient care.

“in late 2022, we started down the path of rapidly rolling out generative AI, first in the context of outpatient care.”

0:10:36 · Seth Hain · Generative AI rollout · 2022

Seth Hain began working across Epic's clinical application suite in January 2020, just as COVID-19 demanded rapid changes such as vaccine administration workflows.

“And so in January of 2020, I started working across the suite of applications”

0:09:06 · Seth Hain · Hain career · 2020

Hain began at Epic working on scalability of Chronicles, Epic's underlying database foundation, and later worked on virtualization and cloud deployment.

“I started by focusing on very low-end scalability, optimizing all the way down to the chipset, the foundation behind Epic, something we called Chronicles.”

0:08:28 · Seth Hain · Hain career

When Seth Hain was a graduate student in Madison, Epic had just signed Kaiser Permanente as a customer.

“they had just signed Kaiser Permanente.”

0:05:44 · Seth Hain · Epic history

Epic planned to open the Curiosity models to researchers on Cosmos in the first quarter of the year following the interview.

“which is planned for the first quarter of next year, to see how they do this also.”

0:20:54 · Seth Hain · Release plan

Epic's event model begins training with no built-in knowledge of medical concepts such as hypertension or Lipitor; tokens map directly to structured events.

“There is no foundational knowledge of what is hypertension or what is Lipitor.”

0:17:38 · Seth Hain · Model design

Through a capability Hain calls Aura, clinicians can place orders in Epic for specialty diagnostics from other companies, with results flowing back into the medical record.

“something we call Aura, placing orders in Epic for specialty diagnostics that come from a different company”

0:00:00 · Seth Hain · Aura

Epic is developing a tool called diagnosis checker that can generate a differential diagnosis and likely follow-up tests.

“something we're calling diagnosis checker, as an example, which is essentially both a differential diagnosis,”

0:22:07 · Seth Hain · Diagnosis checker

Epic's software can generate different summaries of medical records for different roles, such as a physician at a physical or a nurse at shift end.

“We have a capability in the software to generate summaries of medical records.”

0:39:09 · Seth Hain · Chart summaries

Hain says Epic's campus features unusual items, including a Fibonacci spiral staircase and a carousel from the closed Ella's Deli.

“you walk by all of this wild stuff but you you sort of it becomes normal”

0:44:49 · Seth Hain · Epic campus

Chapters

  1. 0:05:44Hain's path from math to Epic

    Explains how a mathematician became head of R&D, from low-level database scaling to clinical applications and generative AI.

    “I was the kid that brought his printout of a computer program to the county fair.”
  2. 0:15:41What Cosmos is and how trust works

    Describes the pooled, de-identified dataset and the governance rules that make health systems willing to contribute.

    “There is no secondary data use of Cosmos.”
  3. 0:19:14Training a model on patient events

    Covers how structured events replace words, the model's size, the 78 evaluations and the scaling-law finding.

    “we studied the scaling laws in the same way that people have for”
  4. 0:26:34Uses, limits and localization

    Lays out planned clinical and operational uses and the open problems of local variation and recommending interventions.

    “you have 20, 30, 40 potential timelines that this patient can go down”
  5. 0:36:38No sepsis comparison yet; silent testing

    Shows the model is undeployed and untested against Epic's existing tools, with silent local trials as the plan.

    “there's an opportunity to run the model silently behind the scenes on the local population”
  6. 0:39:09Evaluating trust in generative AI

    Explains why AI outputs with no human baseline need new evaluation methods.

    “no one was writing those summaries before.”
  7. 0:42:09Lightning round

    Personal answers and predictions, including that AI second opinions will appear in the EHR within five years.

    “Genomics. And I think in a narrow sense, it's being overestimated”
  8. 0:50:22Epic as platform versus one-stop shop

    Hain weighs the case for and against the EHR as the hub for AI and says Epic will be both a builder and a base layer.

    “I think the answer ends up being both in many ways.”

Transcript

0:00:00
Seth Hain

There are many things like new styles of devices, direct relationships between the AI technologies and the scope as an example, or new styles of therapeutics and diagnostic approaches that as patients we think are very, very promising. And as a company, we think it's important to be able to work closely with. And so we see folks as an example through something we call Aura, placing orders in Epic for specialty diagnostics that come from a different company or maybe from a different type of device or a personal device that the patient will be prescribed.

0:00:39

And that information flowing back into the physician medical record to be able to drive insights. and that type of structure that supports all of these different types of AI where it connects best to be at the test, be at the device, etc. And then how do you put it back into workflow? I think there's a role for us and there's a role for many other people in that type of future.

0:01:04
Arjun Manrai

Hi, and welcome to another episode of NEJM AI Grand Rounds. I'm Raj Manrai. I'm here with my co-host Andy Beam and today we're thrilled to bring you our conversation with Seth Hain. Seth is the SVP of R &D at Epic, the company that you're likely very familiar with that good chance makes your organization's electronic health record if you're at a hospital. I mean we had a really wide-ranging conversation with Seth. We got to learn about his background in mathematics, how he started working at Epic, and his work over the past few years on building AI models using massive amounts of data at Epic.

0:01:40

Andy, I really liked some of the shared connections that you had with Seth around interest in math and video games. And then also I think 4-H, which was not on my bingo card for what would come up during this conversation. It was a lot of fun. Yeah, agreed,

0:01:53
Andrew Beam

Raj. This conversation was unexpected for me because I thought that we were going to talk about Epic as the sort of monolithic EHR that everyone knows and has interacted with. But what I really enjoyed was understanding how they think about research and development. So Seth is the head of R &D at Epic. They have a lot of really, really great thoughts about AI, its impact on healthcare, thoughtful ways for training and evaluating healthcare models at scale, pitfalls for how that can be deployed.

0:02:21

And so again, I found the getting to understand how they think about the more researchy side of things to be super enlightening. And I found Seth to be really thoughtful and we had a really, really engaging conversation. The NEJM AI Grand Rounds podcast is brought to you by Microsoft, Viz.AI, Lyric, and Elevent's Health. We thank them for their support. And with that, we bring you our conversation with Seth Hain on NEJM AI Grand Rounds. All right, Seth, well, thank you for joining us on AI Grand Rounds.

0:02:58

We're excited to have you on the show today.

0:03:00
Seth Hain

Thank you for the invitation. Excited to join.

0:03:02
Arjun Manrai

Seth, it's great to have you here. So this is a question that we always get started with. Could you tell us about the training procedure for your own neural network? How did you get interested in artificial intelligence and what data and experiences led you to where you are today?

0:03:16
Seth Hain

So many, so many areas to dive into there. Personally, what led me here, my background was starting in mathematics. So that's what brought me to Madison, Wisconsin. Originally, it was graduate school. And for many years here at Epic, I focused on the back-end architectures that support scaling out the many different applications from MyChart to Epicare in the exam room to the inpatient capabilities. And I think a key aspect of that has always been that at Epic, our focus has been on how do you capture a patient's story and then use that in many different contexts.

0:04:00

And over time, that led to many of the insights that ultimately resulted in what we are building on top of from a foundational model perspective on top of Cosmos and the training that we're doing in that context. And so obviously many, many steps along that path. But I think an important part of that is pulling together both people with a deep understanding of medical records and the journeys that patients are on and an appreciation of the data in that context, combined with building out a team deeply familiar with the pre-training process and beyond for creating foundational models.

0:04:44

That's an abbreviated answer to your question. Happy to dive into the details.

0:04:48
Andrew Beam

Yeah, so you're skipping ahead a couple chapters there. I'd love to hear a little bit more about what made you interested in mathematics. How did a mathematics major end up in an EHR company? Were you interested in medicine? Could you take us to some of those formative data points and help us understand how you got to where you are?

0:05:03
Seth Hain

Yeah, happy to do it. My background in mathematics came from a very young age, and I think as many of us probably got to experience, I had a phenomenal teacher in high school that really laid the foundational interests around mathematics, combined with, kind of as a fun aside, I grew up outside of Lincoln, Nebraska. And I was the odd kid whose dad created a computer science program for 4-H. If you're familiar with 4-H, it's a program where kids tend to raise cattle or pigs or bake cookies and take them to the county fair.

0:05:44

I was the kid that brought his printout of a computer program to the county fair. So I had this combination of at-home programming with a very strong set of teachers around mathematics that led me to sort of this combination. I found myself in Madison for graduate school, and there was a company that was starting to grow. They had just signed Kaiser Permanente. And I thought this could be an interesting opportunity. And in particular, and this remains to the day, Epic is willing to hire folks straight out of school that have a combination of interest and curiosity with an aptitude for quickly learning new ideas.

0:06:29

And so I was lucky that folks took a chance on me in that context. And somehow that has led to now 20 years

0:06:36
Andrew Beam

here. I took a 4-H magic camp one summer, so I can also attest to the impact that 4-H can have on kids early on. Were there other things? So I know that Epic has this large presence in Madison, Wisconsin, and is really great at pulling from deep quantitative talent in that area. Were there other things that you were thinking about at the end of your master's program? What were the set of options that you were thinking about?

0:07:00

And then sort of how did you decide on Epic as your destination? Obviously, it's been a good home there. You've been there, as you said, for two

0:07:06
Seth Hain

decades now. So obviously, you made the right choice. Yeah. At that point in time, there was three different routes I was considering. One was certainly staying in academia, a route that has a lot of opportunity. But I tend to want to and understand a wide variety of spaces. And working in industry, and in particular here at Epic, I have the opportunity to move between areas and continue to grow my breadth of experiences across multiple different domains, which was appealing here.

0:07:36

The other thing I was very interested in was working in an industry where you could help people and you could do so in a collaborative, productive way across industries and including with academia. And Epic met that criteria as well. And so it was an opportunity to work in an industry, solve really hard problems, and do so in a way that my family members could benefit from, as well as the community around now, the whole world. Obviously, that scale was something that happened over that time period, though.

0:08:13
Andrew Beam

Nice. So we're going to talk about what you're working on now in just a bit. You showed up at Epic. You're now, I believe, Senior Vice President of Research and Development. How did you go from Madison, Wisconsin grad to SVP of R &D at the World's Premium at EHR Company?

0:08:28
Seth Hain

It was a journey that involved both learning from a number of folks here and taking opportunities as they came. I started by focusing on very low-end scalability, optimizing all the way down to the chipset, the foundation behind Epic, something we called Chronicles. And over time, that led to working on how do we virtualize the architecture? How do we build out a cloud-based platform for our applications to be deployed in? Essentially, a platform as a service that runs on the hyperscalers, as an example.

0:09:06

And working through those types of steps, building out a series of products that ultimately included building out the artificial intelligence platform or something we call our cognitive computing platform. Interestingly, I was offered the opportunity and took it to gain familiarity with our clinical applications. It's always good to bounce around and have direct experience with end users using the software, particularly both the clinical folks, but then also the back office folks. And so in January of 2020, I started working across the suite of applications that are Essentially everything you walk in for, from an office visit to the urgent care and ED, even to, and this was fun, going on immersions to dental offices for our Wisdom product.

0:09:56

Obviously, January of 2020 was an interesting time to start working with that group. It was an established set of products that suddenly we needed to think about, for example, which clicks in the flow for administering a vaccine are deeply, deeply necessary. And how would you do that if you had a line of cars that needed to happen? Where historically, the vaccine administration activity was very focused on children during while child checks. And so you end up going through those types of pieces as well as getting deep into analyzing things like lab results and those pieces around the COVID tests, as another example.

0:10:36

And then in late 2022, we started down the path of rapidly rolling out generative AI, first in the context of outpatient care. And that led to something we can kind of talk about the trajectory of in more detail, but an intense focus on how to apply generative AI to first help clinicians, directly helping patients and also doing a variety of different research activities to advance medicine, which has been a consistent focus of mine over the last couple of years.

0:11:11
Andrew Beam

Cool. Thanks for that, Seth. So I think that's a great segue to hop into some of the work that you've been doing at Epic recently. In particular, I'd like to focus the discussion on a paper called Generative Medical Event Models Improve with Scale. We talk about scale a lot on this podcast, I'm excited to dig into how you guys have been thinking about it. First, maybe could you set up this paper? What were the goals? What were the key findings?

0:11:34

And then maybe also what were some of the key challenges?

0:11:37
Seth Hain

I think it may be useful to take a step before even the paper, just so the audience has full context of Cosmos, which was a foundational piece that opened the door to make this paper possible and then dive into the findings if that's okay, because I think that the context is useful here. COSMOS is an effort across the EPIC community to bring real-world evidence into the point of care in the context of that visit between a patient and a physician, as an example.

0:12:09

This is an effort that has been underway for almost a decade, at least at this point, and started with a focus on building out a de-identified data set for the purposes of doing real-world analysis, eventually in real time, for example, in workflow. This effort is governed by a variety of health systems, and individuals from the health systems that are participating are elected to a board that create the Governing Council for Cosmos. and Cosmos these days has over 300 million unique patients in it.

0:12:48

When I say unique patients, what I'm referencing is that if an individual goes to multiple different health systems that are part of the Cosmos community, their medical record is stitched together across those so you have that full patient journey and that's what the de-identified record represents in the Cosmos system. And so the first phase after starting to compile that was that health systems participated in it in Cosmos had access to the data set for purposes of research.

0:13:19

And there's been over 130 studies published in over 80 journals built off of that data set. In parallel to that, we started to build it into software as well. That included, for example, connecting physicians that might be seeing patients with a similar constellation of symptoms so that they could discuss that kind of unique situation that was presented to them through the patient with a colleague and identifying those folks, which you can do through this. It also meant generating growth charts for kiddos with kind of a unique, say, disease, not truly unique, a rare disease, if you will, outside of the traditional growth charts.

0:14:02

so them and their parents could see how they compared to other kids in a similar circumstance. And it was those types of applications that led us to wondering if there was a way to make it faster and more efficient to build applications using the Cosmos dataset and do so in a way where it would apply to essentially any diagnosis or a patient at various stages of their medical journey. and make it far more applicable to the broader healthcare community.

0:14:35
Andrew Beam

Can I hop in and ask a question about Cosmos? My experience in healthcare and with hospitals is that they are reluctant to share data with other hospitals and other institutions, but it sounds like if you're using Epic on the back end, then you're contributing to pooling what is by far the largest healthcare dataset in America, you know, and one of the largest in the world. So what's the operational side of this? How do institutions contribute data?

0:15:02

Can you access data from other institutions? Maybe you could just talk a little bit about that.

0:15:06
Seth Hain

It's an important aspect of this. And I would say that foundational to the entire Cosmos effort is trust. And one thing I want to be clear about, while there are organizations that use the Epic software that do not contribute and participate in Cosmos, it is rapid and straightforward for an organization when they're ready and they decide that it makes sense to join. And that includes both a mapping effort, as you might expect, and then the transport into Cosmos is pretty straightforward from a software perspective after we've engineered that.

0:15:41

The key there is an understanding that that data set and the rules of the road that are governed by that elected committee of health system professionals is designed to support in workflow real-world evidence. There is no secondary data use of Cosmos. It's designed for those health systems participating in it to both do research and then use it at the point of care. And that's what establishes the trust.

0:16:09
Andrew Beam

Cool, thanks. So I think we were to the point where we have the database now, and now we can talk about the big model that you guys trained. That's right, that's right. I felt like we needed to set the foundation before we got in. That was very, very good context, yeah. Thanks, Seth, that was great context.

0:16:23
Seth Hain

And I think the key here in regards to the model we trained was the recognition, as I noted earlier, of the value of a patient's story. And in this context, usually folks think about stories or training models around natural language, a story they might read, a novel they might read, a Reddit post they might see, and how in pre-training, a model tries to predict the next word in that story or that Reddit post and then compares it to the actual ground truth of what was in the data set that it's being trained on, right?

0:16:59

That's the general path of a pre-trained model. We identified that you could tell a patient's story, and others have started down this route as well, which are referenced in the paper, through a series of structured medical events, meaning the diagnoses, the orders that are placed, the observations and medications that are part of the medical record, and importantly, the time that passes between those. And rather than predicting natural language words, the question was, could you predict the next event in that patient's story?

0:17:38

And I think this was really, I just want to emphasize this point for the audience. When we start training one of these structured event models, there is no foundational knowledge of what is hypertension or what is Lipitor. They have no idea what that is. The tokens are directly mapped to these structured medical events in the vocabulary and then trained on that sequence of events. And that is what led to us training what is now the largest structured medical event model in the world.

0:18:18

That training cycle ended up ultimately with a billion-parameter model that used over 115 billion discrete medical events in its pre-training process. The paper that you referenced has kind of what I tend to think of as three major sections, and we can dive into those. The first one is around the series of ways to evaluate a model. We ended up building out 78 different evaluations for understanding how a structured medical model could impact everything from diagnosis, say, in the near term as you're going in for an office visit, to prognosis over, say, a year or three years of a chronic disease, to things like operational metrics around length of stay, readmissions, your likelihood of going into the ED.

0:19:14

And so we built out a large eval suite. Obviously, another step of that is then the actual training of the model, how it's structured, how you do the pre-training, et cetera, which is explained in the paper. And that importantly, as loss drops when you're doing the pre-training, it correlates to the impact on those evals that you want from a medical perspective. That's what you would hope for, but it's an important thing to evaluate. And then the third piece of the paper, and maybe we want to dive into it, I think it's ultimately going to prove to quite possibly be the most interesting piece, is that we studied the scaling laws in the same way that people have for

0:20:07

natural language models on these structured medical event models and found that in a very similar and predictable manner, as you scale up the amount of data and the parameter spaces of these models, you can improve the quality of the models. Meaning that if we or others continue to train larger and larger models, they'll get better and better at those types of evaluations we talked about.

0:20:37
Andrew Beam

Awesome. A couple things I'd like to follow up on there. So maybe in the context of the evals, could you talk about how you would imagine this model being used in clinical practice? I have a couple things in mind, but I'd love to hear at least what you guys were thinking as you trained this model.

0:20:54
Seth Hain

Oh, there is so many. And I think one of the things I'm very, very excited about is opening up these models to the researcher community on Cosmos, which is planned for the first quarter of next year, to see how they do this also. But let me give some specifics. I think the first place we will see this model being applied is in many of the contexts where folks built traditionally predictive models for a single given target.

0:21:23

An example of that might be predicting and identifying individuals at high risk of type 2 diabetes complications over the next three years, where this model autoregressively predicts what's to come in that patient's journey, including those complications resulting from diabetes, and then use that in workflow to help prioritize care manager outreach, as an example. This is something people build predictive models for today. But in the context of this model, you can predict many different chronic diseases, as well as the possibility of co-occurrence between different types of outcomes in that context.

0:22:07

So I think that'll be one example. One that I'm very excited about that we looked at quite intensely with those evaluations was what is going to immediately happen as the patient walks in for the office visit. What are the likely diagnoses in this context? What are the likely medications that will be ordered, et cetera? Obviously, that has a number of different ways it can surface into workflow. One of those is through something we're calling diagnosis checker, as an example, which is essentially both a differential diagnosis, essentially that can be generated with a tool like this, as well as likely subsequent labs and tests that can be done to help determine if that diagnosis is accurate or follow-ups based on

0:22:52

it. And so in that context, we see the model helping generate and suggest as it makes sense. Always have with the human physician making the decision at the end of the day, but considerations based on that large data set and doing so in a way that is efficient within the software, which is always part of this is how do you help increase efficiency? The third example of this and the third kind of category of specific cases tends to really be around hospital operations.

0:23:23

Given the volume of challenges facing the healthcare system these days, and in particular, the aging population, which needs increasing amount of care, and candidly, the limitations in regards to access that many health systems, hospitals, and clinics are facing right now, being able to accurately predict things like length of stay, being able to understand the likelihood that somebody will need a more intensive level of care within the hospital, et cetera, is a meaningful set of data points to be able to help plan out everything from staffing to occupancy rates, et cetera.

0:24:03

And I appreciate that in many of these contexts, the things that I'm saying are things people have built individual predictive models for one at a time. Those are intense efforts from a data scientist perspective that includes building out those data sets, et cetera, et cetera, which I'm sure the audience is quite familiar with, but are ultimately around a single target that they end up predicting. Our purpose with building out this generative model was that one model could do all of those predictions and open the door to more complex investigations in the future.

0:24:42
Andrew Beam

Can I ask a question just to make sure I understand correctly? So you have a model trained on 300 million patients and presumably hundreds of different hospitals, academic medical centers, local clinics, anywhere that they're using Epic and they have opted in. Is there a way that the person can say, what's going to happen to this person at my hospital? because it's very hard for me to think about the trajectory averaged across hundreds or thousands of different institutions because all those are going to have different policies, different ways of treating.

0:25:13

There's going to be a lot of like heterogeneity, both in outcome, but how they treat patients. And so sort of thinking about the average trajectory of a patient in a vacuum is a hard

0:25:21
Seth Hain

trajectory for me to think about. No, it's a good question. And a couple of different pieces here that I think are worth noting. And one thing I'll just quick note, Cosmos, the large data set, has 300 million unique patients in it. The largest model to date we've trained on Curiosity has 118 million patients in it. Purpose of the scaling laws is it shows we're going to keep scaling up that compute in the volume. So just a brief aside on that specific piece.

0:25:50

As you note in this context, the current version of Curiosity, or what was formerly known as Comet in some contexts, is essentially a way to generate real-world evidence across the large population in regards to what are the likely events to follow in this patient's trajectory. So when applied in the context of an individual situation, you start with their medical record story and then autoregressively predict the next set of tokens that come. Now, in practice, what this actually looks like, and we can get into this in more detail, is you actually generate that multiple times.

0:26:34

The result of this is that you have 20, 30, 40 potential timelines that this patient can go down, which gives you a sense of the odds of different types of outcomes that can occur in that context. Now, as you noted, this current version of the pre-trained model is across to the full population. It's similar in some sense to doing an analytics query, if you will, against the Cosmos data set to understand outcomes of similar patients in that context, as an example.

0:27:11

We then use software capabilities to localize that to the individual's individual environment in that context and put into place their practices, et cetera, which can inform and overlay that type of additional information. We are interested in the future from a research perspective, what things like what post-training techniques and fine-tuning techniques on this model might be applicable in the context of an individual organization to also account for some of those. But there's different techniques to address that localization, if you will, of the general model that will be both deployed within the software, but then also from a training and research perspective afterwards.

0:27:56
Andrew Beam

Super helpful. I think that localization, I think, makes a ton of sense. One more question, and then I'll throw it over to Raj. My sense is that doctors often don't want predictions, that they want decisions. They want, okay, if this patient is headed towards this outcome, what should I do to steer them away from a bad outcome? And how do you think about that use case? Because my epidemiology sensors start lighting up and confounding, and how do you actually recommend an intervention for a patient, given that those decisions often require very careful analysis, very careful setup to control for things like

0:28:32
Seth Hain

confounding? This is a great and important question. And in part, it gets to how we intend to roll this out. And as I noted earlier, I talked about Cosmos in general and how the first phase of this was getting researchers access to such a large data set for writing papers, exploring how they'd use it in that context. And in a similar way with these curiosity family of models, our first phase is going to be giving researchers access to that.

0:29:08

And that includes a visualization tool so that as a researcher, you could imagine this being for a physician in the future in the exam room, you actually see these 20 trajectories that a patient might go down and commonalities across them so that you can investigate it. As I talk to physicians, we require all developers to go to see their software in action, they often think and appreciate that individual events in a patient's kind of future trajectory might send them down a different route than anticipated.

0:29:43

And they can see that through these multiple timelines that end up getting projected. Now, that doesn't fully answer the question, Andy, that you asked. I think that what we are hoping and what we're trying to encourage is that by, one, publishing this paper and getting it out to the world, and two, creating a set of tools for folks to explore it, they become more familiar with this type of evidence at the point of care, which ultimately leads along with working with HCI colleagues and others on the best ways to embed it into workflow for everyone to be able to consume.

0:30:24

Because you shouldn't have to be a researcher to have an appreciation for this in workflow. But I think there is a need for the science to progress around this and for the education to happen. And we're trying to create

0:30:35
Arjun Manrai

an environment to support that. It's super helpful, Seth. So maybe I can actually just, Before we zoom out from curiosity in Cosmos, I can just ask one follow on to one of Andy's questions. This is about localizing the model and understanding how well it does at a specific institution or how well it performs across institutions. I know it's a ton of work just to train the model right, to do what you guys did in releasing the preprint and the various incarnations from the small to the large model and so on.

0:31:06

But I'm curious if you, in the context of that paper, or if you have planned future work, if you've looked at just empirically the performance of the model across different types of institutions, across different sort of clinical settings, and if you have a sense of where and when the model tends to do well. and maybe that can be somewhat defined by institution type as one of the variables, but maybe you have another sort of natural variable you would go to, like the length of the patient's history or the duration or the frequency in which we can sample their data over the last six months or year.

0:31:41

How do you think about where the model is likely to do well and where it's likely to falter or have a kind of wider and less useful prediction for a given

0:31:50
Seth Hain

patient? No, good question. So on the specifics around localization, this is a question of deep interest to us, but not one that we have fully researched yet. So it's something under trajectory and we have capabilities within the Cosmos data set to be able to compare and contrast this across different types of institutions. I think it also gives us the opportunity to help make sure that patients, regardless of where they're getting seen in some ways growing up in rural Nebraska, I'm excited about the opportunity to use these types of models to help folks in other situations get better and better care.

0:32:34

And I think the type of question you're asking also leads to quality improvement programs that can elicit less, How does it work on different organizations? And more importantly, what are the differences that make this work really well in certain organizations versus others? And how can we share that knowledge across? One of the things we're very interested in right now, and folks that all have read the paper will have noticed this, is both expanding the model to additional populations.

0:33:03

An example being kiddos. We have the opportunity.

0:33:07
Arjun Manrai

Right now it's just adults? It's just adults.

0:33:11
Seth Hain

It's just adults. The population we trained the first model on was over 18 years old. And as we are ramping up for additional training runs, we will be including pediatrics in there as well, which I think is going to be important for a variety of reasons. We work closely with our pediatric institutions across the country and look forward to working with them on research around this. So that's one piece. The other thing that we are working on is expanding the context window.

0:33:41

For audiences not familiar with training transformer-based models, what that basically means is we want to tell more of the patient's story in regards to the prediction. And right now, we have a limited window in regards to the number of structured medical events that can be added. Obviously, we tell them up to the recent pieces. So continuing to expand that context window and then also adding in additional types of demographics data. as an example in expanding the vocabulary.

0:34:14

So I think those will be some of the key pieces you'll see us focused on in addition to obviously continuing to apply those scaling laws.

0:34:23
Arjun Manrai

And maybe as one last sort of follow-on curiosity, I think in the paper, you make a lot of comparisons as one of the baselines, both within the different sizes of the models, right? Towards this big theme around scaling. But I think very interestingly and importantly, you also compare it to supervised versions of the same setup, right? Where you try to have the same predictors and then ask how well like a supervised model would do on the same task to see if this model is learning something that those more sort of traditional approaches to the same prediction challenge would not.

0:34:53

And I think there's a few different comparisons there, different types of supervised models that you're using. I'm curious, one of the models that we've heard a lot about even before this new and interesting work was your efforts for predicting sepsis, for predicting other types of events. Did you do a direct bake-off between the epic sepsis model and this one? I don't know if I caught that in the paper. I'm curious if there's plans to do that.

0:35:17

It's very interesting because there's kind of competing philosophies here, right, for how to actually best predict the clinical outcome and what's also, even to the point of localization, what's likely to be robust enough to survive across different institutions, different ways of coding, different environments as a clinical feature that's stably represented versus something that's institutionally patterned or shortcutted at a given site. So did you do a bake-off with the Epic Sepsis model, or would you do that if you haven't done it already?

0:35:43
Seth Hain

Great question. I think it gets to a key aspect that we have yet to do with curiosity. The short answer is we have not done it yet. The longer answer is that a big part of using models, be those traditional predictive models or these generative AI models, is how you configure them into workflow and how they relate to the specifics of that organization on, say, that floor of that hospital. And so in regards to things like a deterioration index model or a sepsis model used in a production context, a large part of that involves who those predictions go to and at what points from an operational perspective, and then measuring outcomes as a result of that configuration in that context.

0:36:38

And given that we haven't deployed curiosity in any circumstance yet or in that context, we don't have a comparison point at that level. And that's where a lot of those different questions end up coming up. I want to be clear, though. It will be important as folks begin to explore curiosity and deploying it in different contexts that they do have that baseline of impact on the existing workflows. And as we do with all models that can be rolled out across the Epic suite of software, there's an opportunity to run the model silently behind the scenes on the local population in those workflows to understand those comparison points before somebody puts them into practice.

0:37:21

And so we would see that route being what we would head down in this context to answer the important question you asked, Raj. But we don't have that answer today.

0:37:29
Arjun Manrai

Got it. So maybe we could just zoom out a little bit from Curiosity and from your Cosmos work. I imagine that in your role, you're getting pitched both sort of internally and externally all the time about the greatest new generative AI product to integrate into Epic's electronic health record. And I know that there already are several efforts, right, and rollouts of generative AI products that are integrated with Epic. And so I'm going to ask you what I'll just say at the outset is an impossibly hard question.

0:37:59

You can answer whatever you could answer of it, because I don't think anyone has the answer to this question, including regulators who are trying to figure this out. But from your perch, from your perspective at Epic, how do you evaluate the trustworthiness of generative AI products that folks would like to integrate into Epic to serve your users?

0:38:19
Seth Hain

I think this question of trust or worthiness and evaluation, and in particular, the history of where we came from from a machine learning perspective, is a conversation that needs to continue to evolve. And there is a baseline set that needs to be understood around any model or any tool in regards to its accuracy in the context of how it will be used. The interesting thing that we're starting to encounter when applying generative AI into workflows is that it can do things at scale that we never thought it was worth putting time or effort into to do due to limitations of staff, etc.

0:39:09

I'll give you a very simple one from an Epic example. So we have a capability in the software to generate summaries of medical records. It generates different types of summaries in different contexts. Obviously, you want a different summary if you have a patient coming in for a physical after five years than a nurse wants at the end of a shift, as an example. And it can generate those different types of summaries. Now, no one was writing those summaries before.

0:39:38

So you didn't have a baseline of how to compare against that. And so in that regard, we've ended up working with a variety of researchers, including some folks here at the University of Wisconsin, to build out a series of evaluation techniques, including ultimately LLMs as a judge, all of which is part of what we call the AI Trust and Assurance Suite, which is an open source tool for this. And it starts to answer some of these questions you couldn't answer otherwise.

0:40:09

And I think that folks need to take that extra step back to think through what is the ground truth that you're comparing the use of these tools to when thinking through whether or not and how you need to deploy them and how to measure their accuracy and appropriateness in those contexts. It's quite a bit different than, say, historically, I reached out to patients based on ordering their A1C lab values, and now I have a predictive model for their numbers of diabetes complications.

0:40:41

How does this compare? You just don't have the same sort of baseline.

0:40:46
Andrew Beam

Awesome. I think a great discussion, and I think this is a great time to take a quick break and go to the lightning round. How's that sound, Seth?

0:40:54
Seth Hain

Sounds good. Looking forward to it.

0:41:04
Andrew Beam

awesome so uh if you haven't listened to the podcast before these are uh short and sweet questions that usually get short and sweet answers you get to decide which one to take seriously and which ones to take unseriously but we'll just we'll run through them all right let's go all right so the first one is one has become one of my favorite questions and it is what was your first job delivering papers oh nice like a newspaper boy nice Nice.

0:41:28

Cool. How old were you?

0:41:30
Seth Hain

I think I was in fourth grade. Amazing.

0:41:37
Andrew Beam

You've got all of the Americana 4-H paperboy going on for you.

0:41:41
Seth Hain

I had the bag over my shoulder and roll the papers in the basement with the rubber bands. I also simultaneously played paperboy on the old Nintendo system.

0:41:51
Andrew Beam

I was going to say that was my main thing. Running away from the dog, hitting the zombies, doing the dirt bike at the end of each level was a good time. So, yeah.

0:42:00
Arjun Manrai

Amazing. All right, Seth, next lightning round question. What's a technology you think healthcare is overestimating right now?

0:42:09
Seth Hain

Genomics. And I think in a narrow sense, it's being overestimated, but in a broad sense, it's being underestimated. And the real possibilities of kind of personalized therapeutics and where this all might be going, particularly when combined with kind of purpose-built foundational models around biology, I think has tremendous impact. But I think in some ways it has felt like genomics kind of into the layperson perspective is underperforming.

0:42:45
Andrew Beam

Nice. What is your favorite place to visit or go on a vacation?

0:42:52
Seth Hain

My family and I have been going to Portland, Maine the first week after the kids get out of school. And we just love it. And in part, it is the food and in part, it is the walks on the beach with the kids to hear about kind of what they're imagining for that summer. So Portland's a great spot for folks, but I also got to admit, I just got great memories of spending time with family there.

0:43:17

So that's immediately what comes to mind.

0:43:18
Andrew Beam

I will second the Portland main endorsement. It's a great place.

0:43:22
Arjun Manrai

Yeah. And Seth, I think that's actually where we actually first met in person. That's right. That's right. At the Rays meeting too.

0:43:27
Seth Hain

The Rays meeting was excellent. And another example of why I like Portland.

0:43:32
Arjun Manrai

Excellent. All right. The next question. Will medical AI be driven more by computer scientists or clinicians?

0:43:40
Seth Hain

The answer has to be both. I'm sorry. And I think that the reason I say that is computer scientists play a big role. The attention mechanism in transformers and how that helps understand the different parts of a patient's story in regards to predicting what comes next, right? Which is in part what the paper is about, is a computer science technique. But I think we really, really need to think through the ethics of what's being built, how patients will use it, how physicians will use it, and how that will integrate into society in a meaningful and helpful way.

0:44:22

And that needs to be led through joint conversation between deep researchers on both sides.

0:44:29
Andrew Beam

Awesome. So Epic, the Epic campus is normally described as like this Willy Wonka-esque place full of like super creative people. And it's got all this quirk to it. So I guess my next question is, what's the most unusual thing about working at Epic?

0:44:49
Seth Hain

What's the most unusual thing? You know what's strange? You mentioned the campus. one of the things that is incredibly weird is that having been here for 20 years you walk by all of this wild stuff but you you sort of it becomes normal yeah that's that's the really weird part and one of my favorite things is to bring family members here so they can give you another lens on that yep i get i get that's exactly right i get forced to see it through their eyes.

0:45:20

And it's so much fun to see stuff again for the first time. So the most unusual

0:45:27
Andrew Beam

thing is that it seems normal to you, but most people find it highly unusual. Yeah.

0:45:32
Seth Hain

For folks that haven't been here, I'll share my two favorite things on campus. One is a staircase that is a Fibonacci spiral. Nice. And then the other one, we had a deli in town that for 40 years, if it was your birthday and you were a kiddo, you would go there and you get a free ice cream cone and you get a ride on a carousel. And eventually the deli, Ella's Deli, closed. And everybody in the community wondered what would happen to the carousel.

0:46:04

You can guess where it ended up. Amazing. Amazing. Nice.

0:46:08
Arjun Manrai

Very cool. All right. Our last lightning round question, Seth. Will AI second opinions be routinely available in the EHR in the next five years? Yes. All right. I think that will happen.

0:46:21
Seth Hain

We have to figure out how and we need to work together as a community, but I think that will

0:46:25
Arjun Manrai

happen. And honestly, five years feels pretty long on like today's timescale, right? Yeah, you made that question. I feel like I may, I maybe I should have made that months, but anyway. Weeks. Yeah, weeks, seconds. In the next five seconds, Seth, you've survived the lightning round, passed it with flying colors. That was great. We actually just have a few last sort of big picture questions. And I'll start off with the first one. So we want you to sort of give us kind of both sides of the coin here.

0:46:53

And I think you have a uniquely sort of qualified position to answer this question. The first side of this is, what's the strongest case for the EHR being the portal for medical AI? And then the flip question is, what's the strongest case for the EHR not being the portal for medical AI? Yeah. I think there are multiple pieces

0:47:16
Seth Hain

to this question, but for the moment, I'll sort of take a simplified definition of medical AI. And there's sort of two key aspects for the Strongies case. One is the ability and speed of dispersion, the speed of use. It's an important piece in helping scale out the amount of care as the country that we provide. And I think that that will lead to, and we're seeing it lead to very, very rapid adoption. We have over 440 health systems using generated AI in production at this point.

0:47:47

The other thing on the bull case, if you will, on the healthcare technology side being the lead is, in Epic in particular, is around the scaling laws that we talked about with the medical event models. The scaling laws indicate to me that there is something relatively foundational in regards to a patient's story that ends up getting told and ends up creating an opportunity to ask questions in new ways and to understand those trajectories in new ways and the similarities across them.

0:48:21

And I don't think we know what that truly means. If the scaling laws didn't hold, I wouldn't be as bullish on that. I think the best case against is actually what's happening in some of the biological spaces. In particular, if we end up in a circumstance where from the principles of basic science, one is able based on labs or images or other capabilities to take essentially a point in time of a patient and create treatments, that ends up becoming a very powerful part of the way healthcare will be provided.

0:49:00

And I think that there is a possibility that both of these things end up happening. But I do think that that ends up causing a lot of focus into the direct therapeutics and personalized therapeutics.

0:49:14
Andrew Beam

Cool. Maybe just one more follow-up question. Epic seems to have a lot of things going for it in the age of AI, Cosmos being like one of the most unique healthcare data sets there are. I'm sure that you have a lot of computing power to help train models on that. I think one of the things that has changed for me having been in healthcare, machine learning, and AI for 10 years now is that there are lots of really attractive business models and products.

0:49:42

There are companies that are creating auto transcription services where it can record a patient-doctor interaction and turn that into structured data. There are companies that are creating auto billing things to help with medical coding. Is there any reason why that's not all just Epic in five years? It just feels like you guys have all of these resources where you could offer a suite of services across the entire healthcare stack that would be able to service all of these markets all in one shop.

0:50:08

So is that how Epic is thinking about what AI looks for you guys going forward, where you're kind of a one-stop shop for all things medical AI? Or are you, as Raj said, fine to be able to sort of be the base layer that people then build on?

0:50:22
Seth Hain

I think the answer ends up being both in many ways. One thing I appreciate about working here at Epic and about our culture, and you'll see this on the walls of the bathroom if you look at our commandments, which are in every bathroom across the entire campus, is a lot of those are focused on us maintaining our focus of where we can have the most impact. And the scale and breadth of what can be done with AI obviously has an impact directly in the applications we've created today and will continue to create more of in regards to helping patients and physicians take care of one another, right?

0:51:08

And do so in an efficient way throughout the health system. But there are many things like new styles of devices, direct relationships between the AI technologies and the scope, as an example, or new styles of therapeutics and diagnostic approaches that as patients, we think are very, very promising. And as a company, we think it's important to be able to work closely with. And so we see folks as an example through something we call Aura, placing orders in Epic for specialty diagnostics that come from a different company or maybe from a different type of device or a personal device that the patient will be prescribed and that information flowing back into the physician medical record to be able to drive insights.

0:51:56

And that type of structure that supports all of these different types of AI where it connects best to be at the test, be at the device, et cetera. And then how do you put it back into workflow, I think there's a role for us and there's a role for many other people in that type of

0:52:10
Andrew Beam

future. Awesome. Well, Seth, I think that sounds like a great place to end. So let us thank you for coming on the podcast today. Really enjoyed the conversation and it was really fun to learn

0:52:21
Seth Hain

about all the AI stuff happening at Epic. I really enjoyed the conversation. Hope to catch you up on the next set of papers as they come out as well sometime. Take care. Fantastic. Thank you, Seth.

0:52:31
Arjun Manrai

I think so. That was great. Bye.

0:52:34
Andrew Beam

This copyrighted podcast from the Massachusetts Medical Society may not be reproduced, distributed, or used for commercial purposes without prior written permission of the Massachusetts Medical Society. For information on reusing NEJM Group podcasts, please visit the permissions and licensing page at the NEJM website. Thank you.