OpenNashPodcasts

NEJM AI Grand Rounds

AI at the Frontlines: Dissecting Clalit’s Roadmap for the Future of Public Health with Noa Dagan and Ran Balicer

18 Sep 2024 · 1h 11m · Noa Dagan, Ran Balicer, Arjun Manrai, Andrew Beam · 24 quoted facts

▶ Play episodeOriginal source ↗

In brief

Clalit, an Israeli health fund covering more than half the country, argues that its long-running electronic records, a talented data team and a repeatable way of putting predictions into primary care let it change outcomes at scale. Its leaders say the future of public health is predictive: use models to find who needs screening or treatment, and match the simplest effective tool, not the newest technology, to each clinical need.

The central arc runs from research to deployment. Ran Balicer and Noa Dagan describe moving from publications to a platform that feeds risk predictions to over a thousand family doctors. COVID-19 tested this system, producing fast vaccine studies and a rapid risk model. The hepatitis C screening study shows the payoff. The tension is between the hype around large language models and the plain machine learning that mostly does the work.

Key insights

  1. Clalit's edge is talent more than data

    Ran Balicer says Israel's 1994 health law pushed the health funds to digitize early, giving Clalit about 15 years of linked records by 2010. He concluded that talent mattered more after his team won the NEJM SPRINT data challenge, where everyone used the same data.

  2. A reusable platform beats one-off model deployments

    Clalit built CPI, a system that runs prediction models and decision support across many diseases and data types. Dagan says the early extra effort pays off because each new deployment reuses the same pipeline, and it now reaches over 1,500 primary care doctors.

  3. Targeted screening found hepatitis C 100 times more efficiently

    Routine screening of about 50,000 untested people finds roughly 38 cases a year. A machine learning risk model found 38 additional cases among fewer than 500 high-risk people screened, so far fewer tests were needed per case found.

  4. Careful twin matching produced early vaccine effectiveness estimates

    The team paired each vaccinated person with an unvaccinated lookalike and followed both, emulating a randomized trial with observational data. When a twin got vaccinated, the pair was split and a new twin was found. This gave an estimate by February 2021 that later swab data supported.

  5. Use the simplest model that meets the clinical need

    Both guests say they start from the clinical problem and pick from a toolbox running from points-based scores and logistic regression to foundation models. Balicer says plain machine learning, not large language models, is the right answer in about 99% of cases.

Questions

What is Clalit and why is its data considered so valuable?

Clalit is one of Israel's four national health funds. Ran Balicer says its electronic records link primary care, specialist care, hospital care and claims for millions of people, going back to the mid-1990s.

How did machine learning improve hepatitis C screening?

Normal screening of about 50,000 previously untested people finds roughly 38 cases a year. A model that ranked members by risk found 38 more cases after testing fewer than 500 people, which Balicer calls a 100-fold improvement.

What is the CPI platform in plain terms?

CPI stands for Clalit Proactive Preventive Interventions. Noa Dagan says it runs prediction models over patient records to flag who needs attention, then gives family doctors specific advice on what to do.

How did Clalit estimate vaccine effectiveness so quickly during COVID-19?

Balicer describes a 'target trial' approach, which mimics a randomized trial using records. Each vaccinated person was matched to a similar unvaccinated 'digital twin' and followed, giving an early estimate by February 2021.

Do the guests think large language models will replace doctors?

No. Both say large language models are helper tools that could reduce paperwork, and they expect image-analysis models to affect fields like radiology before language models do.

Facts

Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.

The 1997 Hong Kong avian influenza outbreak, according to Ran Balicer, sickened 18 people and killed about one third of them.

“It was an event in which 18 people got sick with avian flu, and one third of them actually died from the virus.”

0:07:42 · Ran Balicer · Avian flu · 1997

After Ran Balicer presented his cost analysis to the Israeli cabinet, the government decided within about ten minutes to buy roughly $100 million of antiviral drugs for a possible flu pandemic.

“10 minutes later, they make a decision to buy drugs in $100 million.”

0:09:14 · Ran Balicer · Antiviral stockpile · 2002

Israel passed the National Health Insurance Law in 1994.

“Israel had a landmark piece of legislation put forward in 1994, the National Health Insurance Law.”

0:20:08 · Ran Balicer · Health insurance law · 1994

According to Ran Balicer, the 1994 law made Israel's four health funds responsible for all their members' medical needs, paid nationally through capitation.

“make the four health funds to become responsible end-to-end for the life of their patients and for all their medical needs throughout their life and paid nationally through capitation.”

0:20:08 · Ran Balicer · Health funds · 1994

By about 2010, Clalit had roughly 15 years of linked longitudinal records covering primary, specialty and hospital care plus claims and provider data.

“we already had like 15 years of longitudinal data, end-to-end, primary care, specialty care, hospital care, claims, and provider data.”

0:20:35 · Ran Balicer · Clalit data · 2010

Ran Balicer says the Clalit Research Institute grew from four people to 70 within a few years of the mid-2010s.

“Because in a few years, we grew from four people to 70.”

0:21:58 · Ran Balicer · Research institute · 2014

Ran Balicer says a Clalit program targeting its lowest-performing, less affluent clinics cut some key health-quality gaps by 60% within three years.

“within three years of an intervention, large scale, we were able to show a 60 % reduction in some of the key measures.”

0:22:28 · Ran Balicer · Health disparities

Ran Balicer says another Clalit program reduced hospital admission rates among patients with multiple chronic conditions by 43%.

“On another program, we were able to reduce, for instance, admission rates on multimorbid patients by 43%.”

0:23:18 · Ran Balicer · Hospital admissions

The Clalit team led by Noa Dagan won first place in the New England Journal of Medicine's SPRINT data challenge in 2017.

“I couldn't have been more proud when the New England Journal of Medicine chose our solution as their first place winner.”

0:24:33 · Ran Balicer · SPRINT Challenge · 2017

A collaboration between Clalit and Harvard's Department of Biomedical Informatics, the Clalit and DBMI Berkowitz Living Lab, was funded by a gift from the Berkowitz family.

“called the CLALIT and DBMI Berkowitz Living Lab, made possible due to a gift given by the Berkowitz family.”

0:16:23 · Noa Dagan · Berkowitz Living Lab

Noa Dagan says Clalit's research institute decided around 2018 to become an innovation center focused on putting models into care.

“Maybe I think that the transformation point was around 2018, when we made this decision to transform from a research institute to an innovation center.”

0:26:32 · Noa Dagan · Innovation center · 2018

CPI, Clalit's in-house platform, stands for Clalit Proactive Preventive Interventions and both hosts prediction models and provides decision support to doctors.

“So SIPA is a platform that stands for Clalit Proactive Preventive Interventions, and it has two major roles.”

0:30:05 · Noa Dagan · CPI platform

Noa Dagan cites a published estimate that primary care physicians would need 26.7 hours a day to deliver all guideline-recommended care.

“if primary care physicians want to do everything by the book, including chronic care, acute care, preventive care, they need to work only 26.7 hours per day.”

0:30:33 · Noa Dagan · Physician workload

Ran Balicer says Israel vaccinated about 2 million people in roughly three weeks early in its COVID-19 vaccination campaign.

“You know, 2 million people in about three weeks.”

0:37:03 · Ran Balicer · COVID-19 vaccination · 2021

Clalit's COVID-19 vaccine studies matched each vaccinated person to an otherwise identical unvaccinated person and followed the pairs, emulating a randomized trial.

“Every vaccinee got somebody who was absolutely identical to them, except that they weren't vaccinated at the time, and we followed them like it was a trial.”

0:38:28 · Ran Balicer · Target trial emulation · 2021

Ran Balicer says Clalit produced the first accurate estimate of real-world COVID-19 vaccine effectiveness by February 2021 and published it in the New England Journal of Medicine.

“we had by February 2021, the world's first very accurate estimate of vaccine effectiveness.”

0:38:28 · Ran Balicer · Vaccine effectiveness · 2021

Ran Balicer was appointed chair of the Israeli government's national advisory team on pandemic response during COVID-19.

“I was appointed as the government's chair of the national advisory team on pandemic response for Israel.”

0:39:04 · Ran Balicer · Pandemic advisory · 2020

Noa Dagan says the multicalibration fairness algorithm Clalit used to adapt its flu model for COVID-19 came from scientists including Guy Rothblum of the Weizmann Institute and Omer Reingold of Stanford.

“We actually knew this algorithm from our work with scientists like Guy Rothlund from the Weizmann Institute and Omer Engel from Stanford, if you know these names.”

0:43:37 · Noa Dagan · Multicalibration · 2020

Within two weeks in March 2020, about 200,000 Clalit patients received phone calls telling them they were predicted to be at high risk from COVID-19.

“And within two weeks, 200 ,000 patients in Clalit received personal phone calls from their clinics, telling them that they are at a predicted high risk”

0:44:11 · Noa Dagan · COVID-19 risk calls · 2020

Ran Balicer says about one in three people with chronic hepatitis C eventually develop cirrhosis after decades of silently carrying the virus.

“in a few decades of silently carrying the virus, one in three will end up with cirrhosis”

0:46:50 · Ran Balicer · Hepatitis C

Ran Balicer says modern hepatitis C treatment was 98.8% effective in one of Clalit's own earlier studies.

“Okay. It's 98.8 effective. That's according to one of our own past studies.”

0:46:50 · Ran Balicer · Hepatitis C treatment

Ran Balicer says routine screening is inadequate for the World Health Organization's goal of eliminating hepatitis C by 2030.

“But it's completely inadequate if we want to reach elimination of this disease by 2030 as the WHO set as a goal.”

0:47:18 · Ran Balicer · Hepatitis C elimination · 2030

In Clalit's prospective study, screening fewer than 500 people ranked highest-risk by a machine learning model found 38 additional hepatitis C patients.

“when we proactively screened less than 500 individuals at top risk, among them, we found 38 additional HCV patients.”

0:00:03 · Ran Balicer · Hepatitis C screening

Ran Balicer says Clalit tests about 50,000 previously untested people for hepatitis C each year under international recommendations, finding around 38 cases.

“So every year, Clalete follows the international recommendations and tests 50 ,000 previously untested people.”

0:47:18 · Ran Balicer · Hepatitis C screening

Chapters

  1. 0:07:07Ran Balicer's route into public health

    Balicer describes starting as a student web entrepreneur, then persuading the Israeli cabinet to stock antiviral drugs. This shows where his confidence in making large system changes comes from.

    “I was a public health resident by accident.”
  2. 0:11:02Noa Dagan's chance-driven path to predictive modeling

    Dagan explains how a high school computer science push and a master's degree during a pregnancy bed rest led her to predictive modeling and to Clalit.

    “it was mostly due to chance”
  3. 0:20:08Why Clalit's data exists and how the institute grew

    Israel's 1994 health insurance law pushed the health funds to digitize early, and Balicer built a research team around that data. He credits the law, not his team, for the data itself.

    “we cannot be credited for creating the data sets”
  4. 0:28:54CPI: moving predictions into primary care

    Dagan explains the decision to focus on community care and to build one reusable system for delivering model outputs to doctors. This is the core of how Clalit implements AI.

    “create a systematic process to inject data-driven insights into the point of care”
  5. 0:38:28COVID-19: matched twins and a rapid risk model

    The guests describe the vaccine effectiveness method and the fast risk model built from a flu model. Both show how the institute worked under pressure.

    “It was to create digital twins, if you'd like.”
  6. 0:42:33Repurposing a flu model for COVID-19 risk

    With no COVID-19 outcome data yet, Dagan's team adapted an existing flu model so that high-risk patients could be called within weeks.

    “ended up actually using a flu prediction model that we developed several years before”
  7. 0:47:55Hepatitis C: a 100-fold screening gain

    This study is the episode's main example of predictive public health: far fewer tests were needed to find each hidden case.

    “That's a 100-fold improvement.”
  8. 0:50:53What counts as AI?

    Dagan argues AI is a toolbox from simple scoring models to language models, and both guests say to choose the simplest tool that meets the clinical need.

    “people forget that AI is a toolbox”
  9. 0:56:38AI and the future of public health

    Dagan counters the host's worry about misinformation, arguing that prediction lets health systems prevent illness instead of reacting. She gives examples at the primary, secondary and tertiary prevention levels.

    “now we can really stop being reactive and start being proactive and preventive about it”
  10. 1:08:32How other health systems could copy Clalit

    The guests say structures like incentives and data availability differ by country, but any system can train people fluent in AI, medicine and public health.

    “you need people who speak three languages, AI, medicine, and public health and epidemiology”

Transcript

0:00:03
Ran Balicer

What if we could use Clalit's insanely wide database to create a machine learning algorithm that would identify the highest risk members of Clalit and will begin screening of those top risk individuals? So we did exactly that and the results show that when we proactively screened less than 500 individuals at top risk. Among them, we found 38 additional HCV patients. So 38 out of less than 500 versus 38 out of more than 50 ,000, that's a 100-fold improvement.

0:00:46

I haven't done many things in my life that have shown 100-fold improvement. And for us, I think this study symbolizes the paradigm shift from the classic public health towards predictive care. So this is what modern day public health could and should look like in the age of AI.

0:01:15
Arjun Manrai

Welcome to another episode of NEJM AI Grand Rounds. I'm Raj Manrai. I'm here with my co-host Andy Beam, and we're delighted to bring you our conversation with Drs. Noa Dagan and Ran Balicer. Noah and Ron are at Clalit Research Institute in Israel. Andy, this was a really interesting conversation. You know, Noah and Ron are both very high-impact researchers, and they've used this truly unique data set, which spans, I believe, more than half of the country of Israel to publish these groundbreaking studies on COVID and many other topics.

0:01:45

But they're also very advanced in implementing predictive models at the point of care to improve care. They've been doing this for years. So things we talk about as hypotheticals and as viewpoints, they have hard data on for more than a decade. All in all, this was really fun and educational. I agree, Raj. And

0:02:04
Andrew Beam

I think this episode hopefully corrected what has been like an overly US-centric bias that we've had on the podcast. I think seeing their work not only on model development, but as you said, also and implementation, they're doing the whole clinical translational pipeline. They have this amazing data set that few others have. They did groundbreaking work during COVID and are really, like, I think at the forefront of clinical informatics in a way that few other people are.

0:02:30

So it was a great conversation. They're both like super energetic and great to talk to. They were even kind enough to humor me with a somewhat tangent conversation on what constitutes AI and what's the distinction between machine learning and statistics. And so all around an extremely enjoyable conversation, and we're excited to share it with the listeners. The NEJM AI Grand Rounds podcast is brought to you by Microsoft, Viz.ai, Lyric, and Elevent Health. We thank them for their support.

0:03:07

And with that, we bring you our conversation with Noa Dagan and Ran Balicer. Noah and Ron, thank you for joining us on NEJM AI Grand Rounds today. We're excited to have you here.

0:03:18
Arjun Manrai

Great to be here. Absolutely. Noah and Ron, this is a question that we like to always get started with on this podcast. So could you tell us about the training procedure for your own neural network? How did you get interested in AI and what data and experiences led you to where you are today? And maybe, Ron, we can start with you and then go to Noah.

0:03:39
Ran Balicer

Sure. It's really great to be here today, Raj. I think my professional career was shaped by kind of constant parallel work in different planes. There were always several streams progressing unsynchronized side by side. And I'm talking about medicine, research and data. I think that the best way to describe how my neural net was shaped is through two specific events that are, I think, still fairly vivid in my memory. And I'll share them with you. The first one, I think, is when I was a medical student.

0:04:13

At that time, Internet just became a thing. I don't know if you remember that time. In parallel to med school, which was a very humbling experience because I was taught by amazing scholars, I began having a second part of life as an internet evangelist. So I became an expert in searching medical literature and information on the web, which was not as intuitive as it is today. But suddenly, I was bringing something else to the daily rounds of the surgical department, a skill that even the feared head of surgery, I'm sure you have one in your memory from your own training, even he couldn't match.

0:04:51

So for me, that was truly transformative. I started making my living as a student from teaching doctors to adapt to the new era. I wrote my first book about internet and medicine and how those two can actually intermix. And then I decided to take it one notch further. And I taught myself how to code HTML. And I created Israel's first web portal for doctors. And eventually I sold it to an internet company. So that was my first entrepreneurial act.

0:05:20

So what this pre-training, if you'd like, what it did to my neural net was the understanding that there's a secret door out of the rat race. That by elevating technology and data, you're no longer bound to your rank in the extremely hierarchical medical letter. So I think that And that and that the fact that information technology is going to basically change everything. So this became crystal clear to me. And from that point on, what I try to strive is to always be the earliest adopter, to take an active role always in trying to kind of use tech to change practice.

0:06:00

So this is one core memory.

0:06:01
Andrew Beam

I was wondering if you could also take us back a little further to your initial conditions and like, how did you even get interested in medicine in the first place? Where did you acquire the background to be able to do this internet evangelism? And can you take us back a little further?

0:06:15
Ran Balicer

Sure. So I think going to med school was a decision made when I was too young to make such difficult decisions. So I was kind of young and stupid and made decisions based on what seems exciting. And medicine seemed the most exciting thing on the university booklet. So I took that one. And as time went by, I think I began understanding better that I was actually fortunate enough to have my calling. And it wasn't that early on.

0:06:42

In terms of my understanding of internet, it was all self-taught. You know, when I had to teach HTML, I went to a guidebook on the internet called The Bare Bones Guide to HTML. And I kind of memorized it and started working step by step. This was all self-taught capacity. I didn't go through formal training, although as a kid, I used to have a computer early on from the age of five. And I actually started programming before I knew English.

0:07:07

So it was kind of funny because I did it word by word. So, yeah, these are all kind of older memories. And when I graduated med school and went into my residency, that's where my second core memory kicks in. I was a public health resident by accident. No, I'm serious. I was sent to have a job interview, and I accidentally knocked on the wrong door. And the guy in that room was curious enough to tell me to come in and sit down and start talking, and the rest is history.

0:07:42

That's how I started my public health residency. So here I am, a young public health resident. And as a resident, I was reading about the 1997 Hong Kong avian influenza outbreak. I don't know if you remember that event. It was an event in which 18 people got sick with avian flu, and one third of them actually died from the virus. It was a really bad bug. I couldn't grasp why no one was talking about this risk of this becoming a pandemic.

0:08:06

And nobody was talking about it. 2002, I think it was. But by then, I already had some MPH training, and I went to play with the numbers. And it seemed like it would be a great economic gain if one would stockpile antiviral drugs for this event, even if a flu pandemic would happen only once every 33 years, which is actually three in a century, which is what you would expect. So I showed this Excel sheet to my mentor.

0:08:30

My mentor at that time is my friend to this day, Professor Itamar Groto. He then became Deputy Director General of the Ministry of Health. And, you know, Itamar said, you know, this is quite convincing. Let's show it to the Surgeon General. So we went to the Surgeon General and says, you know what, that's pretty interesting. Let's show it to the Ministry of Health, to the Director General. So the Director General of the Ministry of Health at that time, Professor Avi Israel, a scholar in public health, he says, you know what?

0:08:51

This is serious. I want to show this to the treasury. So then a few months later, I find myself trembling at a cabinet meeting, the Israeli cabinet meeting with the prime minister. And the prime minister is questioning the minister of health saying, so why do you want to buy drugs in 300 million shekels, in $100 million? And they're trying to explain and they're kind of saying, you know what? Get up and explain the Excel sheet to him.

0:09:14

So I stand up there, young and frightened, and I start talking and he's cross-examining me and yada, yada, yada. 10 minutes later, they make a decision to buy drugs in $100 million. So if you ask what was my take from that specific event, I think it was that with the right scientific liver and the right mentor that supports you and enough perseverance, no matter your rank, your age, you can have a large scale impact. And you don't be afraid to go all the way without you believing.

0:09:45

Don't wait for somebody else to do it. Just be the guy. So when I came to Clalit, I had a strong sense of self-efficacy. I knew you could make a huge systemic change, use mathematical modeling, common sense, and what we call here chutzpah. That's audacity. So just go ahead and do it. And most importantly, I think you need to have someone who believes in you and pushes you forward. So I set up to create a team.

0:10:09

And that's where our story begins, I think.

0:10:13
Arjun Manrai

Ron, that was amazing. That was very, very inspiring. And now I feel like I can kind of go take over everything and I have no excuses to realize dreams. I totally agree. I think you said something at the end there too, which is having a mentor and the right scientific lever to sort of path break and to encourage you and to help you along the journey, I think is very, very critical. And I think multiple of us on this call also have Zach Kohani.

0:10:37

So we have someone in our corner as well. Noah, maybe we can turn to you now. So I'll just repeat the question here. So this is a question that we ask all of the guests, which is, can you tell us about the trading procedure for your own neural network? Maybe you can emphasize this. You could be broad here. How did you get interested in AI and predictive modeling? You know, what led you to the work that you're doing?

0:10:58

What data and experiences brought you to where you are today?

0:11:02
Noa Dagan

Sure. So I really wish I could answer your question with like a big strategic focused decision or a plan, but that was not the case at all. And to be honest, it was mostly due to chance. So maybe in the neural network analogy, it was all in the number in the set seed command in the beginning of the process, which was for me a good number because I'm pretty happy with where I am today. But it was a lot of randomness throughout the process.

0:11:30

So I think it begins with me being a nerd, but not a computer geek kind of nerd at all. I was not into gaming or anything like that, but just the kind of nerd that does what she's told. I think it was my high school days where the high school teachers and headmaster really tried to encourage female students to major in computer science in Porgame, because back then more than 95 % of the students in those classes were males.

0:11:57

So they really, really encouraged female students to pursue these kinds of majors. And I was just, to this day, I don't know how to say no. So I just said yes, and I went into it. And I kind of liked it. So I'm still, to this day, not the computer geek kind of nerd. But I really liked the elegance and aesthetics of writing code, of solving algorithmic riddles, these kinds of things. And so I really had no intention of ever using these skills, but I liked it.

0:12:27

I had fun and I went through my high school doing that, knowing what I really wanted to do is being a doctor. So I think that med school later on was probably the only part of this training process that was not due to chance. I really, this was my lifelong dream to be a physician ever since I can remember myself. And then I was in med school and then chance intervened again. I think it was somewhere in the middle of the clinical years when I was pregnant with my first son.

0:12:59

And I was put, towards the end of the pregnancy, I was put on bed rest. And the length of that bed rest was long enough that it had to translate into me being one year delayed compared to the rest of my class. And I had to, like, skip a year in med school. And I didn't want to completely waste that time and completely waste that year. So I did what everyone else who for any reason had to wait a year during medical school where I studied, which was to do an MPH, a master's in pelvic health.

0:13:28

So I pursued an MPH in that year. And during that time, I actually heard a lecture by a professor visiting from Boston Children's and describing the kind of work he does in predictive modeling. So you can probably guess it was Ben Rice. And it was the first time I've ever met him. And I was completely hooked.

0:13:47
Arjun Manrai

like it was around 2009. Okay, yeah, you just answered it. My question was when I just wanted

0:13:52
Noa Dagan

to index this. So this is 2009. Yeah, so this is and predictive modeling, these kinds of things were not a thing yet. No one talked about AI yet. But it looked, it sounds like science fiction to me. And it was really, really interesting. So I actually went and asked him to be my advisor for the MPH thesis. And this is how I got to know him. And we actually did that thesis together. And then a few years ahead, I completed med school.

0:14:21

I completed my internship. And I was really still, I was sure I wanted to be a physician. But throughout these clinical years, throughout the internship year, I couldn't find a single medical residency that I thought would make me happy to do on a daily basis for the rest of my professional life. And I still think to this day, this is the most noble and interesting and beautiful profession. but something about how medicine is practiced wasn't making me as happy as I thought it would.

0:14:52

So now it was around 2014, and I contacted Ben again. And I asked him, listen, do you know of somewhere in Israel where I can combine my computer science skill, my programming skills with my medical knowledge? And he actually introduced me to Ran. So that's how I met Ran. And Ran was very impressive to this day. He's very impressive. and it was clear that they were doing really interesting research in what was then the Clinic Research Institute, but something about working in a health organization in a basement of a clinic that was more than 100 years old, so you can imagine how it looked like, they actually had a pet mouse in that basement.

0:15:31

So every once in a while, a mouse strolled the corridors and they actually named him and it was basically then when I said, listen, this is really, it looks really, really important and interesting, but I feel that maybe a medical startup will be like a cooler environment to be in. And I ended up finding this kind of startup. I actually negotiated the contract. I was pretty happy. And like for the courtesy of it, I called Ron to say, I'm sorry, but I'm taking the other job.

0:16:00

And to this date, I'm not exactly sure what he told me and what went on in this conversation. but basically I remember one sentence. Ron, I don't know if you remember this call, but he said something like, I can't really explain it in a way that will convince you, but if you'll come, in a few months, you'll know I was right. Again, I don't know what happened there, but I hung on the phone and took the job.

0:16:23

And fast forward 10 years later, I've been working in clinic almost all the time ever since. And I completed my public health residency there and my PhD in computer science. I also did a postdoc in DBMI in the Department of Biomedical Informatics in Harvard with Zach Kohane as my mentor and also Ben Rice. They were my two advisors and I think two true mentors to this day. And this thing was actually part of a pretty beautiful collaboration between CLALIT and DBMI, called the CLALIT and DBMI Berkowitz Living Lab, made possible due to a gift given by the Berkowitz family.

0:17:01

And basically, I think that's how my neural network was trained by mistake. And I think I can also admit by now that, Ron, I think you were right back then in that conversation. So that's the answer for the neural network part of it.

0:17:19
Arjun Manrai

Noah, that was amazing. And I think that's a great point to actually transition to this next part of the discussion. We really want to dive into the work at Clalit and the work that you've done for more than a decade now, including the work that you've led on COVID-19 and on many other topics for the past few years. So I am very privileged, actually, in my own sort of position as academic and running a lab at DBMI at Harvard Medical School that I get to work with both of you.

0:17:45

And so I've been working with you both very closely, and I've really gotten to appreciate how special Clalete's research environment is and how really amazing both the methodologists and the clinicians are and how special your data is. And I can't really think of another example anywhere in the world that is quite like what you've accomplished in terms of just assembling both this amazing team and really investing, I think very early on, and then sustaining that investment in creating this very special data set.

0:18:19

So I'm probably going to get some of the numbers wrong here. I know at least from the projects that we are working on together, that this data spans at least two decades. and it is longitudinal individual level data for millions of patients that represents something like half or more than half of the country of Israel. And this is a dream data set to do many things, to do many epidemiological investigations, but also to ask not only questions about the prevalence and incidence of disease and risk factors and understand those things temporally in this population, but also to even ask nuanced causal inference questions about how effective different drugs or different interventions like vaccines during COVID-19 were and are.

0:19:07

And so that's really maybe where I want to start, and then we can branch off. But what really caught my attention and really made me aware of some of the work that you all are doing and how special this resource is, is the work that you did on COVID-19. This is from 2020 all the way through the next several years. Actually, I'm going to just interrupt myself. Before we talk about that, maybe we can just talk about the circumstances that led you to sort of invest in Clalit.

0:19:35

I think, Ron, you hinted at it. One of the things I learned from both what you said in your opening remarks and Noah's is that you're very persuasive and you're able to accomplish a lot of that. So I'm guessing that that has something to do with how we were able to establish this from the beginning. But maybe you can just tell us a little bit about, you know, before we jump into some of the work that you've done recently, the founding of Clalit Research Institute, what those sort of initial conditions were, and how you got it off the ground and were able to sort of sustain this investment and all the data and the research that you guys are doing.

0:20:06

Sure. So as the setting the

0:20:08
Ran Balicer

ground, I have to say that we cannot be credited for creating the data sets. Okay, we're standing at the shoulders of giants. Basically, I think Israel had a landmark piece of legislation put forward in 1994, the National Health Insurance Law. They did a lot of good things. One of them is make the four health funds to become responsible end-to-end for the life of their patients and for all their medical needs throughout their life and paid nationally through capitation.

0:20:35

So this created really a very good, sound, well-motivated system for all of us. And probably because of that, all of the sick funds decided to go digital pretty quickly. And in digital, not only in their administrative, but also to have something that was unimaginable at that time, electronical health records, mid-90s. And so they went ahead and did all of this. And now, when we went into our endeavor, when we're talking about 2010, we already had like 15 years of longitudinal data, end-to-end, primary care, specialty care, hospital care, claims, and provider data.

0:21:12

So that was truly unparalleled. But Raj, you asked about investment in data, but I think what was even more important was the decision to invest in data science talent. And so at that time in 2010, I was already a few years at Clalit in a policy role. So I decided to kind of put all of those roles aside and established the Clalit Research Institute. And the vision was pretty simple. If we take the best data in the world and we managed to couple it with the best talent Israel could offer will have a leverage point to move mountains, probably on global scale.

0:21:45

And that's probably why I didn't let Noah slip away in that decisive call on Friday, 2014. So, you know, within...

0:21:54
Noa Dagan

So you do remember. Vaguely.

0:21:58
Ran Balicer

Vaguely. But I will say that, you know, it was quite, I think, a pivotal point at the time. Because in a few years, we grew from four people to 70. and this group of amazing young professionals had both the data skills, but also kind of an in-depth understanding of the clinical needs, the kind that you can only get when you're working within a healthcare organization. It's really difficult to do that from an academic position when completely detached from the needs, evolving needs of the organization.

0:22:28

So, you know, early on, we did a lot of cool things with the data. We were changing policy and we could already see measurable impact. We decided to go and reduce health disparities, for instance. So between affluent and less affluent clinics. So within three years of an intervention, large scale, we were able to show a 60 % reduction in some of the key measures.

0:22:47
Arjun Manrai

What was the intervention?

0:22:49
Ran Balicer

So the concept of that intervention was to identify the least successful clinics in achieving their health quality goals, especially with those that were associated with low socioeconomic status, and then go ahead and tell each one of the district heads that they could not just put them under the rug. There was a special focus from the director general on those clinics, least affluent populations with the most difficult life conditions. And people say, you can't change it. You know, the people don't do what we ask of them and it's really difficult.

0:23:18

And at that point we said, no, it's just a matter of determination. And if you want to increase the averages, don't go for the highest achievers and improve them. Go ahead for the lowest achievers and bring them closer to the average. And that kind of approach within three years and, you know, CEO level intervention, 60 % reduction was absolutely amazing. On another program, we were able to reduce, for instance, admission rates on multimorbid patients by 43%.

0:23:43

So, you know, amazing things could happen. And we started collaborating, became a collaborating center for the WHO. So we had a lot of global interest. And at that point, I was asking myself, you know, what is it that's happening here? Is it data or is it talent? And I can tell you when the answer to this question became crystal clear to me. It was in 2017, when actually the New England Journal of Medicine had the global competition called the Sprint Challenge.

0:24:10

I don't know if you guys remember that. They didn't allow scientists to bring their own data, just the skills. The journal brought the data and created an even playing field and everybody had to compete. And at that point, NOAA led our small team. And we created a new way of supporting data-driven individual treatment decisions for, in that case, hypertension. because that's what Sprint is about. With lots of innovation in there, we had new ways of calculating individual number needed to treat.

0:24:33

We had new types of patient involvement interface. And to cut a long story short, I couldn't have been more proud when the New England Journal of Medicine chose our solution as their first place winner. So at that point, I said, yes, it's clear. It's not the data. It's the talent. First and foremost, that's what makes the difference.

0:24:52
Arjun Manrai

Amazing. Amazing. Noah, maybe we can move to some of the work that you're doing in implementing predictive modeling in real clinical care. And what really strikes me is that everyone is talking about this now with AI, large language models, with ChatGPT and its cousins as potential agents to aid in diagnosis. But I think long before this sort of current LLM era, and even before the, let's say, various kind of investments and interest from the broad community in AI, even let's say dating back to 2012 with ImageNet and convolutional neural nets, I think you all were implementing predictive models in real ways that really were changing care, were changing outreach, were changing the way that you delivered information to providers to

0:25:37

reach patients most in need. And I think Ron just hinted at some of that with the sort of district level interventions and reaching out to the less affluent clinics and areas. But maybe you can tell us a little bit about how you see the work evolving and what has led to success in implementing predictive modeling in real clinical care long

0:25:56
Noa Dagan

before our current AI wave that we're in now. Sure. So I think, first of all, Ron and the team, even before I joined, created the first prediction models, I think even maybe 2010. And I think it was more like research focused then. And at some point, you ask what evolved us. I think the decision at some point to transform from research to actual implementation at scale. And the decision that we really want to make an impact, not only through the publications, but through actually transforming care.

0:26:32

So we were actually convinced that it is possible. We were, I think, convinced that data is the key to do that, but we still had limited resources. Maybe I think that the transformation point was around 2018, when we made this decision to transform from a research institute to an innovation center. We had hard choices to make because the resources were still very limited. So we still sat in the same basement. As Ron said, with really great people, I actually think the basement was like a good selection bias for really good people because people who agreed to come and sit in that basement really, really believed in the cause.

0:27:11

So we...

0:27:13
Arjun Manrai

Mission-driven. Mission-driven.

0:27:15
Noa Dagan

Exactly. Today we have much nicer offices, but back then it was like there was a comradity, like where we are, what we do, and we really try to make something that will be influential. So I think the first major decision that we've made was to focus on community care because Because we had 14 hospitals within the clinic network. We could do a lot of innovation work in the hospitals, but maybe it's kind of a surprising choice because most of the innovation that we know today comes out of hospitals.

0:27:48

But for us, we thought that the true clinical impact will come from outpatient care. Because this is where the real medicine should happen. If you want to keep patients healthy, this is where you want to intervene before they go to the hospitals before, let's say for chronic conditions, every admission should be considered as some sort of system failure that is preventable. So if you want to use prediction models at scale, go to that setting where you have millions of patients and you need to choose from those patients who are those patients at risk that you want to intervene with, that you really want to change the course of their disease.

0:28:25

And our first major decision, I think, was to focus on community care setting. And I think that the second major decision back then, which made the difference compared to what we did before that, was to do things in a way that will avoid reinventing the wheel every time for every deployment project. So, like implementing things, deploying things, it's so hard as it is in medicine. There are so many things that make it hard. So, you never want to do the same thing twice.

0:28:54

And we made this decision that we want to really try to use every project and to make sure that every advancement for a specific deployment really paves the way for deploying the next solution. And it's not an obvious decision because sometimes you want to have quick successes and you want to make things happen quickly, but if you really want to do it at scale you have to do it systematically. And those two decisions to focus on community care and to do things systematically led to the decision or to the understanding that what we really need is to create a systematic process to inject data-driven insights into the point of care, specifically primary care, in a way that will be

0:29:36

agnostic to the specific medical domain that will work if it's osteoporosis or cardiovascular or hepatitis C or whatever, and will be agnostic to the type of data, whether it's structured data or images or text or monitoring data, and to the type of model. So machine learning, LLMs, whatever. We didn't think about LLMs back then, but the strategic decision was to basically create something that is agnostic to where the insight comes from and how it was created.

0:30:05

We will have a way to inject it into the point of care. And this is a thing where the true change happened. And this is how a platform we call CPI was born. So SIPA is a platform that stands for Clalit Proactive Preventive Interventions, and it has two major roles. One is to be the house for all of the prediction models we can imagine. So it has a proactive role of identifying who are those patients we want to focus on for preventive care.

0:30:33

I don't know if you know this article that came out a couple of years ago that really neatly showed what we knew for years is the truth, but it really quantified it in a really a nice manner that says that if primary care physicians want to do everything by the book, including chronic care, acute care, preventive care, they need to work only 26.7 hours per day. And so, of course, it's not possible and it's unavoidable that people will go unnoticed for some of the care that they should get.

0:31:03

So these prediction models basically create a really smart spotlight on the right patients for the right conditions. And now we have a place to put them, to put them all and to create this population-based view of who you should focus on for each clinical domain. And the second thing that CPI does is after we focus on a specific patient, because we identify that this patient is at a specifically high risk, we create a decision support of what to do with this patient in order to provide that patient with the best care.

0:31:33

And again, it comes to solve a problem where I think we all now understand that medicine is just becoming super complex and decision-making is becoming really, really hard. In the 90s, let's say we had less than 10 drugs to treat diabetes. Now we have dozens. And for each one of these drugs, you need to know what is the right patients, what is the relative indications and relative contraindications. And it's hard and it's hard not to make any mistakes and not to miss anyone.

0:32:05

So the second thing we do with SIPA is basically to create a very detailed, very individualized, very actionable decision support as to what do we need to do with these patients for numerous clinical domains to the level of an expert clinician consultation. So imagine that every night we have experts, automatic experts, running through the HRs of all the patients in Clalit, identifying all the gaps in care and how we should fix them. And this is, I don't know how to convey the complexity of it, but maybe numbers will help.

0:32:39

So it is to the level of hundreds of clinical features that go into each such clinical pathway. And the number of recommendations that are being created by these pathways are hundreds to thousands of permutations of what is the most accurate recommendations to give to that specific patient. So this platform is being developed in-house by Krelit since then, that strategic decision, plus the amount of years that it took to convince everyone in the system that we need to put money and effort into creating that.

0:33:13

And I think we really succeeded in creating that vision of an interface that is agnostic to anything, to whatever clinical insight you will bring, we'll have somewhere to put it and to deploy it into care. And now we have many domains in that system for numerous clinical chronic conditions, also infection agents, different things that were chosen to be at high degree of priority for us to treat our patients better. And it's currently deployed for, I think today, more than 1 ,500 PCPs.

0:33:45

So I think it is really the way to do things at scale, not to reinvent the wheel for any kind of insight or to any kind of clinical domain that you're interested in, but thinking systematically, even if it takes longer, the effort will pay itself later on.

0:34:04
Arjun Manrai

That's really great. And I think that's a great summary too, and way to think about how to have impact here. So we want to switch topics and switch gears a little bit from integrating predictive modeling into clinical care to some of the research that you've published, the academic research that you've published over the past few years. And so I think Andy's going to talk about some of your more recent work in AI and machine learning, but I want to spend just a few minutes talking about the COVID-19 work that you published from 2020 onwards.

0:34:31

And so Ron said something a few moments ago that really, I think, stuck with me. So both it's the team that you had recruited and you're very well positioned, I think, by the time that COVID-19 hits and we're in 2020 and everyone is confused about what works, what doesn't work, what we should be doing. So you're already well positioned with this stellar team that you have. You're very lucky you have Noah there and many others and you're winning sprint competitions and doing things like that.

0:34:59

So you have a great team. You have this amazing data set and then you have this other or, you know, related ingredient, which doesn't get talked about enough. But I think, Ron, you just mentioned it, which is an understanding of the data. And so that understanding of the data, every time it sort of hops one away from the data creators or what Andy and I like to talk about on the show, which is a data generative process.

0:35:20

You know, what's sort of baked into the data, where it's coming from, what could be confounding, what could be issues with where different institutions record things differently, things like that. that really plague a lot of studies, a lot of observational studies, you are very, very close to the data. So you know the warts, you know the problems, you know how to deal with them. And to me, that is what really stuck out in those COVID papers, just that you were very careful.

0:35:43

You basically did the best you could with non-randomized data, with identifying those sort of potential issues, and then doing either careful sensitivity analysis or careful adjustments to remove the potential confounders and issues with understanding the efficacy of the vaccine in the general population, in some of the early New England Journal of Medicine papers, and then in certain subpopulations and other groups with some of the later work. So that's the way I saw it. I'm wondering if you can just maybe spend a few minutes telling us about what it was like when you were trying to publish that first paper.

0:36:19

I feel like Noah's told me a little bit about this here and there, and I could probably simulate some of it, but maybe I want to hear directly from you. Like what? I mean, you just went on this tear of producing very, very high impact papers, very, very fast. So maybe tell us about the stress levels, the environment, the amount of work, and you know, what you saw as the sort of key ingredients.

0:36:39
Ran Balicer

Maybe we could start with Ron and then go to Noah. So, you know, I think that COVID was truly our make or break. It was like everything we've done until now just has set us for this moment in time and place. And this was like combining all of our capacities, capabilities over the years and making them work at the right time. The setting is that Israel at that time, as you've said, was one of the first countries to vaccinate and was vaccinating faster than anyone else.

0:37:03

You know, 2 million people in about three weeks. And suddenly it was the first place in the world where this data was available. So with this data, we began working, but then we were reading very carefully Miguel Hernan's books. So we knew better than to just jump right in and do simple associations. We knew we had to take causal inference very seriously. We need to emulate the target trial that we're trying to do. And that's non-trivial to do it so quickly and with such complex data.

0:37:32

So this is where, you know, good friends kicked in. Noah talked about the Berkowitz Living Lab collaboration between Harvard DBMI. And so Zach, Zach Ohani again, I think, as we know, one of the most prominent figures in this domain in the US and myself, we're trying to create an ongoing work between the two teams. And at that point, that's where it kicked in because we had the best world-class leaders to support us. We had, in causal instruments, we had Professor Miguel Hernat to work on the paper.

0:38:02

And with epidemiology of infectious diseases, we had Professor Mark Lipstick. What else can you ask for? And in informatics, we had Professor Ben Rice. So, you know, this was really important in giving us the kind of relief that we were in good hands and that we knew what we were doing, because making the wrong decisions here would have had truly detrimental impact. And so what we actually did in terms of the science is we did something new for us.

0:38:28

It was to create digital twins, if you'd like. Every vaccinee got somebody who was absolutely identical to them, except that they weren't vaccinated at the time, and we followed them like it was a trial. And what we did is if somebody from those couplets, the unvaccinated guy got vaccinated, then we decoupled them and had the newly vaccinated guy get a new twin. So this was really computationally heavy, but we had by February 2021, the world's first very accurate estimate of vaccine effectiveness.

0:39:04

And we had that first publication in the New England. And a short while later, we had data from follow-up swab study that we've done in our hospitals to suggest that everything we did in that study was absolutely correct, although it was retrospective. So we ended up indeed, as you suggested, repeating this methodology several times during the pandemic to provide decision makers with real time. And we ate our own dog food. At that time, I was appointed as the government's chair of the national advisory team on pandemic response for Israel.

0:39:38

So my nights were in the cabinet and my days were quite clean. So it was kind of interesting. So in eating our own dog food, I mean that our own data was used for national decision making. And a lot of critical policy decisions were made based on real-time data, based on this data. So it was exciting times,

0:39:55
Arjun Manrai

best of times, worst of times. And Noah, can I turn to you about your experience with those those first few papers?

0:40:04
Noa Dagan

Yeah, sure. I think Ron really described it accurately. I think it was really working under pressure and understanding that the world is waiting for this information. And we really tried to do our best to do the best work we can to provide information that the world waited for. And it was stressful. We didn't sleep for, I think, many nights or slept a very small amount of hours per night back then. It was, I think to this day, one of the major things that we did.

0:40:34

I also have to say that I think the focus was on our vaccine effectiveness studies and our safety studies, other studies regarding those vaccines. But I think the pandemic was also transformative for us for the scale of using data science and prediction tools, regardless of the fact of, Like we're having vaccines, we're doing high-impact studies. I think it was the first time that our prediction models worked at scale to the level of reporting every day to the highest management in Clalit what was going on with them.

0:41:12

So I think even though the world cared about the vaccine effectiveness studies, for us, I think one of the transformative things was actually the work that we did with prediction models back then to the scale that we've never thought possible. The first example was in March 2020, was really the first days of the pandemic. And policymakers in Kledit for the first time came to us to ask for prediction models. So I think until then, we had to preach what prediction models are and to convince them to use them.

0:41:44

This is the first time where I think we all felt that we're dealing with the unknown. And they came to us asking for help and saying that they think that if we have a prediction model that helps to stratify who are those patients who are expected to have a severe condition if they contract the virus, it will really help them. and we said, this is great. And we're really happy that for the first time you're coming to us.

0:42:10

Since then it happened many times, but back then I think it was the first time. We really like to help, but please wait a few months and then we'll be glad to create such a prediction model because right now we have all the features and all the data, but we're missing one crucial ingredient, which is the outcome of interest because we didn't have COVID-19 patients in Israel back then yet. We had maybe a very small amount, but definitely not ones that experienced this severe outcome that we want to predict.

0:42:33

So we said, wait a few months, we'll get back to you. And they really insisted. They said, we really need this thing in order to make decisions now, in order to inform the public now as to who should be careful not to contract this virus. So we ended up actually using a flu prediction model that we developed several years before that to prioritize flu vaccines, to make sure that the highest risk patients will not be not unvaccinated when the winter comes.

0:43:02

And we ended up using that model, but we really want to integrate some COVID-level data into it because we knew something about this disease, right? We knew the case fatality rates that came out of China back then. We knew that, for example, what are the age groups that are more prone to fatality? What are the groups, sex groups, chronic conditions groups, we really wanted to integrate that epidemiological level data back into the flu prediction model that we felt captured some biological tendency to experience severe condition, but we wanted to integrate that into the model as well.

0:43:37

And the way we ended up doing that is to use actually a fairness prediction model that we used in various studies where we wanted to make sure that our predictions are calibrated towards numerous protected groups. So imagine you have these protective variables, you have hundreds or thousands of subgroups, and you want your algorithm to be simultaneously calibrated towards all of these subgroups to make sure that it's fair. We actually knew this algorithm from our work with scientists like Guy Rothlund from the Weizmann Institute and Omer Engel from Stanford, if you know these names.

0:44:11

And they really created a fantastic multi-calibration algorithm for fairness. But we ended up using that algorithm to adjust the flu predictions to those case fatality rates that came out of China back then to make sure that we can actually create something that is logical for COVID. So we ended up using this weird kind of model. And within two weeks, 200 ,000 patients in Clalit received personal phone calls from their clinics, telling them that they are at a predicted high risk to experience this new unknown disease in a severe manner.

0:44:45

So keep your social distancing and please know what social distancing is because no one knew back then. And if you need to contact the clinic, this is how you should do it because we don't want you to come in physically. And a few weeks later, when we actually had enough COVID patients in Israel, we validated the model and we actually were pretty surprised to find out that it worked really well. So for us, it was one of many prediction models that did some really heavy lifting throughout the pandemic for many decisions from national instruction of how to lift the lockdowns and to other prediction models that were used to ensure that high risk patients get Paxlovid in time.

0:45:26

And I think the VE, the vaccine effectiveness studies were really important, but for us, it was also very pivotal in the way, in the scale that we use prediction models for, I think the first time that scale of the entire management in CLI, it's only focusing on these prediction models and how to make care decisions according to them.

0:45:48
Arjun Manrai

Got it. That's great, Noah. Incredibly impressive. And this became also one of the other high-impact papers too, right? There's a Nature Communications paper, I think, that you published with the model. And I'm just going to make a note that we should probably discuss the thread between this and the projects that we're working on together on kidney function and heart disease, because there's probably some interesting parallels. Okay, I want to hand it over to Andy. I think we want to move on to some of the AI work and then some other topics.

0:46:13

I think that's a natural segue to talk

0:46:15
Andrew Beam

about some of your recent non-COVID work. So I'd like to talk about the paper that we published in NEJMAI that you both worked on called Perspective Evaluation of Machine Learning for Public Health Screening, Identifying Unknown Hepatitis C Carriers. So maybe, Noah, could you tee us up on that? Sort of what was the motivation for the study? What did you do and what did you find?

0:46:38
Noa Dagan

Sure. I think we can ask it again and address it to Ron. If that's okay?

0:46:43
Andrew Beam

Yeah, that's fine. Yeah. How about, Ron, could you tee that up for us? Give us the motivation and what you found.

0:46:50
Ran Balicer

Sure. I think most of the people who hear this podcast might be aware of hepatitis C. For those who don't, it's a viral infection that tends to become a chronic infection and gradually messes up with your liver. And in a few decades of silently carrying the virus, one in three will end up with cirrhosis, which is not good. So now we have super effective treatment that could prevent all of this. Okay. It's 98.8 effective. That's according to one of our own past studies.

0:47:18

So it's near perfect treatment. So the only thing is that 50 % of our patients will not be cured by this drug because we don't know that they're carrying the virus and therefore will not be there for them. So every year, Clalete follows the international recommendations and tests 50 ,000 previously untested people. And among them, we generally find 38 patients in this group. Okay. So it's great for those 38 patients. That's really great. But it's completely inadequate if we want to reach elimination of this disease by 2030 as the WHO set as a goal.

0:47:55

So what if we could use Clalit's insanely wide database to create a machine learning algorithm that would identify the highest risk members of Clalete and will begin screening those top risk individuals. So we did exactly that. And the results show that when we proactively screened less than 500 individuals at top risk, among them, we found 38 additional HCV patients. So 38 out of less than 500 versus 38 out of more than 50 ,000. That's a 100-fold improvement.

0:48:40

I haven't done many things in my life that have shown 100-fold improvement. And for us, I think this study symbolizes the paradigm shift from the classic public health towards predictive care. So this is what modern day public health could and should look like in the age of AI.

0:49:02
Andrew Beam

Noah, did you want to add anything on top of that?

0:49:05
Noa Dagan

I'll just add it. I think it's also one of the more satisfying prediction models to develop because usually in medicine, we have these longer outcomes. We predict things 10 years into the future, and it's hard to know whether we got it right or wrong. And this was one of the satisfying cases where each day we could come in, rerun the extraction and see how many patients did we get right last night, how many new identified cases were because we sent them to be screened.

0:49:34

And as a clinician that chose not to be a practicing clinician, it was really feeling the impact of, wow, we've saved those people. We made sure that they will get treatment that could potentially save their lives. So I think that's my favorite part about that project.

0:49:51
Andrew Beam

Awesome. So I do want to use this paper as a jumping off point to talk about future applications of AI in public health. But I think actually I'd like to first ask or first sort of pull back the curtain a little bit here. So I was the handling editor on this paper. So I talked with Noah about it. I talked to you're shocked. I know. And I think there was unanimous agreement that this was an important, high quality study.

0:50:14

What it sparked off for us internally was, is this AI? So I think that this was an interesting test case for us to try and actually, for our own purposes, try and nail down a definition of what we mean when we say artificial intelligence. I mean, like super high quality study, A plus team. I think everyone was all excited about that. But it did spark some internal debate about what we should call AI, what we should call machine learning, and what the taxonomy of this looks like.

0:50:40

So I'm very curious to hear your thoughts on this. I know that trying to define AI is always a hazardous thing to do, but I'd love to hear your thoughts on how you think about that. No, I think you have very strong opinions on this one. So please begin.

0:50:53
Noa Dagan

Yeah, I actually do. And it's interesting to hear the behind the scenes. I didn't know that part of the story. So I do have strong opinions about that because I think the AI, what is AI? It's a definition that is like, I think, well accepted, right? AI is any technique that basically enables computers to mimic human intelligence. Or maybe today we need to also say suppress human intelligence. But I think that people forget that AI is a toolbox.

0:51:21

And it's a wide set of tools from, for me, simple logistic regressions to more sophisticated ML models through foundation models, and if you specifically think about text, now LLMs. But I think that people who are active in this domain really need to develop skills in all of those tools and not to become fanatics of a specific technology, because at the end of the day, I think that the real thing to do is to start from the clinical need, to understand the real clinical need, and then to take the most appropriate tool from your toolbox and to use it.

0:52:01

So we actually make sure that we are well-versed in all of these tools. We create things from logistic regressions and even simpler things than that, even points-based models, to foundation models that we actually develop with great postdocs, for example, from Marinka Zitnik's lab and Zach's lab. So we believe in the full spectrum of these tools, but we, I think, really believe in using the right tool for the right task and not to go with the hype of the technology because if you offer me two different models for the same clinical need and one is simpler and does the work just as well, I will always opt for the simpler model for thousands of reasons of why we should

0:52:43

deploy things that we can understand and trust and monitor. But if that test can only be made possible by a more complex model, we'll use that. But all of the spectrum is AI, as long as you use data to do better care. Ron, I wonder if you agree.

0:53:01
Ran Balicer

I think that's exactly the point. I mean, we feel sometimes people are so excited about the technology that they are not thinking about the real clinical need. They have a tool, and they're looking for the clinical justification. Trying to find the nail for the large language model hammer is not a good way to do what we do. So what we truly aspire to do is to start from the real clinical need and then ask ourselves, what is the best, most simple thing to do?

0:53:27

Hint, hint, not LLM, 99 % of the cases. So whenever you can, good old machine learning is the real answer that you should go for. And don't get me wrong. I mean, we do projects, amazing projects with AI engines and LLMs, but that's not my current knee-jerk model of choice. Let's put it this way.

0:53:48
Andrew Beam

I think that's a very pragmatic answer. And I 100 % agree with a solutions-oriented approach that you should do the most complex model you need, but make it no more complex than it needs to be. I think I have a particular brand of PTSD when it comes to language around models. So I'm trained both as a statistician and a computer scientist. And I've just been at the middle of all of these language wars where is logistic regression machine learning is it statistics and so honestly like depending on what side of the bed i get up on on a given day i may come down on one side of that argument or the other so again i think this paper

0:54:22

for me specifically was a very interesting test case about what the right way to describe the models were i know you and i i think went back and worked on we came to consensus on language and stuff like that but i think I think it did spark a lot of good discussion amongst the editors about how paper should be described and things like that. Again, complete consensus on the quality of the paper and the work. A little bit of a pedantry exercise on our side, I think, though, about exactly how to describe it.

0:54:50

And I see Rod's grinning.

0:54:52
Arjun Manrai

I was just going to say, I was like, I thought AI was just what we haven't done yet. So, you know, yeah, what we're not capable of doing. But yeah, I totally agree. I also know and Ron, I really like the emphasis on simplicity and keeping it as simple as possible while sort of achieving the goal that you're trying to achieve. And I think there is, especially now, there's a lot of energy around sort of technology first approaches as opposed to the needs or the sort of clinical public health goals that I think are really clear.

0:55:22

It's really clear that your mission is to change care and to sort of to do very high quality research. And that dictates the technology that you use.

0:55:31
Andrew Beam

completely agree. So having said that, I often get asked to as someone who works in public health, how will AI affect public health? And I think unfortunately, my knee jerk reaction is usually negatively, where my brain goes is like misinformation. There are just like waves and waves and waves, or huge potential for LLMs to create misinformation machines. But what I think I really like about the work that you guys do is actually it points to a positive use case for this technology for public health.

0:56:01

So could you continue to be the ray of light that you've been so far? And tell me how you think that AI will impact public health going forward.

0:56:07
Noa Dagan

I'm actually really surprised that you said that you think it has a negative impact on public health, because I really think the exact opposite. I'm really sure that this is the way public health will be revolutionized, because I think we can all agree that public health is the best health we can provide because, and not only because Ran and myself are public health physicians, but I think it's the best type of medicine, right? We get more health, more healthy years for less money, fewer side effects.

0:56:38

It's one of these rare cases of a win-win situation all around. So now if we have these automatic tools and we can identify basically who are those patients at risk and what we want to do to provide better care for them. And again, no matter what the tool is, whether it's an LLM that went through your file and identified the care gaps, or if it's a prediction model that identifies that you have a high likelihood of suffering an MI in the next 10 years, now we have all of this automatic information produced for everyone in the population, or everyone we have data for, and now we can really stop being reactive and start being proactive and preventive about it.

0:57:19

And I'm not only talking about the classic primary prevention of really preventing an event before it occurs, like predicting someone will have a osteopoietic fracture, treating that patient, and then preventing that fracture from ever happening. So now we have this counterfactual reality where a person who might have sustained a fracture and maybe died from complications in the next year, now that person goes on to leave for 20 more years of good health. So that's one kind of public health impact that we could have with these kinds of tools.

0:57:53

but it's also secondary prevention of early detection, like the example of hepatitis C that Ron just gave. It's also tertiary prevention, for example, for patients we already know suffer from some conditions, say diabetes. We gave that example earlier of patients who are already suffering from a condition, but we want to make sure that they are being treated by the most appropriate drug. So now you have automatic ways to identify all of those high-risk patients and all of those gaps in care.

0:58:22

And we really have the opportunity, I think, to influence patients' lives at scale. And I have to say, it's not instead of regular medicine and it's not instead of AI for a specific patient, which is a lot of the AI we do. It's kind of creating a safety net for those patients in addition to that wide cast net that will capture all patients and the wide screening criteria that have potentially high sensitivity. now we have all of these tools to shed spotlights on the right patients to make sure that they get the right care i think it's the new age of public health and it's more health that we can provide at scale compared to any individual specific intervention that

0:59:04

we make it's really changing patients lives and really buying them dozens of healthy years so andy i like i really i'm really surprised by your answer, but this is my answer to will public health be changed by AI.

0:59:21
Andrew Beam

So I'm willing, I think I will now revise my answer and going forward say that we are in a golden age of public health enabled by AI. I think that those are very compelling reasons for optimism. So thanks for that, Noah. I think we're going to keep moving along to the lightning round now, if you're both ready to answer some lightning round questions. Let's do it. All right. Yep.

0:59:42
Arjun Manrai

so the rules here are that these are quick kind of rapid fire questions so just give us short answers we'll ask each question to both of you we can maybe start with noah for each of the questions and we'll take turns and we'll kick us off with this first question so noah if you weren't in

1:00:01
Noa Dagan

medicine what job would you be doing oh that's a challenging one so i think i'm doing the alternative job i think i should have been in medicine i thought i should have been in medicine and I'm actually doing the alternative job. And it's hard for me to imagine a counterfactual reality when I'm doing something else because I think I'm really lucky to have my day job as my hobby. And I wouldn't really change that. But if I really have to choose something, I imagine something to do with maybe being like a graphic designer or architect, anything that deals with something that has like neatness to how it's done.

1:00:33

But it's also somewhat like writing a beautiful piece of code or a well-structured paper. I see the aesthetics in all of these things, but I really like creating things graphically. I like to create presentations for that reason. So maybe that, but honestly, I wouldn't change it.

1:00:49
Arjun Manrai

Why the figures are so good in the high impact papers. It all makes sense now. All right, Ron, same question.

1:00:55
Ran Balicer

So I'll keep it short. I think I would have been somewhere in the entrepreneurial world. I would have probably started my own thing and trying to, I have no idea in what domain it would be. And I think there would be a lot of serendipity in choosing where would it be in education, or any other domain. But one thing is sure, I would try to maximize the impacts using technology. But again, any day of the week, any year, I would choose again and again the path that I've taken and nothing else.

1:01:21
Andrew Beam

Nice. Noah, if you were given one of these superpowers for a day, which one would you rather have? Invisibility or the ability to fly?

1:01:30
Noa Dagan

Oh, wow. Fly, I think, every day. I don't have a good reasoning why, but that's the answer.

1:01:39
Andrew Beam

Okay, nice. Ron? I couldn't imagine being invisible. I would choose flight. I think invisibility for me would be a punishment. So I always feel like this is a good test to see if you're secretly an evil supervillain. Because flight is something that we can do now, but invisibility is something that we can't. And you could use it for nefarious purposes. So I think you both passed the are you secretly a supervillain test.

1:02:06
Arjun Manrai

All right, Noah, which type of medical specialty could be most easily replaced by an LLM?

1:02:12
Noa Dagan

Oh, wow. You can't answer that one. But to be honest, I think LLMs can potentially change a lot of things in the way we do stuff. I don't see them really replacing any medical profession soon enough. I'll risk it and say that I think that image analysis models will have a higher impact on specific clinical domains before LLMs will have.

1:02:38
Arjun Manrai

Ron?

1:02:39
Ran Balicer

No, I would say the same thing. These could be ancillary tools for many professions in trying to make physicians' job easier. I think they have the potential of restoring some of the joy of work that we've lost in becoming data entry clerks, which is exceedingly enjoyable by many physicians now spending most of the time doing that. So take that away and people will be happier in any profession. but I don't see LLMs even remotely begin to make the difference.

1:03:05

And again, yes, image recognition would take away some of the technical aspects of radiology, pathology, et cetera, but they have other things they need to do now that their time is away from the technical aspect of looking at pictures. And I think that these will be beautiful professions. Good answers. Good answers. I agree.

1:03:24
Andrew Beam

To avoid order bias, I'm going to go back to Ron first, and then Noah can answer this next question. If you could have dinner with one person dead or alive, who would it be?

1:03:36
Ran Balicer

You know what? I'll choose someone alive. And any day of the week, I'll choose to have dinner with my parents, which are the people who influenced me the most in the world. I think everything that I am is because of what they have put in me. And I'm still to this day trying to kind of satisfy their expectations to some extent, sometimes succeeding, sometimes less. So I enjoyed their time with them and I'm trying to do this as much as I can.

1:04:04

So that would be my choice.

1:04:06
Noa Dagan

At least now I had some time to think while Ron talked. My answer is a bit similar, but also very different because I would choose, I think my grandparents were not alive, but I think, so they were partisans in World War II and they died many years ago. And I think it was before I was mature enough to really understand or fully comprehend what their life story was about. And I think that I would go back and try to make sure I know every piece of detail about it and also to tell them how much I admire them for what they went through.

1:04:41

And I think that also my grandfather was a professor of chemistry and he actually worked his entire life in developing drugs. So it would have been really interesting for me to have a professional conversation with him about what I do today, which I never got to have because I only knew him as a child. So I think those are the people who I really regret not having more mature conversations with.

1:05:08
Arjun Manrai

All right, Noah, this is our last lightning round question. And this one, I do want to start with Noah. And Noah will understand why in a moment. Noah, what is your favorite piece in chess?

1:05:20
Noa Dagan

Wow, so I'm not good at chess. I'm really bad at chess. I have to admit that. So the only thing I can answer about chess is that I wish I would have been a much better player, because first of all, I think I would have liked it and I would have enjoyed it if I was good at it. And the second thing is, I think I would have shared an interest with all of the rest of my household that I currently don't share.

1:05:48

So, and I think this is the reason why you're asking that. And yeah, so I should say Queen or something, right? But the honest answer is I'm just not good at it enough to really choose something meaningful.

1:06:03
Arjun Manrai

All right, fine. We'll let it pass. Rod?

1:06:05
Ran Balicer

I think the answer would be Knight. I think that's the kind of piece that makes the game interesting. and the thought about how you use your night's will is what makes a real player a pro.

1:06:18
Arjun Manrai

I love it. I love it. So the backstory here is that Ron and Andy, Noah's sons actually came over to our house. Noah's family was over at our house and they're very interested in chess and they had a portable chess board with them. And our daughters became fascinated with what they were doing. They're pretty young, but they learned that day and then have not stopped playing chess since meeting Noah's two sons. So we'll let that one pass, though.

1:06:42

I was expecting, I was hoping for Queen or something like that, but all good with that answer.

1:06:48
Andrew Beam

All right. And I apologize, but I'm going to have to hop off at two. Raj, you're in good hands with Raj, and he can take you home. But I have a pretty hard stop at two. Apologies for that.

1:06:58
Arjun Manrai

So Noah and Raj, you passed the lightning round. That was fantastic. We really just have one last kind of big picture question for both of you. And so this is, you know, we've talked a lot about the investment, the recruitment, I think, of very high quality and talented people at Clalit. And this is a question about thinking about how we can emulate that in other places, right? So for other countries and healthcare systems that are looking to emulate Clalit, to produce both the infrastructure that has led to integrating predictive modeling very early on and using it to improve care, but also do very, very high quality research with special data and special teams that you have assembled.

1:07:40

What advice can you give? You know, can we, is there a day that we could do this in the United States? Could we do it in a country like India? Maybe just reflect on that. What do we need to do to really emulate what you've done?

1:07:54
Ran Balicer

So I think the short answer is yes, you could do this in India, definitely. And the longer answer would be there are things that are really hard to change. You can't change the structure, the embedded incentives, the bureaucracy, the risk appetite, the abundance or availability of existing data. They all really differ from country to country. But there are things you can do that are feasible stepping stones that can definitely be true game changers. So one such example for us, I think, is our digital health residency program that is aimed to train the next gen physicians.

1:08:32

our healthcare systems. So the bottom line, I think, is that if you really want to change medicine with AI, you need people who speak three languages, AI, medicine, and public health and epidemiology.

1:08:47
Noa Dagan

And any country can do that. I can add that I think I was the first resident in this program, one of the first two, and I think really shaped it to be something that is a unique set of skills that is what's needed in order to make a change in this domain. So you need to know your medicine. And I think that often people are wondering whether or not you should start with physicians and teach them the other stuff, or you need to start with computer scientists and teach them some medicine.

1:09:18

I think it's really hard to teach medicine. I think that first and foremost, and it's also related to that answer about what is AI, you need to start with the needs. And in order to really understand the needs thoroughly, you need to spend enough time in those settings where it actually happens. And then you need to really understand data. But if you understand medicine, you also understand how the data is formed. And you need to create some or acquire some skills like programming and understanding your toolbox that we've talked about, about like including the full spectrum of AI tools.

1:09:52

And you need some product management training because you want to create some things that will truly be useful. and you need some epi and proper research skills because you really want to be able to measure the impact of what you do. So I think today we have one of the biggest public health residency programs in Israel and we were able to really recruit truly fantastic residents because I think they understand the potential and the impact and everything that this domain has to offer and they're really superstars in their domain.

1:10:25

and it's a really good advice because it's both very feasible, but it's also a true game changer. I think for us it was these people were a game changer for us and with a small amount of people with the really right set of skills, you can really make a difference.

1:10:42
Andrew Beam

Awesome, thanks. Well, Noah and Ron, thank you so much for being on AI Ground Around today. We loved talking to you.

1:10:49
Noa Dagan

Thank you so much for having us. It was really fun. Thank you. This was Retalk. Thank you.

1:10:53
Andrew Beam

That was awesome. This copyrighted podcast from the Massachusetts Medical Society may not be reproduced, distributed, or used for commercial purposes without prior written permission of the Massachusetts Medical Society. For information on reusing NEJM group podcasts, please visit the permissions and licensing page at the NEJM website.