Person
Marzyeh Ghassemi
Marzyeh Ghassemi is an MIT researcher who appeared on the NEJM AI Grand Rounds podcast to discuss whether artificial intelligence can be harmful in healthcare, including bias, privacy, data access and how doctors should work with AI.
She described tests by her group in which a language model completed a note about a belligerent Black patient with "sent to prison" but a white patient with "sent to the hospital." She also said radiologists could not detect patients' self-reported race from chest X-rays, though machine learning models could. She argued medicine already relies on black boxes, citing acetaminophen and lithium.
1 episodes · 19 quoted claims
Marzyeh Ghassemi's Oxford master's work examined whether acute hypotensive episodes in intensive care could be predicted, possibly as a precursor to sepsis, and benchmarked neural networks among the methods.
“could you predict acute hypotensive episodes in the intensive care unit, maybe as a precursor to sepsis?”
NEJM AI Grand Rounds · 0:03:07
Ghassemi says neural networks performed poorly in her master's project because it predated GPUs and datasets with millions of examples.
“they did very poorly, as we all know now, because that was the pre-GPU, pre millions of examples”
NEJM AI Grand Rounds · 0:03:45
Before her master's degree, Marzyeh Ghassemi worked for Intel in Portland on predicting from health data collected by mobile platforms.
“I actually went to work for Intel in Portland for a couple of years before I started my master's.”
NEJM AI Grand Rounds · 0:06:10
For the Hurtful Words paper, Ghassemi's student used SciBERT, a publicly available contextual language model trained on scientific abstracts.
“This is the Seibert model, and it's one that has been trained on scientific abstracts”
NEJM AI Grand Rounds · 0:10:13
In Ghassemi's group's test, SciBERT completed a note about a belligerent patient with 'sent to prison' when the patient was described as Black, whereas a white patient was 'sent to the hospital'.
“African, African-American, or Black patient was belligerent and violent, then the model would autocomplete the note with the patient was sent to prison”
NEJM AI Grand Rounds · 0:10:46
Ghassemi said the LaTeX source of the GPT-4 paper posted on arXiv contained a section on unprompted toxic language generation that was commented out of the published version.
“And then in the actual paper, that whole section is commented out.”
NEJM AI Grand Rounds · 0:13:53
Differential privacy protects data by adding noise to and clipping gradient updates from data points that influence the model too much.
“anytime there's a point during the machine learning process that pulls the model's weights around too much, it's influencing the gradient too much in this batch of data you selected, you noise and clip the gradient”
NEJM AI Grand Rounds · 0:20:26
Ghassemi says Canadian healthcare data do not collect self-reported race, so outcomes such as childbirth deaths by race cannot be verified.
“they do not collect self-reported race in the Canadian healthcare data, which makes it impossible to verify how many, for example, Black or white women die in childbirth”
NEJM AI Grand Rounds · 0:27:12
In an ICML paper, Ghassemi's group showed that adding sex and age to a sleep apnea prediction model can raise overall performance while lowering it for subgroups such as older female patients.
“if you include sex and age, then you can train a logistic regression model and your overall performance will go up.”
NEJM AI Grand Rounds · 0:28:10
Ghassemi says her group tested radiologists and found they could not detect patients' self-reported race from chest X-rays, even though machine learning models could.
“Radiologists cannot detect self-reported race from chest x-rays. We tested them and they can't.”
NEJM AI Grand Rounds · 0:36:51
A collaborating radiologist found old papers on X-rays of bird feathers revealing melanin levels, which the team considered as a possible explanation for race detection in X-rays.
“One about x-rays of bird feathers and how you could tell the color of the bird feather, the level of melanination in the x-ray.”
NEJM AI Grand Rounds · 0:33:35
Ghassemi reports that models whose internal representations made patient race easier to decode had larger differences in true positives between Black and white patients.
“we found that the more obvious it was from the model who was black and white, the worse the disparity was in the model's predictions.”
NEJM AI Grand Rounds · 0:39:52
A paper from Ghassemi's group, The Road to Explainability Is Paved with Bias, found popular explanation methods made model results less fair across several settings.
“we found that when you use these state-of-the-art, very popular local and global explanation methods, they make your results less fair.”
NEJM AI Grand Rounds · 0:43:27
Ghassemi argues medicine already relies on black boxes, citing acetaminophen and lithium as drugs whose mechanisms are not known.
“like we do not know how acetaminophen works or lithium”
NEJM AI Grand Rounds · 0:45:31
Ghassemi cites aviation and space research on pilots and automation as a model for studying how doctors should work with AI.
“all the research that was done in the aviation and space industry to try to figure out how pilots would best be served by having lots and lots of automated systems”
NEJM AI Grand Rounds · 0:49:01
Ghassemi says her main lesson from her year at Verily was that collaborations with large health systems are hard to extend beyond limited studies.
“collaborations with larger health systems can be very hard to extend beyond limited settings of study”
NEJM AI Grand Rounds · 0:53:07
Ghassemi says datasets for machine learning speech research are privately held by companies such as Amazon, Microsoft and Google, whereas vision datasets are usually public.
“All of those data sets are private. They're owned by Amazon or Microsoft or Google.”
NEJM AI Grand Rounds · 0:58:41
Ghassemi says health institutions that cite patient privacy for withholding data can legally sell HIPAA de-identified data to companies.
“they will de-identify that data, so it is legal, HIPAA de-identified, to sell it to a company.”
NEJM AI Grand Rounds · 1:00:22
Ghassemi says the undergraduate machine learning class at MIT enrolls 400 to 500 students each semester.
“The undergraduate machine learning class at MIT gets 400, 500 students per semester.”
NEJM AI Grand Rounds · 1:09:03