Chad Atalla says measuring generative AI is hard because the qualities being tested are abstract and contested and must be measured indirectly.
“the measurement tasks involved in evaluating generative AI systems are often abstract and contested.”
Person
Chad Atalla appears on the Microsoft Research Podcast in an episode on AI testing and evaluation, drawing lessons from pharmaceuticals and medical devices. He discusses how to measure generative AI, how to think about risk, and how to check real-world impacts.
Atalla says measuring generative AI is hard because the qualities being tested are abstract and contested, so they must be measured indirectly. He says the field currently does poorly at measuring real-world impacts, and that evaluation often relies on proxy measures, much like surrogate endpoints in pharmaceuticals. He also says developers should track overblocking, the blocking of benign outputs, alongside safety. Each point below links to the moment it was said.
1 episodes · 10 quoted claims
Chad Atalla says measuring generative AI is hard because the qualities being tested are abstract and contested and must be measured indirectly.
“the measurement tasks involved in evaluating generative AI systems are often abstract and contested.”
Chad Atalla describes a paper proposing a four-level measurement framework for AI evaluation based on measurement theory from the quantitative social sciences.
“a four level framework, a measurement framework that's grounded in the measurement theory from the quantitative social sciences”
Chad Atalla says risk includes both the probability of an event occurring and the severity of its impacts.
“risk is importantly a concept that includes both event and impact. So there's the probability of some event occurring.”
Chad Atalla says the AI evaluation field is currently doing poorly at measuring real-world impacts of AI behavior.
“realistically, right now, the field is not doing a very good job of actually measuring the impacts.”
Chad Atalla says some AI regulation already requires evaluating the base model before it is built into products, with further evaluation once a product and user base are defined.
“For AI systems, we already see some regulation saying you need to start with the base model and do some evaluation of the base model”
Chad Atalla says AI evaluation commonly uses proxy measures in place of studying long-term impact, much like surrogate endpoints in pharmaceuticals.
“This is occurring in the AI evaluation space right now and is often perhaps even the default here, since we're not seeing that many studies of the long-term impact itself.”
Chad Atalla cites social media as an example of long-term mental health harms taking a long time to be understood, arguing for early long-term AI impact studies.
“Think of how with social media it took quite a long time for us to understand that there are some long-term negative impacts on mental health.”
Chad Atalla says phase-three-style multi-year trials are unrealistic as a requirement for vetting each individual AI model before release.
“we are not going to be able to put that on the critical path to getting specific individual AI models or AI systems vetted before they go out into the world.”
Chad Atalla says third-party experts in fields like chemistry and biology are needed to evaluate AI models for specialized risks such as helping make dangerous chemicals or weapons.
“we really need third party experts, people who are experts in chemistry, biology, etc., to come in and evaluate certain systems and models for those specific risks as well.”
Chad Atalla says AI developers should track overblocking, the blocking of benign outputs, alongside safety to manage the trade-off between safety and usefulness.
“It's too easy to produce systems that throw safety to the wind and focus purely on utility or achieving some goal.”