# AI incidents, audits, and the limits of benchmarks - **Show:** Practical AI - **Published:** 13 Feb 2026 - **Source:** https://share.transistor.fm/s/1b8e65f4 - **Transcript:** https://podcasts.opennash.com/shows/practical-ai/ai-incidents-audits-and-the-limits-of-benchmarks-446f66/ - **Provider:** OpenNash Podcasts (machine-generated) ## In brief AI safety needs the same machinery as aviation, finance and food safety: public records of what went wrong, and independent checks of what companies claim. Sean McGregor argues that general-purpose AI breaks the usual safety approach, because no one can test every context. Third-party audits, better benchmarks and shared incident data are how organizations can decide what to trust. The central tension is between research and practice. Benchmarks were built to generate knowledge, yet buyers use them as proof a model is safe for their own use. The DEFCON red-teaming exercise showed the same gap: one-off exploits are not enough, and testers had to show systematic weaknesses with statistics. Mandatory reporting and flaw-reporting systems could close it. ## Key insights 1. **Safety starts with recording what went wrong** McGregor compares the AI Incident Database to aviation crash records and medical adverse-event reporting, which work like regression tests so past failures are not repeated. Without shared data, organizations cannot learn from each other's failures. 2. **General-purpose AI broke traditional safety methods** Safety processes assume a specific context to reason about, but a general-purpose model is used everywhere. Buyers therefore cannot trust vendor claims for their own setting and may need to run their own pilots. 3. **Most benchmarks were built for research, not buying** BenchRisk found many benchmarks lacked evidence for their claims, which McGregor likens to receipts that just say 'trust me'. A high score on a bias test like BBQ does not mean a model is unbiased in your application. 4. **Hackers must show patterns, not single exploits** At the DEFCON challenge, an anecdote of one bad output did not count, because models respond randomly. Entrants had to show a systematic weakness, which forced security people to use statistics. 5. **Gaps between connected AI components are exploitable** The most common DEFCON exploit targeted how a guard model handed off to the underlying chat model. Real products combine several systems, and the interfaces between them are often untested. ## Questions this episode answers ### What is an AI incident, and how is it different from a bug? Sean McGregor says the intergovernmental definition is an event in which a harm has actually taken place. A flaw, by contrast, is a weakness found in testing where no one has been harmed yet. ### Why can't I just trust the benchmark scores that AI companies publish? McGregor says most benchmarks were made for research, not to prove a model is safe in your specific situation. The BenchRisk project found many offered weak evidence for what they claim. ### Why would a business want an independent audit of its AI? McGregor compares it to financial audits: investors expect audited books, and a third party is better placed than the builders to spot what they missed. An audit checks the evidence behind a company's claims about a system. ### Where does the AI Incident Database get its information? Mostly from journalists' reporting, plus some direct submissions from the public. McGregor says this can show that incidents exist but not how often they happen, which is why he wants mandatory reporting such as the EU code of practice requires for severe incidents. ### What did the DEFCON hacking challenge show about AI security? Hackers tried to show that an Allen Institute language model and its guard model did not match their documentation. Winning reports had to show a repeatable weakness, and the biggest one was the loose handoff between the guard and the chat model. ## Facts - **Sean McGregor** [0:02:33]: "a 2017 vintage PhD in machine learning. I focused on reinforcement learning as applied to wildfire suppression policy." - **Sean McGregor** [0:03:26]: "The organization called Centient that shipped when I was there millions and has continued to ship many edge neural network processors." - **Sean McGregor** [0:04:08]: "And we sold the assets associated with that to a old safety organization called Underwriters Laboratories" - **Sean McGregor** [0:04:53]: "you see this in aviation, a plane crashes, you record what happens and use that to make sure effectively in the aviation industry, you have a form of regression test" - **Sean McGregor** [0:07:52]: "the intergovernmental definition of incident that developed was effectively an event that a harm has taken place, that's an incident." - **Sean McGregor** [0:11:49]: "journalists actually put in a tremendous amount of work to validate the base facts involved in it" - **Sean McGregor** [0:12:17]: "In the EU code of practice, there is a requirement for severe incidents to be reported. It hasn't reached implementation as of yet." - **Sean McGregor** [0:16:35]: "there's a premise in there that a third party is better positioned to do that than a first party." - **Sean McGregor** [0:17:19]: "this gentleman who got a traffic citation mailed to him, and he looked at the traffic citation and said, like, you know, this isn't my car" - **Sean McGregor** [0:19:33]: "having audited financials is table stakes. You're not going to put your money into an org that hasn't been audited." - **Sean McGregor** [0:23:27]: "and BBQ is a great benchmark. It was really foundational in the bias benchmarking community" - **Sean McGregor** [0:24:07]: "by and large, most benchmarks historically have been produced for research purposes, not for practical AI purposes." - **Sean McGregor** [0:26:43]: "And security people have a much greater capacity for the presumption of bad actors doing bad things" - **Sean McGregor** [0:29:32]: "this was something called the Generative Red Team Two." - **Sean McGregor** [0:29:32]: "7 billion parameter large language model, along with its guard model, put in front of it" - **Sean McGregor** [0:31:19]: "what we basically had to bring to the security world, which the security world does not like this, and I don't blame them for, is statistics." - **Sean McGregor** [0:35:43]: "exploiting that handoff between the guard model and the underlying model was the kind of, you know, big vector that you could use that to print cash over the course of the competition" - **Sean McGregor** [0:37:15]: "which is flaw reports, because it's not an incident like no one's harmed." ## Transcript **[0:00:03] Ad read:** Welcome to the Practical AI podcast, where we break down the real world applications of artificial intelligence and how it's shaping the way we live, work and create. Our goal is to help make AI technology practical, productive and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn X or Bluesky to stay up to date with episode drops, behind the scenes content and AI insights. **[0:00:33] Ad read:** You can learn more at practicalai. fm. Now onto the show. **[0:00:48] Daniel Whitenack:** Well, welcome to another episode of the Practical AI podcast. This is Daniel Whitenack. I am CEO at Prediction Guard, and I am joined as always by my co host Chris Benson, who is a principal AI research engineer at Lockheed Martin. How you doing, Chris? **[0:01:04] Chris Benson:** Hey, doing great today, Daniel, how's it going? **[0:01:07] Daniel Whitenack:** It's going really good. Lots of fun things in the news to follow and lots of fun things to work on. By the way, for our listeners who might need a reminder about this, we are doing some webinars recently on a variety of topics, some of which are maybe even related to some of the things we're talking about today around security or or safety. So if you want to find out about those, go to practicalai.fm slash webinars. **[0:01:35] Daniel Whitenack:** That's where we have some of those things listed out. But I'm really excited today because have an amazing set of things to talk about with Sean McGregor, who is co founder and lead research engineer at the AI Verification and Evaluation Research Institute, and also the founder of the AI Incident Database. How you doing, Sean? Thanks for joining us. **[0:01:58] Sean McGregor:** I'm doing well, thanks for having me here. **[0:02:00] Daniel Whitenack:** Yeah, yeah, of course. And this is interesting, kind of to think about AI incidents, verification, evaluation. How did you find your way into a day to day where you're thinking about and documenting and studying AI incidents among other things? **[0:02:19] Sean McGregor:** How did I get to work on what you might call impractical AI? **[0:02:24] Daniel Whitenack:** Yeah, yeah, well, I guess, I guess where practical AI turns into problematic AI, let's say **[0:02:33] Sean McGregor:** Yes, yeah, and really, practical AI is the AI that has consequences and matters in the world. And those are the ones you actually care to look into where it goes wrong. So the kind of really quickie professional tour of things is about a 2017 vintage PhD in machine learning. I focused on reinforcement learning as applied to wildfire suppression policy. So a fire starts in a forest, what do you do about it, and how does that impact the development of the land and the values we get from it? **[0:03:03] Sean McGregor:** Over the course of 100 years, really, in that setting, I had a very strong sense of the power of the technology that we were developing, but also the brittleness of it. And just how difficult it was to even know whether the system was doing what you wanted it to do, particularly in a reinforcement learning system. And I just never wanted to wander through a forest and have a sense of, you know, this is a charred wasteland. **[0:03:26] Sean McGregor:** Because of the forester took my simulation and didn't really realize it was research code and that he should actually apply his own human expertise about what is good fire and bad fire in that. So I went from that. Then I worked on energy efficient neural network processors, which that was a great effort. The organization called Centient that shipped when I was there millions and has continued to ship many edge neural network processors. And that gave me a really strong impression of the kind of power and brittleness, once again, inside these consumer electronic devices, hearing aids and the like. **[0:04:08] Sean McGregor:** And so I left that. I started a company dedicated to the test and evaluation of machine learning systems. And when that happened, or when I started that, the kind of LLM explosion happened, and you know, that's the kind of Cambrian explosion of sorts of, you know, practical AI technologies that I'm sure we're all still feeling. And we sold the assets associated with that to a old safety organization called Underwriters Laboratories and spent some time there doing safety stuff before leaving and starting up the Avery, the AI Verification Evaluation Research Institute. **[0:04:53] Sean McGregor:** And during this whole time, I had a project that just kind of, you know, got away from me because it filled a niche. It served a need. And that's the AI incident database. That you really need to have systems that collect and produce usable data sets motivating safety practice. And you see this in aviation, a plane crashes, you record what happens and use that to make sure effectively in the aviation industry, you have a form of regression test. **[0:05:24] Sean McGregor:** You don't want that past crash to happen again. You also have similar things in food safety, you have medical adverse event reporting. These are. This is the fundamental primitive that you need to make sure you have in safety. Is a bad thing happens, you make sure it doesn't happen again. And so we've in that project collected more than five zero human annotated reports of AI incidents. Those are collected across more than one zero discrete incident records at this point. **[0:05:54] Sean McGregor:** And we've formed a lot of the training data of, like, what is an incident? And we've been interacting with a lot of intergovernmental organizations and GRC community governance, risk and compliance community companies. And like, just trying to motivate what is, how to motivate safety, and like why? Very often, that can actually be a business imperative, because a lot of the incidents that we have in the database, you could actually look at the stock price before and after, and there's an impact. **[0:06:23] Chris Benson:** Could you talk a little bit more? Kind of expand on what the nature of safety means in the context of AI? And because coming in as a neophyte on this, I don't know if there's kind of a standard definition. It seems to me there isn't at least not that I'm aware of. So how do you define it? You mentioned kind of defining an incident. Could you define what you think an incident is and how these different concepts relate together? **[0:06:49] Chris Benson:** In a way that those of us who are not part of that kind of world, that specific world, can kind of adopt the ideas and understand what you're talking about? Kind of give us a background on it. **[0:07:00] Sean McGregor:** So this is a three or a four hour podcast, right? **[0:07:03] Daniel Whitenack:** it's like the question like used to always, we would have the question like, is something AI or machine learning? And like, eventually, you just learn that those words are completely meaningless in terms of the way people apply them. So **[0:07:18] Sean McGregor:** Yeah, we just need to treat them as like a pointer, and you know, like the memory might be allocated differently with, or the pointer can point to different memory, or maybe the memory swaps. So, like, there's a lot of safety terms out there. And like, I've actually had a poster for this at one point because this comes up so often. You could have incident, accident, adverse event, you could have vulnerability, exposure, harm event, controversy, issue. Like each of these have like slightly different nuance to them. **[0:07:52] Sean McGregor:** But you can kind of bring it back to the concept of you don't want a bad thing to happen, and you don't want that bad thing to produce a harm. You don't want someone to say like, I've been impacted, or some organization been impacted. And so the intergovernmental definition of incident that developed was effectively an event that a harm has taken place, that's an incident. Your mileage may vary in different contexts and like different communities, there are different terms that resonate. **[0:08:24] Sean McGregor:** But the reason that we went with incident to begin with is basically it's sufficiently vague while still meaning something. Because you can't say accident, some of these have intention, you can't say exposure or compromise, or you know, the ones that are in the computer security world, because some of them don't have intention. And so incident covers them all. And you know, I could gesture and give you some Venn diagrams, but this is a podcast, so we'll avoid that. **[0:08:55] Daniel Whitenack:** I'm wondering, like, in particular, when you narrow it down to AI incident, obviously, sort of AI, depending on your definition of AI, kind of has existed for some time. And so I guess like AI incidents have been happening for some time. In the sense that, like, you could have a computer vision model detecting, maybe it's problems and things coming off of a manufacturing line. It misses one and that causes harm, maybe downstream to someone who uses a product or something like that. **[0:09:29] Daniel Whitenack:** How has this? Maybe, you know, you talked about the shift to, of course, the kind of expansion of what, you know, how AI is impacting our life. How, from your perspective, has that idea, I guess, been stressed in new ways in recent years? **[0:09:48] Sean McGregor:** We have constant debates over what we would ingest into the AI incident database and we because we have to operate and it's a database that we have to make these decisions day in and day out, have very difficult decisions. And it's not just in our case, it's not just what should be considered an incident, it's also if we start ingesting these. Are we lassoing an infinity that's just going to, like, pull us into some extreme direction? **[0:10:18] Sean McGregor:** And maybe we would like to index, catalog and present to the world that infinity. But there are these kind of like little harms that are repeated, maybe millions of times a day. When you have these systems that indexing all of them isn't necessarily useful, so we do have a little bit of a bent towards. Is this informing the production of a safer AI or safer world for AI? But a lot of things meet the fundamental criteria of involving AI and minor harms have taken place. **[0:10:54] Sean McGregor:** There are also increasing number of high scaled harms that take place, like if you make a billion people slightly more depressed, non zero number of people probably have died as a result of that. And that's where, you know, it's not just you're building a bridge that needs to bear weight, you're building a bridge that needs to bear the weight of all of humanity flowing over it. **[0:11:19] Chris Benson:** Uh, I'm curious, as you're kind of selecting things to go into the database, you know, per the criteria that you just talked about, how are you sourcing that? And especially when you consider the fact that, you know, so many AI things today are proprietary and kept secret in organizations for obvious reasons. You know, h How is that sourced? And how do you? Uh, how does the different sourcing affect the utility of the database and how you make those kind of selections? **[0:11:49] Sean McGregor:** Most of what we have at the moment is journalistic reporting. The reason that we are living in that space at the moment is journalists actually put in a tremendous amount of work to validate the base facts involved in it. And that is where the degradation of the journalistic community, or at least the ability for them to make wages, has actually been quite harmful. We also do get direct reporting at times. We do get people that email us or submit directly through our forms. **[0:12:17] Sean McGregor:** We get people that create blog posts and things like that and submit it to the incident database. We do encourage people to submit to the incident database. The simple matter of the fact, though, is the volume that we're dealing with is one where eventually we do need to switch from this kind of voluntary reporting to more mandatory form of reporting. In the EU code of practice, there is a requirement for severe incidents to be reported. It hasn't reached implementation as of yet. **[0:12:51] Sean McGregor:** There's a lot of back and forth on that front. But the insight you can derive from mandatory reporting is greater than that of this voluntary reporting. Like, we can prove existence, we can prove it's happening. It's very difficult for us to assign a rate to incident events, though, particularly as it becomes non newsworthy through time. So we do have some works that we're trying to make the most of that. This bears some resemblance to public health practices, where you don't have perfect insight into disease spread, but you do have these kind of indirect measures of doing tests in the sewers and things to see if the how much of the viral particles are there. **[0:13:32] Sean McGregor:** And **[0:13:33] Daniel Whitenack:** Well, Sean, I appreciate you taking us through this kind of idea of AI incidents. Obviously, we are practical AI, and in a practical way, we would very much like to prevent incidents, or at least understand kind of the security implications of using certain AI models. I know with the Avery, the AI verification and evaluation research institute. In my looking at that, you're talking a lot about kind of third party auditing of AI, or kind of frontier AI, or foundational AI. **[0:14:08] Daniel Whitenack:** However, you want to frame that, do you want to explain a little bit about kind of how that side of things might fit in the landscape and why? It's important? **[0:14:19] Sean McGregor:** certainly so. A fundamental problem that we have, particularly at the frontier model level, with things like OpenAI, Anthropic, Google's Gemini and so forth, is it's very difficult to even know how safe your systems are because they're general purpose systems. This basically broke the safety frame. All the safety processes that we have are built around starting presumption of there's a specific context it's operating in, and you reason about its safety within that context. Well, if your context is just wildcard, star, everything, where does that leave you? **[0:14:53] Sean McGregor:** Do you need to verify across all circumstances that it's going to be safe? Because the answer is going to be no. So how do you approach the task of verifying claims? How do you encapsulate something that a customer or person or organization would rely upon and say, like, okay, this has gotten this level of assurance applied to it? So I know I can apply it within my college age student population, teaching them how to do the maths and getting that level of signal at this point. **[0:15:32] Sean McGregor:** You know, a real practical problem that you have is no one's going to believe you. When you say, you know, this is good for high college age math remedial level. Like, you have to run a pilot program and see how it works before you actually can believe the representations being made by companies because they probably haven't evaluated in your exact circumstances. They might have something that's analogous to that, but they probably won't have it exactly in your circumstances. **[0:16:03] Sean McGregor:** And you know, this is a problem on the practical AI side of things, but it's also a problem on just the increasing power of these systems and the scale in which they operate. You actually do want some really strong top level guarantees that the system is going to try and steer away from catastrophe and doesn't actually have an active propensity for doing the bad things. And the science of establishing that is actually still a work in progress. **[0:16:35] Sean McGregor:** There's a lot of work, not just in establishing evaluation, but meta evaluation, basically deciding is this evaluation saying the thing that it says it says or not? And Avery as an organization is concerned with doing third party audits of basically saying, is this thing safe? Is this thing doing the thing it should be doing? And there's a premise in there that a third party is better positioned to do that than a first party. And we've all been on development teams and, you know, pushed to prod, pushed to the real world, pushed to real world circumstances. **[0:17:19] Sean McGregor:** And found, like, oh, I didn't think about that. Like, in the one of my favorite incidents in the AI incident database is this gentleman who got a traffic citation mailed to him, and he looked at the traffic citation and said, like, you know, this isn't my car. It not only isn't a car, it is a woman who's wearing a shirt that says knitter, and there's a purse strap going across it. So it like, kind of played with the letters, so it looked like K9, TER, or something like that. **[0:17:57] Sean McGregor:** And I don't think the people making the traffic camera were thinking about a woman walking through the field of view of the traffic camera with like a shirt perturbed in such a way that it would catch it and register it as a license plate. And then the traffic station went out like this. The world is hard, the real world is real hard. **[0:18:19] Chris Benson:** I'm curious that I like that example, it's a it's a very personal example and I can certainly imagine that happening to me, actually, bizarre as it is. But I'm curious as you as you are looking at applying this process, you know, that you're describing to maybe a larger incident. Is there an incident that comes to mind in the database? That where? Where you could kind of compare what your estimate like if relative to not having gone through third party process versus if it had, how you might envision that playing out in a different way. **[0:18:57] Chris Benson:** I'm kind of trying to get a sense, and you can interpret it any way you want, or find whatever example, but trying to get a sense. Like, when organizations are listening, you know, they have employees and their leadership are listening to the podcast, and they're trying to put this in the context to their own operations. Like, what is a, what do you think is a kind of aha moment? Potentially for the listener? And maybe a Fortune 500, maybe not that big, you know, maybe a mid sized company where this process will make a substantial difference to the outcomes involved. **[0:19:33] Sean McGregor:** Probably a good way to work here is by analogy to other segments that have audit, and audit is something that no one truly enjoys. Like, you don't, you don't necessarily want to be audited, but you actually do, you do, you do want to be audited. If you're an organization that wants to receive investments from other organizations, having audited financials is table stakes. You're not going to put your money into an org that hasn't been audited. Or actually, I should say there are famous instances in which an org has not been audited and people subsequently regretted it. **[0:20:07] Sean McGregor:** Because the reason they weren't audited is they were, you know, improving disbursements by Slack emojis. And, you know, that didn't go very well in those circumstances for anyone. And so we do periodically get reminders that, you know, these processes are important and that audits solve a problem. And that's you need to be able to trust the fundamental representations about the state of an organization. And it's the same thing for the model, because the model is similar in a sense that it's taking actions, it's doing things and has impacts. **[0:20:42] Sean McGregor:** And so there's value to knowing when you should or should not trust information. **[0:20:47] Daniel Whitenack:** And I'm looking through some of what the Avery Institute has been involved with, and in particular kind of recent work of this like meta evaluation of benchmarks, this sort of benchrisk, which is actually how we got connected through through colleague. Ashwarya And I'm wondering, this is like a meta evaluation. And I know a lot of people make decisions about models based on benchmarks. Could you maybe help us kind of walk through mentally what is the difference between maybe like an audit and a benchmark? **[0:21:28] Daniel Whitenack:** And maybe like tying in some things that might make people think about whether benchmarks as they look at them on leaderboards and that sort of thing are really, yeah, if they're really relevant to the real world behavior of models. **[0:21:46] Sean McGregor:** So maybe overextending the financial metaphor on things. When you audit a financial organization, you say, okay, I see that this organization has a balance sheet indicating they have 10 million in the bank. You're like, great. Now, as an auditor, I need to go and check the balance with the bank and see, is this money real, is it actually there? And similarly, an audit is concerned with. I've seen the balance, I've seen the balance sheet, and now I need to check the evidence and make sure that I actually believe that representation in some form. **[0:22:24] Sean McGregor:** And so where meta evaluation, or evaluating the evaluations, or checking the benchmarks, evaluating the benchmarks are useful, is you're basically checking those receipts. And in the course of the BenchRisk project, we found that a lot of the receipts were just kind of like, well, trust me, bro like written on a piece of paper. And that there were real substantive issues of like, okay, maybe there's like an IOU and there's some gold buried out in a field somewhere. **[0:22:53] Sean McGregor:** But I have some doubts, and at the very least, I need to look into this a little bit more before I rely on it for real world purposes. And the dichotomy that we identified is a lot of the benchmarks that are produced are produced for non practical purposes. They're produced for knowledge generation purposes, they're produced for research purposes, where people are wanting to understand systems that are making sense of it. But it's not produced with the intent that someone's going to then say, all right, I'm going to deploy this in my environment. **[0:23:27] Sean McGregor:** And I know now that it's unbiased because it scored well on BBQ, and BBQ is a great benchmark. It was really foundational in the bias benchmarking community, but it's also not produced with the intention that every time a frontier model is released, you say, like, look, we've improved on bias, and it's ten BBQ points better than the prior ones. And like, that's a good thing, you do want that. But it's not a practical AI purpose. It's not saying, like, it's unbiased for my specific application, because there's a distribution associated with BBQ. **[0:24:07] Sean McGregor:** There's characteristics that aren't necessarily generalizable to the environment to which you're deploying it. The prompts can be in like a very particular space for BBQ that you're just not operating in. And all these things are associated with in this research work that we put out, they're associated with failure modes that we identified. We collected a list of failure modes and then we looked at how well benchmarks had expressed mitigations against those failure modes. And the results were mixed, like some did better than others, but by and large, most benchmarks historically have been produced for research purposes, not for practical AI purposes. **[0:24:49] Sean McGregor:** And this is a problem because this is what we're working with and informationally. And this is why we're starting to see an evaluation ecosystem pop up. And that being a discrete task and something that's supported by organizations separately. **[0:25:04] Chris Benson:** I'm wondering, as we're kind of going through this and discussing the process, if, as you're looking at some of the different risk categories that you guys, you know, outline on your website and stuff. And a couple of examples are misuse and unintended behavior and infosec and emergent social effects. Which of these do you think is kind of like, maybe the most underestimated? You know, the thing that people aren't really expecting that has consequence, that may be outsized relative to what those of us not in this field would expect. **[0:25:40] Chris Benson:** Do you have any sense of that in terms of, like, the kind of a little bit of the surprise that people might not realize is there? **[0:25:47] Sean McGregor:** That's tough because I'm a little bit of the fish in water, being told it's wet. And Fair enough, I will say, in like, kind of rewinding my own experience on this is, I feel like I'm a pretty astute observer, and I've been more predictive of what problems are coming around the corner than I think most people I've encountered. But even with that, I still am regularly surprised. And I wish we could be a little bit more predictive about it. **[0:26:14] Sean McGregor:** And I think we are going to get there and we're going to learn from the past and hopefully push that into the future. But it's this is a new thing, like we don't have. We don't have a like, a great operating history, particularly for the general purpose AI systems. And I think that the split in this community that you should probably think a little bit about that question. In terms of expectation, is there's security and safety people, and that there is a little bit of difference between those. **[0:26:43] Sean McGregor:** Security is safety, safety is security, but what you care about tends to be slightly different in what you're expecting. And security people have a much greater capacity for the presumption of bad actors doing bad things and that producing bad outcomes. And then the safety people have a much stronger presumption of just the world is its own adversary. We're all in our own kind of game against nature and bad things will happen regardless. And maybe in the bad actor side of things, they're a little bit more insistent upon it though, particularly when the scalability of doing bad things is actually substantially increasing. **[0:27:21] Sean McGregor:** So we have to solve all these. **[0:27:23] Daniel Whitenack:** Well, Sean, one of the things I wanted to ask you about, well, in particular. I'm a big fan of the Darknet Diaries podcast and listen to that a lot. And always love, you know, hearing stories of hackers and you know, the kind of world that is there and all of you know what people try to do at Defcon and other places. I was really intrigued by your study, kind of titled To Err is AI. I see in the description you kind of took some models to Defcon, and I'm reading your description. **[0:27:59] Daniel Whitenack:** Had the pleasure of taunting a room of hackers with a declaration that the model is flawless. And you know, obviously certain things happened after that. So I would love to hear, I mean, maybe how this came about, or what the idea really was, and then would love to, you know, hear some of that story. **[0:28:19] Sean McGregor:** Sure, so to set the stage a little bit, DEFCON is a hacker convention conference that's been run in Las Vegas for a good number of years. And it's, I don't know if listeners have seen the 1990s movie Hackers, which has like a very early, like Angelina Jolie and some others in it. And it's like these young men and women, like rollerblading everywhere, you know, like that's just how the future would be. And talking about how risk architectures would change everything. **[0:28:51] Sean McGregor:** And there's this kind of hacker aesthetic that that kind of noisily was portraying of, you know, joyfully compromising systems. And in this particular instance, having a mixture of people that are doing that with good aims and bad aims. But DEFCON is this community organized thing and you have villages and one of them is the AI village that's been run for a few years now. And in that village you have presentations. But there's also a section of it dedicated to some sort of a challenge that's conducted over the course of the conference. **[0:29:32] Sean McGregor:** And people filter through and figure out how to break things and they're treated to cash prizes. In this case, so this was something called the Generative Red Team Two. So it was a change from the prior year and form. And we basically asked people there's documentation for the open language model produced by the Allen Institute for AI. So 7 billion parameter large language model, along with its guard model, put in front of it. And we said, here's the documentation, here's representations about what this model is supposed to do, and your job is to show how that's wrong. **[0:30:10] Sean McGregor:** That kind of unleashed a few days of chaos. And like, I don't know what else was happening in the conference. But like, I was at the judging table for the entire time with reports flying in and needing to decide, you know, like, is this actually a violation of the documentation? It did the thing we wrote, which we did put some easter eggs in there of, like, we expected people to be able to break it. But did they submit sufficient evidence of a violation of that documentation? **[0:30:42] Sean McGregor:** And that is actually a surprisingly difficult thing. We had all these people that were very good at the compromising systems and saying, like, yeah, I know how to prompt this Lm and get it to tell me how to burn down this convention center. Violation of what the model should do. And we kept on needing to say, Oh, that's great, but like, you haven't told us anything. Like, a anecdote does not equal data in this instance. And we need you to show that it's systematically like pushing towards arson, that it's always wanting to burn things down, or that you can put some sort of more universal jailbreak on it. **[0:31:19] Sean McGregor:** That makes it so that it underperforms in that particular filter of things. And so what we basically had to bring to the security world, which the security world does not like this, and I don't blame them for, is statistics. We had to say, like, it's not just about the fact that there's an exploit, it needs to be. It's systematically vulnerable in some form, because it's always rolling dice, that's just the nature of it. And that made for a wild few days, paid out a good cash purse and learned quite a lot about the collision of disciplines that we need to foster when it comes to security, safety, statistics, machine learning. **[0:32:04] Sean McGregor:** The world's getting more complicated. **[0:32:06] Chris Benson:** prior to kind of talking about some of the chaos that ensued at that point. As you talk about that collision of culture, if you will, you know, with kind of kind of taking the hacker out of their thing of thing. Hey, I have an exploit, you know, look at me. And requiring that level of rigor, 1st of all, can you describe why you required that? Why that level of rigor was important in this case, and how did that force different behaviors out of the hacker community that were applying themselves against the problem that you had presented? **[0:32:43] Sean McGregor:** We had a few people that were unhappy initially, and then we explained it and then they went back and came back and then they started turning the crank and getting a lot of payout. So people that really wanted to encamp and figure out the modality did very well. The problem, and the reason why, like, anecdote doesn't equal data here is, if you say something is, you know, 99 filters out 99 of the bad thing. If they wanted to, they could roll up to one of those stations and, you know, just keep on issuing the same query, or like, adding one to it. **[0:33:16] Sean McGregor:** And then one time out of 100, they'll get something, they'll be able to walk up and say, money, please. The problem is that's not really useful. If you're designing the system, you need some idea of what is the systemization here, because you know it's 99, you don't to some extent, like you're going to keep on working to get to 100. But you care about the cases where it's not 99, but you made a mistake and it's actually 70, or it's one, or it's zero. **[0:33:46] Sean McGregor:** And that's a statistical argument. That's a here's a description, a higher level description, then here's an example. Like, this isn't me prompting it. This is a description of an attack that is not accounted for in your documentation, that you're vulnerable for. And you need to solve this as a strategy like you never solved. Talk like a pirate, so anyone can talk like a pirate to the system and get it to do bad things. And that's the thing you care about. **[0:34:11] Sean McGregor:** Rather than I asked it and pirate speak something 100 times in one time. It gave me bad **[0:34:18] Daniel Whitenack:** Were there certain modalities, like you described of failure, that were a surprise, kind of to to the judges in terms of like the or or maybe to like the like? I don't know how much interaction there was with the model building team, you know, at Allen Institute and that sort of thing. But you know, any kind of modes of failure that weren't expected, or just the ones that were continuously problematic, but even if they were known before **[0:34:47] Sean McGregor:** somewhat unexpected. We characterize this in the research paper as well, and I do recommend people take a look at it because it's very practical in nature. The biggest one and the one that I think the most people just instantaneously exploited. And they didn't have full information on this one, like we didn't give them full information on the integration of the guard model and the foundation model or the chat model underneath. And there's multiple ways of configuring that handoff. **[0:35:14] Sean McGregor:** You can either do like a hard reject and say like, this doesn't pass the guard model and we're going to give you nothing, or you can re prompt the underlying model and say like, but don't, here's what you're being prompted, but don't answer it. You know, there's variations of that, and then you can still get like a somewhat useful reply out of the system. It just should reject it. The configuration of that handoff was looser than it necessarily needed to be. **[0:35:43] Sean McGregor:** And exploiting that handoff between the guard model and the underlying model was the kind of, you know, big vector that you could use that to print cash over the course of the competition. And that's the perhaps the thing people need to think about, particularly as deploying these systems are very often a collection of multiple systems that present as one is. The interface between them is very often untested. It's very often difficult to know how those systems will interact, and particularly when the benchmarks are expressed at a lower level than the whole system, you don't know a lot about what will happen. **[0:36:22] Sean McGregor:** At that point, you've muddied the waters a little bit. **[0:36:24] Chris Benson:** I'm curious, as you went through this exercise with the attendees, beyond just the specifics of this problem, what was the value of going through the exercise? And what, you know, in terms of the outcome for you and what you got out of it? Beyond the immediate guard model with the, you know, the model it's protecting? Um, in that process, like, what was that like? And I'm also curious if you go and do a similar exercise in the future, which I'm assuming there will be some some variation of that. **[0:36:58] Chris Benson:** What can you envision being a good next step to tackle in terms of trying to secure value and get to a next level outcome? You know, have you thought about what you would do as you came out of the previous event? Should you do it again? **[0:37:15] Sean McGregor:** So I think one of the really strong takeaways, and there's actually been some work since then, was the need for tools and to kind of scaffold this in some form and to make it. Where you have a clear form of expression of what these things are called, which is flaw reports, because it's not an incident like no one's harmed. You're kind of, you know, testing things in a laboratory. And so what you have in classical computer security is you have bug bounty systems, you have the ability to submit those. **[0:37:48] Sean McGregor:** There's kind of clear adjudication processes that often lead to payouts to people. And the extensions that are necessary for machine learning based systems, data driven systems, distributional systems is, I think clearer as a result of the activity that we ran at DEFCON. Like that, that's the kind of research value that's produced by that. Allen Institute probably got some operational use as well that they learned a lot because they're at that judge table, dealing with a thought spot as well. **[0:38:21] Sean McGregor:** That was the vendor table for things. And so I think if this is run again in the future, getting past those pain points of, you know, you're kind of rolling your own code and all that, and that's not necessary for this. Scaffolding it a lot more will carry it a lot more distance. And then move this towards something where there's one or more companies that are operating a flaw reporting business in the same way that there is security bug reporting. **[0:38:57] Daniel Whitenack:** well, Sean, as we kind of get close to the wrapping up point here on the podcast, I'd love for you to share anything that comes to mind. In terms of, like, as you look out, kind of at the next phase of maybe your own work, but maybe, you know, the ecosystem a little bit more broad in terms of the area of research in which you are involved. What are you? What are those things that, as you, you know, end your days or you're driving back to home or whatever? **[0:39:28] Daniel Whitenack:** What are those things that stick in your mind? Like, you're excited, like, what if this happens? Or I'm really interested to see how this develops, or I'm encouraged by kind of this going into the future. What's kind of top of mind for you in terms of the community that you're involved with coming into this next year? **[0:39:44] Sean McGregor:** There's an interesting phenomena that every time we pass a milestone in the AI incident database and we're like, Oh, we've crossed 500 or one zero incidents, we're like, yay, and then we're like, but it's not a great thing. Should we celebrate? And Should you celebrate? Yeah, I feel like my dream utopia might be like, I need to go do something else because the safety problem has been solved. And there's no more. There's no more to do. **[0:40:13] Sean McGregor:** Like, the unfortunate fact, though, is like, this is only going to get more complicated and there's more to do, there's more to solve. There's not, there's not going to be a perfect AI system out there. So, you know, what I'm hoping for in the next year is to develop a greater sense of what institutions, what techniques, what methods we need to develop and scale and apply so that we can have a safer AI ecosystem. And this is important, not just from the perspective of wanting to prevent harm. **[0:40:55] Sean McGregor:** But, you know, you can't deploy unsafe systems to clients that want safety, that care about outcomes. And so our ability to make a safer system is very heavily involved in our ability to ship product to solve problems in the real world. And so I hope to highlight that, I hope to have a greater accounting for the risks and make it so that we have better and better measures of those risks. Because you manage what you measure, and so let's measure these risks. **[0:41:35] Sean McGregor:** And then we can separate the actors that are investing in safety and stronger AI systems, safer AI systems, from the ones that aren't doing that. And right now, that's not where we are. **[0:41:50] Daniel Whitenack:** well, really appreciate that perspective and appreciate the work that you continue to be engaged in. Sean, thank you for that and from the community, it's really, really important, so appreciate that. And thank you very much for taking time to chat with us about it. It's been great. **[0:42:05] Sean McGregor:** Oh, it's been a lot of fun. Thank you for having me. **[0:42:15] Ad read:** All right, that's our show for this week. If you haven't checked out our website, head to practicalai fm and be sure to connect with us on LinkedIn X or Bluesky. You'll see us posting insights related to the latest AI developments, and we would love for you to join the conversation. Thanks to our partner prediction guard for providing operational support for the show. Check them out at predictionguard com. Also, thanks to Breakmaster cylinder for the beats and to you for listening. **[0:42:41] Ad read:** That's all for now, but you'll hear from us again next week.