Sean McGregor earned a PhD in machine learning around 2017, focused on reinforcement learning for wildfire suppression policy.
“a 2017 vintage PhD in machine learning. I focused on reinforcement learning as applied to wildfire suppression policy.”
Person
Sean McGregor
1 episodes · 18 quoted claims
Sean McGregor worked at an organization called Centient that shipped millions of edge neural network processors, used in devices such as hearing aids.
“The organization called Centient that shipped when I was there millions and has continued to ship many edge neural network processors.”
McGregor's test-and-evaluation company sold its assets to Underwriters Laboratories, an established safety organization.
“And we sold the assets associated with that to a old safety organization called Underwriters Laboratories”
McGregor says aviation records crashes so the same failure is not repeated, which works like a regression test, and he models the AI Incident Database on this idea.
“you see this in aviation, a plane crashes, you record what happens and use that to make sure effectively in the aviation industry, you have a form of regression test”
The intergovernmental definition of an AI incident is an event in which a harm has taken place.
“the intergovernmental definition of incident that developed was effectively an event that a harm has taken place, that's an incident.”
Most entries in the AI Incident Database come from journalistic reporting, because journalists work to validate the base facts.
“journalists actually put in a tremendous amount of work to validate the base facts involved in it”
The EU code of practice includes a requirement to report severe AI incidents, but McGregor says it had not yet been implemented.
“In the EU code of practice, there is a requirement for severe incidents to be reported. It hasn't reached implementation as of yet.”
AVERI performs third-party audits on the premise that an outside party is better placed than the builders to judge whether an AI system is safe.
“there's a premise in there that a third party is better positioned to do that than a first party.”
McGregor cites an AI incident in which a traffic camera misread a woman's shirt text as a license plate and a man received a citation.
“this gentleman who got a traffic citation mailed to him, and he looked at the traffic citation and said, like, you know, this isn't my car”
McGregor argues audited financials are expected by investors, and that this is a model for auditing AI systems.
“having audited financials is table stakes. You're not going to put your money into an org that hasn't been audited.”
McGregor says the BBQ benchmark was foundational for measuring bias in AI models, but it was not designed to certify a model as unbiased for a specific deployment.
“and BBQ is a great benchmark. It was really foundational in the bias benchmarking community”
The BenchRisk project found that most AI benchmarks were built for research rather than for practical deployment decisions, with mixed results on mitigating known failure modes.
“by and large, most benchmarks historically have been produced for research purposes, not for practical AI purposes.”
McGregor says security specialists tend to presume bad actors, while safety specialists tend to presume the world itself can cause harm.
“And security people have a much greater capacity for the presumption of bad actors doing bad things”
The DEFCON AI Village challenge McGregor describes was called the Generative Red Team Two, a change in form from the prior year.
“this was something called the Generative Red Team Two.”
The DEFCON challenge targeted a 7-billion-parameter open language model from the Allen Institute for AI, protected by a guard model placed in front of it.
“7 billion parameter large language model, along with its guard model, put in front of it”
McGregor says the DEFCON challenge required participants to show systematic weaknesses using statistics rather than single exploits.
“what we basically had to bring to the security world, which the security world does not like this, and I don't blame them for, is statistics.”
The most widely exploited weakness at the DEFCON challenge was the loosely configured handoff between the guard model and the underlying chat model.
“exploiting that handoff between the guard model and the underlying model was the kind of, you know, big vector that you could use that to print cash over the course of the competition”
McGregor calls reports of weaknesses found in testing flaw reports, because no one has been harmed, and wants flaw-reporting businesses like bug bounty systems for machine learning.
“which is flaw reports, because it's not an incident like no one's harmed.”