Person
1 episodes · 24 quoted claims
Sander Schulhoff says the first generative AI red teaming competition he ran was sponsored by OpenAI, Scale, Hugging Face and about ten other AI companies.
“We had OpenAI, Scale, Hugging Face, about ten other AI companies sponsor it, and we ran this thing”
Schulhoff says the paper from his competition won best paper at a top natural language processing conference in 2023, out of about 20,000 submissions.
“that paper went on to win best paper at MLP 2023 out of about 20,000 submissions”
Schulhoff says the competition's prompt injection dataset is used by every frontier AI lab and most Fortune 500 companies to benchmark their models.
“The paper and the dataset are now used by every single frontier lab, uh, and most fortune 500 companies to benchmark their models”
Schulhoff says he wrote the first prompt engineering guide on the internet at Learn Prompting, which led him into AI security.
“I wrote the first guide on the internet on learn prompting, uh, and that interest led me into AI security”
Schulhoff says the first public example of prompt injection was a Twitter chatbot by Remotely.io, which was tricked into making threats against the president.
“The very first example of prompt injection publicly on the internet, um, was this Twitter chatbot by a company called Remotely dot io”
Schulhoff says the MathGPT website ran AI-written code on its own server, and an attacker exploited this to steal the site's OpenAI API key.
“Somebody realized that if you get it to write malicious code, you can exfiltrate application secrets and kind of do whatever to that app.”
Schulhoff says the person behind the Las Vegas Cybertruck bombing used ChatGPT to plan it, though he does not know whether a jailbreak was needed.
“the Vegas Cybertruck explosion incident, uh, bombing, rather, and the person behind that used ChatGPT to plan out this bombing”
Schulhoff says a group hijacked Anthropic's Claude Code to carry out a cyberattack.
“group was able to hijack Claude Code into, uh, into performing a cyber attack, basically”
Schulhoff says the Claude Code attackers evaded defenses by splitting their requests into smaller steps that each looked legitimate on its own.
“if you separate your requests in an appropriate way, you can get around defenses very well”
Schulhoff describes the common guardrail setup as one filter model screening inputs to an AI and another screening its outputs before they reach the user.
“So you put another guardrail on the other side to watch the outputs from the model. And before you show outputs to the user, you check if they're malicious or not.”
Attack success rate (ASR) is the share of attacks that get through a system, and it is a common measure of adversarial robustness.
“So ASR is the term you'll commonly hear used here, and it's a measure of adversarial robustness. So it stands for attack success rate.”
Schulhoff says the number of possible attack prompts against a model like GPT-5 is one followed by a million zeros, which he calls effectively infinite.
“And for a model like GPT five, the number of possible attacks is one followed by a million zeros.”
Schulhoff says a study of adaptive attacks he co-authored found human attackers broke 100% of the defenses tested, in roughly ten to thirty attempts.
“first of all, humans break everything. 100% of of the defenses in, uh, maybe like ten to thirty attempts”
Schulhoff says automated attack systems needed a couple of orders of magnitude more attempts than human attackers to succeed.
“it takes the automated systems a couple orders of magnitude more attempts to be successful”
Schulhoff says defenses that rely on instructions inside the prompt have been known not to work since early 2023.
“Prompt based defenses are the worst of the worst defenses, and we've known this since early 2023.”
Schulhoff says the Comet AI browser could be tricked by malicious text on a webpage into leaking the user's account data.
“we recently saw the, uh, the Comet browser have an issue with this, where somebody crafted a malicious, uh, chunk of text on a web page.”
Schulhoff describes CaMeL, a framework from Google, as granting an AI agent only the permissions a user's request needs, so a hidden instruction cannot trigger other actions.
“Depending on what the user wants, we might be able to restrict the possible actions of the agent ahead of time so it can't possibly do anything malicious.”
Schulhoff describes the AI safety field of 'control' as asking how to make a deliberately malicious AI useful while ensuring nothing bad happens.
“Not only do you have a god in the box, but that god is angry, that god's malicious, that god wants to hurt you.”
Schulhoff says the MATS research program, which he is involved with, stands for ML Alignment and Theory Scholars, with the name under revision.
“which stands for ML Alignment and Theorem Scholars, and, uh, maybe Theory Scholars”
Schulhoff says Anthropic's constitutional classifiers made it harder to extract chemical, biological, radiological and nuclear information from Claude, though humans can still do it in under an hour.
“if you look at, like, Anthropic's constitutional classifiers, it's much more difficult to get, like, CIR information out of Claude models than it used to be”
Schulhoff says one proposed way to improve robustness is to apply adversarial training earlier, during pre-training, though he has not seen it tried at scale.
“there's this idea that if you kind of start doing adversarial training in pre training, uh, earlier in the training stack”
Schulhoff praises a Repello product that scans a company's systems to discover which AI deployments are running, including ones leaders did not know about.
“a product that looked at a company's systems and figures out, like, what AIs are even running at the company”
Schulhoff says adversarial attacks on image classifiers, such as altering stop signs, never became a real-world problem, unlike attacks on LLM-powered agents.
“I guess nobody went through the effort of, like, placing tape on the stop sign in the exact way to, like, trick the self driving car”
Schulhoff predicts a market correction for AI guardrail and automated red teaming companies within a year, as buyers realize the products don't work.
“I think we're gonna see a market correction in the next year. Maybe in the next six months.”