2023
Schulhoff says the paper from his competition won best paper at a top natural language processing conference in 2023, out of about 20,000 submissions.
“that paper went on to win best paper at MLP 2023 out of about 20,000 submissions”
Topic
Sander Schulhoff
11 facts from 1 episode, in date order. These are what speakers said on the shows, each linked to the moment it was said. They aren't independently verified.
2023
Schulhoff says defenses that rely on instructions inside the prompt have been known not to work since early 2023.
“Prompt based defenses are the worst of the worst defenses, and we've known this since early 2023.”
Undated
Sander Schulhoff says the first generative AI red teaming competition he ran was sponsored by OpenAI, Scale, Hugging Face and about ten other AI companies.
“We had OpenAI, Scale, Hugging Face, about ten other AI companies sponsor it, and we ran this thing”
Undated
Schulhoff says the competition's prompt injection dataset is used by every frontier AI lab and most Fortune 500 companies to benchmark their models.
“The paper and the dataset are now used by every single frontier lab, uh, and most fortune 500 companies to benchmark their models”
Undated
Schulhoff says he wrote the first prompt engineering guide on the internet at Learn Prompting, which led him into AI security.
“I wrote the first guide on the internet on learn prompting, uh, and that interest led me into AI security”
Undated
Attack success rate (ASR) is the share of attacks that get through a system, and it is a common measure of adversarial robustness.
“So ASR is the term you'll commonly hear used here, and it's a measure of adversarial robustness. So it stands for attack success rate.”
Undated
Schulhoff says a study of adaptive attacks he co-authored found human attackers broke 100% of the defenses tested, in roughly ten to thirty attempts.
“first of all, humans break everything. 100% of of the defenses in, uh, maybe like ten to thirty attempts”
Undated
Schulhoff says automated attack systems needed a couple of orders of magnitude more attempts than human attackers to succeed.
“it takes the automated systems a couple orders of magnitude more attempts to be successful”
Undated
Schulhoff says one proposed way to improve robustness is to apply adversarial training earlier, during pre-training, though he has not seen it tried at scale.
“there's this idea that if you kind of start doing adversarial training in pre training, uh, earlier in the training stack”
Undated
Schulhoff says adversarial attacks on image classifiers, such as altering stop signs, never became a real-world problem, unlike attacks on LLM-powered agents.
“I guess nobody went through the effort of, like, placing tape on the stop sign in the exact way to, like, trick the self driving car”
Undated
Schulhoff predicts a market correction for AI guardrail and automated red teaming companies within a year, as buyers realize the products don't work.
“I think we're gonna see a market correction in the next year. Maybe in the next six months.”
Episodes