Person
1 episodes · 23 quoted claims
AIM Labs demonstrated a vulnerability in Microsoft Copilot called EchoLeak, in which a crafted email caused Copilot to exfiltrate information.
“They did demonstrate a vulnerability on Copilot. Call they called it Echo Leak. So basically, it's the same RAG concept. You send an email, Copilot is just a big RAG.”
The EchoLeak researchers bypassed Microsoft Copilot's regular-expression filtering of markdown syntax using a cleverly crafted markdown form.
“the guys found a way around it to bypass the regular expression that copilot was using.”
Capitella says a main way to exfiltrate data from LLM applications is to make the model output a markdown image whose URL points to an attacker-controlled server and carries stolen data in the query string.
“one of the main vectors to exfiltrate information in LLM applications is to make the LLM produce a a markdown image. And in the URL, uh, you can, you can point the URL to an attacker controlled server”
Donato Capitella presented his team's research at Black Hat in Toronto shortly before recording.
“I am actually just back from Canada, uh, where I was presenting our research at Black Hat, uh, in Toronto.”
Capitella attended a two-day conference in Stockholm called Secure AI, devoted to generative AI security.
“And before that, I was at another conference in Stockholm called Secure AI, a complete two days just focused on gen AI security.”
Capitella says his company's generative AI security work pipeline has tripled in size since the previous conversation.
“I think our pipeline is tripled in size, uh, and we've been doing a lot of research.”
The paper 'Design Patterns to Secure LLM Agents Against Prompt Injection' collects design patterns for defending agents against prompt injection.
“So this paper is called Design Patterns to Secure LLM Agents Against Prompt Injection, and I already like the title of the paper because it's telling you exactly what's in the paper.”
The design patterns paper identifies six patterns, chosen according to the trade-off between security and a use case's usefulness.
“And then they find, uh, they crystallize six design patterns that you can apply, depending on your trade offs between security for that use case”
In the action selector pattern, the LLM only picks a fixed action from the user's input, which Capitella describes as the most secure pattern.
“You are just using the LLM to basically select a fixed action from the user input, so that kind of removes often in that case any anything bad the attacker can do.”
The code-then-execute approach was published by Google under the name CaMeL, which has its own dedicated paper.
“this was published by Google, and I think they call it Camel. There is a dedicated paper to that.”
In code-then-execute, the LLM writes its plan as a Python program and commits to executing it unchanged.
“the LLM agent is prompted to create a plan in the form of, uh, a Python snippet of code, where it's gonna commit to executing that program exactly as it is”
Capitella compares the policy enforcement in code-then-execute to SELinux on the Linux kernel, calling it a reference monitor for LLM agents.
“It's very reminiscent for people in cybersecurity of what SE Linux does on on a Linux kernel. So it's kind of this reference monitor, uh, for for LLM agents.”
Capitella implemented all six design patterns from the paper in one weekend and published them in a Git repository.
“I like this paper so much that one weekend I actually implemented all of the these six design patterns. I think I I put it in a Git repo.”
Capitella's team began penetration testing LLM applications in 2023 and initially did much of it manually.
“when, uh, we started doing pen testing of LLM applications in, uh, 2023, we were doing a lot of stuff manually”
A Reversec colleague wrote a Spikee module using a headless browser, Playwright, to test a chatbot that used web sockets.
“he wrote a Spikee module that's got a play right, so the Spikee module used a headless browser to open the chatbot, send the prompt and read the response.”
On one AWS-based engagement, prompts were submitted by dropping a file into an S3 bucket that triggered a Lambda, with results appearing in another S3 bucket.
“And the way you introduce the prompt is by dropping a file on an S3 bucket, calling a Lambda, and then in another S3 bucket.”
Spikee can run up to 100 variations of each dataset entry using the best-of-n attack, which adds noise to bypass guardrails.
“For each of these entries in the data set, I want you to try up to 100 variations using the uh, best of n attack.”
Capitella's team developed an anti-spotlighting attack that tries to break the spotlighting defence by introducing tags so the LLM cannot tell where data begins and ends.
“the anti spot lighting attack, which is another attack that we developed, where you try to break spot lighting by introducing, uh, tags and strange stuff.”
Spikee uses seeds rather than prebuilt datasets, and customizing a dataset takes about five to ten minutes.
“you have seeds that allow you to build data sets, which can take five to ten minutes to customize.”
A Reversec consultant in the US, named as William Taylor, found a customer-support RAG system in which attacker-written emails could be retrieved into other users' contexts.
“One of our consultants in the US. was doing a test a couple of weeks ago.”
A Hugging Face speaker presented on MCP prompt-injection risks at the Secure AI conference in Stockholm.
“Sean from Hugging Face was talking about it at Secure AI just again in Stockholm a couple of weeks ago.”
Capitella said he believes Sam Altman claimed the previous year that hallucinations would be solved, using this to express skepticism that prompt injection will soon be solved.
“I believe Sam Altman last year said they would have solved hallucinations, and I am not gonna continue,”
Capitella argues that rate-limiting attackers and responding to repeated guardrail triggers parallels password-lockout protections against brute-force guessing.
“Likewise, you don't allow the attacker to send 100,000 prompts per second to explore, to find something that's gonna jailbreak it.”