2024
Dario Amodei says the updated Claude 3.5 Sonnet scored about 50% on the SWE-bench software engineering benchmark.
“the latest model we released, Sonnet 3.5, the new or updated version, it gets something like 50% on Swee Bench.”
Topic
SWE-bench
6 facts from 5 episodes, in date order. These are what speakers said on the shows, each linked to the moment it was said. They aren't independently verified.
2025
Pavel Veller said CodeMie ranked fifth on the SWE-bench Verified list at the time of recording.
“By the way, you guys can go to SWE bench and look at verified list. I believe code me as of now is, uh, takes 5th.”
2025
Dohmke describes the SWE-bench benchmark as pairs of issues and pull requests drawn from a dozen Python repositories.
“which is two zero pairs of issues and pull requests from a dozen Python repos”
2025
Dohmke says that on the multilingual version of SWE-bench, built by its founders, agent scores fall to roughly the 20s to 30 percent range.
“with this multilingual, um, I think the numbers go significantly down”
Undated
Kanjun Qiu describes SWE-bench as a software engineering benchmark based on GitHub issues, on which models scored under 40%.
“if you look at a data set like Sweet Bench, uh, so this is like a software engineering GitHub issues, uh, style benchmark.”
Undated
Sid Pardeshi says turning off thinking drops Claude Opus 4.5's score by five to ten percentage points even on SWE-bench.
“The moment you turn off thinking, five to ten percentage points drops, even on Swee Bench”
Episodes
- 11 Nov 2024#452 – Dario Amodei: Anthropic CEO on Claude, AGI & the Future of AI & Humanity5h 22m
- 10 Jun 2025GitHub CEO Thomas Dohmke on Copilot and the Future of Software Development1h 9m
- 8 Aug 2024Reinventing AI Agents with Imbue CEO Kanjun Qiu48m
- 5 Feb 2026Infinite Code Context: AI Coding at Enterprise Scale w/ Blitzy CEO Brian Elliott & CTO Sid Pardeshi1h 53m
- 14 Apr 2025Orchestrating agents, APIs, and MCP servers42m