Person
1 episodes · 19 quoted claims
In Leash Bio's structure-activity relationship task, each AI agent gets a budget of 100 molecules and is told only the target protein.
“You have a budget of a hundred molecules. We're not gonna tell you anything except the target and, uh, go find the best binder you can.”
Leash Bio reports running about 90,000 agent rollouts in its pre-measured environment.
“And because it's all pre-measured, you can roll out a lot of these. We're up to about 90,000.”
Ian Quigley says the benchmark separates agents well across different model families and capability levels.
“we get great separation of agents across, uh, model families and different levels, that kind of stuff.”
Leash Bio observed agent failure modes including quitting early, choosing molecules that contradict their own hypotheses, and repeatedly retesting the same molecules despite a limited budget.
“agents that give up early, uh, want to buy molecules that don't agree with their SAR hypotheses, ones that keep retesting the same molecules over and over, despite having a limited budget.”
Ian Quigley says weaker models keep making small variations on molecules that will never work instead of moving on.
“they'll just keep running little subtle changes on molecules that aren't ever gonna go anywhere”
Ian Quigley says medicinal chemists who reviewed the agents' reasoning found it plausible for a structure-activity relationship campaign.
“when we get medicinal chemists to look at it, they're like, Yeah, no, those are very plausible thoughts about, you know, how to conduct this particular part of the SAR campaign.”
Leash Bio previously built a game somewhat like a human version of this task, called Name that chemist.
“We did do a game a little bit like this for Name that chemist.”
Ian Quigley says a new frontier AI model has been released roughly every eleven days.
“there's been a new frontier model drop, what, like, once every eleven days.”
Ian Quigley says DeepSeek V4 Flash performs reasonably well on Leash Bio's benchmark.
“They're, uh, Deepseek V four Flash is pretty good, not that bad.”
Leash Bio started an internal project to fine-tune language models to understand SMILES because no existing frontier model handled it well.
“We had started an internal project to do, uh, supervised fine tuning on some of these language models to get them to understand smiles, because none of the existing frontier models were any good at it.”
Ian Quigley says Anthropic and OpenAI models at that time could not reliably convert SMILES strings into non-canonical SMILES strings.
“none of the Anthropic or or OpenAI models at the time could even convert smiles to non canonical smiles very effectively.”
A single molecule can be written as many different SMILES strings, depending on where the graph traversal starts.
“And so there's multiple smiles ways of representing the same molecule.”
Open-source packages such as RDKit are commonly used to convert between SMILES representations.
“there's a couple open source packages people use, like RD Kit, to do this”
Ian Quigley says that around February AI models became able to work with chemistry in a meaningful way, which led Leash Bio to build this environment.
“around February, like a lot of other things, uh, it became clear that they could start to actually work with chemistry in a meaningful way”
Ian Quigley says agent decisions get much worse when information about the molecules is removed, suggesting the agents use it.
“when we remove the information about the molecules, like, these decisions really suffer.”
Ian Quigley says Leash Bio does not use Bayesian hyperparameter optimization as a baseline because he thinks the landscape is too spiky and too high-dimensional.
“We think it's too spiky of a landscape and it's very high dimensional.”
Leash Bio plans to release a set of environments along with many thousands of agent transcripts alongside its paper.
“we're gonna release a bunch of these environments as hardware packages. And then many thousands of transcripts, uh, for people to go play with”
Since its founding, Leash Bio has screened about a thousand proteins against a couple of its chemical libraries.
“We've screened about a thousand proteins, uh, against a couple of these chemical libraries since we started Leash, and we're gonna start to grow that.”
Ian Quigley says it is not yet known whether reinforcement learning progress in these environments transfers to new biology.
“And then we don't yet know if we're able to take reinforcement learning progress from these kinds of environments and move it into new biology yet.”