Latent Space: The AI Engineer Podcast
26 Oct 2026 · 16m · Ian Quigley · 20 quoted facts
Leash Bio has turned its huge library of pre-measured chemistry experiments into a test bed for AI agents. Each agent acts as a medicinal chemist with a 100-molecule budget and tries to find the molecule that binds a protein best. Because every possible answer was measured in advance, the researchers can score both the final result and the quality of the scientific decisions along the way.
The central tension is between the speed and scale of this approach and its limits. Rollouts are cheap, at roughly 90,000 so far, and they show that stronger models drop unpromising chemistry sooner. But agents can only pick molecules that were actually measured. Ian Quigley also says nobody yet knows whether skills learned here will carry over to new biology.
Leash measured a large landscape once, so agents can run huge numbers of 'experiments' against it with instant feedback. Quigley calls this a lab in the loop with amortized latency, meaning the lab cost is paid once and spread across many rollouts.
Since the best molecule in the whole environment is known, the team can see how close an agent got, not just whether it succeeded. That exposes behaviors like giving up early or retesting the same molecules despite a limited budget.
Quigley says more performant models leave unproductive neighborhoods of molecules sooner. Weaker models keep making small tweaks to molecules that will never work, and the team does not yet know why.
Last summer frontier models could not reliably convert between equivalent SMILES strings, which are text codes for molecules, so Leash fine-tuned its own. Around February, Quigley says, models became useful with chemistry, which made this agent work possible.
The environment only offers choices that were actually measured, and Quigley says it is unknown whether reinforcement learning gains here carry to other biology. The team plans to scale from two to about twenty new environments a day to test this.
It tests whether language-model agents can act like medicinal chemists. Given a target protein and a budget of 100 molecules, the agent has to find the molecule that binds best. Every molecule's result was measured ahead of time, so the answers come back instantly.
Real lab-in-the-loop work is slow and has limited budgets, so each rollout can only cover a few choices. Quigley says recording the data once lets agents run effectively unlimited rollouts, which is close to a lab with zero delay.
Quigley says more capable models are much better at abandoning unproductive groups of molecules. Weaker models keep making tiny changes to molecules that won't pan out. He says the reason is not yet known.
SMILES is a way to write a molecule's structure as a single line of text, and one molecule can have several valid SMILES strings. Quigley says early frontier models couldn't convert between these forms well, so Leash fine-tuned its own models before the newer ones improved.
No. Quigley says the agents are limited to molecules Leash actually measured, so they can't try any idea they like. He says the set is still large, and Leash plans to add more environments.
Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.
In Leash Bio's structure-activity relationship task, each AI agent gets a budget of 100 molecules and is told only the target protein.
“You have a budget of a hundred molecules. We're not gonna tell you anything except the target and, uh, go find the best binder you can.”
Leash Bio reports running about 90,000 agent rollouts in its pre-measured environment.
“And because it's all pre-measured, you can roll out a lot of these. We're up to about 90,000.”
Ian Quigley says the benchmark separates agents well across different model families and capability levels.
“we get great separation of agents across, uh, model families and different levels, that kind of stuff.”
Leash Bio observed agent failure modes including quitting early, choosing molecules that contradict their own hypotheses, and repeatedly retesting the same molecules despite a limited budget.
“agents that give up early, uh, want to buy molecules that don't agree with their SAR hypotheses, ones that keep retesting the same molecules over and over, despite having a limited budget.”
Ian Quigley says weaker models keep making small variations on molecules that will never work instead of moving on.
“they'll just keep running little subtle changes on molecules that aren't ever gonna go anywhere”
Ian Quigley says medicinal chemists who reviewed the agents' reasoning found it plausible for a structure-activity relationship campaign.
“when we get medicinal chemists to look at it, they're like, Yeah, no, those are very plausible thoughts about, you know, how to conduct this particular part of the SAR campaign.”
Leash Bio previously built a game somewhat like a human version of this task, called Name that chemist.
“We did do a game a little bit like this for Name that chemist.”
Ian Quigley says a new frontier AI model has been released roughly every eleven days.
“there's been a new frontier model drop, what, like, once every eleven days.”
Ian Quigley says DeepSeek V4 Flash performs reasonably well on Leash Bio's benchmark.
“They're, uh, Deepseek V four Flash is pretty good, not that bad.”
Leash Bio started an internal project to fine-tune language models to understand SMILES because no existing frontier model handled it well.
“We had started an internal project to do, uh, supervised fine tuning on some of these language models to get them to understand smiles, because none of the existing frontier models were any good at it.”
Ian Quigley says Anthropic and OpenAI models at that time could not reliably convert SMILES strings into non-canonical SMILES strings.
“none of the Anthropic or or OpenAI models at the time could even convert smiles to non canonical smiles very effectively.”
A single molecule can be written as many different SMILES strings, depending on where the graph traversal starts.
“And so there's multiple smiles ways of representing the same molecule.”
Open-source packages such as RDKit are commonly used to convert between SMILES representations.
“there's a couple open source packages people use, like RD Kit, to do this”
Ian Quigley says that around February AI models became able to work with chemistry in a meaningful way, which led Leash Bio to build this environment.
“around February, like a lot of other things, uh, it became clear that they could start to actually work with chemistry in a meaningful way”
Ian Quigley says agent decisions get much worse when information about the molecules is removed, suggesting the agents use it.
“when we remove the information about the molecules, like, these decisions really suffer.”
Ian Quigley says Leash Bio does not use Bayesian hyperparameter optimization as a baseline because he thinks the landscape is too spiky and too high-dimensional.
“We think it's too spiky of a landscape and it's very high dimensional.”
Leash Bio plans to release a set of environments along with many thousands of agent transcripts alongside its paper.
“we're gonna release a bunch of these environments as hardware packages. And then many thousands of transcripts, uh, for people to go play with”
Since its founding, Leash Bio has screened about a thousand proteins against a couple of its chemical libraries.
“We've screened about a thousand proteins, uh, against a couple of these chemical libraries since we started Leash, and we're gonna start to grow that.”
A host compared Leash Bio's approach to Lila, which runs experiments and uses the results in rollouts rather than performing them live.
“they also do experiments and then they use that in their rollout, but they're not doing them live.”
Ian Quigley says it is not yet known whether reinforcement learning progress in these environments transfers to new biology.
“And then we don't yet know if we're able to take reinforcement learning progress from these kinds of environments and move it into new biology yet.”
Explains the core setup: an agent with a small molecule budget tries to find the best binder to a target protein. This is the framework for everything that follows.
“You have a budget of a hundred molecules.”
Because the whole landscape is measured, researchers can judge decision quality, not just final scores. This reveals behaviors like early quitting and repeated retesting.
“we also know the results of the entire measured landscape or world.”
The main empirical surprise is that stronger models abandon unproductive chemistry faster, while weaker ones keep tweaking dead ends.
“more performant models are much better at abandoning kind of unproductive neighborhoods”
Quigley discusses how frontier and open-source models compare. He says new frontier releases arrive so often that the team struggles to keep results current.
“there's been a new frontier model drop, what, like, once every eleven days”
Describes how molecules are written as text and why early frontier models struggled with that format. This explains why Leash fine-tuned its own models before agents became viable.
“basically a graph traversal across the molecule, written down into a one dimensional string”
Frames the approach as measuring once and rolling out many times. It also notes the trade-off: agents only have access to what was measured.
“it is a lab in the loop with amortized latency”
Leash plans to expand its set of environments sharply while admitting that transfer to new biology is still unproven.
“we're gonna try to get to maybe twenty of these new environments a day”
Hey there, welcome to Latent Space Science. This is Brandon and RJ, uh, joined with Ian Quigley from Leash Bio. We recorded an episode earlier this summer with Ian, where he talked about the cool things Leash was doing with DNA encoded libraries. And how they could screen millions of compounds against, uh, thousands of proteins in a high throughput way. Around this time, Ian realized that they could do fun things with these data. It essentially allows them to create, like, a virtual sandbox where agents can explore and make decisions in real time, as if they were medicinal chemists and get instant feedback.
But with, like, a huge amount of exploratory rollouts. Since that episode, Ian has and Leash have developed this idea much more fully. And since then, they have and they're now releasing a paper and a data set and a blog post, along with the episode that we just released. We wanted to sit down with Ian, uh, talk a bit more about what he's done and, uh, you know, hear about the updates. Um, anyway, welcome to the podcast.
Thank you, Ian. Yeah, so last time we talked to Ian, you were saying that you had some agents exploring a a sandbox. And you were like, Oh, this is just a fun idea. We don't know what we can do with it, but it's an interesting idea. I think you've since fleshed this out into a full environment for agents to explore, maybe do RL on, you know, scientific reasoning. Can you, uh, talk a bit more about that?
Yes, so the thing that I talked about last time was this idea that we had these large collections of screens and that they were big enough environments and counterfactually dense enough that you could take agents and drop them in there and then give them tasks and they could go do those tasks on a limited budget in this landscape that we had measured completely. And so the flagship problem that we spent a lot of time over the past couple months on is a structure activity relationship problem.
And the idea there is like, okay, I've got some molecules and I know that they bind, maybe they bind to this protein that I want them to. How do I change them to make them better? This is a very common task for medicinal chemists to do. And SAR, as it's sort of abbreviated to, also includes other stuff, like certain drug like properties, you know, does it get metabolized? All these things. We're not doing those right now, we're just doing.
Can we find molecules that bind better to the thing that we're interested in? And so what we do is we take model, we take an agent, like a language model, and then we say, here's a catalog that you could look at. You have a budget of a hundred molecules. We're not gonna tell you anything except the target and, uh, go find the best binder you can. And so different agents try different stuff, they have different search policies, they have different ways of wanting to spend their budgets.
And what they'll do is they'll put out in readable language, uh, as they are hypothesis about why they think these molecules are working or not working. Uh, and then they'll go in and with this limited experimental budget, they'll test those hypotheses. And because it's all pre-measured, you can roll out a lot of these. We're up to about 90,000.
So you said you're doing these on foundation models. I think you have, like, a paper or a little, a preprint or something at this point that you can, uh, share. Do you wanna share that or talk about it?
Yeah, so the preprint will probably be out at the same time that this episode comes out. It is pretty ready. And, uh, what we what we show in there is, uh, that we get great separation of agents across, uh, model families and different levels, that kind of stuff. But another thing that's really interesting is because we have not just the results that you could use as a eval for how the agent's doing the job, we also know the results of the entire measured landscape or world.
And so then you can start to interrogate how good is the agent at making scientific decisions? Because maybe it only got to, like, the thing that's half as good as the best thing in the entire environment. And so there's all kinds of interesting behaviors, like agents that give up early, uh, want to buy molecules that don't agree with their SAR hypotheses, ones that keep retesting the same molecules over and over, despite having a limited budget. And also when it's good to explore versus exploit.
And it turns out that more performant models are much better at abandoning kind of unproductive neighborhoods. And that was actually a really interesting surprise for us. Yeah, and so when the paper comes out, we're gonna release a bunch of these environments as hardware packages. And then many thousands of transcripts, uh, for people to go play with and go see how these agents are being, you know, how their how their science is is That that's cool.
super cool. So do you have a feel for why the smarter models are able to figure out that they're in a bad landscape and move or a bad part of the landscape and move?
We don't know that at all yet. All we know is that they're abandoning these crappy neighborhoods more quickly. Uh, and we definitely see a progression of the like a a more willingness to jump around overall, uh, for more performant models. But they especially abandon the the lousy places fast, and then the really poor ones stick around.
It's not just that they jump more, it's that actually they're some they're figuring something out and saying, Hey, this is not good, let's get out of here.
Exactly, exactly, yeah. And what happens with the the dumber models is that they'll just keep running little subtle changes on molecules that aren't ever gonna go anywhere, right? Like, oh, these two, they have a low score, but I feel good about them, and so I'm gonna keep trying stuff. And then, uh, and then the smarter models will abandon the entire process. We're gonna go somewhere else.
Have you looked at the rollouts to see if it's thinking about whatever?
Yeah, thinking about all kinds of stuff. They'll say, you know, when? When we release all these transcripts, like everyone's look at one of my favorite things to do is just take, you know, large numbers of transcripts and just dump them into LLMs and just have it look at the different failure modes and stuff like that. But you'll see, yeah, I think that this ring is good, and I think that this particular nitrogen is painful, or, you know, whatever.
And they'll, they'll say these things, uh, and then they're very plausible, you know, medicinal chemistry narratives. So it's it's pretty, it's pretty fun.
yeah, that's amazing.
So I'm curious, have you given humans these same questions? I have
to have them look and evaluate, like, you know, not a ton, but because, you know, we have we have a large number of them. But when we get medicinal chemists to look at it, they're like, Yeah, no, those are very plausible thoughts about, you know, how to conduct this particular part of the SAR campaign. And
no, maybe I'm. I'm overthinking. Could you make a game out of this? This is sort of a game, right? And then the the medicinal chemistry, the video game, and then you can no, the hook people in these environments capture, and give them,
you want it as a capture.
yeah, do a medicinal chemistry capture. And then that's yeah, dude,
your, that's your new business model, I, I, I like, just
I like this plan. We did do a game a little bit like this for Name that chemist. Um, we would have to probably boil it down to have a smaller number of building blocks, because right now, to have a chemist go in there, you can imagine that, uh, if you're a medicinal chemist, you might have this suite of things that you might want to try over time. Like, put a methyl here, like, move this around. And when we have these pre-measured environments, we are going to be restricted to the things that we did actually measure.
And so then we would have to be asking the medicinal chemist, Oh, well, you can't do whatever you want, you could just do a lot of stuff. And, uh, and figure and then out a way to
they have to know what stuff to.
they have to know what the stuff is. Yeah, so we'd have to figure out a way to surface it to them in a meaningful way. But I kinda like where you're going with this. It's like, we could definitely shrink it down to a smaller number of options, so that's fun. Yeah, we can do that.
yeah. What model is best? Or the frontier models, what are the best ones?
um, let's see. I was looking at the plot a minute ago. It's it's kind of embarrassing that I forgot it already. Uh, right now, uh, the champion is after six. Yeah, that's
not a not a big shock.
not a big shock. Sonnet five, five is actually pretty good too. Yeah, and and Sol six one is also pretty good. So
and everything five is just a raising pile of crap, right? You know, that five, five, yeah, will, Sonnet five is both yeah,
very expensive and not very performant.
yeah, yeah, but yeah, I think five will be long considered a a big mistake by Anthropic. 5.5 s are pretty good. Yeah, yeah, do you have Opus 5.5?
we don't yet. No, that's that's one thing that has been has actually been a little infuriating, uh, as you guys can appreciate. So when we talked last, two months ago, we're starting these rollouts to see what happened. And then there's been a new frontier model drop, what, like, once every eleven days. Eleven days, yeah, Yeah, so, like, every time, oh, we're gonna ship this, we need to drop this public stuff. And like, some other dude's stupid model shows up.
It's like, Ah,
you have to ship faster than Anthropic.
yeah, yeah, right, good, good luck, Ian. So, yes, so we are excited to put in more in here, uh, and we also have run a large number of of the open source ones as well. So we've thrown a bunch of smaller quins in there and GLMs and Kimmy and Deepseeks and that kind of stuff. So there's a lot of these.
how do they do?
They're, uh, Deepseek V four Flash is pretty good, not that bad. Yeah, maybe I should just send you guys, like, the full chart of how things are going. I think that'd be.
Yeah, well, we can just pull this yeah, we up on the screen. can put Yeah, it into yeah, stick and the
it up there. I, I can s I can set it. It kind of looks a little bit like, you know, like the Arc AGI two plots, where you've got some crappy
I don't know.
models, right, right. It does, it does kind of look like that, and that's kind of why we made it like that. And then, uh, I think one thing I mentioned earlier that has been very interesting to us is about this time last summer. We had started an internal project to do, uh, supervised fine tuning on some of these language models to get them to understand smiles, because none of the existing frontier models were any good at it.
So, like, none of the Anthropic or or OpenAI models at the time could even convert smiles to non canonical smiles very effectively. And so So we were like,
just for non chemists, smiles is a language that you can use, string language that you can use to describe accurately a chemical, right?
Yes, yes, that's right. Uh, I I think we mentioned it a little earlier when we recorded last, but it's a string. And molecules can be forked, or or circles, or or both put together, right? These different things. And so a smiles is basically a graph traversal across the molecule, written down into a one dimensional string. And so you can imagine starting the graph traversal at a different point in the molecule. And so there's multiple smiles ways of representing the same molecule.
And so you, it's nice to be fluent in transferring from one to the other. It's a pretty easy. Uh, there's a couple open source packages people use, like RD Kit, to do this, um, but we wanted the language models to do it just cause it would go faster. And they weren't as good at using tools on their own back then, and nobody could do it. And then, around February, like a lot of other things, uh, it became clear that they could start to actually work with chemistry in a meaningful way, which is a reason we started going in this direction.
that's really interesting because that means that it's internalizing some understanding of the phys, the physical nature of molecules.
I think a little bit, yep, like, these are the tasks that we're having these things do right now. Like, they are iterative tasks where they're looking at molecules and they are learning something about those molecules to make choices to make better molecules next. And when we remove the information about the molecules, like, these decisions really suffer. So they're, they're learning something. This is very interesting, or at least they're using it. I guess that's a better way.
We're not, we're not changing weights here, but they're definitely using the information just to make future choices.
you mentioned all these, this zoo of models, right, that are you've tried. So I wanted to ask, have you just used, like, one of these Bayesian hyperparameter algorithms to to as like a a deterministic baseline for your problem?
We think it's too spiky of a landscape and it's very high dimensional. We're not really, we don't do that, but we mostly think it's not gonna work. I think it's the party line. Yeah,
I know you plan on releasing all this, so I guess we'll put the use. Wait, what are you releasing? You said you're you're gonna have a a preprint blog and then some data rollouts and everything, so we can put all this in the show notes. Is there, uh, anything else? yeah, that's Yeah,
it. Yeah, I think I think that's the big thing, and that's probably something closer to like, I guess, I guess what I would probably say is. We have built this thing where we, if you've got scientific agents and you wanna know how they're behaving, what mistakes they might make, what, how they might learn from physical feedback. You know, if a new training recipe improves scientific decision making, we've got the capabilities to measure these things at scale.
yeah, that's And
really cool. Do you have plans for other similar problems for this same kind of thing?
we do, yes, we have. Broadly, the way that we think about this is it is a lab in the loop with amortized latency. So it's like, go out, you record the thing once, and then you can have a huge number of rollouts through that thing, effectively infinite rollouts. And it. We are thinking hard about other types of biological or scientific questions that you could transmute into these kinds of environments. Because if you're doing conventional lab in the loop, it does take a long time.
And and it, and because those things themselves have limited budgets, it's also not counterfactually dense. So, like, you gotta keep running it and do different choices with every rollout to make the things, to make the agents learn,
right, but with your, your multiplexed strategy, then that, then you can just amortize.
you could just amortize. So it's, it's. It's not quite accurate to say it's a lab in the loop with zero latency, but it's pretty close. You I mean, know, like these
this is the same thing that Lila's doing. I mean, it's
I agree.
not the same thing, but they also do experiments and then they use that in their rollout, but they're not doing them live.
right, right, yeah, yeah. Doing it, doing it in the rollout one step at a time offers a lot more freedom because you can choose almost anything. And here, you're restricted to the number of choices that we made when we pre-measured the environment. That's definitely true, um, but it's like it's still a pretty big number of choices. And, uh, yeah, and then we have right now, we're, we have. We've screened about a thousand proteins, uh, against a couple of these chemical libraries since we started Leash, and we're gonna start to grow that.
I think we're gonna try to get to maybe twenty of these new environments a day. That's a plan for us to try to get to beginning of maybe twenty six or twenty seven. We can, we're doing two a day right now. Uh, so yeah, I think we could probably get to twenty a day without too much lifting. So we'll, we're, we're pushing, we'll see if we can do it.
wow, and so you're gonna have this huge RL environment.
Yeah, yeah, it seemed like it seemed like the thing to do. I mean, kind of informally, it seemed like where a lot of these agents are making the best and the fastest progress is where there's instant feedback that's very scalable. And it's really hard to do that in the physical world. Um, but we think that there's probably a handful of biological questions that you could re-rebuild in this way. Uh, go after them. And even, uh, yeah.
And then we don't yet know if we're able to take reinforcement learning progress from these kinds of environments and move it into new biology yet. Um, but, you know, like, we're trying, we're gonna find out. It's not a lot.
that's that's the obvious. Q Like, how about how does this transfer to other biology? You can imagine if you did it for some sort of covering set, right? That you might be able to get things in between, though, that you could transfer to pretty well.
Yep, yep. And then also one thing that I need to do is, uh, talk to some more decision people. Because a lot of this is about how do you spend a limited experimental budget, and then what can you do to do better at that? Yeah, beyond just medicinal chemistry, right? Just like, Yeah, when do I stay, when do I go? How do I? How do I get better at that? interesting, yeah, you can Yeah,
imagine that if it knows how to connect a decision at that point to a knowledge that it has, then it could learn that skill for other problems.
yes, yes, see, we are. We are incorporating more and more structural tool calls and things like that with this as well. So it's a it's, it's a pretty fun time right now.
yeah, we'll be sure to link to all that cool stuff you have coming out. And, uh, yeah, thanks for taking the time to chat. all right, you, Yeah, or thanks thanks. for taking the time for a follow-up chat. Uh, yeah, I think this will actually appear back to back with the rest of the episode that you
okay, this was great. so, Well, thanks guys. anyway, thanks, It was fun. see ya Oh, it was fun. Yep, talk bye. to you later.