# ⚡ An Eval for AI Scientists That Knows the Answer Key - Ian Quigley, Leash Bio - **Show:** Latent Space: The AI Engineer Podcast - **Published:** 26 Oct 2026 - **Source:** https://www.youtube.com/watch?v=vhPCo-8K78U - **Transcript:** https://podcasts.opennash.com/shows/latent-space/an-eval-for-ai-scientists-that-knows-the-answer-key-ian-quigley-leash-bio-53e231/ - **Provider:** OpenNash Podcasts (machine-generated) ## In brief Leash Bio has turned its huge library of pre-measured chemistry experiments into a test bed for AI agents. Each agent acts as a medicinal chemist with a 100-molecule budget and tries to find the molecule that binds a protein best. Because every possible answer was measured in advance, the researchers can score both the final result and the quality of the scientific decisions along the way. The central tension is between the speed and scale of this approach and its limits. Rollouts are cheap, at roughly 90,000 so far, and they show that stronger models drop unpromising chemistry sooner. But agents can only pick molecules that were actually measured. Ian Quigley also says nobody yet knows whether skills learned here will carry over to new biology. ## Key insights 1. **Pre-measured data makes lab experiments replayable** Leash measured a large landscape once, so agents can run huge numbers of 'experiments' against it with instant feedback. Quigley calls this a lab in the loop with amortized latency, meaning the lab cost is paid once and spread across many rollouts. 2. **Knowing the full answer key grades decisions** Since the best molecule in the whole environment is known, the team can see how close an agent got, not just whether it succeeded. That exposes behaviors like giving up early or retesting the same molecules despite a limited budget. 3. **Stronger models abandon bad chemistry faster** Quigley says more performant models leave unproductive neighborhoods of molecules sooner. Weaker models keep making small tweaks to molecules that will never work, and the team does not yet know why. 4. **Chemistry reasoning improved sharply around February** Last summer frontier models could not reliably convert between equivalent SMILES strings, which are text codes for molecules, so Leash fine-tuned its own. Around February, Quigley says, models became useful with chemistry, which made this agent work possible. 5. **Transfer to new biology is still unproven** The environment only offers choices that were actually measured, and Quigley says it is unknown whether reinforcement learning gains here carry to other biology. The team plans to scale from two to about twenty new environments a day to test this. ## Questions this episode answers ### What is Leash Bio actually testing with these AI agents? It tests whether language-model agents can act like medicinal chemists. Given a target protein and a budget of 100 molecules, the agent has to find the molecule that binds best. Every molecule's result was measured ahead of time, so the answers come back instantly. ### Why is a pre-measured environment better than running real lab experiments? Real lab-in-the-loop work is slow and has limited budgets, so each rollout can only cover a few choices. Quigley says recording the data once lets agents run effectively unlimited rollouts, which is close to a lab with zero delay. ### What do the experiments show about smarter AI models? Quigley says more capable models are much better at abandoning unproductive groups of molecules. Weaker models keep making tiny changes to molecules that won't pan out. He says the reason is not yet known. ### What is SMILES and why did it matter? SMILES is a way to write a molecule's structure as a single line of text, and one molecule can have several valid SMILES strings. Quigley says early frontier models couldn't convert between these forms well, so Leash fine-tuned its own models before the newer ones improved. ### Can the AI use anything it wants, like a real chemist could? No. Quigley says the agents are limited to molecules Leash actually measured, so they can't try any idea they like. He says the set is still large, and Leash plans to add more environments. ## Facts - **Ian Quigley** [0:02:21]: "You have a budget of a hundred molecules. We're not gonna tell you anything except the target and, uh, go find the best binder you can." - **Ian Quigley** [0:02:47]: "And because it's all pre-measured, you can roll out a lot of these. We're up to about 90,000." - **Ian Quigley** [0:03:13]: "we get great separation of agents across, uh, model families and different levels, that kind of stuff." - **Ian Quigley** [0:03:41]: "agents that give up early, uh, want to buy molecules that don't agree with their SAR hypotheses, ones that keep retesting the same molecules over and over, despite having a limited budget." - **Ian Quigley** [0:05:06]: "they'll just keep running little subtle changes on molecules that aren't ever gonna go anywhere" - **Ian Quigley** [0:06:04]: "when we get medicinal chemists to look at it, they're like, Yeah, no, those are very plausible thoughts about, you know, how to conduct this particular part of the SAR campaign." - **Ian Quigley** [0:06:33]: "We did do a game a little bit like this for Name that chemist." - **Ian Quigley** [0:08:02]: "there's been a new frontier model drop, what, like, once every eleven days." - **Ian Quigley** [0:08:49]: "They're, uh, Deepseek V four Flash is pretty good, not that bad." - **Ian Quigley** [0:09:09]: "We had started an internal project to do, uh, supervised fine tuning on some of these language models to get them to understand smiles, because none of the existing frontier models were any good at it." - **Ian Quigley** [0:09:35]: "none of the Anthropic or or OpenAI models at the time could even convert smiles to non canonical smiles very effectively." - **Ian Quigley** [0:09:53]: "And so there's multiple smiles ways of representing the same molecule." - **Ian Quigley** [0:10:19]: "there's a couple open source packages people use, like RD Kit, to do this" - **Ian Quigley** [0:10:19]: "around February, like a lot of other things, uh, it became clear that they could start to actually work with chemistry in a meaningful way" - **Ian Quigley** [0:10:55]: "when we remove the information about the molecules, like, these decisions really suffer." - **Ian Quigley** [0:11:42]: "We think it's too spiky of a landscape and it's very high dimensional." - **Ian Quigley** [0:04:09]: "we're gonna release a bunch of these environments as hardware packages. And then many thousands of transcripts, uh, for people to go play with" - **Ian Quigley** [0:13:49]: "We've screened about a thousand proteins, uh, against a couple of these chemical libraries since we started Leash, and we're gonna start to grow that." - **Co-host** [0:13:42]: "they also do experiments and then they use that in their rollout, but they're not doing them live." - **Ian Quigley** [0:15:07]: "And then we don't yet know if we're able to take reinforcement learning progress from these kinds of environments and move it into new biology yet." ## Transcript **[0:00:04] Host:** Hey there, welcome to Latent Space Science. This is Brandon and RJ, uh, joined with Ian Quigley from Leash Bio. We recorded an episode earlier this summer with Ian, where he talked about the cool things Leash was doing with DNA encoded libraries. And how they could screen millions of compounds against, uh, thousands of proteins in a high throughput way. Around this time, Ian realized that they could do fun things with these data. It essentially allows them to create, like, a virtual sandbox where agents can explore and make decisions in real time, as if they were medicinal chemists and get instant feedback. **[0:00:39] Host:** But with, like, a huge amount of exploratory rollouts. Since that episode, Ian has and Leash have developed this idea much more fully. And since then, they have and they're now releasing a paper and a data set and a blog post, along with the episode that we just released. We wanted to sit down with Ian, uh, talk a bit more about what he's done and, uh, you know, hear about the updates. Um, anyway, welcome to the podcast. **[0:01:06] Host:** Thank you, Ian. Yeah, so last time we talked to Ian, you were saying that you had some agents exploring a a sandbox. And you were like, Oh, this is just a fun idea. We don't know what we can do with it, but it's an interesting idea. I think you've since fleshed this out into a full environment for agents to explore, maybe do RL on, you know, scientific reasoning. Can you, uh, talk a bit more about that? **[0:01:30] Ian Quigley:** Yes, so the thing that I talked about last time was this idea that we had these large collections of screens and that they were big enough environments and counterfactually dense enough that you could take agents and drop them in there and then give them tasks and they could go do those tasks on a limited budget in this landscape that we had measured completely. And so the flagship problem that we spent a lot of time over the past couple months on is a structure activity relationship problem. **[0:01:59] Ian Quigley:** And the idea there is like, okay, I've got some molecules and I know that they bind, maybe they bind to this protein that I want them to. How do I change them to make them better? This is a very common task for medicinal chemists to do. And SAR, as it's sort of abbreviated to, also includes other stuff, like certain drug like properties, you know, does it get metabolized? All these things. We're not doing those right now, we're just doing. **[0:02:21] Ian Quigley:** Can we find molecules that bind better to the thing that we're interested in? And so what we do is we take model, we take an agent, like a language model, and then we say, here's a catalog that you could look at. You have a budget of a hundred molecules. We're not gonna tell you anything except the target and, uh, go find the best binder you can. And so different agents try different stuff, they have different search policies, they have different ways of wanting to spend their budgets. **[0:02:47] Ian Quigley:** And what they'll do is they'll put out in readable language, uh, as they are hypothesis about why they think these molecules are working or not working. Uh, and then they'll go in and with this limited experimental budget, they'll test those hypotheses. And because it's all pre-measured, you can roll out a lot of these. We're up to about 90,000. **[0:03:05] Host:** So you said you're doing these on foundation models. I think you have, like, a paper or a little, a preprint or something at this point that you can, uh, share. Do you wanna share that or talk about it? **[0:03:13] Ian Quigley:** Yeah, so the preprint will probably be out at the same time that this episode comes out. It is pretty ready. And, uh, what we what we show in there is, uh, that we get great separation of agents across, uh, model families and different levels, that kind of stuff. But another thing that's really interesting is because we have not just the results that you could use as a eval for how the agent's doing the job, we also know the results of the entire measured landscape or world. **[0:03:41] Ian Quigley:** And so then you can start to interrogate how good is the agent at making scientific decisions? Because maybe it only got to, like, the thing that's half as good as the best thing in the entire environment. And so there's all kinds of interesting behaviors, like agents that give up early, uh, want to buy molecules that don't agree with their SAR hypotheses, ones that keep retesting the same molecules over and over, despite having a limited budget. And also when it's good to explore versus exploit. **[0:04:09] Ian Quigley:** And it turns out that more performant models are much better at abandoning kind of unproductive neighborhoods. And that was actually a really interesting surprise for us. Yeah, and so when the paper comes out, we're gonna release a bunch of these environments as hardware packages. And then many thousands of transcripts, uh, for people to go play with and go see how these agents are being, you know, how their how their science is is That that's cool. **[0:04:31] Co-host:** super cool. So do you have a feel for why the smarter models are able to figure out that they're in a bad landscape and move or a bad part of the landscape and move? **[0:04:41] Ian Quigley:** We don't know that at all yet. All we know is that they're abandoning these crappy neighborhoods more quickly. Uh, and we definitely see a progression of the like a a more willingness to jump around overall, uh, for more performant models. But they especially abandon the the lousy places fast, and then the really poor ones stick around. **[0:05:00] Co-host:** It's not just that they jump more, it's that actually they're some they're figuring something out and saying, Hey, this is not good, let's get out of here. **[0:05:06] Ian Quigley:** Exactly, exactly, yeah. And what happens with the the dumber models is that they'll just keep running little subtle changes on molecules that aren't ever gonna go anywhere, right? Like, oh, these two, they have a low score, but I feel good about them, and so I'm gonna keep trying stuff. And then, uh, and then the smarter models will abandon the entire process. We're gonna go somewhere else. **[0:05:25] Co-host:** Have you looked at the rollouts to see if it's thinking about whatever? **[0:05:29] Ian Quigley:** Yeah, thinking about all kinds of stuff. They'll say, you know, when? When we release all these transcripts, like everyone's look at one of my favorite things to do is just take, you know, large numbers of transcripts and just dump them into LLMs and just have it look at the different failure modes and stuff like that. But you'll see, yeah, I think that this ring is good, and I think that this particular nitrogen is painful, or, you know, whatever. **[0:05:48] Ian Quigley:** And they'll, they'll say these things, uh, and then they're very plausible, you know, medicinal chemistry narratives. So it's it's pretty, it's pretty fun. **[0:05:57] Co-host:** yeah, that's amazing. **[0:05:58] Host:** So I'm curious, have you given humans these same questions? I have **[0:06:04] Ian Quigley:** to have them look and evaluate, like, you know, not a ton, but because, you know, we have we have a large number of them. But when we get medicinal chemists to look at it, they're like, Yeah, no, those are very plausible thoughts about, you know, how to conduct this particular part of the SAR campaign. And **[0:06:17] Host:** no, maybe I'm. I'm overthinking. Could you make a game out of this? This is sort of a game, right? And then the the medicinal chemistry, the video game, and then you can no, the hook people in these environments capture, and give them, **[0:06:26] Co-host:** you want it as a capture. **[0:06:26] Host:** yeah, do a medicinal chemistry capture. And then that's yeah, dude, **[0:06:30] Co-host:** your, that's your new business model, I, I, I like, just **[0:06:33] Ian Quigley:** I like this plan. We did do a game a little bit like this for Name that chemist. Um, we would have to probably boil it down to have a smaller number of building blocks, because right now, to have a chemist go in there, you can imagine that, uh, if you're a medicinal chemist, you might have this suite of things that you might want to try over time. Like, put a methyl here, like, move this around. And when we have these pre-measured environments, we are going to be restricted to the things that we did actually measure. **[0:06:59] Ian Quigley:** And so then we would have to be asking the medicinal chemist, Oh, well, you can't do whatever you want, you could just do a lot of stuff. And, uh, and figure and then out a way to **[0:07:06] Host:** they have to know what stuff to. **[0:07:07] Ian Quigley:** they have to know what the stuff is. Yeah, so we'd have to figure out a way to surface it to them in a meaningful way. But I kinda like where you're going with this. It's like, we could definitely shrink it down to a smaller number of options, so that's fun. Yeah, we can do that. **[0:07:19] Host:** yeah. What model is best? Or the frontier models, what are the best ones? **[0:07:22] Ian Quigley:** um, let's see. I was looking at the plot a minute ago. It's it's kind of embarrassing that I forgot it already. Uh, right now, uh, the champion is after six. Yeah, that's **[0:07:32] Co-host:** not a not a big shock. **[0:07:34] Ian Quigley:** not a big shock. Sonnet five, five is actually pretty good too. Yeah, and and Sol six one is also pretty good. So **[0:07:42] Host:** and everything five is just a raising pile of crap, right? You know, that five, five, yeah, will, Sonnet five is both yeah, **[0:07:50] Ian Quigley:** very expensive and not very performant. **[0:07:52] Host:** yeah, yeah, but yeah, I think five will be long considered a a big mistake by Anthropic. 5.5 s are pretty good. Yeah, yeah, do you have Opus 5.5? **[0:08:02] Ian Quigley:** we don't yet. No, that's that's one thing that has been has actually been a little infuriating, uh, as you guys can appreciate. So when we talked last, two months ago, we're starting these rollouts to see what happened. And then there's been a new frontier model drop, what, like, once every eleven days. Eleven days, yeah, Yeah, so, like, every time, oh, we're gonna ship this, we need to drop this public stuff. And like, some other dude's stupid model shows up. **[0:08:24] Ian Quigley:** It's like, Ah, **[0:08:26] Co-host:** you have to ship faster than Anthropic. **[0:08:28] Ian Quigley:** yeah, yeah, right, good, good luck, Ian. So, yes, so we are excited to put in more in here, uh, and we also have run a large number of of the open source ones as well. So we've thrown a bunch of smaller quins in there and GLMs and Kimmy and Deepseeks and that kind of stuff. So there's a lot of these. **[0:08:48] Host:** how do they do? **[0:08:49] Ian Quigley:** They're, uh, Deepseek V four Flash is pretty good, not that bad. Yeah, maybe I should just send you guys, like, the full chart of how things are going. I think that'd be. **[0:08:57] Host:** Yeah, well, we can just pull this yeah, we up on the screen. can put Yeah, it into yeah, stick and the **[0:09:00] Ian Quigley:** it up there. I, I can s I can set it. It kind of looks a little bit like, you know, like the Arc AGI two plots, where you've got some crappy **[0:09:09] Host:** I don't know. **[0:09:09] Ian Quigley:** models, right, right. It does, it does kind of look like that, and that's kind of why we made it like that. And then, uh, I think one thing I mentioned earlier that has been very interesting to us is about this time last summer. We had started an internal project to do, uh, supervised fine tuning on some of these language models to get them to understand smiles, because none of the existing frontier models were any good at it. **[0:09:35] Ian Quigley:** So, like, none of the Anthropic or or OpenAI models at the time could even convert smiles to non canonical smiles very effectively. And so So we were like, **[0:09:43] Co-host:** just for non chemists, smiles is a language that you can use, string language that you can use to describe accurately a chemical, right? **[0:09:53] Ian Quigley:** Yes, yes, that's right. Uh, I I think we mentioned it a little earlier when we recorded last, but it's a string. And molecules can be forked, or or circles, or or both put together, right? These different things. And so a smiles is basically a graph traversal across the molecule, written down into a one dimensional string. And so you can imagine starting the graph traversal at a different point in the molecule. And so there's multiple smiles ways of representing the same molecule. **[0:10:19] Ian Quigley:** And so you, it's nice to be fluent in transferring from one to the other. It's a pretty easy. Uh, there's a couple open source packages people use, like RD Kit, to do this, um, but we wanted the language models to do it just cause it would go faster. And they weren't as good at using tools on their own back then, and nobody could do it. And then, around February, like a lot of other things, uh, it became clear that they could start to actually work with chemistry in a meaningful way, which is a reason we started going in this direction. **[0:10:44] Co-host:** that's really interesting because that means that it's internalizing some understanding of the phys, the physical nature of molecules. **[0:10:55] Ian Quigley:** I think a little bit, yep, like, these are the tasks that we're having these things do right now. Like, they are iterative tasks where they're looking at molecules and they are learning something about those molecules to make choices to make better molecules next. And when we remove the information about the molecules, like, these decisions really suffer. So they're, they're learning something. This is very interesting, or at least they're using it. I guess that's a better way. **[0:11:21] Ian Quigley:** We're not, we're not changing weights here, but they're definitely using the information just to make future choices. **[0:11:25] Co-host:** you mentioned all these, this zoo of models, right, that are you've tried. So I wanted to ask, have you just used, like, one of these Bayesian hyperparameter algorithms to to as like a a deterministic baseline for your problem? **[0:11:42] Ian Quigley:** We think it's too spiky of a landscape and it's very high dimensional. We're not really, we don't do that, but we mostly think it's not gonna work. I think it's the party line. Yeah, **[0:11:50] Host:** I know you plan on releasing all this, so I guess we'll put the use. Wait, what are you releasing? You said you're you're gonna have a a preprint blog and then some data rollouts and everything, so we can put all this in the show notes. Is there, uh, anything else? yeah, that's Yeah, **[0:12:04] Ian Quigley:** it. Yeah, I think I think that's the big thing, and that's probably something closer to like, I guess, I guess what I would probably say is. We have built this thing where we, if you've got scientific agents and you wanna know how they're behaving, what mistakes they might make, what, how they might learn from physical feedback. You know, if a new training recipe improves scientific decision making, we've got the capabilities to measure these things at scale. **[0:12:32] Ian Quigley:** yeah, that's And **[0:12:33] Co-host:** really cool. Do you have plans for other similar problems for this same kind of thing? **[0:12:38] Ian Quigley:** we do, yes, we have. Broadly, the way that we think about this is it is a lab in the loop with amortized latency. So it's like, go out, you record the thing once, and then you can have a huge number of rollouts through that thing, effectively infinite rollouts. And it. We are thinking hard about other types of biological or scientific questions that you could transmute into these kinds of environments. Because if you're doing conventional lab in the loop, it does take a long time. **[0:13:09] Ian Quigley:** And and it, and because those things themselves have limited budgets, it's also not counterfactually dense. So, like, you gotta keep running it and do different choices with every rollout to make the things, to make the agents learn, **[0:13:24] Co-host:** right, but with your, your multiplexed strategy, then that, then you can just amortize. **[0:13:30] Ian Quigley:** you could just amortize. So it's, it's. It's not quite accurate to say it's a lab in the loop with zero latency, but it's pretty close. You I mean, know, like these **[0:13:38] Co-host:** this is the same thing that Lila's doing. I mean, it's **[0:13:41] Ian Quigley:** I agree. **[0:13:42] Co-host:** not the same thing, but they also do experiments and then they use that in their rollout, but they're not doing them live. **[0:13:49] Ian Quigley:** right, right, yeah, yeah. Doing it, doing it in the rollout one step at a time offers a lot more freedom because you can choose almost anything. And here, you're restricted to the number of choices that we made when we pre-measured the environment. That's definitely true, um, but it's like it's still a pretty big number of choices. And, uh, yeah, and then we have right now, we're, we have. We've screened about a thousand proteins, uh, against a couple of these chemical libraries since we started Leash, and we're gonna start to grow that. **[0:14:17] Ian Quigley:** I think we're gonna try to get to maybe twenty of these new environments a day. That's a plan for us to try to get to beginning of maybe twenty six or twenty seven. We can, we're doing two a day right now. Uh, so yeah, I think we could probably get to twenty a day without too much lifting. So we'll, we're, we're pushing, we'll see if we can do it. **[0:14:33] Co-host:** wow, and so you're gonna have this huge RL environment. **[0:14:37] Ian Quigley:** Yeah, yeah, it seemed like it seemed like the thing to do. I mean, kind of informally, it seemed like where a lot of these agents are making the best and the fastest progress is where there's instant feedback that's very scalable. And it's really hard to do that in the physical world. Um, but we think that there's probably a handful of biological questions that you could re-rebuild in this way. Uh, go after them. And even, uh, yeah. **[0:15:07] Ian Quigley:** And then we don't yet know if we're able to take reinforcement learning progress from these kinds of environments and move it into new biology yet. Um, but, you know, like, we're trying, we're gonna find out. It's not a lot. **[0:15:19] Co-host:** that's that's the obvious. Q Like, how about how does this transfer to other biology? You can imagine if you did it for some sort of covering set, right? That you might be able to get things in between, though, that you could transfer to pretty well. **[0:15:31] Ian Quigley:** Yep, yep. And then also one thing that I need to do is, uh, talk to some more decision people. Because a lot of this is about how do you spend a limited experimental budget, and then what can you do to do better at that? Yeah, beyond just medicinal chemistry, right? Just like, Yeah, when do I stay, when do I go? How do I? How do I get better at that? interesting, yeah, you can Yeah, **[0:15:51] Co-host:** imagine that if it knows how to connect a decision at that point to a knowledge that it has, then it could learn that skill for other problems. **[0:16:00] Ian Quigley:** yes, yes, see, we are. We are incorporating more and more structural tool calls and things like that with this as well. So it's a it's, it's a pretty fun time right now. **[0:16:07] Host:** yeah, we'll be sure to link to all that cool stuff you have coming out. And, uh, yeah, thanks for taking the time to chat. all right, you, Yeah, or thanks thanks. for taking the time for a follow-up chat. Uh, yeah, I think this will actually appear back to back with the rest of the episode that you **[0:16:21] Ian Quigley:** okay, this was great. so, Well, thanks guys. anyway, thanks, It was fun. see ya Oh, it was fun. Yep, talk bye. to you later.