Reinventing AI Agents with Imbue CEO Kanjun Qiu

You're listening to Gradient Dissent, a show about making machine learning work in the real world, and I'm your host, Lukas Biewald. Today I'm talking with Kanjun Qiu, who is the founder of Imbue, a research company that focuses on building agents. And what agents mean to them is software that can essentially execute arbitrary code on a machine and gives the power of a software engineer to anyone that wants access to them. It's a super big vision, but they're also doing a lot of practical research in the open to get to that vision.

And we go deep on a few of the things that they're working on. Including questions like, why are benchmark performance so much better than the actual performance of LLMs that we see in the real world? And how to make evaluations improve by asking questions in a more specific, tight way? I hope you enjoy this one. All right, well, Kanjun, I think we need to start by, um, you telling us what? For those who don't know what Imbue does?

Um, you describe yourselves as as like a a research institution, but um, obviously I know it's it's more specific than that.

That's a great question, yeah. So we describe ourselves as a research com research company. Um, so in view, we build AI agents, and these are, you know, when we think about what an agent is, it's a system that really helps your computer do things for you, uh, in the world. And our belief is in this current revolution in AI, it's going to enable a computer that is much more molding to you, much more personal. Um, if we think about our computers today, we don't think about it.

But like, we have to micromanage them. We're always in front of them, glued to our screens every day, cause they're not that smart. We don't trust them to go off and and do things for us. And so at Inview, what we're trying to do is build systems that can actually go off and accomplish our goals, help us accomplish our to dos, our goals. And we get there by starting with actually code. And the reason for this is actually because we have a pretty different way of thinking about what an agent is.

Um, an agent, we think of an agent as this intermediary layer between you and your computer. So it talks to you in natural language, it talks to your computer, and it does that by writing code. That's the kind of most powerful way of talking to your computer. And so a way to think about what an agent is doing is is actually letting you, as a user, basically write arbitrary software on your computer. And we believe that if we really make these agents, um, uh, powerful, then everyone in the world is able to write software.

And software becomes essentially make much cheaper. Uh, low, zero marginal cost.

Hm, hm, and and you're are you're pre product, right? or or you? You don't have a public product like, and you're, you're still like building it internally. Is that? Is that right?

That's right, yeah, so we work on a product internally.

And can you talk about, like, how that, like, how that's going, or or, or what that does today?

Yeah, I can, uh, at. So, at the moment, the product works with us to write code. And um, really, what its goal is is to let me express myself at a more conceptual level. So, and in the beginning, it works primarily for programmers. And this is because today's models, they're not good enough to, like, write code fully by themselves. So if I'm not an engineer and it produces bad code, I can't go in and edit it, and so as an engineer, I can go in and edit it and make sure that it actually works.

Over time, our goal is to make the capabilities better so that it can more, uh, less and less technical. People are able to use it over time to help, like, accomplish our to dos. So concretely, what that means is it works with me to accomplish to dos in my code base, uh, whether that's writing functions or things like that. Um, and we focus a lot on both writing the code and verifying the code. So when it comes to agents, one of the biggest issues is with verification and trust.

Like, how do I trust that this thing actually did the thing? I hoped without any crazy edge cases, or like, really minor err, uh, really subtle errors.

Hm, and uh, uh. One thing that's striking about your company is you're you're doing a lot of, um, really high quality publishing, like it seems like you're almost doing, um, you know, research at, you know, out in the open. And and I wanna get into a whole bunch of the research projects that you've, that you've published. Um, but but how do you? how do you think of that? How does that support the the company's mission?

Yeah, so fundamentally, kind of. The mission of the company is to make the power of computing much more accessible to everyone, um, and much more powerful as well. And, you know, as I mentioned, we micromanage our computers today. Um, I think computing's potential is really vastly underexplored. And um, part of why we publish is to, kind of, you know, in the in this, in this, um, frame of. We want to use computing to really empower people, we want to dist, distribute the power of computing to people.

Um, part of why we publish is to enable a greater ecosystem around model building. Uh, so that, like more open source models can be available to more people, um, so that more people have access to do fine tuning and reinforcement learning on top of those models. In order to get to the actual use cases that they're trying to get to, um, so that we don't end up with like super power, uh, concentrated power, we have more decentralized, distributed, democratized power of computing.

I think with the current AI revolution, one of the risks that no one really talks about is that, like everyone talks about safety and alignment, but no one really talks about the risk of power concentration. And that's one of the biggest kind of effects that's happening right now.

Hm, hm, do you think about what you're doing in? In contrast to other code generation Lm products that, you know, folks listening to this might be might be aware of, like, um, Codium or um, Copilot or or things like that? Like, do, do, do you view those as as like, similar, or it's a total different, totally different category of thing?

Yeah, I would say it's somewhat orthogonal, uh, so one way to think about the spectrum of coding products. Um, maybe there are two things one is think about the spectrum of coding products right now. Uh, on the one hand, it's like autocomplete in line, like copilot, codeium. On the other hand, you have something that's like an autonomous agent, like, um, demin or uh, things like like, uh, factory, ai. And the copilot like inline things, they kinda they work well.

Um, we use them all the time, we use them alongside our own product, uh, but there's not much power in them. Uh, you know, I can't ask them to do something totally different, I can't ask them to generate whole functions or test them or things like that. Uh, on the on the other end of the spectrum, with demin or these other tools, you know, I can give it. Like, write a web page that does X, and then I'll wait two days and it'll come back and it'll write a web page that did y, but kind of does x.

And it's not quite what I wanted. Um, and so there's actually a middle layer that we've discovered is kind of the way I want to be collaborating with my computer. Uh, when it comes to actually driving forward my projects, it turns out that, like most of the work that is super important to me. I don't want to be asynchronous, I actually want it to be done right now. It's like, in the critical path, and so we basically try to help you solve the critical path problems as much as possible.

So one way to think about, like a primary thing that we do is like, generate functions, generate functions that are correct, that are test like tested. Not with unit tests, but where you can verify it by giving it inputs and and looking at its outputs and like, trust that the function is correct. Um, or like, write wait, sorry, whole modules.

I didn't quite, I didn't quite follow that. So so how would you verify that it's correct? Uh, that that kind of sounds like a unit test, doesn't it? To to try an input and see what the output is, or I'm not sure I understand the distinction you're making there.

yeah, yeah, it it, um, it's kind of an implementation detail distinction. So, uh, a way to verify its imp, uh, its correctness is to give me as the user. Like a bunch of sample inputs and then run those inputs through the function. And I look at the sample outputs and I say, Ah, it's missing this edge case, so like, fix it to account for this edge case. That's like oh, I a way of

see. So that's sort of like the workflow working of like, engaging with the

exactly. It's a way of working with the model. As opposed to like, oh, I wrote a function and now I'm going to write unit tests in order to test the function in the piece of software.

Hm, I see, I see. And y, you know, my experience with these? Um, you know, this is probably dated, but you know, the the sort of like J kind of generating bigger blocks of code. Is that, you know it? It can work like phenomenally well in sort of smaller projects, where I can really like, feed it all in. But it's really challenging with really, you know, kind of large codebases, like, how, how do you think about that?

Yeah, one actually pretty interesting thing that we've seen is that in larger codebases, and it's it's not like a super massive code base, but in larger codebases. The user usually knows what files or functions they're referencing, or classes they're referencing. And so as a user, being able to tag like, Hey, this file is relevant, come like, look at this file, or like, Um, uh, I want to change the parameters to this function and like, look at everywhere where it's called, um, and like, modify the return values or something.

Um, there as a user, because I'm driving it, I have a lot more context on what's going on, and so I can. Actually, we see that users will like pull in the context themselves.

Hm, hm. And do you think that, like using this product yourselves has has made your organization, uh, more efficient at at coding?

Yeah, it's actually we're finally getting to the point where we're building it with it, which is really fun. Um, and you know, I would say we are pretty, uh. Part of why we don't have a public product yet is because we just don't think it's good enough yet. Um, but we see a line of sight to a place where it actually can, like, greatly increase our own productivity. Right now, it is pretty magical. It like we're using it to write functions for itself, implement new features, things like that.

It doesn't work for everything. We're pretty calibrated to what it does and doesn't work on, and we have a bunch of ideas for basically how to improve the capabilities. And so and I guess we kind of do that in a full stack way.

what? W What would you need to see to to feel good about putting it out into the world?

Yeah, it's a good question, we're not sure yet. Um, the the biggest thing that we look at internally is like, are you are people using it every day? Um, and two, like, what percent of the things that they give, uh, do they accept? So, like, you kick off a bunch of agents to do to, like, fill in a bunch of functions or to dos, like, what percent of those do the person actually accept? Um, and those numbers, at some point, will feel like, okay, this feels good, it feels really powerful.

Um, and we have some other stuff on the horizon that's kind of beyond what I just talked about here that we're exploring as well. I think, you know, if you think about what we're doing, what we're doing is not helping people write code, it's like helping people, like, write code. As a way toward robust agents. And so, um, that's kind of a different end goal. Um, and so it's not like, oh, let's just try to get as many users as possible for this thing right now.

More, the question we ask ourselves is okay, this thing works this well for writing code for itself right now. Um, what's next in order to get to agents? And we have this concept internally of like a system that can write entire trustworthy software systems. Once a system can write a whole trustworthy software system, that's definitely an agent. So, like, what's the path from here to there? Um, and so at some point there, we'll productize something, um, but really, it's kind of like on the roadmap to general agents.

And do you think that, um, like, do you imagine like specific use cases that might become feasible? 1st? Or do you sort of feel like it all happens all at once? Use cases for Like for like, um, allowing people like, y, you're saying, like, you know, your goal is to get the software, your software to. I think, if I understood correctly, to to generate kind of complete applications. And I, I sort of imagine there's there's a lot of different, you know, applications like I, I, you know.

I'm occasionally able to get GPT four to generate, you know, simple applications like M. My friend wanted to know, like, which side of the airplane to sit on to be out of the sun, you know, based on like, lat, long coordinates. And it like, nailed that one, you know, like, 1st, 1st try. But, you know, like, hopefully, uh, to replicate weights and biases would, um, would take a little longer. Um, I guess, like, how do you do?

Do you have some sort of like sorting of the complexities applications? Do you have like initial applications in mind that you think you might go out with? 1st? Like, how do you? How do you think about that?

Yeah, definitely, I would definitely. Um, I would reframe what we're trying to do a little bit differently. We're actually n not trying to, as a product, help people build whole applications. That's nice, that's a nice side effect. Um, if someone really wants to use it for that, they can go for it, uh. But what we're really trying to do is help people accomplish their to dos or accomplish their goals. And so a to do might be like, Um, I want all of the Slack messages that some team member has pinged me for reviewing a document on in the last seven days.

Can you like, pull them all into my Notion doc and then also take a 1st stab at it or something like that? Um, that's like a thing that's on my to do list. It has to write software to accomplish that thing. That software might not be robust, maybe it's a one time thing, but maybe that software does need to be robust. Ideally, it would be able to do something like that. Um, to your question of is there a spectrum?

there's definitely a spectrum, so there's, uh, the spectrum is kind of from like, smaller to larger, or more hard coded to less hard coded, or, you know, something like that. Um, so right now I can pretty easily use it actually to write like really small apps. Um, right now it's like, V mediocre at API integrations, like, not good at testing them, um, but I could see a future in which, like, that's, you know, that's like a s very solvable problem.

Um, once I start getting to, uh, th. This is why the user base matters. Like, if I'm a software engineer, I can use these things right now for like, building the Slack bot that I just said. Um, it's gonna still take some time, maybe it'll take like a day or two days or something like that. If I really wanted to get it done, um, where? And I think like, that's kind of like. The initial set of use cases is like people who are fairly technical, spending, you know, some amount of time building, like small to medium sized things, or using it in their everyday workflow.

But as we get to less technical people, that's where, uh, we start needing better capabilities. So for the same app, like make a Slack bot that does X. If I'm less technical, it might take me much longer, it might take me more than two days. Um, or it may. I may never be able to do it, cause I don't know how. Um, so in that situation, maybe things like a pre built library of components or something like that that the agent already has access to, um, would allow that kind of unlock that next step.

Uh, so we do think about basically kind of the sequencing. Is really like a user type of user expansion and also complexity of to do item expansion.

But so you imagine starting with kind of the most technical users and expanding to less technical users. Did I understand that right? I see, even yeah, although, though maybe a less technical user might have a, you know, a lower bar for for usefulness, right?

yeah, we definitely have seen that. Uh, like, uh, there's a. There's a class of engineer who will not be excited to use these tools because they want to understand every function that's written. Or they're working on a type of application. Like a lot of our model training code, like, we just can't use this at all. Um, like PyTorch, uh, just like too finicky, uh. There are too many details, too easy to introduce bugs that you'll regret for a long time.

Um, and so for that kind of stuff, I think that's like a late adopter, uh, like, super late adopter set of things, uh, category of things. But we have seen, like one of our team members, uh, who's on the recruiting, uh group here. She built a Hacker News, uh, kind of watcher bot, uh, because there are certain types of candidates that we would love to hire. And they post certain types of things on Hacker News. And so she'd like to be flat, uh, notified when that happens.

Um, and she last coded like ten years ago, and she was able to throw something together. It was super cool. And so that's like an example of a less technical person. She's got a use case. It doesn't have to be a robust piece of software, it's already starting to work. So, yeah, to your point, it's definitely it's not like a perfect, you know, mix of things, but I think the general thing is, uh, we wanna make the systems better at like, more robust.

Uh, we kind of call it last mile reasoning. So all of these things around, like figuring out if it's correct, uh, like it. Being aware of whether it needs to ask you questions, or whether it's making the right decisions, or whether it's writing correct code. Like all of that is necessary to, like, make someone be able to do this without micromanaging the process.

And is that the things you described like, do, do you feel like that is sort of like the breakthroughs needed to to make a product that you're proud of? It's like it's like verifying correctness and understanding where you need to ask questions. Are there, like other pieces that that need to be improved to to to get something out the door?

Yeah, it's a good question. Um, I. From our perspective, those are the primary two things, is like verification and ambiguity. So, uh, so that's kind of like reasoning about whether I did the thing right. As a human, I do that all the time, constantly, and then ambiguity is like reasoning about, like, did I? ha, do I have the right information for this? Like, where do I need to go learn more before I start doing something? Okay, now that I did something, I'm now confused about the goal.

Maybe I should like, go back and fix the goal, um, or I should ask the user about the goal, uh, or something like that. So those are, it's not perfectly clean, but those are the two categories.

And I guess how are you like approaching these categories? Like, is it is the main thrust of the improvements through? Um, I could imagine, like, a completely prompt oriented um strategy. Um, is, is that what you're doing? Is that like where you think the the meat of the improvements will come from? Or what? What do you think? Uh,

That would be, uh, if, if only, um, but no, that would be much cheaper. Uh, um, no. Unfortunately, just writing good prompts doesn't, like, help a ton with verification. Uh, the current public models are not super good at verification, uh, for a variety of reasons. But um, we have found that so you can do some, uh. We find that, like, RL is helpful for a bunch of stuff, uh, on top of either, um, own models or open source models.

Um, we try to work on directions toward verification and ambiguity, like better verification, better ambiguity. That are complementary to what public models are going to improve at automatically. And like, so public models, they'll get better at code generation in various ways. Um, we can prompt them to improve their quality of code, um, we can fine tune them to improve their quality of code, to be in our style guide, et cetera. But the internal research is much more around, yeah, how do we get these like last mile pieces correct?

and that piece of is is verification through through custom models. Did I? Did I understand that right?

Yeah, it's a good question, so verification under verification, there are a ton of pieces. Um, it's not so simple, it's not just custom models. Um, you know, the thing I mentioned about the user looking at the inputs and outputs, uh, that's like an example of verification, but it's at the user level. Then there's like verification at the like model level. So doing reinforcement learning to make the model better at verifying, uh, whether the output generated was correct.

Um, there's also verification at the like code evaluation compiler level. Um, there's like, you know, various forms of verification. Um, so yeah, it's like a bucket under which research and product development falls.

And you say that, um, that, you know, like open source LMS and and, oh, you know, public LMS are not good at, um, at this kind of verification. I'm imagining this. The verification task here is like, I wanted, you know, I wanted this, like, is, is this like? is this like giving me what I wanted? Is that like J? A good summary of, like, what verification is? Like? You're sort of looking at an input and output and checking if it, if it is Basically, like matching the intention.

yes, I wanted And and this, did yeah,

it give me what I wanted?

and why? why would? um, yeah, why? like, why would it? l like a general purpose LLM not be good at that?

Uh, yeah. So part of it probably is that they're not. There's not a lot of V data sets around verification on the internet, so there's a lot of code on the internet. Uh, but there's not a lot of like, okay, this was the prompt that the user had. And then here is like the code that was generated, and then here's like an evaluation of whether the code is correct. And here's like a explanation of how the code had these bugs, and then now here's like a regeneration.

You know, that kind of trace doesn't really exist, so you could create that kind of trace and, um, fine tune your models on it or do RL. Um, that's one potential hypothesis. Yeah,

Hm, I was gonna ask, So, you know, I saw on y, you know, on your website, you, you open sourced. It seemed like you not just open sourced the model, but you were really kind of open about how you trained a, um, a foundation model. Can can you describe, you know, what that was and what was the was the purpose there to do this kind of verification?

oh, it's a great question. So, yeah, recently we open sourced, we published three big blog posts, um, and we were open sourcing. Essentially, how do you set up hardware from scratch? So we have, you know, 4096 GPUs, um, and how do you set that all up from scratch? Uh, in order to train 70 billion parameter model, um, we also released an optimizer, so this is a optimizer that we used that lets us train like 7 billion parameter models.

And then predict performance once we scale up to 70 billion parameter model. That saves us a lot of money and time. Um, it also caused us to, like, not have lost spikes, et cetera. And then finally, we, uh, released a bunch of cleaned up benchmarks and evaluations. So on the verification and ambiguity piece, the benchmarks and evaluations are actually the main driver of, uh, like, main piece of work that kind of, like, expresses some of that work that we do internally.

Um, people are least excited about benchmarks, um, but what we found actually that was interesting is, uh, when we look at evaluations, we can actually improve the performance of the model on an evaluation question by reducing the level of ambiguity in the question. So if I'm much more specific about what I'm asking for, the model does do better. Um, and so

Wait, could you? that's like, Could you give an example of that?

Hm, yes, um, well, like a very a. A trivial example might be something like, book me a flight to Paris. There are a lot of open questions about booking a flight to Paris. Uh, like, when should that flight be? and do you care about layovers? and things like that? Um, and like, if I've said, Uh, book me a flight to Paris, that is, uh, has no layovers and is on August 3rd. Then the types of responses that come back from the model will be more relevant.

So this also applies in code generation. So if I say something like, write a function that, um, like, uh, so if I say something like, write a function that solves scrabble. So that's like a very underspecified problem. The model might produce a function, uh, but its inputs and outputs might not be what you expected. Um, the more specific, what we find is that the more specific that you are with the specification, the more likely it is to produce.

Uh, like a function that has inputs and outputs that are what you expected, and also account for like edge cases that you specified. And when you're evaluating, this is like a really boring point, but when you're evaluating against your own evaluation set or public evaluation sets. Like, there are often tests, uh, for these code, uh, function generation problems, and those tests, uh, kind of try to capture a bunch of edge cases. Um, and those edge cases aren't necessarily specified in the task, so it's easy to specify the edge cases and then be able to pass those tests.

But okay, like, backing up, I thought it was interesting. It seemed like you, you fine tuned a model, um, previously, and then you, you trained your own model. And I, I guess, can you describe this? Uh, the thinking behind, like, Hey, what made you decide? Hey, we really need to, like, make this investment in training our own, you know, 70 billion parameter model, which is not a small undertaking, even if you do have, you know, libraries that that lower the cost.

And kind of like, what the the differences were in performance.

Yeah, so there are a bunch of reasons for training our own model. Um, one is that it turns out that data quality matters a lot. So, um, and for our particular use cases, like the data we feed in matters a lot. So one thing we found early on was our hyperparameter optimizer was like, wow, books matter a lot more. Like, please weight the books dataset really highly. Um, and we were like, Oh, maybe we could acquire more books, like scan them and, like, you know, legally acquire them, the text of these books.

And then that improves, uh, performance on certain reasoning evaluations, or for certain code evaluations, like we can see that adding certain types of code, um, datasets will help improve performance. Um, so I think, like, basically, if you're optimizing for particular evaluations or particular, uh, types of tasks, it's often useful to do some kind of. Maybe not full pre training, but at least mid training and fine tuning. Um, so mid training is kind of like a post pre training stage where you train on a bunch of, like, a large quantity of data and it.

You can actually improve task specific performance that way.

And I guess, um, uh, can you say like, how much it cost you roughly to to train the 70 billion parameter model? Ooh, I mean, you're pretty open with the specs, do you? D? Does that? Do you have like a word I think someone magnitude you feel?

can, someone can derive from the specs how much it cost.

All right, fair enough, I'll, I'll give it a shot myself. And, uh, refer to the notes, um, but. But I guess like one of the knocks on sort of like, or one of the challenges, like of like, building your own model is like to sort of stay at, you know, at the state of the art. You probably need to do it again and again and again, right? As, like, you know, hardware gets cheaper and these models expand.

Like, do you plan to do this again? Like, what were the results? Sort of like, different enough that that you would definitely kind of continue to invest in in having your own foundation models here.

yeah, that's a great question. So w. The way we think about it is like pre training, mid training, fine tuning, and RL are all levers for solving the problems we wanna solve. Sometimes, like pre training is the best lever, sometimes RL is the best lever. Um, for the current problems that we're trying to solve, like, we actually think post training is more effective. But, uh, the post training is identifying problems in the pre train model, and so we will have to do pre training again.

Probably, um, in order to, like, solve some of these problems that we're seeing in post training, yeah, but I'm not 100 sure yet. It's like that that stuff is definitely still research.

I see, and you know it, it sort of seemed like you. You spent quite a lot of effort, like building technology that's sort of important, you know, for doing this. And and also on on cutting edge. And yeah, I think, you know, kind of other companies might not have open sourced it. Was there like discussion internally about, you know, should we open source all this tooling? Like, was there some tooling that you kind of like, held out to sort of maintain a, um, a competitive advantage?

Like, how, how did you think about that?

Well, we didn't open source the model, so we did open source all of the tooling around. You know, a lot of the tooling that we use, like hardware scripts for checking, uh, the health of, uh, GPU nodes and also the optimizer, the for the optimizer. What we hope to do is to help researchers be able to run experiments, uh, at a small scale and be able to trust. Like, oh, if I scale this up, actually my results will hold, or if I scale this up, my results will not hold.

And they can do that on, you know, eight nodes or or four nodes instead of on 512. Um, so that, uh. A lot of what we hope to do is to cultivate much more of a C, like an open community around it. I think one of the things that we see today is there's really a centralization and concentration of, like, the both the knowledge and the capabilities to train these large models. I don't think that has to be the case at all.

And there are a lot of really talented researchers, um, and some efforts out of Stanford, actually to try to pull together a community around this. Um, so Percy Liang is working on this, and, uh, yeah, I think like a robust open source community, kind of like the whole open source software movement. Hm, maybe it's not open source, maybe it's like some other form of, like, open model building. I think will be really healthy for the ecosystem.

And so we hope to, like, contribute to that, contribute knowledge to that, contribute other stuff to that. I think there's been very few, like fully open source, like no one's really open sourced the training process for 70 billion model. Um, and we learn a ton of things like training at that scale, um, and like, about how you avoid lost spikes and things like that train, avoid training instabilities and that kind of thing. Like, if there's another research group that is wanting to train that scale up model, then, like, you know, it seems like most have not successfully done that yet.

Like, we hope that this infrastructure will be helpful. Um, it benefits us, obviously, like we can pull in improvements, and we often do pull in papers from the outside world into our own, our own work. So,

Although, you know, when you talk to people about, um, open source models like, um, you know, Meta comes to mind here, who's probably less clear about the the training process. But, you know, opens up the model. They they say a lot of the same things that, you know, you just said around having like a community around the model. Like, you know, it's a little different, like, helps with sort of like testing and like understanding what it does and all this.

Like, so what do you think is like different about your business or about your mindset? That you thought, okay, like, we don't want to actually open source, um, the model. Even though we'll give you a pretty clear blueprint on how to, um, to make this model yourself,

yeah, so in view is all about directly empowering people, um, and when I say empower, it means like a person or a group of people feel like, okay. I know how to like customize this to my needs, I know how to like make this my own so that I can use it. Um, I think, uh, and and our product, our culture, like everything, is around directly empowering people and kind of decentralizing power, like distributing power. Um, I think open sourcing a model's weights is very different from open sourcing the process for creating one.

Even though creating a model might be expensive, it might get cheaper in the future. Um, like, the process is much more empowering, even if you read about the process and you understand it. And now you apply it to some mid training run or some other type of run that's not full pre training. Like, that's much more empowering than like, okay, I've g been given this black box. And now I, I guess I can do some fine tuning on top of it, which is great.

Um, and and like, use it for my own use case. But it doesn't necessarily help cultivate a community of people who feel like, yeah, I can do this, I know how to do this. Like, let's like, create kind of like, um, a seniors, or like a a like, um, community of practice around training these things. So I think philosophically it's quite different. It's not just like, oh, we're going to release models, um, just so that, uh, you know, other people can look at it.

And for marketing purposes, um, it's rather like we really want to directly empower people.

I see, and um, and I guess, do you? Do you plan to continuously train new models like like this? Like, will you? will you make a bigger one?

We will, if we have to. I think we try to avoid pre training models as much as possible, to be honest. Um, it's a lot of work it is, you know, as you said, like, it's a lot of work to keep training the latest model. And so so long as really good open source models keep coming out, we're gonna use them. Um, and like, apply all of our post training stuff on top of them and kind of, like, get a bunch of stuff for free.

Um, but you know, it's probably likely that at some point that might taper off, and so at that point we'll probably do pre training again. Um, but that's like, kind of a more of, like, a strategic, like, pragmatic decision. Yeah, ultimately, I think our focus is really on the end goal of making agents. So I think, you know, whether we're doing pre training or mid training, or fine tuning or RL, it's all for like, okay, how do we actually get these systems to like robustly reason?

So like, verify that they're correct? And um, be able to, kind of don't really know exactly how to articulate it, but like, uh, this kind of like last mile reasoning, like being able, being able to know, like, okay, yeah, I, I did properly reason through this.

Hm, hm, it's kind of funny, right? Because it's it's like, hard to find these days a benchmark data set where models are are. Well, at least like, you know, the biggest models, like, um, you know, GPT four and and others, um, you know, perform significantly worse than the like, a human benchmark. I mean it, it, it's actually kind of, kind of striking. So like, w like, where are the like? If you wanted to create a benchmark that these models would fail on, uh, how would?

How would you go about constructing that? Or, or sort of like, what's what's sort of left in the in the power of the models? Between, um, you know, between where, where they are now, and and where you need them to be to to do the sort of like the the kind of agents that that you're talking about,

Yeah, so, um, public benchmarks, they don't really capture, like, as a person, you know, it's kind of interesting. What you said is like, yeah, looking at public benchmarks, these models seem to do just as well as people. Um, but then when I'm trying to use them to actually, like, write software or like, help me do a thing, it does not do that. Question is why? Basically, what's the difference between what the benchmark evaluation is capturing and like, what this real world task is?

Um, we do a lot of like manual benchmark, like evaluation creation internally, based on tasks that users actually try to use it for. And like, making a benchmark eval is actually pretty tricky. Um, it's kind of why, like, it's kind of like SAT multiple choice questions, like an SAT multiple choice question set is like, not really indicative of the real world. Um, a person who's able to answer a bunch of SAT multiple choice questions wouldn't necessarily be effective inside a company operating.

There's a lot more context and kind of judgment, and like, um, understanding the situation, understanding edge cases, uh, being able to, like, do multiple iterations on the thing. Just like the real world has way more complexity than what these benchmarks capture. So, yeah, I think there's this like big gap between what they're actually useful for and what the benchmarks capture. Um, maybe a more useful answer to your question is like, an example of a type of evaluation that people don't do a lot of is like mocking up APIs that you can now call.

Um, or, um, we have a data set we call wild code, where it's a bunch of code, it's in the wild. Uh, we pull code, uh, functions that have tests that are well tested, and then, uh, you know, we run the test suite, tests pass. But we try to get them, and they're well commented, well documented functions. We try to get the model to write the function in order to pass the tests, and all the public models get zero out of, like, literally 0% on this data set.

Uh, and so there's definitely a gap here. Um, I think another example is like, if you look at a data set like Sweet Bench, uh, so this is like a software engineering GitHub issues, uh, style benchmark. It's not the perfect benchmark, it has all these issues. But again, these models get, you know, less than 40, certainly not human level. Um, so yeah, I think there's a lot more kind of complexity.

do you think a human, like a really good programmer, would, um, would do, would get more than zero on this wild wild code? Um, benchmark? I mean, certainly, I would be nervous if I could, if I could, uh, pass all the tests on a on a well specified um function. I'm not

Yeah, w. We do test them internally, so the answer is yes, humans can do this. Uh, Maybe not they need some context,

all humans.

not all humans. Yeah, yeah, we have a bunch of data sets in between, basically, like this data set is fully solved, and this data set is completely unsolved. Um, and so there's like, uh, other data sets where clearly humans can, uh, solve most of these problems, uh, but the model is not very good at it. If you, um, you know, sometimes the model can solve it with like one zero tries or something, but

Oh, I see, yeah, cause

ideally, I would.

if if you have the tests, you can keep trying. I see, um, that'd be, I guess, more my style of, um, of software development. exactly, just just uh, Test and check, yeah,

test and check, test driven development,

yeah, yeah, test driven development, yeah, um, okay, well, I, you know, I wanted to ask you too. I mean, um, I guess, like, a maybe a very small number of companies like you, you know, have raised, um, like, really significant amount of money, you know, before the before the product. And I'm, I'm sure that irritates the hacker news crowd, and, you know, maybe, you know, some, some entrepreneurs who who feel, um, who feel underfunded. But um, I, I.

I would love it if you could walk me through your thought process in in raising such a large round. And also the the investor thought process and kind of what got them comfortable. Um, making such a large investment in the company pre um, pre traction.

yeah. So ultimately, for us, we think about things over the very long term, because what we're trying to do is really give power to people. And, uh, I think it's easy for, like that goal to get sidetracked by optimizing for revenue or things like that. Like the goal of actually making these systems accessible to regular people, uh, that takes invention and a lot of creativity. So there's actually a good analogy from the 60s, uh, about personal computing, so in the 60s, people were really excited about the supercomputer.

They were like, supercomputer is the future, everyone in the future is gonna do time sharing on supercomputers, and supercomputers are a good analog. They're like these centralized systems that you use and you time share on. And it took a small group of people at Xerox Park to invent, like the mouse and the GUI, and the desktop and windows and files, and all of these concepts that we as people understand today. Um, and when the personal computer 1st came out, people like, made fun of it.

They were like, this is like a hobbyist, stupid hobbyist thing, right? It's like, not real, there's no market for this. Yeah, Yeah. Wasn't um, yeah, so

there, but yeah, yeah, that makes you, you were sense. Yeah,

there, I wasn't there, but like that, that's kind of the reception, um. And so basically, our hypothesis is it takes like very dedicated effort to invent the primitives needed and the uh, capabilities needed to make this ac accessible to regular people. And that's a huge market opportunity. Like, obviously, once we do that, then that's like a like. Tons of people can now totally modify their computers, computing completely changes, it becomes something that molds to you. Maybe you have totally different operating systems, whatever it is.

Um, software doesn't look like what it does today, when software is super cheap to build. Like, um, you can build software for yourself, for your community, for your grandmother, et cetera, for your friends. Um, so that world is kind of what we're going toward is, uh, really trying to give these superpowers to to people. And so the hypoth, so the um. Thinking behind raising a lot of money is like, okay, well, we're gonna have to train models and we want to have the runway to be able to get there.

Um, and we saw compute as an asset, like compute is an asset that we can use, so it's actually a pretty good investment. Um, and so from an investor perspective, like a company that's able to set up compute and figure out what to do with it is like, actually not a bad investment. Um, and then that company being able to, like, train their own models and have good results on, like, reasoning benchmarks like, that's an even better investment.

And that company, like working on agents in order to, like, try to make everyone be able to have them, like, that's a huge potential opportunity. So from an investor perspective, that makes sense. We also, though, didn't raise from venture investors for this reason of, like, really wanting to focus on the long term. So, um, you see, our investors are Nvidia, Eric Schmidt, uh, nonprofit, Haotester Institute.

Do you? Did you feel? Um, we can take this out if you don't feel comfortable asking. But I'm just

That's okay.

curious as a as a fellow entrepreneur, did, did you feel afraid at all? Like, I, you know, I, I feel like I look at some of these companies that did that. And it sort of feels like there's these really kind of public humiliations that that I think of, and I I think it would make me, yeah, feel kind of afraid. Um, th did that? uh, was that something you experienced at all in in in doing this?

Or, Yeah, did um,

I experience fear? Definitely, like, it's definitely a scary amount of money to both raise and have. Like, um, you know, even the interest on this amount of money covers a lot of our our burn. Um, but it felt important, like it was definitively the right thing to do. We had the opportunity to do it, our strategy made sense, like we were acquiring an asset that had value. Um, and like, Yeah, it feels it felt correct, um, but it was definitely a really big bet, and I think in retrospect, definitely the right bet.

okay, well, what? uh, to take things in a totally different direction? Um, you know you, you have a podcast too, where you you interview really technical, interesting people. And and I was kind of curious, um, what, what prompted you doing that? and and also what? What do you feel like you've been been learning from these interviews? Like, is was there other particular things that, um, that really stand out that you've kind of taken from this process of, um, interviewing all these very, very technical people?

yeah, 100. So we have a podcast called Generally Intelligent, and we interview researchers mostly, um, and these are mostly machine learning researchers. Although we also recently interviewed a researcher who studies power and power in technology and how technology affects power distribution. The really interesting thing about interviewing researchers, we ask them, basically, where did they start out? how did their ideas change over time? And I think one of the, you know, main things I know from scientific publishing is the paper is just a kind of snapshot into a very sur surface level of your ideas.

All of the core intuitions, uh, what I call the generating functions of your ideas, are inside of you. And by doing this podcast, basically, what I understand is basically what are people's generators for their ideas, what are their hypotheses, theories? Um, the way that they see the world, how does that lead them to have the ideas that they have? And that's super helpful for us, um, in terms of triangulating, like, huh, okay, this is a direction or a hypothesis, or kind of like an an inkling of a a thing that might work, or like, this is a direction.

That, you know, even though this person is excited about it, clearly has no promise. Um, like, I can get a deep sense of that from a long, detailed technical conversation, uh, where I definitely can't get a deep sense of that from reading all of their papers.

Interesting, So in that conversation, you're able to kind of assess if their ideas are are good or not.

It's more I can assess their intuitions and what, um, what is driving them, and, uh, kind of their own level of confidence in some of these ideas. Um, and get some sense of kind of what feels potentially promising to investigate. Um, it's not. It's not like a judgment of like, oh, your ideas are bad or something like that. But you know, sometimes ideas will have to be expressed confidently in a paper. But the person actually has a lot of doubts about the idea.

They understand, like it cannot be scaled, it's only works for toy settings. Um, you know, we shouldn't use it for XYZ, or, um, they might have a way of seeing their paper. That's actually not what the paper talked about. Um, and so they have some core insights about what the paper's actually about that they don't express in the paper. I've seen this a lot. Um, I think the yeah, the paper writing format that we have is not in machine learning, especially, is like, not necessarily useful, um, for expressing these deeper intuitions, because well, totally, they feel very speculative.

especially, especially if you're not used to it. I think it's like, I think, you know, if you've done it a lot, you can kind of read between the lines and and find this. But it's an incredibly weird um for. I mean, I I find the blog posts often much easier to parse, cause it's a more freeform. Um,

yeah, and even the blog posts, uh, both like reading between the lines and the paper, and also the blog posts. I think it's still hard to really understand a person's intuitions that are driving their conclusions, like, what is their view on the world that's causing this? Um, and I think we get a a good sense of that from the interviews,

that's so cool. So you do these interviews for your own understanding of of research.

yeah, of the field.

Oh, very cool. Is it? Do you have a do you have a favorite? Um, interview?

ooh, I shouldn't judge my podcast guests, I'm just kidding, uh,

One of your favorite interviews that I should, that I should watch? 1st,

hm. I actually think Jim fans, uh, on Foundation models to embodied agents was pretty fun and interesting. Uh, they're cool and Noam Browns,

that they sound fun.

yeah, Noam Brown's is on, uh, so Noam did. Cicero, which is, uh, human level performance on diplomacy. Hm, you're And he has a bunch of thoughts on reasoning and, uh, just like, super interesting intuitions, so that's also a really good one. hm. very cool, Yeah,

awesome. Well, um, really appreciate the conversation, appreciate your time, thank you so much.

yeah, thank you, Lucas. This is super fun.

Thanks so much for listening to this episode of Gradient Dissent. Please stay tuned for future episodes.