Inside Cursor: The future of AI coding with Co-founder Sualeh Asif

You're listening to Gradient Dissent, a show about making machine learning work in the real world, and I'm your host, Lukas Biewald. Sualeh Asif is the CPO and co founder of Cursor, one of the best loved and most exciting and popular AI products out there. It helps you with coding, helps you use LLMs to do coding. I use it all the time and and I really love it. And I was just excited to ask him about how he built such a great product.

I found his answers super interesting and I hope you enjoy this interview. All right, well, thanks so much for, um, for taking the time to talk. I guess maybe this is a softball question, but I was really interested in just hearing the story of Cursor. Like, how, how you, how you started it and what the moment was where it really started to take off. Because, you know, now it's like one of the most loved products, I think out there.

Oh, I think that story comes from, uh, we had been really interested in sort of scaling laws, and, uh, back in college, sort of, I had, I had gone on and, uh, worked on a sort of search engine type company with a friend. Uh, and there we were really bo- bullish on language models, uh, because it felt like language models could really compress all the world's information. And there should be this end to end index of, uh, of searching the internet, instead of sort of many of the heuristics we have coded in over the years.

It felt like you could, sort of, that should be the end to end way of doing things, and so scaling laws, you know, doing the search engines, you know, training, training large models. At the time, I think copilot was the 1st really big moment for us, where, like, it was this project that was truly magical. Uh, it was fast, uh it, it felt like kind of new, you, uh. But then, you know, copilot did not improve much over the coming year or two.

And, uh, for us, when we saw GPT four for like, we thought the ceiling for what a really, really great product was possible at that moment was really high. And then it was like, pretty clear that, like, as the models got much better, like, you know, scaling laws progress, um, models got much better. The product that can be built in the future is e even higher ceiling. Uh, and that was like, you just it, it was this sort of super attractive thing to go do it.

And you know, we're all sort of coders at heart. And we wanted to be building things that we use every day. And you know, cursor was originally built for ourselves in in many ways. Uh, it was, uh, and it, it was sort of fun seeing that, you know, everyone else really liked. It was definitely built for ourselves. Um, and we were sort of experimenting, so a lot of the early culture of the company was experimenting with various different ways of using the models.

Should there be a document that you're sort of typing things out and the model is coding things? Should there be, uh, you know what if you wanna do this, you know? Next action prediction of like, you're in a location, what should be the edit? Maybe you should be the model should be telling you where to go next. You should be able to make, uh, edits over your entire repository. And, uh, some of those things have taken, you know, a year, year and a half, uh, several iterations, and some of them, you know, we've continued building on.

So now, like, some of the core parts of the product is the, you know, this next action prediction thing. Where, uh, it predicts your next edit at the cursor's location and then where you should be going next. And that people really, really love that feature. Uh, and then we're, you know, sort of working our way towards. You should just be able to make any edit you want across the entire repository, like code base wide, uh. And obviously there's sort of hurdles along the way that we'll talk about.

Um, some easy, some, you know, sort of still quite difficult. Like, models still sort of struggle with what exactly the architecture of the repository is. If you, uh, you know, ask, what is the architecture of the repository? That is really quite difficult because it requires sort of looking at potentially billions of tokens, uh, tens of billions of tokens. And say, asking the question, what is really going on? As opposed to like, you could like, list the function names, right?

Like, uh, but that doesn't really tell you, you know, that, know what is exactly going on?

Well, totally, I wanna dive into that as much as you're comfortable sharing, but I guess I wanted to ask you, um, you know? One of the surprising things that I learned in my, you know, background research on you is I I think you guys came from using Vim, not VS Code. Is that? Is that right?

I, I, I. All of us were really early users of Wim. Uh, we did eventually, you know, had used VS Code. Uh, probably the last one to. There was a couple of us. Amon and Arvind probably were the last to switch over from from Wim to VS Code. And and the trigger there was was GitHub copilot,

Oh, I see, so GitHub copilot actually pulled you over or pulled in the end.

so I, I. I had switched over before, but then, you know, like Amon and Arvind, only switched over after I see GitHub copilot became it was. It was this was just the kender feature, right? in in some ways, it was the kender feature.

totally. And and uh, why doesn't something like Vim actually have, um, something like what you guys built? Like? It seems like a lot of smart coders, you know, like to use it. Is there something about it, like a graphical interface that lends itself to to this kind of structure, like coding with an AI?

I think for us, uh, versus code is, for one, it's pretty clear it's the most loved platform on the internet. Uh, for for coders, it's it's the thing that sort of is the ipso facto, it's, it's, it's the default. Um, and uh, we wanted to sort of incrementally evolve it towards, uh, the world where you're starting to automate coding, uh. And the cursor of, you know, one year from now should look very different from the cursor of today.

Uh, which should look, which means almost by default, it should not look like exactly like versus code. But, you know, in looking very different, you wanted to start from a place where you. You didn't wanna have a text box, uh, to code, because, you know, coders still want to type characters, right? Uh, you, you wanna be able to edit your entire repository at a higher level, uh. But at some point, if you find that, there's a change that you can, like, quickly execute in ten keystrokes.

Uh, we wanna let you be able to dive into the details. Uh, you know, at any point in time, you're sort of maybe editing some pseudocode representation, uh, you know, maybe a year from now, right? Like, humans are editing a pseudocode representation, and that's, you know, really quick to edit. And the model is sort of working for you in the background. But you're writing some kernel and you wanna go in and toggle some of the indices.

It's it's much easier to do it by hand, than that. You always, I think developers will want this ability to go in. And you know, unless we truly believe that everything is going away, it's like you really, really want the fine grain control,

yeah, yeah, that makes sense. Um, you know, one thing that that strikes me from what you were saying earlier about, um, like, observing that copilot was really great. And there's all these, you know, all this opportunity. And how do you kind of work with these AI models? Is, I think a lot of people, other people thought that at the same time. So, you know, like, you know, you had this idea that I think many people had, including a bunch of, like YC companies and other products that I saw.

And it seemed like Cursor emerged as kind of the winning one among these, right? So it seems like there's great product execution here, which I'm always really interested in. Like, do you have a sense for what you were doing differently than your competitors, that that made your product work so well? Was it like, was it like certain decisions, or was it like a process?

you know, why? Why? Questioning it all is really hard. I don't know. Um, I, it's, it's. It's very hard to tell exactly what we did, right. Uh, I think it was a bunch of things where, uh, we always tried to push the ball as much as possible without, uh, being. We always wanted to be the most useful product at any moment in time. Uh, so, like at the frontier, um, it's very easy to over promise and under deliver, uh.

And a lot of what we have tried to do is to, uh. We didn't ship the agent until we were very confident it was something that was really useful, uh, and we had probably done three agent prototypes before that. We didn't ship because some version of the the model would just lose track and you could make something that was, uh, c. Could help you in the short term and and really hurt the hurt. What people think of as a reliable product in the long term.

Uh, maybe that is, that is part of it. It was always, uh. But then also, I think we've been 1st to a lot of the inventions that people really like. So, um, you know, more recently, the the f. The ability to jump to the next location that should be edited, um, is something we've had for closer to eight, eight months, ten months, a year. Uh, and we hopefully will release a a much more upgraded version of it, uh, soon, that'll be like, quite a bit better.

And, uh, only recently, you know, other people have tried to to do that. So we've always tried to be like, think of what's coming and at least have a prototype out. Uh, as soon as we think it's, it's something that's really useful, there's the tap to jump feature saves, there's the apply features, yeah, and we also, like, you know, we've done this at scale also, so I think that that has benefited. Uh, where? For example, for our custom tab model, we do something like 100 million requests a day, uh, and and and quickly growing it, uh.

I I think part of doing it well had been able to do it reliably for for lots and lots and lots of people.

do you think that any of the the data that you have, or the feedback that you have from users is part of your success? Or are you more making decisions through your own experience?

it's, it's it. The the data has definitely been enormously useful. Uh, I think, you know, the the feedback loops that people consider obvious are indeed extremely useful. You wanna be the company that, uh, feeds an an extremely good product that everyone loves, and then that definitely helps in making the next version even better. Uh, it helps in small ways, it helps in training models, it helps in, yeah, where the small ways are. You know you, you understand how how people are using your product, what is the most important thing to ship at any moment, and then in big ways, in like training models and, uh, improving just the core workflows.

You can, for example, like, technically speaking, one one loop in the apply use case is, uh, you know, you train your 1st version of an apply that is quite a bit bigger. And, uh, you then deploy it for all users. You get lots and lots of data and then you can distill a slightly smaller model, uh, that gets faster, that people use it even more, and you then distill an even smaller model. And you can, like, keep compressing the models down because you're sort of generating the data that allows you to do that.

But also, yeah, it's just it's this, this like feedback loop, and then the, you know, some of the things get faster. So, like, for now, up to, like, I don't know. A thousand or 2,000 line file apply feels effectively instant, and that's what we want it to feel right, right. We want it to feel like apply is this deterministic, you know? They they figured out some deterministic algorithm to, like, place the blocks, but that's not actually what's happening.

It's it's a model that is actually rewriting the entire file and a lot of, you know, a lot of the improvements have been making the model smaller. There's obviously improvements in just making the inference much faster with, uh, when doing these speculative edits. So yeah,

For something like, um, the agents that you talked about it, like, you had some iterations where it wasn't, you know, useful enough to ship. Like, how did you know that it wasn't good enough to ship? Like, how did you think uh, I about that?

didn't use it on a day to day basis. I think I think these things are are really quite easy to figure out. Uh, if you know you're coding 10 h a day in cursor, right? Like you're, you boot up the editor, you're making the improvements, and you're seeing it on a daily basis. And you, if the devs themselves can't, don't use it every single day. It's probably not something that everyone else will wanna use. I mean, there's obviously corner cases to this thing, uh, where we're not the perfect coders, uh.

But you know, a thing like an ing is such a general feature that if you're not using it, it's almost certainly not useful.

that actually leads to another question I had, which is, you and your co founders have this background in sort of, um, competitive coding, right? Like, you know, does that? Do you think that's an advantage for you? Cause? I could imagine that that might sort of put you at sort of the forefront of, like, wanting to be efficient in coding. But I could also imagine that you might have idiosyncrasies in the way that you wanna write code.

That might be different than your general Um user.

I think we're not only competitive coders, like we did competitive math in college. Because, you know, that's sort of part of the background. It's it's always really hard to distinguish, you know, what part of your identity is is the most important. But, uh, many of us had worked at sort of software companies before, stripe and the like. And you had some idea that production coding was very different from, uh, and then people had actually be built products.

So I think Michael had spent quite a bit of time building these, uh, high performance games, uh, and you know, we had done modeling work, so we had we had seen quite a wide variety of coding. Um, sort of bringing back to, like, does competitive programming really, really affect how you did coding, uh, on a day to day basis? Like, not really it. It was just, yeah, I th. I think we knew what engineering was, we were sort of doing day to day engineering, and you could see if the agent was helpful.

And in this case, it was very clear that, for example, our early iterations were not really that useful. Uh, it was was very slow, uh, I mean, one of the most important things that has changed there is the length of the context windows that you can do on every single, uh, keystroke, on every single request. Uh, you know, when the models started out, you were sort of doing these 4k, 8k context windows. And even if the models, you know, slightly supported them, the models were not very good at using the large context windows.

And now that you can sort of easily do, as the cost curve has gone down for the language models, you can do requests on the order of, like, you know, mid hundreds of thou, like, you know, 50, zero tokens, 60, zero tokens. Uh, reliably, that has, like, enormously helped. Uh, one way, one intuition to have here is the model can't even feed your current file like it'll not be very useful. Let alone like, read the rest of your repository or do searches, or lots of things that you expect a basic agent to be able to do.

Uh, that wouldn't work if your 8k tokens, 8k tokens, what can you even fit in there? Uh,

one. Another interesting thing that that you guys said in in your interview with Lex Friedman is that you kind of wanted the experience of using a code editor to be fun, which I thought was kind of a cool idea. Like, a little bit surprising, right? It seems like such a utilitarian thing. It kind of reminded me that when I, um, I remember when I switched in cursor from my default Lm from Sonnet to O one. I actually think I started coding a little less, like I was actually having a lot less fun because there, like, the latency was higher.

It kinda took me a little while to, like, realize that. But I actually like, I like, for some reason, sonnet had lower latency and it made it just like, a lot more fun. And I was like, you know what, I just need to go back to the, you know, the Lm where I was like, just enjoying, like, writing code. So I do actually kind of relate to what you're saying, but I'm kinda curious, like, how, how?

The idea of like, f a fun experience shows up in your, um, utilitarian feeling application.

I think it just, I mean, there's, there's always this sort of end metric, right? The end metric is how much we are enjoying using the model. And, uh, it's. It's been very clear that we enjoy using Sonic more than 01:00. And part of it is, uh, there's a few things, so one is, I think Sonnet is, uh, extremely, uh, is is even at scale, like reliably, quite fast, uh. And I think we want to ship models that are even faster, uh, that are better than Sonnet, that are, uh, much longer context windows, uh.

That could make, you know, edits reliably over a much larger set of your code base. Uh, for exactly the same reason, because it becomes much more fun, so it's it's in some sense, it's this, uh, hard to pin down feeling, but in some sense, you know what really affects it. Like, you will get bothered if the if you have to explain to the model again and again what you're doing, or you will get bothered if, uh, the model doesn't really understand that.

Like, you had some recently viewed file open and the model doesn't see it, and that does sort of annoy it, just it's just straight up annoying. So so you can turn it into some methodical, uh thing that you can, like, track down, uh. But some of the inventions are just like, you know, wouldn't it be more fun if blah blah blah happened? Like, wouldn't it be more fun if you were, uh, coding? And the model would?

Just once you started doing it in factor, it could just you could tab, tab, tab through the entire thing, like ten tabs. What would that take? And then once you think, like, oh, you know, ten tabs make me would make me feel really, really happy. You can then sort of reverse engineer the, uh, the exact thing, uh, like the modeling work that you would have to do, so like, do what? What size of the model you want to train, uh, how much time you want to spend, sort of pre training, post training and rl-ing the models to, uh, to be able to consistently do the same behavior again and again.

Uh, and you could, in another another example, right? Concrete example is, uh, you could always over train the tab models to be annoying, uh, so part of just, you know, if you were to only worry about making sure that every single time it does the edit, you would over predict. Like, sometimes you really want to, like, you're writing against some kernel, you want to spend some time thinking, and you don't want the tab model bothering you.

And that's the thing you would only care about if you're making it fun and enjoyable. As opposed to something that like, yeah, so something that's like, uh, you know, obviously just always over predicting.

But this is a pretty subjective experience that you probably couldn't pull from user data. So, like, how do you work through that internally? Do you ever have, like, a difference of opinion with yourselves around what's the more fun, um approach?

um, I think some of these transitions are subjective, but, uh, I think if you take it out these, they're, they're not always that controversial at at the end of the day. You're trying it out, uh, there, there's. There's always some intuition where you might like over trigger in some direction, but for the most part, I I think, like, if, you know, like you. There's not that much argument, or is sonnet more fun or or is oh one more fun?

I mean, sonnet is sort of arguably better. And hopefully there'll be like more more models that are optimized towards sort of keeping you in the flow. I think that you, you need a broad cat, two categories of models, you need the category of models that is, uh, RL. Towards being fast and super large context windows. And just make edits across your entire code base and make you feel like you're sort of breezing through things. And you want a category of models that is sort of trained for being extremely careful, uh, for reviewing every single small thing before they make the edit.

Uh, you know, may maybe you do a bunch of research, they, they then make the edit in the background for you, like, uh, and then come back to you with a PR. And in that case, the thing that will be fun is if if they're more more correct than not, and, uh, then fast is not the only thing that's fun. It's it's, it's being correct, or how they write out, like how they prove to you that they're doing the right thing.

I guess? A as you kind of build a a bigger brand and you build, you know, trust with with users like me. Why are you even asking me what model I wanna use? Like, I'm, I'm sort of aware of the different models, but I would sort of trust you more to know what's gonna be fun and useful for me. Like, why, why Yeah, I are you even exposing

think I think I think you're kind

that?

of right. Yeah, I think you're kind of right. We've been, oh, we've been. Part of building the trust was always showing exactly what we're using, and I think it's you're probably correct that we should have a default mode and you should use the default and you should feel happy. But there's always if, if you are, if you are the kind of person that wants a do mode and, uh, want to perfectly fine tune every single thing.

You should be able to do that, and then there should be just simple defaulting, uh, so there should be a release in a week or two that fixes all of this for you.

Oh, here's something I I've been wondering about, uh, myself quite a bit. Like, do you think that? Do you think there are best practices in changing the structure of my own code base, or the way that I should code? To make your product work even even better? Like, for example, we have one engineer that's been sort of ha letting, like putting in, letting the Lm, like, put in notes inside the code base of like, helpful things to kind of help I think that's help it understand like the one the code base.

one of the things that that we that we've been sort of speculating about. We don't actually have a really, uh, correct solution there, uh. But this idea of like, maybe there should be a readme. ai. md, uh, in every folder, uh, with the idea being, you know, at any point in time, if you ask changes around any folder. The model should be able to look up what's the nearest place where there's an architecture, uh, written down that it can, uh, sort of on.

On the technical side, the thing to understand the models are much faster at reading tokens than humans, and like, orders of magnitude faster than sort of ingesting these tokens, uh. But humans have, for example, like, some small things memorized, uh, so there is obviously small differences between how we code, where the model is starting from scratch every time. So cursor tab in our code base is named CPP for being copilot plus plus. And, uh, the model will always sort of needs needs to be reminded that, uh, whenever you're searching for something that says copilot plus plus or something, you sh what you actually really need to.

Or whenever I say cursor tab, you should actually search for copilot plus plus or something like that. So there there are these facts and rules that are quite important. Uh, how I don't want the default to be so a. It would be better if everyone sort of changed their way of coding. Uh, I I think the obviously better approach is we just figure it out. Uh, you know, we should just spend all the time and energy we need and all the compute we need to really nail down, nail down the architecture that we that you have, like, really figure out all the facts and rules.

I don't know if I have any like, interesting, controversial ideas for how that should be done. Uh, someone was joking that, you know, maybe we should email you ten rules in the morning and you'll just like, yes or no on the ten rules. And hopefully you'll build up a corpus over time like you. You want a system that allows you to add rules and then prune bad rules. Like sometimes there will be like, if you just ask the model to look at a PR and give you some rules, sometimes it'll come up with bad rules.

And you need a way of pruning them out. Uh, so, like, what is the minimal set of rules, uh, such that, you know, all your PRs become much easier? Uh, does the model need to look at all of the rules? I mean, we're still sort of figuring it out, but I think I think there's something important at the core of this. That is both in terms of, like, how humans would change, and also in terms of what we should change, just to, like, make the defaults much better.

Uh, because not every single person will change,

of course. But, for example, like, do you think smaller file sizes are better because the model can more easily, like, navigate the code hierarchy, or do you think that creates, um, complexity?

there's always some trade off. So, like, the funny joke is that like, sometimes, uh, people will sort of keep adding to the same file more and more until the model can't edit it anymore. And then you just ask the model to refactor that file for you, cause you, you just like, sort of, you know, in in cursor terminology, you know, composing the file more and more, uh. And it seems pretty clear to me that there is obviously some advantage to the model.

Seeing all the context relevant to the current task in the same file, and also that, uh, for future tasks, it'll be easier. If the model, the file was smaller, uh, I think. Infrastructure wise, we will also make it possible for you to sync all of these files to, uh, a remote server. So, like, we will have a copy, uh, like a big enough copy of your code base at some point. So right now, we're extremely privacy conscious.

And that means like, we try to make sure that the that we never store any code past the life of your request. Ideally, in the future, we can store at least some part of it, uh, in a private way. That allows the model to very quickly do, uh, reliable edits. So you shouldn't have to sort of make these round trips for making every single small edit and that that feels quite bad.

Hm, what else? So, y, you, actually, you were telling me that you run infrastructure, um, a. Also, can you talk about, like, what the interesting, um, infrastructure trade offs are at, Uh, at Cursor?

Um, we, we've built lots of different pieces of infra. There's sort of the traditional company infrastructure, but then there's also a lot of things. Uh, the one that we've been sort of very public about is our indexing infrastructure that, uh, we spent a lot of time optimizing and running at quite enormous scales, like billions of files, uh, per day kind of infrastructure. And for that, we run our own inference. So, uh, for all the models that sort of embed your files, we run, you know, an enormous amount of, uh, just you see, like, really large pipelines.

So like, if you're, uh, some some big company and you have like 400 files or four or 500 files. You want the ability to, while the user is coding, effectively feel like it's being instantly synced across, uh, to the server, uh, while the model is using the embeddings to like search the code base or edit the code base, et cetera, et cetera, uh. So scaling that has been, uh, quite a challenge. And, uh, I think there's been this sort of broad category of databases that are being built on top of S3.

And we're like a big believer in this approach of you should build your UM database. Slash, uh, store of St. Like there, like the I. I don't think there's like a a cl like. The usual term is sort of, you know, separation of storage and compute, or disaggregated, you know, storage databases. But, uh, so the classic example of this is, uh, we use Turbopuff. Turbobuffer stores most of the vectors on an S3, uh, sort of path.

And then, uh, they have a write ahead log, and you sort of write to this write ahead log, the write ahead. When there's some compaction process, it compacts the write ahead log, um, back into, back into the database. And then there's sort of new challenges we've been dealing with with with this indexing infrastructure. So we've been thinking about is there a way in which you can support shared, shared code bases? So if you, you know, all the people at YC, batch C's have a really big code base.

Uh, hopefully in the future, that, you know, you will be able to spin up background, uh, models, editing your code base. And so, you know, we want thousands, if not tens of thousands of of sort of, uh, clients that are connecting to to that code base. And we don't wanna have 10,000 copies of the YC batch C's code base. Uh, for, you know, most of which are not being utilized. Uh, so, can we have a shared trunk?

And then, uh, every single person can have their sort of branch off that trunk? Uh, that sort of architecture is still, uh, you know, we're working on it. It, it's. It's not exactly easy to do, because, you know, how do you easily branch this vector database? At the end of the day, you want to be able to query both the trunk and your section and merge them in a way that you still get the correct top k chunks.

And that's that's not

So when I fire up cursor, it's like quietly indexing all the files that are in my yes, the so we project.

try to, you know, yes, exactly. So when you fire up cursor, it quietly indexes every single thing. Uh, whether, uh, as long as you both a allow us, and you know if it's default turned on, uh. One really popular cursor use case is like, you open up a GitHub repo, you clone it, and then you fire up cursor in that GitHub repo. And now you can quickly ask questions about it. And we try our best to, like, make it effectively instant to to index these really, really large codebases.

Um, there's still obviously, if you clone like LVM, which is 120 files like that will take us a bit longer. Uh, but so, like, for example, an in interesting infrastructure question, uh, for you know, the listeners, or whoever you know, is like, pondering about is like, should you? How should you allocate these token capacity? So we at any point in time have a fixed number of GPUs, which means we have a fixed num amount of token capacity.

Should you be, uh, you know, you want to index LVM or weights and biases? And that's a really large code base, and then there's a bunch of people that have a number of smaller code bases. Should the number of smaller code bases always be allowed to go through? And you should be slow, or should you take a lot of the capacity in the beginning? And everyone else gets like a smaller chunk in the in the hope that, like, you shouldn't get a really bad experience.

And that kind of question is still sort of hard to run. SCALE

Hm, well, how do you think about So everyone, that?

everyone, um, currently, currently, we try to like, keep keep both both sides, like, relatively happy, uh, so peop, you know, there's. You can boost up your capacity a bunch on the next thing. But I'm still like, looking for better answers. Hope, I think there's probably. We haven't spent like that much time thinking about it, but hopefully there's like a really good answer to like, how do you make people happy? Um, there, there's no serverless GPUs, right?

Like, there's no, uh, great serverless option. Because at the end of the day, the amount of compute we're spending is still fixed. Uh, you know, Lucas indexes the the amount of compute is just like the amount of compute you index your code base, plus the amount of compute for like, every single other person that we're indexing. Uh, so, in in an ideal world, that'd be this phenomenal serverless thing where you could boost up your capacity. And then, you know, uh, people can use that, that capacity and then we can boost it down again.

Uh, which is what would happen in CPU land, and that sort of infra has not been built for GPU land. Yet.

Is indexing the main thing that your GPUs are doing? Cause you're also running lots of models, too, like, how do you think yeah, so so about that?

we run the tab model, like indexing is a very small percentage of, uh, of our GPUs. I mean, we run the tab models and, uh, hopefully we'll be running much larger models in the future. And yeah, they, they, they far and away dominate most of the compute cost

I see, so the model running everything. the tab, the tab model Yeah, we, is what you mean.

we, we, we, yeah, so tab model is like hundreds of millions of calls per day. Uh, the uh, big models we're running have, you know, thousands of requests every, uh, you know, okay, so without going into the detail, you know, thousands of requests going on. And so, uh, we're. We're scaling up these models as fast as we can, and, uh, they, they definitely take up far more compute. It kind of makes sense. Also, they're like, larger, uh, you know, one intuition to have is again, you're doing tens of thousands of tokens of inference per per keystroke per person.

Uh, which is both really cool and also really scary. As you're running the inference, uh, like, obviously, caching really helps, uh, but it's still scarier, scarier than running a server.

Has there been any surprises as as you've, like, scaled up this ML infrastructure? Like, is, I mean, you gotta be one of the fastest scaling, you know, ML companies ever? Like, have there been any like kind of pitfalls, or like, I don't know, like, what's that experience been like? Smooth?

Um, there've been glitches, uh, but I think, like, again, the team is really, really talented and, uh, we've sort of gotten over it.

Nice, um, what about, uh? I mean, we're talking like, you know, maybe two weeks after Deepseek came out. And then obviously caused investors to, like, change their mind about Nvidia stock. Did it, like, update your beliefs It's been at all?

really weird to me because I think we're, uh, like, both on the lex pod, but also, uh, before that, we've been pretty public about using DeepSeek in many ways. And, uh, we used to use their 1.5 series models and then switched over to their V2 series models. So it was. It was like, big shock to the system to me that, like everyone was sort of like, going hoo ha, like, this is some new thing, you know?

They've been producing phenomenal work for a while. Uh, their their models, like I used to joke, like they were one of the three or four or five companies that you would trust. If you like, produce good models, where, like, the numbers wouldn't feel like they were juiced up, uh, in a way that there were certain models that felt like their numbers had been a little bit too juiced up, like by by juiced up. I mean, like they would be really high for evaluations, but then if you used the model in practice, you would never like using the model.

And it's just very specific to sort of the evaluations, uh, but DeepSeek, I felt like, was very honest about things and produce has been producing really good models. So we've been running the Deepseek V2 model, uh, for eight or ten months now, probably twelve months, something like that, on our, you know, own inference, uh, and that's the model. We've scaled up to hundreds of millions of calls.

Interesting, how did you choose it? Was it like, just the the best

Oh, we, we knew they were,

picture?

yeah, it was the best, it was the they. They had been producing extremely good open code models. Uh, we have our own sort of post training stack, and, uh, we do RL and stuff, but, uh, for just picking a really well pre trained base, uh, DeepSeek does a phenomenal, phenomenal job. Uh, the data they train on is really good, and the model is both quite knowledgeable, quite smart, and also quite cheap to run for the tab in particular.

And, you know, I think in general, I'm really excited about DeepSeek V3. I think DeepSeek V3 is actually a really well pre trained base for large models. And I suspect it will be very, very useful for making these custom applications.

So, I mean, you've obviously launched, um, you know, like agents, and and it's it's pretty cool, but it's also kind of, you know, contained in, like, how many iterations steps the agent will will do and things like that. Like, do you like, where do you see kind of agents? Um, you know, going in the near term, I mean, like, obviously, inference's getting a lot cheaper, it seems like you could go much broader if you wanted to.

Like, like, what are you I think we're super thinking?

focused on it. I think as people have been getting better at doing RL, the model have been getting better at both thinking and also being extremely coherent. So I think one of the things that is talked about less is how coherent the models have gotten over producing tens of thousands of tokens of output, which, uh, they were not before. I think they would sort of go into delusional mode after a couple thousand tokens immediately, and now they, they've gotten quite a bit more coherent.

And that comes from doing RL and, uh, really, really good post training. And I think agents were bottlenecked by that particular aspect of coherency. Uh, one of the things that makes the Sonnet experience really magical for using, uh, in an agent, is that it's so coherent over such a long, uh, period of time. Like over tens of tool calls, and you know, I suspect as the tasks get harder and harder, you would need to be coherent over hundreds, if not thousands, of tool calls.

And working on it,

One of the things that we

yeah, I I think t are are, generally, like, again, like, back to the mission of the company, the mission of the company is sort of what is the uh? We wanna automate as much of coding as possible, and while still having the developer in the in the front seat. And automating coding in the short term involves, you know, allowing developers to, just like in the cases where they wanna sit back and let the model code doing that.

But in the cases where they wanna drive the editor to, like, make code. Like, I don't know. You're doing race and biases and you want to, like, switch your GRPC thing to some other TLS package in Rust. Like, you should just be able to tell the model, like, I wanna switch my GRPC thing to to use, you know, Rust TLS instead of something else. And the model should just get it and be able to make these large scale code base wide changes.

And that requires the model to be, uh, s, have some some agent type things. Because you're never gonna w sit down and write out exactly the spec of your code base. The the thing that the agent really helps with is you don't have to sit down and explain. Like, yeah, we are YDB, we make this, you know, uh? We have a backend that is written in Rust and Go, the Rust hooks up to going this way.

And f, you know, for our library, we use this and the model should just go and figure it out.

you know, my own experience of playing with agents, which is much diminished compared to yours, is that, you know, when it breaks is kind of the challenge to debug. Like, have you built any systems internally for just even looking at like, Okay, what is the agent doing? Like, you know, why did it get in a weird loop here? Like, what's happening? Like, how do you visualize that?

Oh, we're building our own infra for it for now. Uh, I suspect that there will be phenomenal products in the future that will make this much easier. Uh, for now, we, uh, same thing with building prompts, so we use this internal library called Preempt Um, and the way we built Preempt was it was well suited to our own needs at the time. And I think for the same sort of agent infrastructure, we'll be building our own infrastructure in the short term.

And I suspect in the long term, there will be somewhat phenomenal, uh, you know, dev tools that will come up. To make it much easier to both inspect the chains, be able to stop at any point and restart the chains, uh, be able to debug them in production when something weird goes wrong, all sorts of things that you would need to be able to run like a production system at scale.

Is is the agent evaluation like more, also like a? It sounds like it's more of a vibes based approach than like specific metrics, So currently, yeah, right?

it's, it's pretty, pretty vibes based. I suspect it'll be vibes based in the short term and become as as we get better at shipping these, it'll become more and more sort of, um, daily metrics, and you'll be much more operational with it.

Hm, when you look at like something like a Devon, or like these sort of like, completely automated, like no programmer like approaches. Do you view that as like competitive or interesting, or like, what? What is your your

I think they're interesting.

take there?

And in the medium term, if if you can actually take your hands off and let the model drive your entire editor, or drive, let the model drive the entire editing process. Uh, I am totally open to it, uh, but in in the case where it's like, not really useful and kind of boring and not really that fun. Like, we just wait,

we just what, just wait,

we, we just wait until it gets good enough, like we keep training the models. And, uh, at some point it will get good enough and then it'll be really fun to use. I think in general, over a one to two year timeframe, I expect that the way people will code will change, and I don't think people, I think in the short term, that seems really scary. But I think it'll be this gradual process and it'll be ex extremely natural to everyone coming in that the way the way coding is changing.

You, you, I think, like, for example, the change from not having a copilot to a copilot was extremely natural. In retrospect, like it was not something that that was like, scary to anyone, right? It was this thing that, like, sort of predicted the other thought, and you were like, Wow, this is phenomenal, and you just sort of started using it. And then the change from going from sort of this copilot to this, um, you know, foreground agent interface, where the model that does edits across multiple different files.

And you're like, Oh, I want to switch this to use Rust TLS, and I want to, you know, make sure that you always use HTTP2 and blah blah. Like, the model gets it and it reads all the files and it makes the changes, and you can immediately review the changes very quickly and tell that they're correct. And that was also pretty natural. I don't think there was any way in the middle where, like, people felt like disoriented.

And I think that's sort of going to background things. It'll be all these things are always, you know, more more gradual than one would expect. You would've expected in 2020 that, like, if I said, the way you'll be coding is, you'll sort of start talking to the computer and it'll make changes to random files. And you'd be like, kind of freaked out, you'd think, Oh, it's gonna add all these bugs, it's gonna be impossible to review.

Like, I really enjoy coding, why the fuck am I doing this? You know that, like all all these things, would've seemed scary, uh, and and yet, you know, five years in, four years into the language model, uh, you know, language model journey of products, like things feel quite natural. So, like, 2021, copilot, 2025 are in now. And at any point in time, you know, making the change has not felt very disorienting, which Well, like, maybe maybe in one chapter it would have, but right now it's not really that disorienting.

well, it feels like a lot of fun to me. I mean, like, it's like, I guess, like when I like, connect the dots from like, 2020 to now, I'm It's like, it's like,

gotten better, it's like, it's, it's, it's gotten better, right? It's like, Yeah, well, suddenly gotten like,

it's, I guess where, like, I'm going is like, you know, when I, when I look a few years out, it's I. I have no idea. But it's hard not to see that. Like a world where you, you wouldn't really be doing anything that looks like programming a few years out, right? Or or were you?

like, more people will be coding, more people will be making much more difficult things, like things that are considered much more difficult. Uh, be it lower level things, be it, uh, larger projects, uh, even for their side projects, uh. I think people are usually very conservative with their side projects because they're like, Ah, you know, I probably won't have that much time. I think people will get much less conservative with these side projects. I, I'm generally just extremely optimistic in the medium term.

Yeah, yeah, do you feel like at all? Like, I mean, I guess 1st of all, like, I mean, don't you think? It's a totally different world where everyone can do these like monster side projects? So easily, like, uh, it that that seems like software has a very different feeling, even like doing a software company, seems like it might be hard to like, you know, have like, a protected advantage as much, right? When it's easy to build this stuff,

I I can't philosophize over that. I think I'm not really scared of people having medium sized, uh. I tend to think of these things as like, experimentation becomes much more, uh, much more natural. I think a lot of the things that large changes are usually scary at companies, because a large change requires changing so many pieces and changes. So much time that you want to s plan out everything upfront. And then planning is really hard because you can't really foresee, like, how your production system will look if you do XYZ.

Then everything becomes much more scary. And then you add more meetings and it becomes more formal, and then, and everything just becomes worse and worse over time. And I understand it, right? If you're doing monkey your database transition, boy, do you want to plan out every single small detail, uh, and then you want to argue over every small, single small detail, uh. But if you can start sort of prototyping these things really quickly, uh, maybe it becomes less talking, more coding.

You know, you have much cleaner, concrete artifacts. Um, if you're in PyTorch and you want to do a small API change in PyTorch, it'll take you a day. You probably want to debate out the hell out of it. If you're in PyTorch and you know, you can have a prototype in three days. Uh, maybe you should just argue over the prototype now.

is that how you do? You think about cursor?

Um, hopefully more and more so. Yeah, uh, I mean, there's still things that are scary, but, uh, definitely, I think I found myself thinking, it's it's just much better to argue over the co. I. I suspect that that change will continue.

Hm, awesome, well, I guess, um, one one final question, um, if something comes to mind when you think, um, if you were sort of outside of of cursor and kind of like, fresh eyes into this, you know, kind of world of um. AI applications and and LLMs that kind of working for so many different things, um, is there something else that kind of excites you that you wish you had time to to think about?

personally for me, I've, I've always wanted sort of like, a really good, uh, reading experience, uh, both I. I. I like to spend my time, sort of, uh, free time, either, you know, reading or or spending, uh, time, even reading codebases. Uh, I think it's sort of this underrated aspect of of coding that, like, you know, all of us produce these, some of these artifacts that we pour our, you know, many years of our life into, uh, Reddit, someone has poured their life into Reddit, and I really want to go read and understand Reddit.

What were the hard decisions, what were the easy decisions? Uh, and, uh, I think, both for reading books, for reading papers and for reading codebases, we haven't discovered the the final, optimal AI tool. I think hopefully cursor will contribute to at least reading codebases. But maybe, you know, someone makes it, uh, easier to read books or to read papers. I'll be really happy. Uh, like, reading papers is still like quite an arduous process. I mean, PDF viewers, I, I don't love the current PDF viewer.

Still, like, you click a thing and it'll jump you to the final thing, it's it, it, it. It feels, uh, like, a lot more primitive than it should be. Um, and you know, I've recently been reading papers by just pasting them into, you know, one of these sort of chat apps, and, uh, things are getting better. I I think in general, like, uh, it feels like there's a lower, like a lot of low hanging fruit in lots of different areas of life.

Okay, I gotta ask, Do you have any? What are your top, um, recommended reading codebases?

Um, well, so, as I just mentioned, Reddit Reddit is quite good. If you haven't read it, um, it's, it's relatively small, uh, and still is still quite fun. Probably that's the one I'd be I'd most recommend people because it's a thing that is used by everyone. And, uh, it's just really, really well written. SQLite, for sure, also if you haven't read SQLite, um, again, very well written, it's it's, it's this coherent document by a very small number of people, uh, and then I think mostly I would just recommend.

Like software that you use, you should try to go read the software that you use. I mean, some things are harder, but like, I don't know. If you're a fan of Ghosty, the terminal, maybe you should go go, uh, spend a weekend trying to read Ghosty. Or like, if you're a fan of PyTorch, maybe you should go look into why PyTorch does what it does. Uh, I think there's a lot of choices that you can sort of criticize on the outside.

And you people underappreciate the tremendous amount of work that people say on the PyTorch team have put in to make PyTorch, like, really, really easy for you to use. And this like magical experience where all the just sort of gradients flow naturally. That, uh, that has taken like many tens of thousands of engineering hours, I don't know if it's in the hundreds of thousands or the millions, but it's like a lot of engineering hours.

interesting. Well, thank you so much. I really appreciate your time. Thanks so much for listening to this episode of Gradient Dissent. Please stay tuned for future episodes.