The Future of AI in Coding with Codeium CEO Varun Mohan
Gradient Dissent · 23 May 2024
Source: https://wandb.ai/site/resources/podcast
Transcript: https://podcasts.opennash.com/shows/gradient-dissent/the-future-of-ai-in-coding-with-codeium-ceo-varun-mohan-cc9f8a/ (via OpenNash Podcasts, machine-generated)

You're listening to Gradient Dissent, a show about making machine learning work in the real world, and I'm your host, Lukas Biewald. Varun Mohan is the co-founder and CEO of Codeium, a tool that helps you interactively generate code that I actually use almost every day. We talk about whether the models of today are good enough to really write meaningful code for you, how humans might change how we code to increase engineering velocity. The massive effort it took him to get these models running in production for his company, and the lessons that he took away from working in the AV industry.

This is a really in-depth interview, and I hope you enjoy it as much as I enjoyed doing it. All right, Varun, so I'm excited to talk to you today. It's, uh, my favorite kind of guest, where you make an AI product that I actually use in my daily life. So I I feel very informed about it. Um, do you wanna talk about your company and what it does?

Yeah, so I guess maybe I'll start with a little bit of an introduction. Um, before working on this, I worked at this company called Neuro, uh, where I led large-scale, offline deep learning infra. Um, and then afterwards, actually, the company didn't even start out building Codeium. It was this company called Exafunction, and we built GPU virtualization software. So the rough idea was we took large-scale GPU workloads and we made them actually run on CPUs, and we would transparently offload the computations onto a variety of remote machines and pack the computations on as few GPUs as possible.

We also ended up sort of building GPU compiler technology, so technology to make the make models run even faster for autonomous vehicles. We were managing upwards of 10,000 GPUs in the public cloud for a handful of autonomous vehicle customers. And in the middle of 2022, we realized that a lot of companies were starting to reach out to us to effectively run large-scale transformer models for these generative workloads. And we had the perfect infrastructure to go out and do that.

But we felt at the time that purely running deep learning inference for transformer models was not where most of the value we could provide would be. And we thought that a lot of the value would be in the application layer because there are so many other aspects to building a great app, right? As you can imagine, you need to train a model, you need to curate data, you need to actually validate that the model in production is running properly.

And then take your findings and then improve the system. And we were all early users of GitHub Copilot, and we believed that that was just the tip of the iceberg on what the future of these products could look like. And that's where we started with with Codeium. And we started with, I think, a little bit of an ambitious goal in our launch in the end of 2022 that we've made the product entirely free. Um, which, to be honest, at the at the time sound reason sounded reasonable, but it forced us to build really best-in-class infrastructure, uh, to make sure we could support a large workload.

And just to run through some statistics, in the beginning of 2023, we had less than a thousand users, and by the end of 2023, we had over 500,000 users that used the product. We process between fifty and 100 billion tokens of code every day, which I think is puts us in one of the top five to ten generative AI applications with respect to text in the world. Um, and when developers use Codeium, over 45% of their software is actually generated by Codeium.

Wow, over 45% of the software is generated by Codeium. Yeah, that's right. So of, and w- you know, if you look at the space right now, I think there's a variety of ways you can look at these metrics. One is like, potentially acceptance rate, but to be honest, we realized very quickly that that was a little bit of a bogus metric. You can just imagine, let's say you are a code AI tool like Codeium.

Um, you could just show suggestions whenever you have a curly brace, uh, n and you lied, right? You don't need to show any complex suggestions, and I could have 100% acceptance rate. But instead, we wanted to set ourselves up to have a metric that is in some sense unjuicable. It's a metric that actually measures how much value are we driving to the developer. So what we actually measure is not only what lines are they accepting, but at a character level attribution of what ultimately makes its way into the code base.

And this has forced us i if I were to go into the product uh development cycle to actually develop a product that works rather than the product that games a bunch of metrics.

Well, maybe since you started by telling the story, you know, as a as a founder, a CEO, I always, um, am interested to hear about pivots, and it, you know, kind of raises my anxiety level just, um, you know, imagining it. But but what was that like in 2022? I mean, it sounds like you had some active customers, like, um, you know, kind of what gave you the conviction to switch to a totally new mode?

Was it like something that you kind of started to work on on the side and it started to pick up steam? Or did you, like, completely, like, one day show up and be like, Hey guys, stop, and we're gonna, you know, tell all our customers goodbye? Or h How did you think about it? How did you implement it?

Yeah, so actually, that's a that's a great question. I think one of the things and I talk about this a lot, um, to to other folks in in the company. I think one of the big things a startup can do better than a, I think, a larger company is be like, incredibly truth-seeking. And just the the real honest answer was we were at seven figures in revenue at the time, but I didn't know how we could effectively scale this a ton.

Like, as you can imagine, having only autonomous vehicle customers is is not a great place to be, especially in the middle of, uh, 2022. And I think we realized that generative models were going to be, uh, probably the future of how people trained a lot of deep learning models. And I think effectively what we believed was in the past, when people would train bespoke models to sort of satisfy a workload. Instead, a lot of these single, like, zero-shot use cases of generative models would replace those applications, um, in a lot of ways.

So I think in that case, we could have easily just been, hey, we will provide inference for generative AI models. But one thing that we genuinely believed was, I guess there were two facts that we sort of saw in real time. Um, so, first thing is, if all we were doing was providing inference for generative AI models, we thought we would become a massive commodity. Because before that, what we were basically doing is we were enabling sort of workloads that were across many different deep learning architectures, right?

We were running graph neural neighbors, we were running ResNets, we were running top-down lidar point segmentation models and all these other things. So there was a lot more, I guess, differentiation there. But if all we were doing was, let's say, running Llama two, I thought it would become a race to the bottom for us as a business. Um, one. And I guess the second thing that we sort of saw was we believed as a small team we could build a best-in-class app.

And I guess, you know, we thought it was like the coming of the internet. It was like the next coming of the internet. And we thought if a technology as transformation like this came out, we should first focus on building app applications rather than building infrastructure. Um, to sort of start with. And I think you're completely right there. It was a it was a big transition, and we didn't make it a slow transition of, oh, why don't we validate the hypothesis?

Because our belief was if we weren't full and full on in on working on this, we would just fail, right? We had never trained a large language model before this, uh, but we needed to figure it out as soon as possible and, um, and just make the application work.

But I mean, seven figures of revenue is no joke, right? Like you probably had customers really rely on you, like, you know, how long was it between sort of realizing, like, hey, you know, I don't think this is gonna be the business that I want it to be, and this is where I wanna go to actually, like, implementing that.

Yeah, so I think I think what we effectively did was we we gave companies, uh, like, sort of a roll down period of of like, hey, here's what we're going to do to support you guys. But afterwards, effectively give away a lot of the software for free afterwards. Um, for the companies we were working with. Um, and uh, yeah, I guess I guess it is it is concerning, but in some ways, we are like a venture-backed company.

And I guess the trade off for being a venture-backed company is the expectations are not that we are just a seven figure revenue company, uh, in the in the long term. And I think, frankly speaking, we I personally didn't know how to grow our revenue tremendously beyond what we had at the time. Um, so I guess in in that scenario, we had two paths one of certain slow death or one of uncertain high variance or high value outcome, basically, but still high probability of death.

How big was the team when you made this transition?

So we were only eight people at the time. So one thing as a company that we've always done, probably compared to most companies, is we've remained fairly lean as a company. Um, quite late. I'm I'm pretty averse to hiring people, or my philosophy on hiring people is we only hire someone when we ourselves did something and we didn't do it that great, but it kind of worked. Uh, but the assumption is not if we cannot even get it 10% of the way there, someone else comes in and solves the problem for us.

Like, we never hired a sales rep, um, at at that point, actually.

And and I guess, um, how does how does Codeium work? So did you train your own model? Do you use, like, an open source model? How? how does how does that work?

so I think it's it's a great question. So for for us, um, we started out our our 1st thing was just we have to ship very quickly. And we basically, after starting working on it within a month and a half, we shipped a product and put it on HN. And as you can imagine, HN tore us apart, uh, as as Hacker News usually does. Um, but I think the the key thing, though, is in the beginning, our goal here was just build the best product we can possibly build.

And we realized latency and and s and acceptance rate at the very beginning was going to be a very important important metric for us. And we originally, actually, to get a product up and running, we used Salesforce's CodeGen product. Um, in the in the end of 2022, I'm not sure if you're familiar with that, you you probably are. Um, but very quickly, what we realized was actually that product wasn't cutting it. And the the fundamental reason is because actually a lot of the times when people are write typing code, they're typing code in the middle of a line, right?

And this entire paradigm is called fill in the middle. And we quickly realized actually the fill in the middle capabilities. It, first of all, it didn't exist. And that caused the product to actually not be good enough for people to want to use it. So, out of necessity, we were already forcing ourselves to kind of train a model. We very quickly realized that we needed to, actually. And then after training our own model, we saw a material improvement in usage and retention of the product, right?

And a significant uptick in in sort of growth of the product. Um, so this is once again, I think of like, we did the pragmatic approach of we're not a model company, we are a product company. And I guess that that. But in the process, we realized actually we needed to train our own models. And very quickly, for instance, I think, sale, uh, I think it was, there was a model called StarCoder that came out, but StarCoder actually didn't have great fill-in-the-middle properties when it came out.

It didn't support as many languages, actually, when it came out, um, as well. And we realized actually, once again, one of the driving value adds for Codeium is not just can it fill in the code on the current file, can it use actually parts of other files to generate even higher quality code. So we actually realized we needed to tune our models, or even pre-train our models to use context to generate a better, to have a better model.

And actually, maybe this is a I'm I'm getting in the weeds here, so sorry for this. go, get get But fill in the into the weeds middle of an idea

as to why.

is actually not something you can you can train on top of a model. It is something that actually needs to be a part of the pre-training objective of the model itself. Which is like a weird, it's kind of weird, like that, that that is not the case. It's fundamentally, actually, because it's taking your your, um, your data set, and it actually, like, cuts it up into a bunch of pieces to create a suffix, prefix, and middle.

But the way in which you're cutting the suffix, prefix, and middle doesn't, it doesn't tokenize as nicely as normal. Dots, so the model needs to see a lot of it, right? Because usually when you see a piece of code, you know, and let's say you have return x, you never see, like, just the token r e t u. That isn't a token you basically see. You will usually see r e t u r n, right? So because of that, that actually meaned meant that we needed to train our models to satisfy a bunch of objectives to make even the autocomplete product as good as possible.

And I think, you know, most people at this point probably don't think much about autocomplete. It's probably an assumption that it already exists, but a lot of engineering needed to go in there to make a high-quality product.

That's really interesting that that kind of takes me back to, like, BERT models, right? Like, I feel like everyone does, you know, predicting the next word or next token without using the context. But I feel like in in the early days of NLP, people would always kind of predict, um, the thing that was in the middle. I mean, that does sound like a quite different training

No, that's actually true.

regime. Do you, um, is there any else anything else that's different about your models? Like, do you feel like that's like, Um, yeah, maybe what else is, um, unusual about the model that you're training yourself?

Yeah, so maybe I can talk a little bit about why do we even exist? Like, hey, we have this free product. I s I believe our free product is significantly better than Copilot. So I would love if if other folks obviously use the product. We have like a five star rating on the VS Code marketplace, and I I think we have like a a loyal group of people that like using our product. Um, but

I'm one of them, by the way, I am a fan. I I I mean, I use it every day when I, well, I guess I don't write code every day, but I I use whatever it. Um, you code, you're, I when I, when mean, that's I code, I I have it get running. Yeah,

I'm glad to hear it. Um, but I think, you know, there's obviously this whole enterprise side where because we train our own models, we're able to self enable companies to self-host the product. Um, which is, I think, a big thing, just because a lot of companies don't send their code outside. Like, actually, if you look at the largest companies at GitLab and and Atlassian's Bitbucket, and even if you look at GitHub, most of the companies are actually self-hosting their code, uh, on these on these platforms.

A vast majority of GitLab's revenue is self-hosted revenue. Um, actually, so I think I think for us, a there's a capability for us to deliver that. But b also, I think the differential value that we try to provide is how do we deliver products that are, like, contextually relevant. So we we provide enterprises with two real offerings. Um, an ability to leverage context, and when we say leveraging context, it's not just we're wrapping a vanilla model and we're just putting in context.

Our models are tuned to actually leverage context and generate better suggestions based on that. And by the way, by leveraging context for even something like autocomplete, even outside of chat, we we found 27% uptick in amount of code accepted. So it was a massive, massive win. And obviously, that sounds trivial. That sounds incredibly obvious that using other parts of the code base would give you a higher amount of code accepted. But that was one thing where we try to build differentially better products, uh, from that perspective.

And then secondarily, actually for larger companies, if the company has a very bespoke language, we enable them to continuously tune the model on their on their private data, right? And, uh, we part of, uh, us having an, uh, yeah, an infrastructure background enables us to do that on the same piece of hardware that that we sort of have that they sort of have, right? Um, or the same same piece of hardware that they're deploying inference on.

Um, I think maybe some of the other key pieces of of differentiation that we have beyond just being contextually aware is I just think for our product, if you look, we have another functionality at the company called Command, right? And what Command does is it takes a chunk of code and it's able to edit it or generate a lot of code. And what we actually realize is we try to hold our feet to the fire. And our goal here is not to just be some self-hosted product, it's to be the best product on the market.

And what that means is in the very beginning, we actually try open AI, where possible, we try open AI, we try GPT 3.5, GPT four. And we actually very quickly realized that we we couldn't use that for some of the applications that we have. And I'll give you an example here, and hopefully this goes to show how if you're application oriented rather than model oriented, it actually changes the calculus of how you think about building product.

And for the command model, we initially used GPT 3.5. So the user user provides an instruction and we modify the code. What we realized for GPT 3.5 is it's actually really hard to steer. The model is, like, very hard to steer. So if you give it an instruction, you know, this is a little bit of exaggeration, but it instead first responds, Thank you, sir. Here's a bunch of chunks of code, and it, like, responds with a variety of chunks of code.

So it's actually, first of all, you need to parse out where the chunks of code are, right, which is now starting to get into heuristic territory. And I'll have an analogy to this about autonomous vehicles, but it's a bunch of heuristics. You need to wait for all the chunks of code to show up, so you cannot stream the response back, right? Uh, correctly, because you need to wait for the entire instruction to come back. So now, from the user perspective, the latency is incredibly high.

And then what we did instead was we later used GPT four because it it actually is more steerable, but then the latency on a per token basis is too high for users to want to use the product. So very quickly, we actually had to train our own command model, which actually does edits, edits at scale. And very quickly, the uptick in usage on the product started to start to go up precipitously, right? So this is one of those things where we quickly realized, and you could try even Copilot's, uh, product on editing editing code.

Ours is gonna be a significantly snappier experience, and it's gonna be a product that is like, you could follow up, you can edit code incredibly quickly. And this is one of those things where we learned qu- learned very fast of how do we actually build this product correctly? And we just look at our usage data, like, are users genuinely l like enjoying using this product? That's like, ultimately, the only thing we should be focused on.

that makes sense. I mean, do you how big do you make your model? Like, are you trying to keep it small so it can, like, operate at that speed and operate for free? Like, is it smaller even than a a GPT 3.5?

Yeah, so actually, this is a this is an interesting thing. Um, for for our free product, we are obviously like, there's a limit to how big we can we can make the model. But actually, for for our our team's product, which is enterprise class, we're starting to roll out a model that is larger than 3.5 on every keystroke that the user is typing. And I'm using the leak of how big 3.5 is.

I'm not, yeah, I won't share what the what that what that leak fundamentally was, but we actually are. My viewpoint is not that small is is better, by the way. That's like, I feel like a little bit of cope. I think a larger model is always more capable, um, than what a smaller model is more capable of. The real question, though, is what is a user willing to use, right? That's actually what matters, because ultimately, when you look at these systems, like even at ChatGPT, it is not correct all the time.

But there is a reason why the user is willing to use it, which is that it is correct enough given the latency that you're giving for the user. Like, instead, if 3.5, when it was rolled out, or ChatGPT, when it was rolled out in December of the year of 2022, actually came out and was a twenty times slower product, it would not have taken off the same way it did, right? As many people would not have wanted to use this product.

And I guess this is where sort of the the product sector of LLMs really matters. Like, I think of three distinct things that matter is about an LLM product. It's that the quality matters, obviously, like if the quality is too low, it doesn't really matter. And I'll give you a classic example of this. Pre-large language models, autocomplete was not a good enough product. You were not able to generate suggestions with high enough quality that users wanted to use the product.

The second piece is is the latency of the product. If the latency is, like, incredibly high, the quality better be perfect. And that is a big thing that I think a lot of Twitter demos don't really understand, right? You can ship a demo that works one in a hundred times, but then the reality is, like, you will erode developer trust very, very quickly. Right? Well, And then the for the sorry, I'm third

surprised. Oh sorry, let me fi let you finish, and then

No. Y- I mean, you could go just where where do you go?

uh, yeah, and I I. One of the things you said that kind of surprised me, just as a as a user, I think of your of your free product, is that like 45% of the code being auto-generated. My my experience of your product is that it's, I think it's a little more cautious about, um, making suggestions than Copilot. I mean, that's one man's, you know, observation. I I actually feel like if I, you know, just put a brick on the tab key, I wouldn't, um, you know, actually end up with like 45% auto-generated.

So I, I'm like, a little, I'm just kind of confused on. M. Maybe other people are using it a different way, or kind of getting more suggestions than than I do.

Yeah, I think I think, but to be honest, this might have come down to, and I think this is a real point. There's a lot of variability on person-to-person, the way in which you use the product. Um, you could, you could imagine, actually, like, may maybe if I give you an example. There are some people that kind of use the product in such a way where they are working with AI to make sure that they generate the most amount of code possible.

They will over-comment their code to make sure that when they get multi-line suggestions, they're getting the perfect multi-line suggestion for themselves, right? And I think I think it kind of depends on the way in which you're developing. We've met people that percentage code written is, like, quite low, right? Um, but what we're looking at is a is a is a swab of, like, everyone that uses the product, and we're actually noticing that much. I guess maybe if I was to give you a a particular example of, like, how we measure this, we actually measure inside the company two metrics, uh, for some a product like Autocomplete.

We measure this metric called percentage code written. Obviously, that's fine, but the second metric that we're able to tune is this thing called characters per opportunity, right? And characters per opportunity is effectively measuring on every keystroke that the user types, what is the expected number of keystrokes that get accepted by our product, right on it, like, effectively that. And why is this like an interesting metric? It's interesting because if we decide not to show a suggestion, we are basically reducing the expected number of key, uh, characters that could get accepted.

And if we show a very long suggestion that is mostly crap, then we will also, we will also hurt ourselves. So the goal here is like we internally try to optimize this as much as possible. And interestingly, for us, the characters per opportunity that we sort of notice is significantly higher than one. But the reason why it being higher than one means that still it's less than 50% of code written is people do edit the suggestions that we generate, they go out and delete a lot.

So I don't know empirically if that's sort of what you're seeing, then I I believe it, uh, but that's

No, I don't know. I that's kind of the mean,

ways in which we measure it inside the company.

we could screen share and see if I'm using your product, uh, properly. But um, I, I also find myself I don't really use the feature. I think there's a cool feature you have of, like, you know, highlighting text and then, like, talking about it, um, which somehow, just I'm not in the habit of using, but Y- you it, it know, we we can be quite I. I actually find useful.

that people that are r written software for a longer time, like, don't use chat nearly as much. So this is like an interesting thing where a lot of the folks at our company didn't use chat a ton. And then suddenly, after deploying the product, we realized how much value was generating. And I guess, like, when it comes to, like, businesses that we work with, we've noticed chat. And because chat knows your code base, it's helping developers onboard onto, like, large code bases way more quickly.

You know, we work with some Fortune one hundred s, where the number of repositories at these companies is like fifty to 100,000, which is mind-boggling. So most people have no idea what's going on in any of the software, which is very different than I would say. Like, probably a company like yours or mine, where a lot of folks have a good understanding of the piece of code that they're working on. Um, so I totally believe that, by the way,

So how do you like, um, incorporate a gigantic codebase into your model for a customer?

yeah, so we do. We do sort of two things here, and and, you know, sort of I will, I can go into to each one. There's this notion of if what they're writing is a very specific domain specific language, or a language that is not very public, right? So we have a customer that writes a lot of OCaml, and there's more OCaml inside this customer than exists outside in the public.

Seriously, wait, who is the customer? Uh, I'll We could probably share with you, but send them some OCaml fans.

yeah, yeah, so I I'll share with you after after the after, uh, after the conversation. no, no problem. If Um,

they wanna be mentioned, we'll put them in the notes. That's a wild, uh,

yeah, um, so we work with, and effectively, what we realized is actually the default vanilla model that we gave them was not good enough. And effectively, we needed to train on a lot of their internal OCaml. And the reason why that was possible on the same piece of hardware is that actually training on a S on a on a data set that is like, let's say, 100,000 or a million lines of code is only 10 million tokens.

Whereas the amount of tokens that are used in the pre-training data sets for is like trillions of tokens, right? So we're talking about five, if not closer to six orders of magnitude difference in the sides of of the workloads. So it's actually doable to do on, like, a, you know, multi, uh, a sort of eight x A100 on the same piece of hardware, right? Which is actually an interesting observation. And, uh, that that we were able to build out for, uh, for a lot of these companies.

And the second way that we sort of provide personalization is with sort of both keyword search plus in, uh, plus embedding search, right? I mean, this should be, I think everyone has heard RAG more times than than they would like to hear. Um, but e effectively, what we have is we can maintain for a developer a local index of their local changes and a remote index that stores like a point in time snapshot of of embeddings metadata across their code base.

Um, including like dependencies between particular files, like we can even look at commit data to see what files were committed at roughly the same periods of time to sort of see if there's relevance between these pairs of files. So, um, we're able to take these pieces of information to to deliver as much context as possible. Whenever a user types a query, even when they write an autocomplete, when we generate an autocomplete for them,

wait. But how do you incorporate? How do you incorporate their code base, like their entire code base, into an autocomplete generation? Like, is that happening for me? Like, I, I don't actually know if yeah. So if it does that.

interestingly, for autocomplete and I, we've never gotten into the weeds here. So I like, this is this is telling that, um, that we're doing it on this podcast. So for autocomplete, we do it asynchronously, so you can imagine one of the nice things for autocomplete that we're able to take advantage of is like, as you're typing, you're kind of on the same file. So as you're typing, we're kind of taking the cursor of where you are.

Um, the potential utility functions that we think could be relevant based on this, and we're doing offline sort of embedding search to, like, bring in as much context as possible. And the answer is yes, and by the way, that is also true when you write a command in real time, it is actually trying to find all the utility files it can find. So if you open up a new file and you try to write a unit test, it will pull the context, um, from the original function so that you can actually write out the unit test, right?

Um, so that is actually something that we're doing today. For chat, we do it synchronously, and for command, we do it synchronously because the user is willing to wait after they type an instruction. But, for instance, we cannot potentially like, this is a crazy thing. If we add a couple hundred milliseconds of latency to autocomplete, we've noticed a a massive decline in sort of the usage of the product, right? So this is one of those things where us owning the stack end to end, you know, we don't use things like Tensor RT, actually.

Um, and part of the reason why we don't use it is not because we don't like Nvidia, we we love Nvidia. I think the reason is we want a system that is tunable, where we can control performance, right? Autocomplete is a product looks actually quite different than an arbitrary chat product, actually.

I mean, one of the things I talk about, you know, with my friends a lot, is like, you know, should we be changing our coding style to kind of help tools like yours be more successful? And I often wonder about that, like, do you find yourself changing the way you write code? Like, should I be splitting things into smaller files, maybe so that it's like more, more organized somehow for these products like yours?

You know, I would really hope that our answer is, and, you know, w- this entire experience, what it's shown me is being a product that is passive and and goes into the workflow as as seamlessly as possible is the product that wins. It is like the barrier to entry if you want to change user behavior is really high, and you better deliver a best-in-class product. And I think we have some exciting products that are coming down the line there.

Um, but you need to, you need to truly believe that you are g- like your product is gonna change developers minds instantly if you wanna change their behavior. So m the one way in which I've noticed I've changed the way I develop software is like, I over-comment it now. I like, because one of the things about autocomplete that is very different than chat is like, which which actually sort of keeps the ceiling for autocomplete not incredibly high.

Like, it is impossible, even with an infinitely smart model, to get an eighty or 90% acceptance rate for autocomplete. And the fundamental reason is because you don't have all the intent information, right? Some of the intent is in the human's brain, and a lot of it is not in the keystrokes that you're typing or the cursor movements. Like, you just don't have enough information. So what I find myself doing is trying to be more intent-driven, tell the model more about how I'm thinking about it, how I wanna solve the problem.

Because then I now have, like, a frontier, like in my head, I have a frontier visualization of, okay, given this much information, here's how good I can expect the model to perform. And effectively, now I'm like, kind of working with the model, uh, to to get the best suggestion. I don't know if that sort of made sense there, but that's sort of the way my no, that totally view changed.

resonates. I think I think I kind of intuitively do the same, the same thing as you, for sure. Um, I almost like, want a button where I can be like, Hey, come on, model, like, I'm waiting for you here. Like,

Yeah, like, throw more compute S and give me a better suggestion, right? Yeah,

One question, you know, I'm in like a s CEO group, also with like a bunch of kind of SaaS CEOs that are like, I would say, like, more oriented, uh, mostly less technical than me. And they're like, Hey, like, you know, I hear these like coding tools, like, you know, h like throughout these things, like, you know, 45%, um, you know, code generated. Like, why are my engineers, like, you know, twice as efficient? Uh, and I like, think about for myself, like, am I twice as efficient?

Like, am I like 1.5 x efficient? It's really hard for me to like, even like, make some kind of assessment, you know, of myself. Like, I almost feel like I'm like, just a little bit, like, maybe lazier when I'm writing code. Like, I'm just sort of like my brain is like adapting to, like, using, um, less brain power as I do it. I'm, I'm not sure. I'm. I'm curious if you have any sense of like, um, are we making efficiency gains with tools like yours?

yeah, so look, I. We try to. One of the things that we try to do for customers and just every user is we try to be the company that is not a hype machine. Like, we will build the best technology we can build, and given the frontier of what's capable, we will always do that. Um, but just be very but I'm I'm honestly, upfront.

I'm. I'm. All you picking this up for you, man? Like, like, what do you? what do you? What's your experience with your own product? Like, do you feel like No, so you're way more I'll, I'll tell you, efficient?

I'll tell you this. I think software development is not just writing code, um, like, I think it's it's partially that, obviously, like, you know, maybe one pessimistic way to look at it is that if you give your engineers more time and you don't actually ratchet up the amount of tickets that they need to solve or the amount of products that they need to build, you will still get the same output. Um, and and it will just be like, stolen value I i in some sense.

But look, I think the the other way to look at it is. We've worked with some of the largest companies and they're measuring the percentage code written to be somewhere in the forty to fifty range. But they're actually looking at the sort of PR cycle time on a per developer basis, and they're re they're looking at numbers that are between fifty and twenty percent, 20% faster, right? For these companies. Um, Does that does that and

fit with your own experience?

yeah, so my sus, my suspicion, and is it's very much task dependent. So let me give you an example. Um, our developers need to write software for Windows, okay, and this is a thing that will be very unheard of for Silicon Valley. Uh, but a lot of our developers write software in Windows. In fact, like, you will see a lot of Silicon Valley products that never build, uh, solutions for Windows. Despite the fact that over 50% of all software developers are on Windows, and that's the same for our our own usage base, right?

So we need to deliver, uh, develop for Visual Studio, which requires us to write code in C, right? And no one at the company has ever written C before. And this notion of actually being able to write software and quickly iterate on an extension that is in a completely brand new language is not something I think we would have been able to undertake quickly enough without tools like this. So this is actually, like, brand new value that we're able to generate, right?

I think for a startup, actually, this is this is whether or not this is a good thing to say or not, it is. I think these tools are not nearly as valuable as they are for a large company, simply because the sheer amount of code isn't that large. So the amount of time it takes to onboard at a startup is going to be way lower and is going to be way lower than a large company that has written billions of lines of code, right?

And the the sh the surface of code that you might need to touch is significantly larger. But I'm just giving you an example of, like, a use case where I could not have seen us being able to do it nearly as quickly, and now it is actually feasible, right? Um, developers that previously had never written a lot of C now can, and, uh, most of our inference stack is in C now, right?

It's funny, I I do feel like it's a magical experience when I like encounter a language I'm less familiar with. Like, I, um, I was doing some robotics with a friend over the weekend, and so I was writing C, which is not something I've done in, like, you know, over a decade. And it was just like, man, like, th- this is making me probably like a hundred times more productive than I would be if I, you know.

But. But then I kind of wondered, like, okay, if I was actually doing this in a real-life code base where someone kind of cared about the quality, is it like encouraging me to get like, overconfident? And am I like, am I writing like, really buggy code that, um, Well, I I shouldn't do. be writing? I, I'm just I wasn't sure.

I think it's look, I think it's a little bit of both. I think one of the things that I I think we are currently this entire space is forgetting is like, yeah, barfing out a lot of code is not the real goal here. It is like code that is like tested and reviewed, and probably if it's in C, there's a lot of bugs there can be. Are there use after free issues? Are you actually freeing things up?

Is the m is the multi-threaded properties of your code actually safe, right? Is this actually thread safe? And by the way, like, the more smart you are in C with atomics and stuff, you're more likely to write more buggy code. So the more sort of bravery you have, that's probably a dumb idea, unless it really, really matters on the performance side. Do the dumb thing in C is probably the right right mental model. Um, so I totally agree with you, and that's sort of why actually, as a company, we're trying to actually extend the software development lifecycle beyond just writing code.

We want to actually provide value, even in the code review stage, right, and the code investigation state. How are you? Like, how are you actually looking through code? We don't actually believe that purely writing code is all the time saved. In fact, when you go to, like, the largest companies, the amount of time that people are actually spending only writing software is probably around twenty.

Do you try to filter out, like, poorly written code, like, you know, some of my experimental stuff in your, um, training process?

Yeah, so okay, here's what we basically do. So, first of all, we don't train generative models on any of our user data. Um, so that's that's one thing. Um, obviously, like, I think I think that helps, uh. But also at the same time, then we don't need to worry about all this experimental stuff. Um, I think what we do use user data for is knowing if we're getting better as a product. So every hour we get mil, we get over a million pieces of feedback from our users on is our product getting better.

And are these kinds of suggestions getting accepted with more likelihood than other kinds of suggestions, right? Which is helping us actually iterate on the product, uh, very quickly. In terms of what you were saying, which is like, can we actually make sure that our suggestions are high quality? There's a limit, there's actually a limit, and this is the honest answer, which is that these models are confidently wrong in a lot of cases, they are very confidently wrong.

And I think, you know, one of our competitors, obviously the biggest one, Copilot, came out with a post where they were like, Hey, we don't output security vulnerabilities. And we, they said, we'd never output a SQL injection, we never output a path injection, a password leak. And we were able to generate all of these in the product. So the way I like to look at it is don't get rid of deterministic tools that work to serve a purpose.

Large language models aren't at a state right now where they can do that right now because of the non-determinism. If you have tools that actually validate for SQL injections, all the uh, like, sort of mass security vulnerabilities, keep all of those tools. In fact, what I believe is those tools are a big gr are like a helpful ingredient to large language models, right? They're a great way to remediate bugs potentially at scale. Um, but that's sort of my my take on this.

I think, you know, if humans still write bugs, LLMs are basically still writing bugs still. So,

You know, I think one of the things I'd attribute to, you know, Weights and Biases early success was, um, we cared a lot about the quality of the documentation. I mean, this is something that I think, you know, even my co-founders, um, you know, thought was really important. We treated the docs like, you know, like, almost like a product, and we've really tried to kind of maintain that level of quality. I don't know if we're always, um, successful, but it is kind of interesting, you know, using these tools.

I find myself going to library's documentation quite a lot less. So, I mean, I, I guess, like one, do you think, like, you know, maybe documentation methods change or or get less important? And then the other thing is, you know, I often see it using, um, you know, deprecated functions. And I really wish I could, like, you know, tell it, hey, once and for all, like, don't, don't do it this way anymore. Like Weights and Biases has, like, a new, um, you know, better way of doing things.

Like, can you you see any, like, kind of path to that? Like, I hate to sort of get stuck in the way that we, yeah, do things two or three years ago and have to sort of, like, support that, you know, forever. Because these tools just kind of keep learning from the, um, median way that people do things.

yeah, I think, you know, to your point there, it's. It's pretty obvious that these pieces of code, for instance, like, let's say, let's say you, uh, there's like, maybe this is a bad example, you know? Go has now templates, right? They actually have generics, and we want to be able to use Go generics, but the usage of Go is like, it's using pre-generic go. My suspicion is the the big thing that's going to end up happening is this is gonna be a context problem.

Where ultimately, when you write software, it's going to need to know the dependencies that are most relevant and then pull the right documentation. My suspicion is actually writing really good product documentation is still going to be what you, what you should basically do. Um, I would argue probably the best place to put it now is GitHub also. Um, just because it's open, and now that's probably what a lot of companies are crawling, right? I if anything, like a lot of open source code.

Um, but yeah, I think I think this is a big problem right now. And my suspicion is like, this is something which we w we constantly get asked, right? Our Java version is not, is not 1.7, it's 1.8. Like, give me the streaming implementation. I'm just giving you an example there, right? Um, and I think I think this is something that we'll need to come to terms with and grapple with, and I think it's context, that's probably the right solution there.

Well, I mean, I think another thing that EV everyone's wondering about is kind of where does this go, right? Like, these models are clearly like getting better and better. It's sort of hard to exactly quantify that. It sounds like, you know, you have some methods to do that. But like, you know, when you think about sort of the next generation of these models, um, you know, what do you? What do you imagine is happening there? Like, do you think that, um, you know, writing code as a skill goes away anytime soon?

I think, you know, first of all, I'm like a big tech like technological optimist. I love technology. Like before this, I think very like anyone who worked in AV probably is they believe in the future. Um, I think one of the things that AV genuinely made me feel, though, is is technology will improve way, way more quickly than you believe. But also at the same time, you cannot set deadlines on real R and D. Like every year that I was at an AV company, the next year AV was going to be solved.

But that is not to say that I'm, I don't believe in the future. Actually, if you look in the meantime, in 2018, the amount of teraflops in a consumer grade GPU was ten teraflops. By the end of 2022, with the forty nineties, you got to 660 teraflops with FP eight. So we had two orders of magnitude increase in the amount of compute per consumer grade GPU. But so the benefit there is technology is getting so much better.

But that also means if you were to build an AV system all from scratch in 2022, it looks remarkably different than the one you built in 2017. It it's not the same, it's not the same kind of car. You can now run models that were never feasible in the past. And where am I really getting with this? I think what I'm getting with is technology is going to improve tremendously. But the products and the capabilities that are going to be possible, we w- we're not gonna have control over that, right?

In other words, we should be very honest with ourselves. Like, I personally do not believe everyone will be using an AI engineer in the next month or two, right? Or in the next three months. And the reason is actually because, like, when you look at at every company, there have been a lot of companies that try to solve in the AV analogy, level five autonomy for the last ten years. And we're just now starting to get there, right, with with Waymo, I think, starting to scale.

But ultimately, like you, you don't have control over that. And but I, I do suspect that in the next coming years, the models are gonna become more and more capable, capable of of higher and higher reasoning. And what that's gonna enable developers to do is is even for us. And we're gonna start launching these, make changes that are more than just in a single file. We can start making larger scale changes and get to a path to a PR.

Like, our goal is the same as everyone else. I wanna generate entire PRs. I don't want our goal is not to just auto complete sort of ten, fifteen lines of code, right? But we just need to be very honest with what the timelines are. And I suspect in that world, by the way, I think that there will be more developers than there are today. It'll be the same as when we move from assembly to Python. More people are able to leverage technology to build more.

And if you go to, like, the largest companies, they're mostly software start. They wanna do a lot with technology because technology can actually improve their bottom line, it can accelerate revenue growth for them. So my suspicion is that we will have more people that are maybe they're not called software developers like ten years from now, but people that are like manipulating technology. Uh, I think we are going to have way more of them than we have today.

Well, okay, but you're, you're really, um, uh, I feel like making, uh, an easy claim here of saying we won't have something like this in the next like month or or two months. Like, you know, but let's like roll forward, like a year, to, like, you know, be concrete. Like, um, do you think that, you know, we should expect to see, like, multi-file auto-generated PRS? Like, do you think a lot of the PRs at Weights and Biases, um, in a year from now, will be generated by, um, code?

yeah, so here's okay, maybe let me be more concrete, uh, with this. I think what's gonna happen is you will see AI AI modify multiple files, but it will still be interactive with the developer, right? And I think the fundamental reason is is actually human behavior. How long, Lucas, are you willing to wait for an AI to stumble across doing a bunch of stuff and potentially twelve hours later, tell you with twenty chance that it got the task done?

How many? And all the while, you need to chat with it along the way. A lot Well, that's of people

a big caveat at the end. I mean, if it, like, went through all our GitHub issues and like, tried to fix them with like twenty chance success, I, I would be like, Yeah, that's great, you know?

but then, but then here's the here's the right the actually, that's a that's a good format, oh, uh, sort of formulation. But I think that's not revolutionary, and the reason why that's not revolutionary is what you actually want to do as a business is you wanna finish something by the end of the quarter. And you probably cannot have something be every twenty four hours. I have like a ten chance of a developer walking through with something and just getting a a task done right.

It it's not, it's not something

Kinda similar to how we operate. I don't, I don't know about you, but sometimes I feel like,

okay, okay, but like, you get my point though, right? What you do want, in my view, what I think is a more rich philosophy is what happens if we could take every developer that currently exists and multiply their capabilities. Rather than have something that is like non-deterministically taking, uh, autonomously finishing some fraction of tasks, and likely the easier tasks. Why not just be a bar raiser? Why not just make it so that every developer, the junior developer, is able to more autonomously ramp on a code base and write way more software, right?

That is more tested. And the senior developer, that is ultimately the person that is has a lot of knowledge at the company and truly just wants to build apps. I don't think any senior developer likes answering questions all day or all night, but that's the ultimate end state of where these developers go. Because they built up so much bespoke knowledge at these companies, right? What happens if you're then able to multiply the amount of output that they're able to generate?

And look, maybe the answer is you automate part, parts of it. You're able to generate a ten file, ten file change. The only question, though, that I believe is, I don't think by the end of the year, we will be able to consistently generate ten file changes at a speed, at a speed that's fast enough that a user just commits the commits. The code, I think what's gonna happen is we're gonna generate a ten file change.

The developer will go in and modify, and very quickly, a new plan will rapidly get created. And I think that level of interactivity, if we look at all the products that have worked so far, as long as the product is not perfect, optimize for speed and latency and interactivity. And I think that's that's a more I I believe that's where we're probably going. Um, and and that's where we're building towards as well as a company.

It's kind of interesting, you know, I? I would think, like, um, if it didn't have to interact with me and could do something useful, I might be much more tolerant of of higher latency, right? Like, the latency is low because we're kind of working together. But if if it was like doing things while I slept, that were useful, I think I might not mind that's true. So it too much.

if you could, if you could treat it as you could do simple tasks while you sleep and it goes off and does them with twenty chance, I think that's, I think that's, that's reasonable. Um, I just think it's a question of is that as as valuable? And I'm not saying it's an either-or, by the way, I'm not saying it's an either-or. Is that as valuable as potentially making your developer, each of your developers, two times more efficient?

And I would argue, actually, the the latter is something that's more deterministic and actually way more valuable to your business. Because it actually just means that you get things done twice as fast, because speed is not something you can buy.

What do you think has to happen to get developers twice as efficient? Like, what, what are you? Yeah, what are you working on?

no, I think I think we need to redesign with the way in which we write software. And this is sort of why I was saying writing software is not the only thing that needs to get done. You need to be accelerating PR reviews, you need to accelerate the way, the speed at which you can run code. Like, I think execution is a very important aspect. But if your execution signal is your CICD pipeline, and that is the way these agents operate, how slow is your CICD pipeline?

Let's be honest here, right? For a lot of companies, it's like thirty minutes to an hour. Imagine if the only signal that these things have is running a thirty minutes to an hour or running a unit, running a unit test where the build process takes an hour. So you might need to reinvent the way in which you can test micro changes, like dynamically build or or compile small chunks of code and be able to unit test them incredibly quickly.

Because I actually believe, as long as the system is not perfect, optimize for for iteration, iteration speed, right? I I think, I think Lucas, like, probably most people criminally underestimate how long they are willing to wait and watch a bad system operate autonomously. That is not a product that is gonna be sticky. Like me, myself, if I saw this thing run two commands incorrectly in a row and I could do nothing but just watch it run two commands incorrectly, I'm gonna rage quit.

Interesting, do you, um, do you plan then to kind of take feedback from unit test and incorporate it in the product? Cause that does seem like it would be a big level up here.

100%, yeah, we believe actually this is why, like, we actually take outputs from the terminal. We are actually capable of of leveraging terminal outputs, um, as as part of the context. One of the UN, one of the weird things about unit tests, I will say, like, you know, I don't know how much about your code base. A lot of cases, unit tests and tests in general are harder than the code you write itself, right? Let's say you're unit testing a database, you need to mock the entire database, you need to mock all the underlying data structures, you need to encode.

If it's a multi-threaded piece of software, this is not easy to test, right? So I think, in a lot of ways, if we can leverage unit test, that's great. I think we just need to be honest about, like, how much code is truly well, like, properly unit tested right now. Um, it's probably not incredibly high, right? Or like, has a proper integration test as well. Um, at a company, but you're totally right. Like, leveraging unit tests as a feedback loop is one of the things that I believe is gonna be the most important thing for agents to do.

But the other thing I believe needs to happen for agents to work is one step of an agent needs to be good enough and quick enough. Otherwise, I don't think agents are gonna be the way in which we're gonna write software.

Well, look, we're we're getting a little bit close to to time here. And and one question that I ask, um, everybody, and I'm always really curious about, especially someone like you that has something really in, um, production that's working. Is like, what? What have been the surprising pitfalls of, like, taking this core Lm technology and getting it into user hands? Like, the more specific you can get about, like, what your like, you know, stack looks like, and you know any sort of like mistakes in the way you designed it that you've learned from along the way.

I'm I'm sure listeners would really appreciate.

I'm trying to, I'm really just trying to think, um, about this. I think the biggest mistake is to think that the product is just a a model. The sheer amount of engineering we had to do to, uh, to deploy any of these features is is herculean. And I And why think,

is that? Wait, so but why is that right? Cause there are, you know, there's like, you know, there's like wrappers that you can use, like, um, you know, Llama CPP, or, you know, there's, there's a whole bunch now. And

yeah, I'll give you an example.

um, you know, yeah, tell me what's what's hard, what, what really Yeah, so happens here?

okay for us, we we actually have a a crazy amount of volume of autocomplete requests, uh, every second. We probably process, I don't even know what the number is. Every time a person types a keystroke, we're going out and doing hundreds of trillions of computations, um, hundreds of trillions of computations. We realized actually very quickly that actually, if we didn't batch aggressively, we would be hurting performance. And I'm getting into the weeds here. We didn't we.

We realized we would need to buy way more GPUs, and then, uh, on top of, on top of that, if we batch too heavily, we hurt latency. What is the smooth trade-off, right? Uh, of this? And then maybe even a a simpler thing, right? When we show a single versus multi-line tradition, how do we know when to show a single line? how do we know when to show a multi-line? how do we know when to be more aggressive?

We actually turn on a tremendous amount of experiments to see can we turn on multi lines all the time? And we actually realize that users actually, in a lot of cases, get out of flow state. How do we measure what flow state is? And I could just go down a l rabbit hole of of of every, everything we need to do from a product side to build the best product possible. And part of it is the modeling, like, obviously, everything with the inference infrastructure is the modeling, um, ultimately, but it's all in service of building a great, a great product.

And every time you ask a question, which is like, how do we do this? There is a good deal of infrastructure going to test a hypothesis, right?

And how often do you roll out new models?

Yeah, so, um, we generally speaking, try to roll out new models every quarter or so. Uh, and and sometimes this might be we wanna change some of the some of the data on top. We realize actually a new type of data would be very valuable, uh, to improve, improve quality, like taking some other type of context, maybe putting in stack traces in as input, um, to make sure that we could use terminal output and and those kinds of things.

So I think whenever we have new ideas, we're constantly trying to validate hypotheses and and ship them out. And this is also why, like for us, we had to build infrastructure to experiment on top of models. And this probably sounds basic, but even when we change models, sometimes the tokenization changes, and the tokenization, though, lives on the client. And our client needs to live on a variety of operating systems. We need to support Windows, Mac, um, Linux, across ARM and X86, right?

And each of those operating systems has different amount of compute. Sometimes people run on a Chromebook that has no compute on it, and then even the tokenization takes a long time on those machines. So none of these problems are that sexy, right? They're very unsexy, and um, but at the same time, like problems we needed to solve to just get this thing running and working.

Totally. Um, can you speak to any products that are coming? Can you get us excited about, uh, new things you're thinking about?

Yeah, so I think we will be very like, in in short order, launching something that is is doing what we discussed before. Um, I believe it agents are are like, I genuinely believe in agents. I know it sounds from all this that I don't, uh, but we're going to be figuring out products that can that can actually, like, operate on top of very large codebases and make what we believe are high confidence changes fairly quickly on top of many files.

Um, and we think that this is gonna be a big unlock for developers. And on top of that, starting to hit on more, more of the software development lifecycle, which includes the ways in which you review software. Um, I think we will be starting to make progress there too, uh, which is outside of the ID ID as well. And, you know, interestingly, this is probably something else that Silicon Valley is not very familiar with. Most companies don't use GitHub, actually, very few companies actually use GitHub.

Um, which is, uh, which is something that I didn't realize until actually embarking on this journey.

Really? What do they use? GitLab?

Yeah, GitLab, Bitbucket, Bitbucket has month millions of monthly active users, right? Which is, uh, which is crazy. Perforce, Garrett, Subversion, Mercurial, Harvest. No way I mean, I could go through subversion? No, come on, no. A lot of companies like, use Subversion. It's it's you would. So, one of the interesting things, and I'll end with this because I don't know if this is interesting. Source code management tools are some of the most sticky pieces of software out there, which is actually good, like it's a good if you're in the business of a source code management tool.

Because the CICD pipeline is coupled there, people write comments in there and they don't wanna lose that. And a lot of companies actually have many, many repositories. So imagine having many, many tiny Jira instances that you need to move from Jira to linear. It's gonna be annoying. Once you pick one, it is actually really, really annoying to move off. And some companies, when they get bigger, they have many business units, each business unit has its own source code management tool.

when you talk about, like, developing an agent, do you imagine, like, letting something loose on my computer to kind of, like, look at all the files, run some code, maybe, like, you know, go into my version control and like, check something out and try it? It would Is that need to be some does that sound about right?

combination of local and remote, yeah, because the local has more most of the recent changes and probably some of the build settings. And then, but even if we do both local and remote, actually, you know, when we test our software, um, we actually, it's, it's actually kind of cool the way our eval works. We take a bunch of and I'll tell you how this it tell you this just because it'll give you a sense of this.

We take a bunch of open source code we fi that we don't train on. We find the unit test that d all the unit tests, we find the functions that make the unit test pass, and we start stripping the function. And why is this a good unit test, and why is this a good eval? What we're then testing is is our model and our ability to retrieve context able to make the unit test pass. So it's now checking not just can it parrot the output, but can it apply context well?

Are we good at retrieving context? Is the model actually capable of our reasoning and does it make the unit test pass? And we run those in in sort of gVisor containers because the goal is like, we don't want this thing to rm rf, right? We don't want this

Really?

to remove my entire hard disk. So we would need to build infrastructure to do that also if we do it on the user's local machine. Obviously, that has its own annoyances because we'd need to do this in an operating system agnostic way. Um, so yeah, these are all problems that we need to solve, and once again, like, I think they're exciting problems and that we that that I'm I'm hungry for us to go out and solve.

It's just, uh, yeah, it's gonna be hard work.

Very cool, Vir, thanks for going into the details with me. I hope other people enjoy this as much as, uh, I enjoyed talking to you.

Yeah, Lucas, thanks a lot for the time.

Awesome, yeah, have a good one. Thanks so much for listening to this episode of Gradient Dissent. Please stay tuned for future episodes.