Exafunction, the predecessor of Codeium, managed more than 10,000 GPUs in the public cloud for a few autonomous vehicle customers.
“We were managing upwards of 10,000 GPUs in the public cloud for a handful of autonomous vehicle customers.”
Person
Varun Mohan
1 episodes · 19 quoted claims
Codeium grew from fewer than 1,000 users at the start of 2023 to over 500,000 users by the end of 2023.
“in the beginning of 2023, we had less than a thousand users, and by the end of 2023, we had over 500,000 users that used the product.”
Varun Mohan says that when developers use Codeium, over 45% of their software is generated by it.
“and when developers use Codeium, over 45% of their software is actually generated by Codeium.”
Exafunction had seven figures in revenue when it pivoted, but Varun Mohan did not see how to scale it substantially.
“we were at seven figures in revenue at the time, but I didn't know how we could effectively scale this a ton.”
The company had only eight people when it pivoted to building a coding product.
“So we were only eight people at the time.”
Codeium shipped its first product and posted it on Hacker News within about six weeks of starting work.
“within a month and a half, we shipped a product and put it on HN.”
Codeium initially used Salesforce's CodeGen model to get its product running before training its own.
“And we originally, actually, to get a product up and running, we used Salesforce's CodeGen product.”
Codeium found that using context from other parts of a codebase raised the amount of autocomplete code accepted by 27%.
“by leveraging context for even something like autocomplete, even outside of chat, we we found 27% uptick in amount of code accepted.”
Varun Mohan says most of GitLab's revenue comes from self-hosted deployments.
“A vast majority of GitLab's revenue is self-hosted revenue.”
Codeium found GPT-4 more steerable than GPT-3.5 for code edits, but its per-token latency was too high, so it trained its own edit model.
“And then what we did instead was we later used GPT four because it it actually is more steerable, but then the latency on a per token basis is too high for users to want to use the product.”
Varun Mohan says a customer dataset of 100,000 to a million lines of code is only about 10 million tokens, tiny next to the trillions used in pre-training.
“training on a S on a on a data set that is like, let's say, 100,000 or a million lines of code is only 10 million tokens.”
Codeium observed a large drop in usage when autocomplete latency increased by a couple hundred milliseconds.
“If we add a couple hundred milliseconds of latency to autocomplete, we've noticed a a massive decline in sort of the usage of the product”
Varun Mohan says some of Codeium's largest customers measure percentage of code written by the tool at roughly 40 to 50%.
“We've worked with some of the largest companies and they're measuring the percentage code written to be somewhere in the forty to fifty range.”
Varun Mohan says Codeium receives over a million pieces of user feedback each hour, used to judge product quality rather than to train generative models.
“every hour we get mil, we get over a million pieces of feedback from our users on is our product getting better.”
Varun Mohan says GitHub Copilot published a post claiming it did not output security vulnerabilities, which Codeium says it could reproduce.
“one of our competitors, obviously the biggest one, Copilot, came out with a post where they were like, Hey, we don't output security vulnerabilities.”
Varun Mohan says CI/CD pipelines at many companies take thirty minutes to an hour, which slows feedback for coding agents.
“Let's be honest here, right? For a lot of companies, it's like thirty minutes to an hour.”
Codeium tries to release new models about once per quarter.
“we generally speaking, try to roll out new models every quarter or so.”
Codeium's evaluation uses open source code it does not train on, strips functions that unit tests cover, and checks whether the model can restore code that passes the tests.
“We take a bunch of open source code we fi that we don't train on.”
Codeium runs its code evaluations in gVisor sandbox containers so generated code cannot damage a machine, such as by running rm -rf.
“And we run those in in sort of gVisor containers because the goal is like, we don't want this thing to rm rf, right?”