2023
Josh Albrecht says Imbue tested how well existing large language models detect bugs, and found a model that tended to say yes to any line it was asked about.
“we were trying to see how well existing LLMs would do at detecting bugs.”
Topic
Imbue
21 facts from 2 episodes, in date order. These are what speakers said on the shows, each linked to the moment it was said. They aren't independently verified.
2024
Imbue says it works across the whole stack, from its own hardware and infrastructure through model training and evaluation to user interface design.
“We do everything from setting up our own hardware, building our own infrastructure, pre training, fine tuning, RL evaluations, data generation, cleaning, like UI user experience, like the whole thing.”
2024
Imbue built CARBS, a cost-aware hyperparameter tuner that Josh Albrecht describes as a cost-aware Pareto region Bayesian search.
“carbs is cost aware Pareto region Bayesian search or something like that. But basically, it's a hyperparameter tuner that is cost aware.”
2024
Imbue created its own internal versions of open-source NLP benchmarks to avoid contamination from training data and to check question correctness.
“We've actually taken all of the open source NLP benchmarks and made our own internal versions of these systems to make sure that they're not contaminated by the training data”
2024
Josh Albrecht says one of Imbue's researchers collaborates with Greg Yang, whose MuP work concerns how to parameterize language models.
“like the MuP work by Greg Yang, et cetera. We have, uh, one of our researchers is a collaborator of his”
2024
Jamie Simon, a researcher at Imbue, has published work on the fundamentals of learning, including a notion of learnability.
“there's some interesting work by one of our researchers, uh, Jamie Simon, who on kind of like, what's actually happening in the fundamentals?”
2024
An Imbue paper on self-supervised learning found that learning proceeds in a stepwise fashion, one piece at a time.
“we had another paper about, like, self supervised learning, where you can see, like, oh, there's this sort of like stepwise nature as it learns”
2024
Imbue holds a weekly paper club on Fridays to review state-of-the-art research.
“We have a paper club, you know, every Friday, where we're looking at, like, the state of the art stuff that's coming out”
2024
Josh Albrecht said in May 2024 that Imbue did not yet have a product, having chosen to invest in deeper understanding first.
“That's why, you know, we don't necessarily have a product yet, we're working on it.”
2024
Josh Albrecht says Imbue's evaluation data show most system failures come from under-specified user requests rather than model mistakes.
“most of the time when systems fail, it's actually from like under specification, and not from, oh, the model, like messing it up.”
Undated
Imbue used 4,096 GPUs to train a 70-billion-parameter model.
“we have, you know, 4096 GPUs, um, and how do you set that all up from scratch? Uh, in order to train 70 billion parameter model”
Undated
Imbue released an optimizer that lets it train 7-billion-parameter models and predict performance at 70 billion parameters.
“we also released an optimizer, so this is a optimizer that we used that lets us train like 7 billion parameter models.”
Undated
Imbue built a dataset called wild code made of well-tested, well-documented functions drawn from real code.
“we have a data set we call wild code, where it's a bunch of code, it's in the wild.”
Undated
Kanjun Qiu says public models scored 0% on Imbue's wild code dataset.
“all the public models get zero out of, like, literally 0% on this data set.”
Undated
Kanjun Qiu says Imbue's investors include Nvidia, Eric Schmidt and a nonprofit institute, rather than venture investors.
“our investors are Nvidia, Eric Schmidt, uh, nonprofit, Haotester Institute.”
Undated
Imbue's hyperparameter optimizer indicated that books data should be weighted heavily in training.
“our hyperparameter optimizer was like, wow, books matter a lot more. Like, please weight the books dataset really highly.”
Undated
Kanjun Qiu defines mid-training as a stage after pre-training that uses a large quantity of data to improve task-specific performance.
“mid training is kind of like a post pre training stage where you train on a bunch of, like, a large quantity of data”
Undated
Kanjun Qiu says Imbue currently finds post-training more effective, though it exposes problems that may require pre-training again.
“we actually think post training is more effective. But, uh, the post training is identifying problems in the pre train model”
Undated
Imbue runs a podcast called Generally Intelligent that interviews mostly machine learning researchers.
“we have a podcast called Generally Intelligent, and we interview researchers mostly”
Undated
Kanjun Qiu says Imbue chose not to open-source its model's weights while releasing tooling and process details.
“Even though creating a model might be expensive, it might get cheaper in the future.”
Undated
Kanjun Qiu says interest earned on Imbue's raised funds covers a large part of its spending.
“even the interest on this amount of money covers a lot of our our burn.”
Episodes