Person
1 episodes · 19 quoted claims
Imbue says it works across the whole stack, from its own hardware and infrastructure through model training and evaluation to user interface design.
“We do everything from setting up our own hardware, building our own infrastructure, pre training, fine tuning, RL evaluations, data generation, cleaning, like UI user experience, like the whole thing.”
Imbue built CARBS, a cost-aware hyperparameter tuner that Josh Albrecht describes as a cost-aware Pareto region Bayesian search.
“carbs is cost aware Pareto region Bayesian search or something like that. But basically, it's a hyperparameter tuner that is cost aware.”
According to Josh Albrecht, CARBS runs hundreds of experiments at different scales, starting small and then scaling up to learn how hyperparameters change.
“it's tried hundreds of experiments at different scales, so it tries at a really small scale and it sees, like, okay, for a really small scale like.”
Imbue created its own internal versions of open-source NLP benchmarks to avoid contamination from training data and to check question correctness.
“We've actually taken all of the open source NLP benchmarks and made our own internal versions of these systems to make sure that they're not contaminated by the training data”
Josh Albrecht says Imbue's perplexity-based evaluation idea came from a paper asking whether emergent properties of language models are a mirage.
“are emergent properties of language models a mirage? or something like that was the title of the paper.”
Josh Albrecht explains that the apparent sudden emergence of abilities in language models disappears on a log scale, indicating the metrics were poor.
“if you look at their performance on a log scale, it's linear. So what was really happening is just our metric was not very good”
Josh Albrecht says DeepSeek Coder significantly outperforms Llama models of the same size and some much larger ones, which he attributes to heavy training on code.
“I think that model still significantly outperforms Llama of a model that's the same size and even of of much larger models.”
Josh Albrecht says one of Imbue's researchers collaborates with Greg Yang, whose MuP work concerns how to parameterize language models.
“like the MuP work by Greg Yang, et cetera. We have, uh, one of our researchers is a collaborator of his”
Josh Albrecht says that if the theoretically right parameterization is used, the learning rate no longer needs retuning as models are scaled up.
“the practical implication is like, you no longer need to tune the learning rate as you scale them up.”
Jamie Simon, a researcher at Imbue, has published work on the fundamentals of learning, including a notion of learnability.
“there's some interesting work by one of our researchers, uh, Jamie Simon, who on kind of like, what's actually happening in the fundamentals?”
A paper by Jamie Simon argues that a network of a fixed size can only learn a limited number of things.
“there's this notion from one of his papers about learnability, like a network of a fixed size can only learn so many things.”
An Imbue paper on self-supervised learning found that learning proceeds in a stepwise fashion, one piece at a time.
“we had another paper about, like, self supervised learning, where you can see, like, oh, there's this sort of like stepwise nature as it learns”
Josh Albrecht says Imbue tested how well existing large language models detect bugs, and found a model that tended to say yes to any line it was asked about.
“we were trying to see how well existing LLMs would do at detecting bugs.”
Imbue holds a weekly paper club on Fridays to review state-of-the-art research.
“We have a paper club, you know, every Friday, where we're looking at, like, the state of the art stuff that's coming out”
Josh Albrecht said in May 2024 that Imbue did not yet have a product, having chosen to invest in deeper understanding first.
“That's why, you know, we don't necessarily have a product yet, we're working on it.”
Josh Albrecht says Imbue's evaluation data show most system failures come from under-specified user requests rather than model mistakes.
“most of the time when systems fail, it's actually from like under specification, and not from, oh, the model, like messing it up.”
Josh Albrecht praises GitHub Copilot for keeping its suggestions short, which makes them easy to review.
“One thing that I really like about Copilot, uh, just as an example, is that it keeps it short, right? So it's easy to review.”
Josh Albrecht argues that an agent able to write software can perform tasks beyond the actions pre-programmed for it, linking agents, coding and reasoning.
“if it can write software now, it can do some huge set of things, and even things that you never intended or programmed in the first place.”
Josh Albrecht predicts that robust AI reasoning will cause more labour displacement and disruption in the next year or two than people expect.
“Now we're talking about a lot more like labor displacement and disruption than we were before.”