Imbue used 4,096 GPUs to train a 70-billion-parameter model.
“we have, you know, 4096 GPUs, um, and how do you set that all up from scratch? Uh, in order to train 70 billion parameter model”
Person
Kanjun Qiu
1 episodes · 17 quoted claims
Imbue released an optimizer that lets it train 7-billion-parameter models and predict performance at 70 billion parameters.
“we also released an optimizer, so this is a optimizer that we used that lets us train like 7 billion parameter models.”
Kanjun Qiu says Percy Liang is working on an effort at Stanford to build a community around open model building.
“Percy Liang is working on this, and, uh, yeah, I think like a robust open source community”
Imbue built a dataset called wild code made of well-tested, well-documented functions drawn from real code.
“we have a data set we call wild code, where it's a bunch of code, it's in the wild.”
Kanjun Qiu says public models scored 0% on Imbue's wild code dataset.
“all the public models get zero out of, like, literally 0% on this data set.”
Kanjun Qiu describes SWE-bench as a software engineering benchmark based on GitHub issues, on which models scored under 40%.
“if you look at a data set like Sweet Bench, uh, so this is like a software engineering GitHub issues, uh, style benchmark.”
Kanjun Qiu says Imbue's investors include Nvidia, Eric Schmidt and a nonprofit institute, rather than venture investors.
“our investors are Nvidia, Eric Schmidt, uh, nonprofit, Haotester Institute.”
Kanjun Qiu says a small group at Xerox PARC invented the mouse, graphical interface, desktop, windows and files.
“it took a small group of people at Xerox Park to invent, like the mouse and the GUI, and the desktop and windows and files”
Kanjun Qiu says that in the 1960s people expected everyone to time-share centralized supercomputers.
“in the 60s, people were really excited about the supercomputer.”
Imbue's hyperparameter optimizer indicated that books data should be weighted heavily in training.
“our hyperparameter optimizer was like, wow, books matter a lot more. Like, please weight the books dataset really highly.”
Kanjun Qiu defines mid-training as a stage after pre-training that uses a large quantity of data to improve task-specific performance.
“mid training is kind of like a post pre training stage where you train on a bunch of, like, a large quantity of data”
Kanjun Qiu says Imbue currently finds post-training more effective, though it exposes problems that may require pre-training again.
“we actually think post training is more effective. But, uh, the post training is identifying problems in the pre train model”
Noam Brown worked on Cicero, an AI that reached human-level performance in the game Diplomacy.
“Noam did. Cicero, which is, uh, human level performance on diplomacy.”
Imbue runs a podcast called Generally Intelligent that interviews mostly machine learning researchers.
“we have a podcast called Generally Intelligent, and we interview researchers mostly”
Kanjun Qiu names Jim Fan's Generally Intelligent interview on foundation models and embodied agents as a favorite.
“Jim fans, uh, on Foundation models to embodied agents was pretty fun and interesting.”
Kanjun Qiu says Imbue chose not to open-source its model's weights while releasing tooling and process details.
“Even though creating a model might be expensive, it might get cheaper in the future.”
Kanjun Qiu says interest earned on Imbue's raised funds covers a large part of its spending.
“even the interest on this amount of money covers a lot of our our burn.”