Google put its entire search index into memory in 2001, according to Jeff Dean.
“So in 2001, we introduced, we put our entire index in memory.”
Person
This page collects what Jeff Dean said on the Latent Space: The AI Engineer Podcast, in an episode called "Owning the AI Pareto Frontier". He covered Google's early search systems, the history of neural networks at Google, the Gemini AI models, and the computer chips and techniques used to run them.
Dean says Google put its entire search index into memory in 2001. He also says Flash models from a new Gemini generation have matched or beaten the previous generation's Pro model, and that he believes general models will win out over specialized ones in most cases. Each fact below comes with his exact words and a link to the moment.
1 episodes · 26 quoted claims
Google put its entire search index into memory in 2001, according to Jeff Dean.
“So in 2001, we introduced, we put our entire index in memory.”
Jeff Dean says Google's index update rate went from about once a month to updating any page in under a minute.
“So it used to be once a month. And then we went to a system that could update any particular page in like sub one minute.”
Jeff Dean gave a talk at a web search and data mining conference in 2009 about the origins of Google search.
“I actually gave a talk in, I guess, a web search and data mining conference in 2009.”
Google redesigned its search and retrieval system through four to six generations between about 1999 and 2004 or 2005, according to Jeff Dean.
“But we went through sort of four or five or six generations of redesigning of the search and retrieval system from about 1999 through 2004 or 2005.”
Jeff Dean says distillation was motivated by image specialists: treating about 50 specialist models as an ensemble performed much better but was impractical to serve.
“you get much better performance if you then treat that whole set of maybe 50 models you've trained as a large ensemble.”
Jeff Dean says that across multiple Gemini generations, the Flash model of a new generation has matched or beaten the previous generation's Pro model.
“for multiple Gemini generations now, we've been able to make the sort of Flash version of the next generation as good or even substantially better than the previous generation's Pro.”
Jeff Dean says Google uses its Flash model in search products such as AI Mode.
“We're using it more in our search products, AI mode, AI reviews.”
Jeff Dean says Gemini's long-context capabilities first arrived with Gemini 1.5.
“some of the long context capabilities of the Gemini models that came, I guess, first in 1.5”
Jeff Dean says Google is pushing context lengths of one to two million tokens.
“We're trying to push the frontier of 1 million or 2 million context lengths.”
Jeff Dean says Google's ranking systems start with a huge number of pages and progressively filter them with more sophisticated algorithms.
“our ranking systems would be built to start with a giant number of web pages in our index.”
Jeff Dean's 'Latency numbers every programmer should know' lists roughly eight to ten metrics such as the time for a cache miss.
“I mean, this has like sort of eight or ten different kinds of metrics that are like how long does a cache miss take?”
Jeff Dean says moving data across the same chip can cost about a thousand picojoules, far more than a multiply.
“moving data from the sram on the other side of the chip not even off the off chip but on the other side of the same chip can be you know a thousand picajoules”
Jeff Dean describes speculative decoding as predicting several tokens ahead, which raises the effective batch size and amortizes the cost of loading weights.
“You predict eight tokens out, and that enables you to sort of increase the effective batch size of what you're doing by a factor of eight”
Jeff Dean says a TPU chip design can take about two years before it reaches a data center.
“that design might take two years before it even lands in a data center.”
Jeff Dean says he named Gemini for the twin teams merging, and also for NASA's Gemini program preceding Apollo.
“there's also the NASA interpretation of the early Gemini project being an important thing on your way to the Apollo project”
Jeff Dean says that before Gemini, Google Research and the Brain team had separate large language model efforts, alongside multimodal work and DeepMind's own models.
“at the time, we had efforts within Google Research and in the brain team in particular on large language models.”
Jeff Dean says he met Andrew Ng in a Google micro kitchen when Ng had just started consulting, which led to scaling up neural nets at Google.
“I actually bumped into Andrew Ng, who's a Stanford faculty member, and I knew him from, I'd given talks at Stanford a couple years before”
Jeff Dean wrote an undergraduate thesis on parallel neural network training in 1990.
“I did an undergrad thesis on neural network training, parallel neural network training, back in 1990 when I got exposed to neural nets.”
Jeff Dean says an early Google paper on sparsely activated models was titled 'Outrageously Large Neural Networks'.
“we did an early paper on this where we really scaled up, you know, outrageously large neural networks.”
Jeff Dean says Transformers gave a 10x to 100x improvement in compute cost to reach a given quality level compared with LSTMs.
“Transformers similarly gave you a 10x to 100x improvement in compute cost to a given quality level versus, say, LSTMs at the time.”
Jeff Dean says Gemini turned a YouTube highlight video of 18 sports moments into a table of events, dates and descriptions.
“it was like a YouTube highlight video of 18 memorable sports moments across the last 20 years or something.”
Jeff Dean says Google's math Olympiad result in the latest year came from a single unified model close to the production model rather than specialized systems.
“switching to a single unified model that is roughly the production model with a little bit more inference budget”
Jeff Dean says models were struggling with simple GSM8K word problems about two years earlier, before reaching Olympiad-level math.
“two years ago we were struggling with gsm 8k problems right like you know fred has two rabbits he gets three more rabbits”
Jeff Dean says he believes general models will win out over specialized ones in most cases.
“I mean, I think general models will win out over specialized ones in most cases.”
Jeff Dean describes DeViSE as an early model that fused a language model with an image model so it could label unseen categories such as microscope.
“I did some work on an early model that fused together a language-based model where you have nice word-based representations and then an image model”
Jeff Dean says Kalamong, an example low-resource language, is spoken by about 120 people and has no written text.
“which is truly low resource because it's only spoken by, I think, 120 people in the world and there's no written text.”