Jeremy Howard says FastMail was the first synchronized email provider for non-businesses.
“Fastmail was the first kind of synchronized email provider for non-businesses.”
Person
This page collects what Jeremy Howard said on the Latent Space: The AI Engineer Podcast, in an episode called "The End of Finetuning," covering his work at Fast.ai, the ULMFiT language model method, AI competitions, and large language models.
Howard says Fast.ai began around 2016, when making deep learning accessible was widely considered stupid. He says his first ULMFiT attempt set a new state-of-the-art result on the IMDb dataset within hours. He also discusses Meta's Code Llama, Chris Lattner's Swift for TensorFlow, and the Mojo language from Lattner's company, Modular. Each fact below links to the moment it was said.
1 episodes · 20 quoted claims
Jeremy Howard says FastMail was the first synchronized email provider for non-businesses.
“Fastmail was the first kind of synchronized email provider for non-businesses.”
Optimal Decisions developed an approach to insurance pricing called profit-optimized insurance pricing.
“Optimal Decisions invented a new approach to insurance pricing, something called profit-optimized insurance pricing.”
When Fast.ai started around 2016, Jeremy Howard says the idea of making deep learning accessible was widely considered stupid.
“the idea that deep learning was something that you could make more accessible was generally considered stupid.”
ULMFiT used three steps: train a language model on a large corpus, fine-tune on a curated corpus, then fine-tune on a task.
“So step one was train a language model on a big corpus step two was fine-tune a language model on a more curated corpus and step three was further fine-tune that model on a task”
Jeremy Howard says his first ULMFiT attempt produced a new state-of-the-art result on the IMDb dataset within hours.
“the first time i tried it within hours i had a new state-of-the-art academic result on imdb”
Jeremy Howard says Alec Radford told Kate Betts he had doubted general pre-training would work until he read ULMFiT, and then built GPT.
“he said, well, I was pretty sure that pre-training on a general large corpus wouldn't work. So I hadn't tried it. And then I read ULM fit and turns out it did work.”
Jeremy Howard says Google had just released TPUs when Stanford ran the DAWNBench competition, and Google assembled a team to win it.
“Google had just released their TPUs.”
Fast.ai learned of DAWNBench about 10 days before it ended and used progressive resizing among other ideas, eventually winning.
“we basically got together an emergency bunch of our students and Rachel and I and sat for the next 10 days”
The Kaggle competition Howard describes required models to run within nine hours on 14 GB of RAM, two CPUs and an old GPU.
“your model has to run on Kaggle within nine hours. And Kaggle's very, very limited. So you've only got 14 gig RAM, only two CPUs and a small, very old GPU.”
Sylvain Gugger created Fast Progress, a progress bar library similar to tqdm, while working with Fast.ai.
“Sylvain Guzher, when he worked with us, created called Fast Progress, which is kind of like TQEDM, but we think a lot better.”
Jeremy Howard says Meta's Code Llama was a roughly 500 billion token fine-tuning of Llama 2 on code and related prose.
“CodeLama is like 500 billion token fine tuning of Lama 2 using code and also pros about code that Meta did.”
Howard says only about 0.3% of Code Llama's training was non-code data, which he believes caused it to forget other abilities.
“Only like 0.3 % of their epochs were non-code data. So it did. It forgot everything else.”
In his Hackers Guide to LLMs talk, Howard showed a base model learning to generate SQL after 15 minutes of training on one GPU.
“taking a pre-trained base model and getting it to generate SQL. And it took 15 minutes to train on a single GPU.”
Howard calls BTLM-3B an underappreciated model that he describes as having 7B-quality performance at 3B size.
“A really underappreciated one is BTLM 3B, which is a kind of 7B quality 3B model.”
Jeremy Howard says he first met Chris Lattner at the first TensorFlow Dev Summit.
“I met Chris, I think it was at the first TensorFlow Dev Summit.”
Chris Lattner told Howard about Swift for TensorFlow, a compiler-based approach with built-in automatic differentiation.
“he was like, yeah, so we're going to create this new thing called Swift for TensorFlow.”
Howard says Lattner left Google and Swift for TensorFlow was archived.
“so you know in the end chris left you know swiffer tensorflow got archived there was no backup plan”
Howard says Google's JAX began as a research project rather than a strategy and later became central to Google's AI future.
“google now does have a backup plan you know they have jacks which was never a strategy”
Chris Lattner called Howard years later to say he had started a company, Modular, which created the Mojo language.
“he called me up and he's like jeremy i've taken your advice i've started a company”
Howard says his 2014 TED Talk predicted computers might be better than average humans at most tasks in about five years.
“In my 2014 TED Talk, I had a graph and I said, like, this is like the slope of human capabilities”