OpenNashPodcasts

Person

Jeremy Howard

This page collects what Jeremy Howard said on the Latent Space: The AI Engineer Podcast, in an episode called "The End of Finetuning," covering his work at Fast.ai, the ULMFiT language model method, AI competitions, and large language models.

Howard says Fast.ai began around 2016, when making deep learning accessible was widely considered stupid. He says his first ULMFiT attempt set a new state-of-the-art result on the IMDb dataset within hours. He also discusses Meta's Code Llama, Chris Lattner's Swift for TensorFlow, and the Mojo language from Lattner's company, Modular. Each fact below links to the moment it was said.

1 episodes · 20 quoted claims

Optimal Decisions developed an approach to insurance pricing called profit-optimized insurance pricing.

“Optimal Decisions invented a new approach to insurance pricing, something called profit-optimized insurance pricing.”

Latent Space: The AI Engineer Podcast · 0:19:31

When Fast.ai started around 2016, Jeremy Howard says the idea of making deep learning accessible was widely considered stupid.

“the idea that deep learning was something that you could make more accessible was generally considered stupid.”

Latent Space: The AI Engineer Podcast · 0:03:39

ULMFiT used three steps: train a language model on a large corpus, fine-tune on a curated corpus, then fine-tune on a task.

“So step one was train a language model on a big corpus step two was fine-tune a language model on a more curated corpus and step three was further fine-tune that model on a task”

Latent Space: The AI Engineer Podcast · 0:11:40

Jeremy Howard says his first ULMFiT attempt produced a new state-of-the-art result on the IMDb dataset within hours.

“the first time i tried it within hours i had a new state-of-the-art academic result on imdb”

Latent Space: The AI Engineer Podcast · 0:11:40

Jeremy Howard says Alec Radford told Kate Betts he had doubted general pre-training would work until he read ULMFiT, and then built GPT.

“he said, well, I was pretty sure that pre-training on a general large corpus wouldn't work. So I hadn't tried it. And then I read ULM fit and turns out it did work.”

Latent Space: The AI Engineer Podcast · 0:13:12

Fast.ai learned of DAWNBench about 10 days before it ended and used progressive resizing among other ideas, eventually winning.

“we basically got together an emergency bunch of our students and Rachel and I and sat for the next 10 days”

Latent Space: The AI Engineer Podcast · 0:28:04

The Kaggle competition Howard describes required models to run within nine hours on 14 GB of RAM, two CPUs and an old GPU.

“your model has to run on Kaggle within nine hours. And Kaggle's very, very limited. So you've only got 14 gig RAM, only two CPUs and a small, very old GPU.”

Latent Space: The AI Engineer Podcast · 0:29:25

Sylvain Gugger created Fast Progress, a progress bar library similar to tqdm, while working with Fast.ai.

“Sylvain Guzher, when he worked with us, created called Fast Progress, which is kind of like TQEDM, but we think a lot better.”

Latent Space: The AI Engineer Podcast · 0:29:58

Jeremy Howard says Meta's Code Llama was a roughly 500 billion token fine-tuning of Llama 2 on code and related prose.

“CodeLama is like 500 billion token fine tuning of Lama 2 using code and also pros about code that Meta did.”

Latent Space: The AI Engineer Podcast · 0:35:11

Howard says only about 0.3% of Code Llama's training was non-code data, which he believes caused it to forget other abilities.

“Only like 0.3 % of their epochs were non-code data. So it did. It forgot everything else.”

Latent Space: The AI Engineer Podcast · 0:35:36

In his Hackers Guide to LLMs talk, Howard showed a base model learning to generate SQL after 15 minutes of training on one GPU.

“taking a pre-trained base model and getting it to generate SQL. And it took 15 minutes to train on a single GPU.”

Latent Space: The AI Engineer Podcast · 0:53:26

Howard calls BTLM-3B an underappreciated model that he describes as having 7B-quality performance at 3B size.

“A really underappreciated one is BTLM 3B, which is a kind of 7B quality 3B model.”

Latent Space: The AI Engineer Podcast · 0:53:51

Chris Lattner told Howard about Swift for TensorFlow, a compiler-based approach with built-in automatic differentiation.

“he was like, yeah, so we're going to create this new thing called Swift for TensorFlow.”

Latent Space: The AI Engineer Podcast · 0:46:16

Howard says Google's JAX began as a research project rather than a strategy and later became central to Google's AI future.

“google now does have a backup plan you know they have jacks which was never a strategy”

Latent Space: The AI Engineer Podcast · 0:49:59

Chris Lattner called Howard years later to say he had started a company, Modular, which created the Mojo language.

“he called me up and he's like jeremy i've taken your advice i've started a company”

Latent Space: The AI Engineer Podcast · 0:49:23

Howard says his 2014 TED Talk predicted computers might be better than average humans at most tasks in about five years.

“In my 2014 TED Talk, I had a graph and I said, like, this is like the slope of human capabilities”

Latent Space: The AI Engineer Podcast · 1:02:58