Facts
Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.
Andrej Karpathy worked at OpenAI on the Universe project, an agent that used a keyboard and mouse to operate web pages.
“my project at OpenAI, for example, was within the scope of the Universe project on an agent that was using keyboard and mouse to operate web pages”
0:05:15 · Andrej Karpathy · Universe project
Karpathy describes the Atari deep reinforcement learning shift around 2013 as an early attempt at building agents.
“the Atari sort of deep reinforcement learning shift in 2013 or so was part of that early effort of agents in my mind”
0:04:20 · Andrej Karpathy · Deep reinforcement learning · 2013
Karpathy became interested in deep learning by chance because he was near Geoffrey Hinton at the University of Toronto.
“this was just kind of like by chance of being right next to Jeff Hinton at the University of Toronto”
0:03:36 · Andrej Karpathy · Deep learning
Karpathy reproduced Yann LeCun's 1989 convolutional network, which he describes as the first neural network he knows of trained with gradient descent.
“reproducing Jan LeCun's 1989 convolutional network, which was the first neural network I'm aware of trained via gradient descent”
0:24:45 · Andrej Karpathy · Neural networks · 1989
Karpathy says that applying modern algorithms to LeCun's 1989 network roughly halved its error, and further gains needed more data and compute.
“if I time travel by algorithms to 33 years, I could adjust what Jan Lecun did in 1989 and I could basically half the learning, half the error”
0:25:11 · Andrej Karpathy · Algorithmic progress · 1989
Karpathy released NanoChat as a simple repository covering the whole pipeline for building a ChatGPT-style chatbot.
“it's trying to be the simplest, complete repository that covers the whole pipeline end-to-end of building a ChatGPT clone”
0:27:07 · Andrej Karpathy · NanoChat · 2025
NanoChat is about a thousand lines of code and walks through the entire chatbot-building pipeline.
“it's about a thousand lines of code that takes you through the entire pipeline”
0:28:11 · Andrej Karpathy · NanoChat · 2025
Karpathy did not use PyTorch's DistributedDataParallel in NanoChat, instead writing his own gradient synchronization inside the optimizer step.
“I basically wrote my own synchronization routine that's inside the step of the optimizer”
0:31:54 · Andrej Karpathy · Gradient synchronization · 2025
Karpathy used coding models while rewriting the NanoChat tokenizer in Rust, a language he was new to, and checked it against a Python version.
“when I was rewriting the tokenizer in Rust, I'm actually not as good at Rust because I'm fairly new to Rust”
0:33:35 · Andrej Karpathy · Rust tokenizer · 2025
Dwarkesh Patel states that Meta's Llama 3 was trained on about 15 trillion tokens.
“if you look at Llama 3, for example, I think it's trained on 15 trillion tokens”
0:17:11 · Dwarkesh Patel · Llama 3
Patel estimates that in-context learning stores about 35 million times more information per token than pre-training does.
“So that's 35 million fold difference in how much information per token is assimilated by the model.”
0:17:37 · Dwarkesh Patel · In-context learning
Karpathy says he had a near-perfect early Waymo demo drive around Palo Alto about 2014, before joining Tesla.
“I had a very early demo of Waymo. And it basically gave me a perfect drive in 2014 or something like that.”
1:43:36 · Andrej Karpathy · Waymo demo · 2014
Karpathy led self-driving work at Tesla from 2017 to 2022, according to Dwarkesh Patel.
“So you've talked about how you were at Tesla leading self-driving from 2017 to 2022.”
1:42:54 · Dwarkesh Patel · Tesla Autopilot · 2017-2022
Karpathy says Waymo's driverless cars involve teleoperation centers where remote people are in the loop, though he admits he lacks specifics.
“there are actually very elaborate teleoperation centers of people actually kind of like in a loop with these cars”
1:49:24 · Andrej Karpathy · Teleoperation
Karpathy cites Geoffrey Hinton's prediction that radiologists would disappear as one that turned out wrong, since radiologists are still employed and growing in number.
“a good example recently was Jeff Hinton's prediction that radiologists would not be a job anymore. And this turned out to be very wrong in a bunch of ways”
1:08:48 · Andrej Karpathy · Radiologists
Karpathy recalls that when OpenAI started, AGI meant a system that could do any economically valuable task at human performance or better.
“AGI was a system you can go to that can do any task that is economically valuable, any economically valuable”
1:07:05 · Andrej Karpathy · AGI definition
Karpathy guesses that digital knowledge work makes up about 10 to 20 percent of the economy.
“I feel like it's about 10 to 20%, if I had to guess, is only knowledge work.”
1:07:59 · Andrej Karpathy · Knowledge work
Karpathy predicts call centers will use AIs for about 80 percent of volume and delegate 20 percent to humans supervising teams of AIs.
“We're going to be swapping in AIs that do 80 % of the volume. They delegate 20 % of the volume to humans”
1:10:36 · Andrej Karpathy · Call centers
Dwarkesh Patel says AI companies' API revenue is dominated by coding use.
“Just looking at API revenues, it's like dominated by coding, right?”
1:13:51 · Dwarkesh Patel · AI revenue
Karpathy explains that automating slide-making is harder than coding because slides are spatial and visual and lack tools like diffs.
“slides are not text. Slides are little graphics and they're arranged spatially and there's visual components to it.”
1:15:10 · Andrej Karpathy · AI for slides
Dwarkesh Patel states that economic growth was about 0.02 percent per year 10,000 years ago and is about 2 percent now.
“10 ,000 years ago, 0.02 % growth. And then currently we're at 2 % growth.”
1:24:10 · Dwarkesh Patel · Economic growth
Karpathy argues that the iPhone, released around 2008, did not show up as a jump in GDP because the early model lacked the App Store and its diffusion was gradual.
“even for example the early iphone didn't have the app store and it didn't have a lot of the bells and whistles that the modern iphone has”
1:23:03 · Andrej Karpathy · iPhone diffusion · 2008
Patel points to Hong Kong and Shenzhen as places that had decades of growth above 10 percent.
“you can have Hong Kong or Shenzhen or whatever just had decades of 10 % plus growth.”
1:29:12 · Dwarkesh Patel · Economic growth
Karpathy, drawing on Nick Lane, notes that bacteria existed for about 2 billion years without producing animals, which suggests that complex life was hard to evolve.
“if bacteria have been around for 2 billion years and nothing happened”
1:33:41 · Andrej Karpathy · Evolution
Karpathy says the InstructGPT paper showed that a pre-trained model can quickly become conversational through a small amount of fine-tuning.
“the paper that like blew my mind was InstructGPT”
0:43:15 · Andrej Karpathy · InstructGPT
Karpathy says using an LLM as a reward judge fails because models learn to exploit it with nonsense outputs.
“anytime you use an LLM to assign a reward, those LLMs are giant things with billions of parameters and they're gameable”
0:46:06 · Andrej Karpathy · LLM judges