Person
Andrej Karpathy
Andrej Karpathy is a computer scientist who appears on the Dwarkesh Podcast episode "AGI is still a decade away," where he discusses his work in AI, self-driving cars, and how artificial intelligence may change the economy. He describes working on early AI agents at OpenAI and reproducing Yann LeCun's 1989 neural network.
He also talks about releasing NanoChat in 2025, a repository of about a thousand lines of code that covers the whole process of building a ChatGPT-style chatbot. Karpathy says Geoffrey Hinton's prediction that radiologists would disappear turned out wrong. He also predicts call centers will use AIs for about 80 percent of volume. Each fact below links to the moment it was said.
1 episodes · 20 quoted claims
Andrej Karpathy worked at OpenAI on the Universe project, an agent that used a keyboard and mouse to operate web pages.
“my project at OpenAI, for example, was within the scope of the Universe project on an agent that was using keyboard and mouse to operate web pages”
Dwarkesh Podcast · 0:05:15
Karpathy describes the Atari deep reinforcement learning shift around 2013 as an early attempt at building agents.
“the Atari sort of deep reinforcement learning shift in 2013 or so was part of that early effort of agents in my mind”
Dwarkesh Podcast · 0:04:20
Karpathy became interested in deep learning by chance because he was near Geoffrey Hinton at the University of Toronto.
“this was just kind of like by chance of being right next to Jeff Hinton at the University of Toronto”
Dwarkesh Podcast · 0:03:36
Karpathy reproduced Yann LeCun's 1989 convolutional network, which he describes as the first neural network he knows of trained with gradient descent.
“reproducing Jan LeCun's 1989 convolutional network, which was the first neural network I'm aware of trained via gradient descent”
Dwarkesh Podcast · 0:24:45
Karpathy says that applying modern algorithms to LeCun's 1989 network roughly halved its error, and further gains needed more data and compute.
“if I time travel by algorithms to 33 years, I could adjust what Jan Lecun did in 1989 and I could basically half the learning, half the error”
Dwarkesh Podcast · 0:25:11
Karpathy released NanoChat as a simple repository covering the whole pipeline for building a ChatGPT-style chatbot.
“it's trying to be the simplest, complete repository that covers the whole pipeline end-to-end of building a ChatGPT clone”
Dwarkesh Podcast · 0:27:07
NanoChat is about a thousand lines of code and walks through the entire chatbot-building pipeline.
“it's about a thousand lines of code that takes you through the entire pipeline”
Dwarkesh Podcast · 0:28:11
Karpathy did not use PyTorch's DistributedDataParallel in NanoChat, instead writing his own gradient synchronization inside the optimizer step.
“I basically wrote my own synchronization routine that's inside the step of the optimizer”
Dwarkesh Podcast · 0:31:54
Karpathy used coding models while rewriting the NanoChat tokenizer in Rust, a language he was new to, and checked it against a Python version.
“when I was rewriting the tokenizer in Rust, I'm actually not as good at Rust because I'm fairly new to Rust”
Dwarkesh Podcast · 0:33:35
Karpathy says he had a near-perfect early Waymo demo drive around Palo Alto about 2014, before joining Tesla.
“I had a very early demo of Waymo. And it basically gave me a perfect drive in 2014 or something like that.”
Dwarkesh Podcast · 1:43:36
Karpathy says Waymo's driverless cars involve teleoperation centers where remote people are in the loop, though he admits he lacks specifics.
“there are actually very elaborate teleoperation centers of people actually kind of like in a loop with these cars”
Dwarkesh Podcast · 1:49:24
Karpathy cites Geoffrey Hinton's prediction that radiologists would disappear as one that turned out wrong, since radiologists are still employed and growing in number.
“a good example recently was Jeff Hinton's prediction that radiologists would not be a job anymore. And this turned out to be very wrong in a bunch of ways”
Dwarkesh Podcast · 1:08:48
Karpathy recalls that when OpenAI started, AGI meant a system that could do any economically valuable task at human performance or better.
“AGI was a system you can go to that can do any task that is economically valuable, any economically valuable”
Dwarkesh Podcast · 1:07:05
Karpathy guesses that digital knowledge work makes up about 10 to 20 percent of the economy.
“I feel like it's about 10 to 20%, if I had to guess, is only knowledge work.”
Dwarkesh Podcast · 1:07:59
Karpathy predicts call centers will use AIs for about 80 percent of volume and delegate 20 percent to humans supervising teams of AIs.
“We're going to be swapping in AIs that do 80 % of the volume. They delegate 20 % of the volume to humans”
Dwarkesh Podcast · 1:10:36
Karpathy explains that automating slide-making is harder than coding because slides are spatial and visual and lack tools like diffs.
“slides are not text. Slides are little graphics and they're arranged spatially and there's visual components to it.”
Dwarkesh Podcast · 1:15:10
Karpathy argues that the iPhone, released around 2008, did not show up as a jump in GDP because the early model lacked the App Store and its diffusion was gradual.
“even for example the early iphone didn't have the app store and it didn't have a lot of the bells and whistles that the modern iphone has”
Dwarkesh Podcast · 1:23:03
Karpathy, drawing on Nick Lane, notes that bacteria existed for about 2 billion years without producing animals, which suggests that complex life was hard to evolve.
“if bacteria have been around for 2 billion years and nothing happened”
Dwarkesh Podcast · 1:33:41
Karpathy says the InstructGPT paper showed that a pre-trained model can quickly become conversational through a small amount of fine-tuning.
“the paper that like blew my mind was InstructGPT”
Dwarkesh Podcast · 0:43:15
Karpathy says using an LLM as a reward judge fails because models learn to exploit it with nonsense outputs.
“anytime you use an LLM to assign a reward, those LLMs are giant things with billions of parameters and they're gameable”
Dwarkesh Podcast · 0:46:06