Facts
Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.
Eliezer Yudkowsky published an article in Time magazine calling for a moratorium on further AI training runs shortly before this interview was recorded.
“So yesterday when we're recording this, you had an article in Time calling for a moratorium on further AI training runs.”
0:00:51 · Dwarkesh Patel · Time article
Eliezer Yudkowsky says his parents tried to raise him Orthodox Jewish and that it did not take, and that he took his ethos from the science fiction books in their library instead.
“my parents tried to raise me Orthodox Jewish, and that did not take at all.”
0:12:44 · Eliezer Yudkowsky · Upbringing
Yudkowsky notes that OpenAI has not disclosed the architecture of GPT-4, so outsiders cannot be sure it was built simply by stacking more transformer layers.
“OpenAI has very correctly declined to tell us what exactly goes on in there in terms of its architecture.”
0:37:50 · Eliezer Yudkowsky · GPT-4 architecture
Yudkowsky says the Machine Intelligence Research Institute ran a Visible Thoughts Project that aimed to build a dataset encouraging language models to think out loud, and that it lacked funding and staff.
“Muri has something called the Visible Thoughts Project, which is like probably like did not get enough funding and enough personnel and was going too slowly”
0:51:46 · Eliezer Yudkowsky · Visible Thoughts
Yudkowsky created a Manifold prediction market asking whether, by 2026, anyone will understand something inside a large language model that would have been unfamiliar to AI scientists in 2006.
“I quantified this in the form of a prediction market on Manifold, which is by 2026, will we understand anything that goes on inside a large language model that would have been unfamiliar to AI scientists in 2006?”
1:00:43 · Eliezer Yudkowsky · Interpretability bet · 2026
Yudkowsky and Paul Christiano found one concrete disagreement in predictions: Christiano gave 8% and Yudkowsky 16% to an AI winning gold on International Mathematical Olympiad problems by 2025.
“what we ended up with was Paul's 8 % versus my 16 % for an AI getting gold on International Mathematics Olympics problem set by, I believe, 2025.”
1:46:14 · Eliezer Yudkowsky · Prediction bet · 2025
Yudkowsky says the Hanson–Yudkowsky 'Foom' debate on AI takeoff was won by Gwern Branwen, who sits further toward Yudkowsky's side than Yudkowsky did.
“I think that the Hansen-Yudkowsky-Foom debate was won by Warren Branwen.”
1:48:41 · Eliezer Yudkowsky · Foom debate
Yudkowsky says AI progress repeatedly surprised him by moving faster than expected in 2015 to 2017, and that it slowed somewhat around 2019 and 2020.
“I think that 2015, 16, 17 were the years at which I'd noticed I'd been repeatedly surprised by stuff moving faster than anticipated.”
2:10:23 · Eliezer Yudkowsky · AI timelines · 2015-2017
Yudkowsky says he wrote the LessWrong Sequences mainly to create people able to take over his work, because his health was poor and worsening.
“I tried really, really, really hard to create a new crop of people who could do all the stuff I could do to take over because I knew my health was not great and getting worse.”
2:04:13 · Eliezer Yudkowsky · LessWrong Sequences
Yudkowsky says Project Lawful is the one piece of his fiction in which a character literally gives lectures, after arriving on another planet and having to explain science to its people.
“There's only one piece of fiction of mine where there's literally a character giving lectures because he arrived on another planet and now has to lecture about science to them”
3:44:53 · Eliezer Yudkowsky · Project Lawful
George C. Williams wrote Adaptation and Natural Selection, a book that started what is called the Williams revolution by arguing that natural selection does not produce the nice properties biologists had expected.
“And the Williams revolution, as it's sometimes called, is when George Williams wrote Adaptation and Natural Selection, a very influential book saying like, that is not what this optimization criterion gives you.”
3:59:07 · Eliezer Yudkowsky · Evolutionary biology
Yudkowsky says about a quarter of Nobel laureates were students or grand-students of other laureates, which he takes as a sign that science is passed on through apprenticeship rather than written down.
“We have like half of the, what was it, a quarter of the Nobel laureates being the students or grand students of other Nobel laureates because we never figured out how to teach science.”
4:00:48 · Eliezer Yudkowsky · Science training
Yudkowsky cites Nate Silver's argument that Donald Trump had to clear six stages, each less than 50% likely, to become the Republican nominee, as an example of why splitting an event into stages can mislead.
“Nate Silver arguing that Trump needed to get through six stages to become the Republican presidential candidate, each of which was less than half probability.”
3:00:08 · Eliezer Yudkowsky · Multiple-stage fallacy
Yudkowsky describes himself as the inventor of logical decision theory and says it shows how a smart AI could trick its creators with a values handshake.
“Speaking as the inventor of logical decision theory, if the rest of the human species had been keeping me locked in a box, and I have watched people fail at this problem.”
1:10:16 · Eliezer Yudkowsky · Decision theory
Yudkowsky says the Center for Applied Rationality tried to train people to be saner using neuroscience but failed, never even acquiring an fMRI machine, partly for lack of funding.
“This is the dream that the Center for Applied Rationality failed at. It's not easy. You know, they didn't even like get as far as buying an fMRI machine.”
1:40:04 · Eliezer Yudkowsky · Rationality training
Yudkowsky cites an interview in which Ilya Sutskever said that predicting the next token requires predicting the world that produces the text.
“call back to Ilya's recent interview that I retweeted where he points out that to predict the next token you need to predict the world that generates the token”
0:52:52 · Eliezer Yudkowsky · Next-token prediction
Yudkowsky uses text containing hashes followed by the strings they hash to show that predicting internet text does not stop being useful at human-level intelligence.
“somewhere on the internet is a list of hashes followed by the string hashed. This is a simple demonstration of how you can go on getting lower losses by throwing a hypercomputer at the problem.”
2:53:36 · Eliezer Yudkowsky · Text prediction
Yudkowsky describes a LessWrong post about 'pebble sorters', aliens who value correct heap sizes, to illustrate how an agent's preferences can shift as it gets smarter.
“describing a species of aliens who think that a heap of size 7 is correct and a heap of size 11 is correct but not 8 or 9 or 10.”
2:15:22 · Eliezer Yudkowsky · Pebble sorters
Scott Aaronson wrote in a recent blog post that accepting the practical orthogonality thesis would mean education and knowledge cannot be seen as tools for moral improvement.
“If you really accept the practical version of the orthogonality thesis, then it seems to me that you can't regard education, knowledge, and enlightenment as instruments for moral betterment.”
2:14:58 · Dwarkesh Patel · Orthogonality thesis
Yudkowsky says he has had a couple of conversations with Demis Hassabis and found him much easier to talk to than other AI lab leaders.
“I've had a couple of conversations with like Demis Asabas, who struck me as like much more the sort of person who is possible to have a conversation with.”
3:26:07 · Eliezer Yudkowsky · AI lab leaders
Yudkowsky says he began working on the AI safety problem in 2001 because he expected it to become an emergency later, and that almost nobody else started then.
“I mean, so like also like there's just like me starting to work on this problem in 2001 because it was like super predictable going to turn into an emergency later.”
3:36:14 · Eliezer Yudkowsky · AI safety history · 2001
Yudkowsky proposes $100 billion in prizes for AI interpretability research, saying it is a kind of alignment work whose results can be checked.
“Let's offer $100 billion in prizes for interpretability. Let's get all the hotshot physicist graduates kids going into that instead of wasting their lives on string theory or hedge funds.”
1:02:10 · Eliezer Yudkowsky · Interpretability prizes