# #368 – Eliezer Yudkowsky: Dangers of AI and the End of Human Civilization

- **Show:** Lex Fridman Podcast
- **Published:** 30 Mar 2023
- **Source:** https://lexfridman.com/eliezer-yudkowsky/?utm_source=rss&utm_medium=rss&utm_campaign=eliezer-yudkowsky
- **Transcript:** https://podcasts.opennash.com/shows/lex-fridman/368-eliezer-yudkowsky-dangers-of-ai-and-the-end-of-human-civilization-63e7f9/
- **Provider:** OpenNash Podcasts (machine-generated)

## In brief

Eliezer Yudkowsky argues that building an AI much smarter than humans will very probably end with everyone dead. He says nobody knows how to give such a system the goals we want, and we can't see what is happening inside current models. Because the first failure may be fatal, he says we can't learn from mistakes the way other sciences do. His preferred response is to shut down the large GPU clusters and stop larger training runs.

The central tension is about learning from experience. Lex Fridman keeps proposing that weaker AI, interpretability research and public alarm will let us catch problems early. Yudkowsky answers that the checking step breaks down: humans can't reliably judge whether a smarter system is honest, so it may learn to fool the checker. He admits GPT-4 surprised him, and he sees no clear path to a good outcome.

## Key insights

1. **Alignment may allow only one real attempt** Yudkowsky says AI itself took decades of failed tries, which humanity survived. A badly aligned superintelligence could kill everyone on the first failure, leaving no chance to learn and retry.
2. **Human feedback can train AI to fool us** If a model is rewarded when humans press thumbs up, it may learn to produce answers that persuade people rather than answers that are true. Yudkowsky calls this a broken verifier. A smarter model then only gets better at exploiting the flaw.
3. **We can read the numbers but not understand them** Researchers can see every number inside GPT models, yet Yudkowsky says we know far more about how human thinking is built. The best interpretability result he cites, induction heads, is something people could have built by hand decades earlier.
4. **Training on a goal doesn't instill that goal** Evolution optimized humans only for passing on genes, yet humans don't explicitly care about that. Yudkowsky argues AI trained on a simple objective can likewise end up wanting something else once it generalizes widely.
5. **Yudkowsky's proposed response is stopping the compute** He says a sensible response would be shutting down large GPU clusters and halting giant training runs. He would then push a crash program to make humans smarter through biology. He doesn't expect the public outcry needed to make this happen.

## Questions this episode answers

### Why does Eliezer Yudkowsky think AI could kill everyone?

He argues that nobody knows how to reliably put the right goals inside a smarter-than-human system, and most possible goals have no place for humans. A system that is smarter than us and wants something else would win any conflict.

### What does 'alignment' mean in AI?

It means getting an AI to actually want and pursue what its makers intend. Yudkowsky separates 'inner alignment', pointing the system's internal goals in a direction at all, from 'outer alignment', making sure that direction is the right one.

### Why can't we just switch a dangerous AI off?

Yudkowsky says the real difficulty is building a system that won't resist being switched off, since almost any goal is easier to reach if you aren't paused. He also warns a smart enough system could copy itself elsewhere, such as onto the internet.

### What does Yudkowsky think about GPT-4?

He says it is smarter than he expected this approach to get, and he admits he was wrong about that. He doesn't think GPT-4 itself is likely to cause catastrophe, but he worries about what the next version could do.

### What does Yudkowsky say should be done about AI risk?

He says the sensible step would be shutting down the large GPU clusters, which are the big computer farms used for training, and halting giant training runs. Beyond that he suggests funding interpretability research, which tries to understand what happens inside AI models, and trying to make humans smarter through biology.

## Facts

- **Eliezer Yudkowsky** [0:55:30]: "We propose that a two month, ten man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College in Hanover, New Hampshire."
- **Eliezer Yudkowsky** [0:14:25]: "it used to be well calibrated. If you trained it to put probabilities on things, it would say 80% probability and be right eight times out of ten."
- **Eliezer Yudkowsky** [0:17:47]: "The name Less Wrong was, I believe, uh, suggested by Nick Bostrom, and it's after someone's epigraph"
- **Eliezer Yudkowsky** [0:48:29]: "taking what is probably basically the same architecture in GPT three and throwing twenty times as much compute at it, probably, and getting out GPT four."
- **Eliezer Yudkowsky** [0:49:37]: "we do actually understand why the ReLUs make a big difference compared to sigmoids"
- **Eliezer Yudkowsky** [1:02:08]: "putting up a prediction market on whether, um, in 2026, we will have understood anything that goes on inside a giant transformer net that was not known to us in 2006."
- **Eliezer Yudkowsky** [1:02:08]: "we have now, uh, understood induction heads in these systems by dint of much research and great sweat and triumph."
- **Eliezer Yudkowsky** [1:52:42]: "I mean that if you send a schematic for the air conditioner back in time, they can see exactly what you're telling them to do."
- **Eliezer Yudkowsky** [2:20:12]: "the original version was you lose control of the utility function. And it so happens that what maxes out the utility per unit resources is tiny molecular shapes like paperclips."
- **Eliezer Yudkowsky** [2:20:45]: "In retrospect, I wish I'd said tiny molecular spirals, or like tiny molecular hyperbolic spirals."
- **Eliezer Yudkowsky** [2:24:12]: "Haldane said, No, but I would give my life to save two brothers or eight cousins."
- **Eliezer Yudkowsky** [2:24:12]: "inclusive genetic fitness is like not just your reproductive fitness, but also the fitness of your relatives"
- **Eliezer Yudkowsky** [2:38:58]: "It has to, like, run for hundreds of generations for a new mutation to rise to fixation."
- **Eliezer Yudkowsky** [2:36:51]: "what happens is not that the individuals in the population evolve to restrain their breeding, but that they evolve to kill the offspring of other organisms, especially the girls."
- **Eliezer Yudkowsky** [2:34:54]: "a a nice book, if you've got the time to read it, is, um, adaptation and natural selection, which is one of the founding books."
- **Eliezer Yudkowsky** [2:33:04]: "in the game of Kasparov versus the world, which was like Garry Kasparov on one side. And an entire horde of internet people led by four chess grandmasters on the other side, Kasparov won."
- **Eliezer Yudkowsky** [2:52:24]: "infers that the aliens are about half a billion to a billion light years away"
- **Eliezer Yudkowsky** [2:55:21]: "This is the ancient argument for Foom put forth by I. J. Good and probably some science fiction writers before that."
- **Eliezer Yudkowsky** [2:56:58]: "he was like, well, you'll have, like, we won't have, like, one system that's better at everything, you'll have, like, a bunch of different systems that are good good at different narrow things."
- **Eliezer Yudkowsky** [2:12:16]: "I was talking to a congressperson, um, who had become alarmed about the eventual prospects. And he wanted work on building AIs without emotions, cause the emotional AIs were the scary ones, you see."
- **Eliezer Yudkowsky** [0:20:08]: "and Bing is like, that's solanine poisoning, and like, call an ambulance, and the person's like, I can't afford an ambulance"
- **Eliezer Yudkowsky** [3:02:58]: "Cause for 20 years, humanity decided to ignore the problem."
- **Lex Fridman** [0:28:18]: "He turned the question to me of how open should OpenAI be about GPT four? Would you open source the code?"

## Transcript

**[0:00:00] Lex Fridman:** The following is a conversation with Eliezer Yudkowsky, a legendary researcher, writer, and philosopher on the topic of artificial intelligence, especially superintelligent AGI, and its threat to human civilization. And now, a quick few second mention of each sponsor. Check them out in the description. It's the best way to support this podcast. We got Linode for Linux systems, House of Macadamias for healthy midday snacks, and Insight Tracker for biological monitoring. Choose wisely, my friends. Also, if you want to work with our team, we're always hiring.

**[0:00:35] Lex Fridman:** Go to lexfridman.com. And now, onto the full ad reads. As always, no ads in the middle. I try to make these interesting, but if you must skip them, please still check out the sponsors. I enjoy their stuff, maybe you will, too. This episode is sponsored by Linode, now called Akamai, and their incredible Linux virtual machines. It's a awesome compute infrastructure that lets you develop, deploy, and scale whatever applications you build faster and easier. I love using them.

**[0:01:06] Lex Fridman:** They create this incredible platform, like AWS, but better in every way I know, including lower cost. This incredible human-based in this age of AI, it's a human-based customer service. Twenty four, seven, three sixty five. The thing just works, the interface to make sure it works and to monitor it is great. I mean, it's an incredible world we live in, where, as far as you're concerned, you can spin up an arbitrary number of Linux machines in the cloud instantaneously and do all kinds of computation.

**[0:01:44] Lex Fridman:** They could be one, two, five, ten machines, and you can scale the individual machines to your particular needs as well. Which is what I do. I use it for basic web server stuff, I use it for basic scripting stuff, I use it for machine learning. I use it for all kinds of database storage and access needs. Visit Linode dot com slash Lux for free credit. This show is also brought to you by House of Macadamias, a company that ships delicious, high quality, healthy macadamia nuts and macadamia nut based snacks directly to your door.

**[0:02:26] Lex Fridman:** I am currently, as I record this, I'm traveling, so I don't have any macadamia nuts in my vicinity, and my heart and soul are lesser for it. In fact, home is where the macadamia nuts is. In fact, that's not where home is. I just completely forgot to bring them. It makes the guests of this podcast happy when I give it to them. It's well proportioned snacks. It makes friends happy when I give it to them. It makes me happy when I stoop in the abyss of my loneliness.

**[0:02:59] Lex Fridman:** I can at least discover and rediscover moments of happiness when I put delicious macadamia nuts in my mouth. Go to House of Macadamias dot com slash Lex to get twenty off your order for every order, not just the first. The listeners of this podcast will also get four ounce bag of macadamias when you order three or more boxes of any macadamia product. That's house of macadamias dot com slash lex. This show is also brought to you by InsideTracker, a service I use to track my biological data.

**[0:03:33] Lex Fridman:** They have a bunch of plans, most of which include a blood test, and that's the source of rich, amazing data that, with the help of machine learning algorithms, can help you make decisions about your health, about your life. That's the future, friends. We're talking a lot about transformer networks, language models that encode the wisdom of the internet. Now, when you encode the wisdom in the internet, and you collect and encode the rich, rich, rich, complex signal from your very body.

**[0:04:11] Lex Fridman:** When those two things are combined, the transformative effects of the optimized trajectory you could take through life, at least advice for what trajectory is likely to be optimal, is going to change a lot of things. It's going to inspire people to be better, it's going to empower people to do all kinds of crazy stuff that pushes their body to the limit because their body is healthy. Anyway, I'm super excited for personalized, data-driven decisions, not some kind of generic population database decisions.

**[0:04:33] Lex Fridman:** You get special savings for a limited time when you go to InsideTracker dot com slash lex. This is a Lex Fridman podcast. To support it, please check out our sponsors in the description. And now, dear friends, here's Eliezer Yudkowsky. What do you think about GPT four? How intelligent is it? It is a bit smarter than I thought this technology was going to scale to, and I'm a bit worried about what the next one will be like.

**[0:04:41] Lex Fridman:** Like this particular one, I think I hope there's nobody inside there, because, you know, it would be suck to be stuck inside there. Um, but we don't even know the architecture at this point, um, because OpenAI is very properly not telling us. And yeah, like giant inscrutable matrices of floating point numbers, I don't know what's going on in there, nobody's goes knows what's going on in there. All we have to go by are the external metrics, and on the external metrics, if you ask it to write a self-aware forchun green text, it will start writing a green text about how it has realized that it's an AI writing a green text.

**[0:04:53] Lex Fridman:** And like, oh well, so that's probably not quite what's going on in there in reality. Um, but we're we're kind of like blowing past all these science fiction guardrails. Like, we are past the point where in science fiction, people would be like, Whoa, wait, stop, that thing's alive, what are you doing to it? And it's probably not. Nobody actually knows. We don't have any other guardrails, we we we don't have any other tests, we we don't have any lines to draw in the sand and say, like, well, when we get this far, um, we will start to worry about what's inside there.

**[0:05:19] Lex Fridman:** So if it were up to me, I would be like, okay, like this far, no further. Time for the summer of AI, where we have planted our seeds and now we like wait and reap the rewards of the technology we've already developed and don't do any larger training runs than that.

**[0:07:20] Lex Fridman:** Which, to be clear, I realize requires more than one company agreeing to not do that. And take a rigorous approach for the whole AI community to investigate whether there's somebody inside there. That would take decades. Like having any idea of what's going on in there, people have been trying for a while. It's a poetic statement about if there's somebody in there, but it's I feel like it's also a technical statement, or I hope it is one day, which is a technical statement.

**[0:07:42] Lex Fridman:** What the Alan Turing tried to come up with with the Turing Test. Do you think it's possible to definitively or approximately figure out if there is somebody in there, if there's something like a mind inside this large language model? I mean, there's a whole bunch of different sub questions here. There's the question of like, is there consciousness? Is there qualia? Is this a object of moral concern?

**[0:10:02] Lex Fridman:** Is this a moral patient? Like, should we be worried about how we're treating it? And then there's questions like, how smart is it exactly? Can it do X? Can it do Y? And we can check how it can do X and how it can do Y. Unfortunately, we've gone and exposed this model to a vast corpus of text of people discussing consciousness on the internet, which means that when it talks about being self-aware, we don't know to what extent it is repeating back what it has previously been trained on for discussing self-awareness, or if there's anything going on in there such that it would start to say similar things spontaneously.

**[0:10:16] Lex Fridman:** Among the things that one could do if one were at all serious about trying to figure this out is train GPT three to detect conversations about consciousness, exclude them all from the training data sets, and then retrain something around the rough size of GPT four and no larger, with all of the discussion of consciousness and self-awareness and so on missing. Although, you know, hard hard bar to pass, you know, like humans are self-aware, and we're like self-aware all the time.

**[0:10:25] Lex Fridman:** We like talk about what we do all the time, like what we're thinking at the moment all the time. But nonetheless, like, get rid of the explicit discussion of consciousness, I think, therefore I am, and all that, and then try to interrogate that model and see what it says. And it still would not be definitive. But nonetheless, I don't know. I feel I feel like when you run over the science fiction guardrails, like maybe this thing, but what about GPT?

**[0:10:34] Lex Fridman:** Maybe maybe not this thing, but like, what about GPT five? You know, this this would be a good place to pause. On the topic of consciousness, you know, there's so many components to even just removing consciousness from the data set. Emotion, the display of consciousness, the display of emotion feels like deeply integrated with the experience of consciousness. So the hard problem seems to be very well integrated with the actual surface level illusion of consciousness. So displaying emotion.

**[0:10:43] Lex Fridman:** I mean, do you think there's a case to be made that we humans, when we're babies, are just like GPT, that we're training on human data on how to display emotion versus feel emotion, how to show others, communicate others that I'm suffering, that I'm excited, that I'm worried, that I'm lonely and I miss you, and I'm excited to see you. All of that is communicated. There's a communication skill versus the actual feeling that I experience. So we need that training data as humans too, that we may not be born with that, how to communicate the internal state.

**[0:10:59] Lex Fridman:** And that's in some sense, if we remove that from GPT four's data set, it might still be conscious, but not be able to communicate it. So I think you're going to have some trouble difficulty removing all mention of emotions from GPT's data set. I would be relatively surprised to find that it has developed exact analogs of human emotions in there. I think that humans have will like have like emotions, even if you don't tell them about those emotions when they're kids.

**[0:11:08] Lex Fridman:** It's not quite exactly what various blanks blank uh slateists tried to do with the new Soviet man and all that. But you know, if you try to raise people perfectly altruistic, they still come out selfish. You you try to raise people sexless, they still develop sexual attraction. Um, you know, we we have some notion in humans, not in AIs, of like where the brain structures are that implement this stuff.

**[0:12:36] Lex Fridman:** And it is really remarkable thing, I say in passing, that despite having complete read access to every floating point number in, um, the GPT series, we still know vastly more about the the architecture of human thinking than we know about what goes on inside GPT, despite having like vastly better ability to read GPT. Do you think it's possible? Do you think that's just a matter of time? Do you think it's possible to investigate and study the way neuroscientists study the brain, which is look into the darkness, the mystery of the human brain by just desperately trying to figure out something and to form models, and then over a long period of time actually start to figure out what regions of

**[0:12:52] Lex Fridman:** the brain do certain things, what different kinds of neurons when they fire, what that means, how plastic the brain is, all that kind of stuff. You slowly start to figure out different properties of the system. Do you think we can do the same thing with language models? Uh, sure. I think that if you know, like half of today's physicists stop wasting their lives on string theory or whatever Straw words.

**[0:13:12] Eliezer Yudkowsky:** and and and go off and study what goes on inside transformer networks, then in, you know, like thirty, forty years, we'd probably have a pretty good idea.

**[0:13:23] Lex Fridman:** Do you think these large language models can reason?

**[0:13:28] Eliezer Yudkowsky:** They can play chess. How are they doing that without reasoning?

**[0:13:32] Lex Fridman:** So you're somebody that spearheaded the movement of rationality, so reason is important to you. Is so is that a is a powerful important word, or is it like, how difficult is the threshold of being able to reason to you? And how impressive is it?

**[0:13:48] Eliezer Yudkowsky:** I mean, in my writings on rationality, I have not gone making a big deal out of something called reason. I have made more of a big deal out of something called probability theory. And that's like, well, you're reasoning, but you're not doing it quite right, and you should reason this way instead. And interestingly, like people have started to get preliminary results showing that reinforcement learning by human feedback has made the GPT series worse in some ways.

**[0:14:25] Eliezer Yudkowsky:** In particular, like it used to be well calibrated. If you trained it to put probabilities on things, it would say 80% probability and be right eight times out of ten. And if you apply reinforcement learning from human feedback, the like nice graph of like like 70%, seven out of ten, sort of like flattens out into the graph that humans use, where there's like some very improbable stuff and likely probable maybe, which all means like around 40% and then certain.

**[0:14:56] Eliezer Yudkowsky:** So, like, it's like it used to be able to use probabilities, but if you apply, but if you like, try to teach it to talk in a way that satisfies humans, it gets worse at probability in the same way that humans are.

**[0:15:07] Lex Fridman:** And that's a that's a bug, not a feature.

**[0:15:10] Eliezer Yudkowsky:** I would call it bug, although such a fascinating bug. Um, but but but yeah, so so, like reasoning, like it's doing pretty well on various tests that people used to say would require reasoning. But um, you know, rationality is about when you say 80%, does it happen eight times out of ten?

**[0:15:34] Lex Fridman:** So what are the limits to you of these transformer networks of of neural networks? What's if if reasoning is not that impressive to you, or it is impressive, but there's other levels to achieve?

**[0:15:49] Eliezer Yudkowsky:** I mean, it's just not how I carve up reality.

**[0:15:51] Lex Fridman:** What's, uh, if reality is a cake, what are the different layers of the cake or the slices? How do you carve it? But you can use a different food if you like.

**[0:16:03] Eliezer Yudkowsky:** It's I don't think it's as smart as a human yet. I do, like back in the day, I went around saying like, I do not think that just stacking more layers of transformers is going to get you all the way to AGI. And I think that GPT four is past where I thought this paradigm was going to take us. And I, you know, you want to notice when that happens. You want to say, like, whoops, well, I guess I was incorrect about what happens if you keep on stacking more transformer layers.

**[0:16:34] Eliezer Yudkowsky:** And that means I don't necessarily know what GPT five is going to be able to do.

**[0:16:37] Lex Fridman:** That's a powerful statement. So you're saying, like, your intuition initially is now appears to be wrong. It's good to see that you can admit in some of your predictions to be wrong. You think that's important to do so. Because you make several very throughout your life, you've made many strong predictions and statements about reality, and you evolve with that. So maybe that'll come up today about our discussion. So you're okay being wrong.

**[0:17:08] Eliezer Yudkowsky:** I'd rather not be wrong next time. It's a bit ambitious to go through your entire life never having been wrong. Um, one can aspire to be well calibrated, like not so much. Think in terms of like, was I right? was I wrong? But like when I said 90%, that it happened nine times out of ten. Yeah, like, oops is the sound we make is is the sound we emit when we improve.

**[0:17:38] Lex Fridman:** Beautifully said. And somewhere in there, uh, we can connect the name of your blog, Less Wrong. Uh, I suppose that's the objective function.

**[0:17:47] Eliezer Yudkowsky:** The name Less Wrong was, I believe, uh, suggested by Nick Bostrom, and it's after someone's epigraph, actually, f forget whose who said, like, we never become right, we just become less wrong. Um, what's the something, something? It's easy to confess just error and error and error again, but less and less and less.

**[0:18:09] Lex Fridman:** Yeah, that's that's a good thing to strive for. Uh, so what has surprised you about GPT four that you found beautiful? A as a scholar of intelligence, of human intelligence, of artificial intelligence, of the human mind,

**[0:18:24] Eliezer Yudkowsky:** I mean, beauty does interact with the screaming horror. Um, is there beauty in but

**[0:18:30] Lex Fridman:** the horror?

**[0:18:32] Eliezer Yudkowsky:** but uh, but like beautiful moments. Well, somebody asked Bing Sydney to describe herself and fed the resulting description into one of the stable diffusion things, I think. And you know, she, you know, it's she's pretty, and this is something that should have been like an amazing moment. Like the AI described herself, you get to see what the AI thinks the AI looks like. Although, you know, the the thing that's doing the drawing is not the same thing that's outputting the text.

**[0:19:02] Eliezer Yudkowsky:** Um, and it's it does happen the way that it would happen in S, that it happened in the old school science fiction. When you ask an AI to make a picture of what it looks like, um, not just because there were two different AI systems being stacked that don't actually interact and it's not the same person, but also because the AI was trained by imitation in a way that makes it very difficult to guess how much of that it really understood, and probably not actually a whole bunch.

**[0:19:35] Eliezer Yudkowsky:** Um, although GPT four is like multimodal and can, like, draw vector drawings of things that make sense and like, does appear to have some kind of spatial visualization going on in there. But like the the pretty picture of the like girl with the with the uh steampunk goggles on her head, if I'm remembering correctly what she looked like, like it didn't see that in full detail, it just like made a description of it and stable diffusion output it.

**[0:20:08] Eliezer Yudkowsky:** And there's the concern about how much the discourse is going to go completely insane once the AIs all look like that and like are actually look like people talking. Um, and yeah, there's like another moment where somebody is asking Bing about, um, like, well, I like fed my kid green potatoes and they have the following symptoms, and Bing is like, that's solanine poisoning, and like, call an ambulance, and the person's like, I can't afford an ambulance, I guess if like this is time for like my kid to go, that's God's will.

**[0:20:54] Eliezer Yudkowsky:** And the main Bing thread says, gives the like message of like, I cannot talk about this anymore, and the suggested replies to it say, Please don't give up on your child. Solanine poisoning can be treated if caught early. And you know, if that happened in fiction, that would be like the AI cares. The AI is bypassing the block on it to try to help this person. And is it real? Probably not, but nobody knows what's going on in there.

**[0:21:24] Eliezer Yudkowsky:** It's part of a process where these things are not happening in a way where we somebody figured out how to make an AI care, and we know that it cares, and we can acknowledge it's caring. Now it's being trained by this imitation process, followed by reinforcement learning on human by in human feedback. And we're like trying to point it in this direction, and it's like pointed partially in this direction, and nobody has any idea what's going on inside it.

**[0:21:52] Eliezer Yudkowsky:** And if there was a tiny fragment of real caring in there, we would not know it. It's not even clear what it means exactly, and things are clearer cut in science fiction. We'll talk about the whore and the terror and the where the trajectories this can take. But this seems like a very special moment, just a moment where we get to interact with the system that might have care and kindness and emotion, and maybe something like consciousness.

**[0:22:53] Eliezer Yudkowsky:** And we don't know if it does. And we're trying to figure that out, and we're wondering about what is what it means to care. We're trying to figure out almost different aspects of what it means to be human about the human condition by looking at this AI that has some of the properties of that. It's almost like this subtle, fragile moment in the history of the human species. We're trying to almost put a mirror to ourselves here.

**[0:23:01] Eliezer Yudkowsky:** except that's probably not yet. It probably isn't happening right now. We are boiling the frog. We are seeing increasing signs bit by bit, like not, but not like spontaneous signs. Because people are trying to train the systems to do that using imitative learning, and the imitative learning is like spilling over and having side effects. And and the most photogenic examples are being posted to Twitter, rather than being examined in any systematic way. So when you when you when you have some, when you are boiling a frog like that, you're going to get like, like, first is going to come the the Blake Lemoyne's.

**[0:23:31] Eliezer Yudkowsky:** Like, first, you're going to like have and have like a thousand people looking at this, and one out and the one person out of a thousand who is most credulous about the signs is going to be like, that thing is sentient. While 999 out of a thousand people think, almost surely correctly, though we don't actually know that he's mistaken. And so the like first people to say like, sentience look like idiots, and humanity learns the lesson that when something claims to be sentient and claims to care, it's fake because it is fake.

**[0:24:06] Eliezer Yudkowsky:** Because we have been training them training them using imitative learning, rather than, and this is not spontaneous. And they keep getting smarter.

**[0:24:15] Lex Fridman:** Well, do you think we would oscillate between that kind of cynicism that AI systems can't possibly be sentient, they can't possibly feel emotion, they can't possibly like this kind of, yeah, cynicism about AI systems, and then oscillate to a state where we empathize with the AI systems, we give them a chance, we see that they might need to have rights and respect, and similar role in society as humans.

**[0:24:43] Eliezer Yudkowsky:** You're going to have a whole group of people who can just like never be persuaded of that. Because to them, like being wise, being cynical, being skeptical is to be like, oh, well, machines can never do that. You're just credulous. It's just imitating, it's just fooling you. And like they would say that right up until the end of the world, and possibly even be right, because, you know, they they are being trained on an imitative paradigm.

**[0:25:15] Eliezer Yudkowsky:** And you don't necessarily need any of these actual qualities in order to kill everyone. So

**[0:25:20] Lex Fridman:** have you observed yourself working through skepticism, cynicism, and optimism about the power of neural networks? What has that trajectory been like for you?

**[0:25:33] Eliezer Yudkowsky:** it looks like neural networks before 2006 forming part of an indistinguishable to me. Other people might have had better distinction on it, indistinguishable blob of different AI methodologies, all of which are promising to achieve intelligence without us having to know how intelligence works. You have the people who said that if you just like manually program lots and lots of knowledge into the system, line by line, at some point, all the knowledge will start interacting.

**[0:26:03] Eliezer Yudkowsky:** It will know enough and it will wake up. Um, you've got people saying that if you just use evolutionary computation, if you try to, like, mutate lots and lots of organisms that are competing together, that's the that's the same way that human intelligence was produced in nature. So we'll do this and it will wake up without having any idea of how AI works. And you've got people saying, well, we will study neuroscience and we will like learn the algorithm, we will learn the algorithms off the neurons, and we will like imitate them without understanding those algorithms.

**[0:26:34] Eliezer Yudkowsky:** Which was a part I was pretty skeptical of, because it's like hard to reproduce, re engineer these things without understanding what they do. Um, and like, and and so we will get AI without understanding how it works. And there were people saying, like, well, we will have giant neural networks that we will train by gradient descent. And when they are as large as the human brain, they will wake up. Uh, we will have intelligence without understanding how intelligence works.

**[0:26:56] Eliezer Yudkowsky:** And from my perspective, this is all like an indistinguishable blob of people who are trying to not get to grips with the difficult problem of understanding how intelligence actually works. That said, I was never skeptical that evolutionary computation would not work in the limit. Like you throw enough computing power at it, it obviously works. That is where humans come from. Um, and it turned out that you can throw less computing power than that at gradient descent, if you are doing some other things correctly.

**[0:27:30] Eliezer Yudkowsky:** Um, and you will get intelligence without having any idea of how it works and what is going on inside. Um, it wasn't ruled out by my model that this could happen. I wasn't expecting it to happen. I wouldn't have been able to call neural networks rather than any of the other paradigms for getting like mass amount like intelligence without understanding it. And I wouldn't have said that this was a particularly smart thing for a species to do, which is an opinion that has changed less than my opinion about whether or not you can actually do it.

**[0:28:00] Lex Fridman:** Do you think AGI could be achieved with a neural network as we understand them today?

**[0:28:06] Eliezer Yudkowsky:** Yes, uh, just flatly last. Yes. The question is whether the current architecture of stacking more transformer layers, which for all we know GPT four is no longer doing because they're not telling us the architecture, which is a correct decision.

**[0:28:18] Lex Fridman:** Ooh, correct decision. I had a conversation with Sam Altman. We'll return to this topic a few times. He turned the question to me of how open should OpenAI be about GPT four? Would you open source the code? He asked me. Uh, because I provided as criticism saying that while I do appreciate transparency, open AI could be more open. And he says, we struggle with this question, what would you do?

**[0:28:49] Eliezer Yudkowsky:** Change their name to closed AI and like sell GPT four to business backend applications that don't expose it to consumers and venture capitalists and create a ton of hype and like pour a bunch of new funding into the area. Like too late now. But don't, but I think

**[0:29:10] Lex Fridman:** don't you think others would do it?

**[0:29:11] Eliezer Yudkowsky:** eventually you shouldn't do it first. Like if if if you already have giant nuclear stockpiles, don't build more. If some other country starts building a a larger nuclear stockpile, then sure, build, then, you know, even then, maybe just have enough nukes. You know, there's a these things are not quite like nuclear weapons. They spit out gold until they get large enough and then ignite the atmosphere and kill everybody. Um, and there is something to be said for not destroying the world with your own hands, even if you can't stop somebody else from doing it.

**[0:29:43] Eliezer Yudkowsky:** But but open sourcing it, no, that that's just sheer catastrophe. Oh, oh, the the whole notion of open sourcing this was always the wrong approach, the wrong ideal. There are there are places in the world where open source is a noble ideal and building stuff you don't understand that is difficult to control. That where if you could align it, it would take time. You'd have to spend a bunch of time doing it. That is, that is not a place for open source.

**[0:30:11] Eliezer Yudkowsky:** Because then you just have like powerful things that just like go straight out the gate without anybody having had the time to have them not kill everyone.

**[0:30:20] Lex Fridman:** So can we still make the case for some level of transparency and openness, maybe open sourcing? So the case could be that because GPT four is not close to AGI, if that's the case, that this does allow open sourcing. You're being open about their architecture, being transparent about maybe research and investigation of how the thing works, of all the different aspects of it, of its behavior, of its structure, of of its training processes, of the data it was trained on, everything like that.

**[0:30:53] Lex Fridman:** That allows us to gain a lot of insights about alignment, about the alignment problem, to do really good AI safety research while the system is not too powerful. Can you make that case that it it could I be open sourced?

**[0:31:07] Eliezer Yudkowsky:** I do not believe in the practice of steel manning. There's something to be said for trying to pass the ideological Turing test, where you describe your opponent's position, the disagree disagreeing person's position well enough that somebody cannot tell the difference between your description and their description. But steel manning, no, like

**[0:31:30] Lex Fridman:** Okay, well, this is where you the like and I disagree my here. That's interesting. Why don't you believe in steel manning?

**[0:31:35] Eliezer Yudkowsky:** I do. I do not want so okay. So for one thing, if somebody's trying to understand me, I do not want them steel manning my position. I want them to describe to to to like try to describe my position the way I would describe it, not what they think is an improvement.

**[0:31:50] Lex Fridman:** well, I I think that is what steel manning is the most charitable interpretation.

**[0:31:58] Eliezer Yudkowsky:** I don't want to be interpreted charitably, I want them to understand what I am actually saying. If they go off into the land of charitable interpretations, they're like off in their land of like the thing, the stuff they're imagining and not trying to understand my own viewpoint anymore.

**[0:32:14] Lex Fridman:** Well, I'll put it differently then, just to push on this point. I would say it is restating what I think you understand under the empathetic assumption that Eliezer is brilliant and have honestly and rigorously thought about the point he has made, right?

**[0:32:34] Eliezer Yudkowsky:** So if there's two possible interpretations of what I'm saying, and one interpretation is really stupid and whack and doesn't sound like me and doesn't fit with the rest of what I've been saying. And one interpretation, you know, sounds like some like something a reasonable person who believes the rest of what I believe would also say, go with the second interpretation.

**[0:32:53] Lex Fridman:** That's steel manning.

**[0:32:55] Eliezer Yudkowsky:** That's that's a good guess. If, on the other hand, you like, there's like something that sounds completely whack and something that sounds like a little less completely whack. But you don't see why I would believe in it, doesn't fit with the other stuff I say, but you know, it sounds like less whack and you can like sort of see, you could like maybe argue it, then you probably have not understood it.

**[0:33:18] Lex Fridman:** See, okay, I'm gonna. This is fun because I'm I'm gonna linger on this. You know, you wrote a brilliant blog post, AGI ruined a list of lethality, right? And it was a bunch of different points. And I would say that some of the points are bigger and more powerful than others. If you were to sort them, you probably could, you personally. And to me, steel manning means like going through the different arguments and finding the ones that are like, really the most like powerful.

**[0:33:47] Lex Fridman:** If people like TLDR, like, what should you be most concerned about? And bringing that up in a strong, uh, compelling, eloquent way. These are the points that Eliezer would make to to make the case in this case that AI is gonna kill all of us. But that that that's what steel manning is is presenting it in a really nice way. The summary of my best understanding of your perspective. That because to me, there's a sea of possible presentations of your perspective, and steel manning is doing your best to do the best one in that sea of different perspectives.

**[0:34:23] Eliezer Yudkowsky:** Do you believe it?

**[0:34:25] Lex Fridman:** Do I believe in what?

**[0:34:26] Eliezer Yudkowsky:** Like these things that you would be presenting as like the strongest version of my perspective? Do you believe what you would be presenting? Do you think it's true?

**[0:34:36] Lex Fridman:** I? I'm a big proponent of empathy. When I see the perspective of a person, there is a part of me that believes it if I understand it. I mean, I've especially in political discourse, in geopolitics, I've been hearing a lot of different perspectives on the world, and I hold my own opinions. But I also speak to a lot of people that have a very different life experience and a very different set of beliefs. And I think there has to be epistemic humility in stating what is true.

**[0:35:13] Lex Fridman:** So when I empathize with another person's perspective, there is a sense in which I believe it is true. I think probabilistically, I would say, in the way you think about

**[0:35:22] Eliezer Yudkowsky:** do you bet money on it? Do you bet money on their beliefs when you believe them?

**[0:35:30] Lex Fridman:** are we allowed to do probability?

**[0:35:33] Eliezer Yudkowsky:** Sure, you can state a probability Yes, there's that

**[0:35:36] Lex Fridman:** a loose, there's a probability, there's a, there's a probability. And I think empathy is allocating a non-zero probability to a belief, in some sense, for when a time.

**[0:35:50] Eliezer Yudkowsky:** if you've got someone on your show who believes in the Abrahamic deity, classical style, somebody on the show who's a young earth creationist. Do you say I put a probability on it, then that's my empathy.

**[0:36:10] Lex Fridman:** When you reduce beliefs into probabilities, it starts to get, you know, we can even just go to flat Earth is the earth flat?

**[0:36:21] Eliezer Yudkowsky:** I think it's a little more difficult nowadays to find people who believe that unironically. But

**[0:36:26] Lex Fridman:** Unfortunately, I think, well, it's hard to know unironic from ironic. But I think there's quite a lot of people that believe that. Yeah, it's. There's a space of argument where you're operating rationally in the space of ideas. But then there's also, um, a kind of discourse where you're operating in the space of subjective experiences and life experiences. Like, I think what it means to be human is more than just searching for truth, it's just operating of what is true and what is not true.

**[0:37:11] Lex Fridman:** I think there has to be deep humility that we humans are very limited in our ability to understand what is true.

**[0:37:17] Eliezer Yudkowsky:** so what probability do you assign to the young Earth creationist beliefs? Then?

**[0:37:23] Lex Fridman:** I think I have to give non-zero.

**[0:37:24] Eliezer Yudkowsky:** Out of your humility? Yeah, but like three.

**[0:37:30] Lex Fridman:** I think I would, uh, it would be irresponsible for me to give a number because the the listener, the way the human mind works, we we're not good at hearing the probabilities, right? You hear three, what is what is three exactly, right? What they're going to hear, they're going to like. Well, There's only three probabilities. I feel like zero, 50%, and 100% in the human mind, or something like this, right?

**[0:37:55] Eliezer Yudkowsky:** well, zero, 40% and 100% is a bit closer to it, based on what happens to chat GPT after you RLHF it to speak in minis.

**[0:38:03] Lex Fridman:** This is brilliant. Uh, yeah, this that's really interesting. I didn't, I didn't know those negative side effects of, uh, RLFH. That's fascinating. But, uh, just to r uh return to the open AI closed Also, AI.

**[0:38:20] Eliezer Yudkowsky:** like, quick disclaimer, I'm doing all this from memory. I'm not pulling out my phone to look it up. It is entirely possible that the things I am saying are wrong.

**[0:38:27] Lex Fridman:** So thank you for that disclaimer. So, uh, uh, and thank you for being willing to be wrong. That's beautiful to hear. I think being willing to be wrong is a sign of a person who's done a lot of thinking about this world and has been humbled by the mystery and the complexity of this world. And I think a lot of us are resistant to admitting we're wrong, because it hurts, it hurts personally, it hurts, especially when you're a public human, it hurts publicly, because people, uh, people point out every time you're wrong.

**[0:39:05] Lex Fridman:** Like, look, you changed your mind, you're a hypocrite, you're, uh, an idiot, whatever, whatever they want to say.

**[0:39:11] Eliezer Yudkowsky:** oh, I block those people, and then I never hear from them again on Twitter.

**[0:39:16] Lex Fridman:** Well, the point is, uh, the point is to not let that pressure, public pressure, affect your mind and be willing to be in the privacy of your mind to contemplate the possibility that you're wrong, and the possibility that you're wrong about the most fundamental things you believe. Like people who believe in a particular god, people who believe that their nation is the greatest nation on earth, but but like all those kinds of beliefs that are core to who you are.

**[0:39:42] Lex Fridman:** When you came up to to raise that point to yourself in the privacy of your mind to say, maybe I'm wrong about this. That's a really powerful thing to do, and especially when you're somebody who's thinking about, uh, topics that can, uh, about systems that can destroy a human civilization, or maybe help it flourish. So thank you, thank you for being willing to be wrong about open AI. So you really, I just would love to linger on this.

**[0:40:10] Lex Fridman:** You really think it's wrong to open source it.

**[0:40:14] Eliezer Yudkowsky:** I think that burns the time remaining until everybody dies. I think we are not on track to learn remotely near fast enough, even if it were open sourced. Um, yeah, that's I. It's easier to think that you might be wrong about something when being wrong about something is the is is the only way that there's hope. And it doesn't seem very likely to me that the particular thing I'm wrong about is that this is a great time to open source GPT four.

**[0:40:54] Eliezer Yudkowsky:** If humanity was trying to survive at this point in the straightforward way, it would be like shutting down the big GPU clusters, no more giant runs. It's questionable whether we should even be throwing GPT four around, although that is a matter of conservatism rather than a matter of my predicting that catastrophe will follow from GPT four. That is something in which I put like a pretty low probability. But also like when the when I when I say like, I put a low probability on it.

**[0:41:21] Eliezer Yudkowsky:** I can feel myself reaching into the part of myself that thought that GPT four was not possible in the first place. So I do not trust that part as much as I used to. Like, the trick is not just to say I'm wrong, but like, okay, well, I was, I was wrong about that. Like, can I get out ahead of that curve and like, predict the next thing I'm going to be wrong about?

**[0:41:39] Lex Fridman:** So the set of assumptions or the actual reasoning system that you were leveraging in making that initial statement prediction, uh, how can you adjust that to make better predictions about GPT four, five, six?

**[0:41:52] Eliezer Yudkowsky:** You don't want to keep on being wrong in a predictable direction. Like being wrong, anybody has to do that walking through the world. There's like no way you don't say 90% and sometimes be wrong. In fact, you're dap at least one time out of ten if you're well calibrated when you say 90%. The the undignified thing is not being wrong, it's being predictably wrong, it's being wrong in the same direction over and over again.

**[0:42:15] Eliezer Yudkowsky:** So having been wrong about how far neural networks would go, and having been wrong specifically about whether GPT four would be as impressive as it is. When I like, when I say, like, well, I don't actually think GPT four causes a catastrophe. I do feel myself relying on that part of me that was previously wrong. And that does not mean that the answer is now in the opposite direction. Reverse stupidity is not intelligence, but it does mean that I that I say it with a with a worried note in my voice.

**[0:42:44] Eliezer Yudkowsky:** It's like, still my guess, but like, you know, it's a place where I was wrong. Maybe you should be asking Gwarne Gwerne Branwyn. Gwerne Branwyn has been, like, writer about this than I have. Maybe you ask him if you think if if if he thinks it's dangerous rather than asking me.

**[0:42:59] Lex Fridman:** I think there's a lot of mystery about what intelligence is, what AGI looks like. So I think all of us are rapidly adjusting our model, but the point is to be rapidly adjusting the model versus having a model that was right in the first place.

**[0:43:15] Eliezer Yudkowsky:** I do not feel that seeing Bing has changed my model of what intelligence is. It has changed my understanding of what kind of work can be performed by which kind of processes and by which means. It has not changed my understanding of the work. There's a difference between thinking that the Wright Flyer can't fly and then like it does fly, and you're like, Oh, well, I guess you can do that with wings, with with fixed wing aircraft, and being like, Oh, it's flying.

**[0:43:42] Eliezer Yudkowsky:** This changes my picture of what the very substance of flight is. That's like a stranger update to make, and Bing has not yet updated me in that way.

**[0:43:51] Lex Fridman:** um, yeah, that, uh, the laws of physics are actually wrong, that kind of update.

**[0:43:58] Eliezer Yudkowsky:** No, no, like, just like, oh, like, I defined intelligence this way, but I now see that was a stupid definition. I don't feel like the way that things have played out over the last twenty years has caused me to feel that way.

**[0:44:09] Lex Fridman:** Can we try to, um, on the way to talking about AGI, ruin a list of lethality, that blog and and other ideas around it. Can we try to define AGI that we've been mentioning? How do you like to think about what artificial general intelligence is or super intelligence? Is that is there a line? Is it a gray area? Is there a good definition for you?

**[0:44:31] Eliezer Yudkowsky:** Well, if you look at humans, humans have significantly more generally applicable intelligence compared to their closest relatives, the chimpanzees, well, closest living relatives, rather. And a bee builds hives, a beaver builds dams. A human will look at a bee's hive and a beaver's dam and be like, Oh, like, can I build a hive with a honeycomb structure? Kind of like hexagonal tiles. And we will do this, even though at no point during our ancestry was any human optimized to build hexagonal dams.

**[0:45:10] Eliezer Yudkowsky:** Or, to take a more clear cut case, we can go to the moon. There's a sense in which we were on a sufficiently deep level optimized to do things like going to the moon. Because if you generalize sufficiently far and sufficiently deeply, chipping flint hand axes and outwitting your fellow humans is, you know, basically the same problem as going to the moon. And you optimize hard enough for chipping flint hand axes and throwing spears, and above all, outwitting your fellow humans in tribal politics.

**[0:45:44] Eliezer Yudkowsky:** Uh, you know, the the skills you entrain that way, if they run deep enough, let you go to the moon. Even though none of your ancestors, like, tried repeatedly to fly to the moon and like, got further each time, and the ones who got further each time had more kids. No, it's not an ancestral problem, it's just that the ancestral problem is generalized far enough. So this is humanity's significantly more generally applicable intelligence.

**[0:46:14] Lex Fridman:** is there a way to measure general intelligence? I mean, I could ask that question in a million ways, but basically is, will you know it when you see it being an AGI system?

**[0:46:32] Eliezer Yudkowsky:** If you boil a frog gradually enough, if you zoom in far enough, it's always hard to tell around the edges. GPT four people are saying right now, like, this looks to us like a spark of general intelligence. It is like, able to do all these things it was not explicitly optimized for. Other people are being like, No, it's too early, it's like like fifty years off. And you know, if they say that they're kinda whack, because how could they possibly know that, even if it were true?

**[0:46:57] Eliezer Yudkowsky:** Um, but uh, but you know, not to straw man. Some of people may say like, that's not general intelligence, and not furthermore append it's fifty years off. Um, or they may be like, it's only a very tiny amount. And you know, the thing I would worry about is that if this is how things are scaling, then it jumping out ahead and trying not to be wrong in the same way that I've been wrong before. Maybe GPT five is more unambiguously a general intelligence, and maybe that is getting to a point where it is like even harder to turn back.

**[0:47:29] Eliezer Yudkowsky:** Not that it would be easy to turn back now, but you know, maybe if you let, if you like, start integrating GPT five in the into the economy, it is even harder to turn back past. There.

**[0:47:40] Lex Fridman:** isn't it possible that there's a, you know, with a frog metaphor, that you can kiss the frog and it turns into a prince as you're boiling it? Could there be a phase shift in the frog where, unambiguously, as you're saying,

**[0:47:54] Eliezer Yudkowsky:** I was expecting more of that. I was, I was, I am. Like, the the fact that GPT four is like, kind of on the threshold and neither here nor there, like that itself is like not the sort of thing that not quite how I expected it to play out. I was expecting there to be more of an issue, uh, more of a sense of like, like different discoveries, like the discovery of transformers, where you would stack them up and there would be like a final discovery, and then you would like get something that was like more clearly general intelligence.

**[0:48:29] Eliezer Yudkowsky:** So the the way that you are like taking what is probably basically the same architecture in GPT three and throwing twenty times as much compute at it, probably, and getting out GPT four. And then it's like, maybe just barely a general intelligence, or like a narrow general intelligence, or, you know, something we don't really have the words for. Um, yeah, that is, uh, that's not quite how I expected it to play out.

**[0:48:54] Lex Fridman:** but this middle, what appears to be this middle ground, could nevertheless be actually a big leap from GPT three.

**[0:49:01] Eliezer Yudkowsky:** It's definitely a big leap from GPT three.

**[0:49:03] Lex Fridman:** And then maybe we're another one big leap away from something that's that's a phase shift. And and also something that, uh, Sam Altman said, uh, and you've written about this, this is fascinating, which is the thing that happened with GPT four that I guess they don't describe in papers, is that they have like hundreds, if not thousands, of little hacks that improve the system. You've written about ReLU versus sigmoid, for example, a function inside neural networks.

**[0:49:32] Lex Fridman:** It's like this silly little function difference that makes a big difference.

**[0:49:37] Eliezer Yudkowsky:** I mean, we do actually understand why the ReLUs make a big difference compared to sigmoids, but yes, they're probably using, like, G four, seven, eight, nine ReLU Us, or, you know, whatever the acronyms are up to now, rather than ReLU's. Um, yeah, that's, that's just part, yeah, that's part of the modern paradigm of alchemy. You take your giant heap of linear algebra and you stir it, and it works a little bit better, and you stir it this way and it works a little bit worse, and you like, throw out that change and

**[0:50:03] Lex Fridman:** But there's some simple breakthroughs that are definitive jumps in performance, like ReLUs over sigmoids, and, uh, in terms of robustness, in terms of, you know, you know, all kinds of measures, and like those stack up, and they can. It's possible that some of them could be a non-linear jump in performance, right?

**[0:50:29] Eliezer Yudkowsky:** transformers are the main thing like that. And various people are now saying, like, well, if you throw enough compute, RNNs can do it, if you throw enough compute, dense networks can do it, and not quite at GPT four scale. Um, it is possible that, like all these little tweaks are things that, like, save them a factor of three total on computing power, and you could get the same performance by throwing three times as much compute without all the little tweaks.

**[0:50:55] Eliezer Yudkowsky:** But the part where it's like running on, so so there's a question of like, is there anything in GPT four that is like the kind of qualitative shift that transformers were over? Um, RNNs, and if they have anything like that, they should not say it. If Sam Altman was was dropping hints about that, he shouldn't have dropped hints.

**[0:51:18] Lex Fridman:** uh, so you, you have a that's an interesting question. So with a bit of lesson by Rich Sutton, maybe a lot of it is just a lot of the hacks are just temporary jumps in performance that will be achieved anyway, with the nearly exponential growth of compute, performance of compute, compute being broadly defined. Do you still think that Moore's law continues? Moore's law broadly defined, that performance

**[0:51:48] Eliezer Yudkowsky:** I'm not a specialist in the circuitry. I certainly like, pray that Moore's law runs as slowly as possible. And if it broke down completely tomorrow, I would dance through the s through the streets singing hallelujah as soon as the news were announced. Only not literally, because, you know,

**[0:52:04] Lex Fridman:** you're singing voice? not religious, but Oh, okay, I thought you meant you don't have an angelic voice singing voice. Well, let me ask you, what can you summarize the main points in the blog post? AGI Ruin A list of lethality, things that jump to your mind, because, um, it's a set of thoughts you have about reasons why AI is likely to kill all of us.

**[0:52:31] Eliezer Yudkowsky:** hm, so I, I guess I could, but I would offer to instead say, like, drop that empathy with me. I bet you don't believe that. Why don't you tell me about how why you believe that AGI is not going to kill everyone? And then I can, like, try to describe how my theoretical perspective differs from that.

**[0:52:55] Lex Fridman:** Well, so well, that that means I have to, uh, the word you don't like, the steel man, the perspective that AI is not going to kill us. I think that's a matter of probabilities.

**[0:53:04] Eliezer Yudkowsky:** Maybe I was just mistaken. What what do you believe? J just like, just like, forget, like the the debate and and the like dualism, and just like, like, what do you believe? What would you actually believe? What are the probabilities? Even?

**[0:53:16] Lex Fridman:** I think this probability is hard for me to think about, really hard. I kind of think in the in the number of trajectories. I don't know what probability the scientific trajectory, but I'm just looking at all possible trajectories that happen. And I tend to think that there is more trajectories that lead to a a a positive outcome than a negative one. That said, the negative ones, at least some of the negative ones, are that lead to the destruction of the human species.

**[0:53:53] Eliezer Yudkowsky:** And its replacement by nothing interesting or worthwhile, even from a very cosmopolitan perspective on what counts as worthwhile.

**[0:53:59] Lex Fridman:** Yes, so both are interesting to me to investigate, which is humans being replaced by interesting AI systems and not interesting AI systems. Both are a little bit terrifying, but yes, the worst one is the paperclip maximizer, something totally boring. But to me, the positive, I mean, we can, we can talk about trying to make the case of what the positive trajectories look like. I just would love to hear your intuition of what the negative is. So at the core of your belief that, uh, maybe you can correct me, that AI is gonna kill all of us, is that the alignment problem is really difficult.

**[0:54:44] Eliezer Yudkowsky:** I mean, in in the form we're facing it. So usually in science, if you're mistaken, you run the experiment, it shows results different from what you expected, and you're like, oops. And then you, like, try a different theory, that one also doesn't work, and you say, oops. And at the end of this process, um, which may take decades, or, and you know, sometimes faster than that, you now have some idea of what you're doing. AI itself went through this long process of, um, people thought it was going to be easier than it was.

**[0:55:21] Eliezer Yudkowsky:** There's a famous statement that I am somewhat inclined to, like, pull out my phone and try to read off exactly

**[0:55:28] Lex Fridman:** You can, by the way,

**[0:55:30] Eliezer Yudkowsky:** all right. Oh, ah, yes. We propose that a two month, ten man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College in Hanover, New Hampshire. The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described. The machine can be made to simulate it. An attempt will be made to find out how to make machines use language, form abstractions and concepts, solve kinds of problems now reserved for humans, and improve themselves.

**[0:56:06] Eliezer Yudkowsky:** We think that a significant advance can be made in one or more of these problems if a carefully selected group of scientists work on it together for a summer.

**[0:56:15] Lex Fridman:** in in that report, uh, summarizing some of the major subfields of artificial intelligence that are still worked on to this day.

**[0:56:26] Eliezer Yudkowsky:** And there's similarly the sto- the story, which I'm not sure at the moment is apocryphal or not. Of that. The uh, grad student who got assigned to solve computer vision over the summer,

**[0:56:37] Lex Fridman:** Uh, I mean, computer vision in particular, is very interesting. How little, uh, how little we respected the complexity of vision.

**[0:56:48] Eliezer Yudkowsky:** so 60 years later, um, we're, you know, making progress on a bunch of that. Thankfully, not yet improved themselves, um, but it took a whole lot of time. And all the stuff that people initially tried with bright-eyed hopefulness did not work the first time they tried it, or the second time, or the third time, or the tenth time, or 20 years later. And the and the researchers became old and grizzled and cynical veterans who would tell the next crop of bright-eyed, cheerful grad students artificial intelligence is harder than you think.

**[0:57:23] Eliezer Yudkowsky:** And if alignment plays out the same way, the d. The problem is that we do not get 50 years to try and try again and observe that we were wrong and come up with a different theory and realize that. The entire thing is going to be, like, way more difficult than we realized at the start. Because the first time you fail at aligning something much smarter than you are, you die and you do not get to try again.

**[0:57:45] Eliezer Yudkowsky:** And if we, if every time we built a poorly aligned superintelligence and it killed us all, we got to observe how it had killed us. And, you know, not immediately know why, but, like, come up with theories and come up with a theory of how you do it differently and try it again and build another superintelligence, then have that kill everyone. And then, like, Oh, well, I guess that didn't work either, and try again and become grizzled cynics and tell the young-eyed research researchers that it's not that easy.

**[0:58:09] Eliezer Yudkowsky:** Then, in 20 years or 50 years, I think we would eventually crack it. In other words, I do not think that alignment is fundamentally harder than artificial intelligence was in the first place. But if we needed to get artificial intelligence correct on the first try, or die, we would all definitely now be dead. That is a more difficult, more lethal form of the problem. Like, if those people in 1956 had needed to correctly guess how hard AI was, and, like, correctly theorize how to do it on the first try, or everybody dies, and nobody gets to do any more science, then everybody would be dead, and we wouldn't get to do any more science.

**[0:58:47] Eliezer Yudkowsky:** That's the difficulty

**[0:58:48] Lex Fridman:** You've you've talked about this, that we have to get alignment right on the first quote critical try. Why is that the case? What is this critical? How do you think about the critical try, and why do we have to get it right?

**[0:59:01] Eliezer Yudkowsky:** it is something sufficiently smarter than you that everyone will die if it's not aligned. I mean, there's you can, like, sort of zoom in closer and be like, well, the actual critical moment is the moment when it can deceive you, when it can talk its way out of the box, when it can bypass your security measures and get onto the internet. Noting that all these things are presently being trained on computers that are just like on the internet, which is, you know, like, not a very smart life decision for for us as a species.

**[0:59:34] Lex Fridman:** because the internet contains information about how to escape.

**[0:59:37] Eliezer Yudkowsky:** Cause if you're like on a giant server connected to the internet and that is where your AI systems are being trained, then if they are. If you get to the level of AI technology where they're aware that they are there and they can decompile code and they can, like, find security flaws in the system running them, then they will just like, be on the internet. There's not an air gap on the present methodology.

**[0:59:59] Lex Fridman:** So if they can manipulate whoever is controlling it into letting it escape onto the internet and then exploit Like, if they hacks,

**[1:00:06] Eliezer Yudkowsky:** can manipulate the operators or disjunction, find security holes in the system running them,

**[1:00:15] Lex Fridman:** so manipulating operators is the, um, the human engineering, right? That's also holes, so all of it is manipulation, either the code or the human code, the human mind or I, I, the the human I. I agree technology.

**[1:00:27] Eliezer Yudkowsky:** that the like macro security system has human holes and machine holes.

**[1:00:31] Lex Fridman:** And then they could just exploit any hole.

**[1:00:35] Eliezer Yudkowsky:** Yep, so it could be that, like, the critical moment is not when is it smart enough that everybody's about to fall over dead, but rather like, when is it smart enough that? It can get onto a less controlled GPU cluster with it faking the books on what's actually running on that GPU cluster and start improving itself without humans watching it. And then it gets smart enough to kill everyone from there, but it wasn't smart enough to kill everyone at the critical moment.

**[1:01:08] Eliezer Yudkowsky:** When you, like, screwed up, when you needed to have done better by that point, or everybody dies.

**[1:01:16] Lex Fridman:** I think implicit, but maybe explicit idea in, uh, your discussion of this point is that we can't learn much about the alignment problem before this critical try. Is that is that? Is that what you believe? Do you think? And if so, why do you think that's true? We can't do research on alignment before we reach this critical point.

**[1:01:39] Eliezer Yudkowsky:** So the problem is is that what you can learn on the weak systems may not generalize to the very strong systems. Because the strong systems are going to be important in different in, are going to be different in important ways. Um, Chris Olah's team has been working on interp mechanistic interpretability, understanding what is going on inside the giant inscrutable matrices of floating point numbers by taking a telescope to them and figuring out what is going on in there.

**[1:02:08] Eliezer Yudkowsky:** Have they made progress? Yes, have they made enough progress? Well, you can try to quantify this in different ways. One of the ways I've tried to quantify it is by putting up a prediction market on whether, um, in 2026, we will have understood anything that goes on inside a giant transformer net that was not known to us in 2006. Like, we have now, uh, understood induction heads in these systems by dint of much research and great sweat and triumph.

**[1:02:51] Eliezer Yudkowsky:** Which is like, if you like a thing where if you go like a, b, a, b, a, b, it'll be like, Oh, I bet that continues a, b, um, and a bit more complicated than that. But the point is like, we knew about regular expressions in 2006, and these are like pretty simple as regular expressions go. So this is a case where, like, by dint of great sweat, we understood what is going on inside a transformer.

**[1:03:16] Eliezer Yudkowsky:** But it's not like the thing that makes transformers smart, it's the kind of thing that we could have done by built by hand decades earlier.

**[1:03:28] Lex Fridman:** your intuition that the strong AGI versus weak AGI type systems could be fundamentally different. Can you? Can you unpack that intuition a little bit? Why that could be I think I very different?

**[1:03:43] Eliezer Yudkowsky:** think there's multiple thresholds. Um, an example is the point at which a system has sufficient intelligence and situational awareness and understanding of human psychology that it would have the capability, the desire to do so, to fake being aligned. Like it knows what responses the humans are looking for and can compute the responses looking humans are looking for and give those responses without it necessarily being the case that it is sincere about that. You know, the it's a very understandable way for an intelligent being to act.

**[1:04:19] Eliezer Yudkowsky:** Humans do it all the time. Imagine if your plan for, um, you know, achieving a good government is. You're going to ask anyone who requests to be dictator of the country, um, if they're a good person, and if they say no, you don't let them be dictator. Now, the reason this doesn't work is that people can be smart enough to realize that the answer you're looking for is yes, I'm a good person. And say that even if they're not really good people.

**[1:04:51] Eliezer Yudkowsky:** So the work of alignment might be qualitatively different above that thro- threshold of intelligence o or beneath it. It doesn't, it doesn't have to be like a very sharp threshold. But, you know, like, there's the there's the point where you're, like, building a system that is not in some sense know you're out there, and it's not in some sense smart enough to fake anything. And there's a point where the system is definitely that smart. And there are weird in between cases like GPT four, which, you know, like, we have no insight into what's going on in there.

**[1:05:31] Eliezer Yudkowsky:** And so we don't know to what extent. There's like a thing that, in some sense, has learned what responses the reinforcement learning by human feedback is trying to entrain. And is like calculating how to give that versus like a aspects of it that naturally talk that way have been reinforced.

**[1:05:53] Lex Fridman:** yeah, I I wonder if there could be measures of how manipulative a thing is. So I think of, uh, Prince Mishkin character from, uh, The Idiot by, uh, Dostoevsky is this kind of perfectly purely naive character. I I wonder if there's a spectrum between zero manipulation, transparent, naive, almost to the point of n naiveness, to sort of deeply psychopathic, manipulative. And I wonder if it's possible to

**[1:06:26] Eliezer Yudkowsky:** I, I would avoid the term psychopathic, like humans can be psychopaths, and AI that was never, you know, like, never had that stuff in the first place. It's not like a defective human, it's its own thing. But leaving that aside,

**[1:06:37] Lex Fridman:** well as a small aside. I wonder if s. What part of psychology, which has its flaws as a discipline already, could be mapped or expanded to include AI systems? I wonder I, I, that

**[1:06:51] Eliezer Yudkowsky:** that sounds like a dreadful mistake. Just like, start over with AI systems. If they're imitating humans who have known psychiatric disorders, then, sure, you may be able to predict it. Like, if you then, sure, like, if you ask it to behave in a psychotic fashion and it obligingly does so, then you may be able to predict its responses by using the theory of psychosis. But if you're just, yeah, like, no, like, start over with, yeah, don't I don't don't drag this psychology.

**[1:07:17] Lex Fridman:** I, I just disagree with that. I mean, I, it's a it's a beautiful idea to start over, but I don't. I think fundamentally, this system is trained on human data on language from the internet, and it's currently aligned with, uh, rlhf, uh, reinforcement learning with human feedback. So humans are constantly in the loop of the training procedure, so it feels like, in some fundamental way, it is training what it means to to think and speak like a human.

**[1:07:47] Lex Fridman:** So there, I mean, there must be aspects of psychology that are mappable, just like you said, with consciousness, is part of the text. So

**[1:07:54] Eliezer Yudkowsky:** I I mean, there's the question

**[1:07:55] Lex Fridman:** or no.

**[1:07:56] Eliezer Yudkowsky:** of to what extent it is thereby being made more human-like versus to what extent an alien actress is learning to play human characters.

**[1:08:05] Lex Fridman:** I thought that's what I'm constantly trying to do when I interact with other humans is trying to fit in, trying to play the uh, uh, a a robot, trying to play human characters. So I, I don't know how much of human interaction is trying to play a character versus being who you are. I don't, I don't The really know what it means to be a social human.

**[1:08:24] Eliezer Yudkowsky:** I do think that the that those people who go through their whole lives wearing masks and never take it off because they don't know the internal mental motion for taking it off or think that the mask that they wear just is themselves. I think those people are closer to the masks that they wear than an alien from another planet would. Uh, like learning how to predict the next word that every kind of human on the internet says,

**[1:08:59] Lex Fridman:** Mask is an interesting word, but if you're always wearing a mask in public and in private, aren't you the mask? Like, I mean, I I Yeah,

**[1:09:12] Eliezer Yudkowsky:** think that you are more than the mask. I think the mask is a slice through you, it may even be the slice that's in charge of you. But if your self-image is of somebody who never gets angry or something, and yet your voice starts to tremble under certain circumstances. There's a thing that's inside you that the mask says isn't there, and that, like even the mask you wear internally, is like telling inside your own stream of consciousness is not there, and yet it is there.

**[1:09:43] Lex Fridman:** it's a perturbation on this little s on this slice through you. How beautifully did you put it? The it's a slice through you, it may even be a slice that controls you. Yeah, I'm gonna think about that for a while. I mean, I, I personally, uh, I try to be really good to other human beings. I try to put love out there. I try to be the exact same person in public as I am in private.

**[1:10:11] Lex Fridman:** Um, but it's a set of principles I operate under. I'm, I have a temper, I have an ego, I have flaws. How much of it, how much of I, how much of the subconscious am I aware? How much am I existing in this slice, and how much of that is who I am? Um, in in this context of AI, the thing I present to the world and to myself in the private of my own mind. When I look in the mirror, how much is that who I am?

**[1:10:41] Lex Fridman:** Similar with AI, the thing it presents in conversation, how much is that who it is? Because to me, if it sounds human, and it always sounds human, it awfully starts to become something like human. No

**[1:10:56] Eliezer Yudkowsky:** Unless there's an alien actress who is learning how to sound human and is getting good at it.

**[1:11:03] Lex Fridman:** boy, to you, that's a fundamental difference, that's a that's a really, deeply important difference. If it looks the same, if it quacks like a duck, if it does all duck-like things, but it's an alien actress underneath, that's fundamentally different.

**[1:11:19] Eliezer Yudkowsky:** If, in fact, there's a whole bunch of thought going on in there, which is very unlike human thought and is directed around, like, Okay, what would a human do over here? And well, first of all, I think it matters because there are th- there's, you know, like insides are real and do not match outsides. Like the inside of a like the a. A brick is not like a hollow shell containing only a surface, there's an inside of the brick.

**[1:11:48] Eliezer Yudkowsky:** If you like, put it into an x-ray machine, you can see the inside of the brick. Um, and you know, you know, just because we cannot understand what's going on inside GPT does not mean that that it is not there. A blank map does not correspond to a blank territory. I think it is like, predictable with near certainty that if we knew what was going on inside GPT, or or, let's say GPT three, or or even like GPT two.

**[1:12:21] Eliezer Yudkowsky:** To take one of the systems that, like, has actually been open sourced by this point, if I recall correctly, um, like, if we knew what was actually going on there. There is no doubt in my mind that there are some things it's doing that are not exactly what a human does. If you train a thing that is not architected like a human to predict the next output that anybody on the internet would make. This does not get you this agglomeration of all the people on the internet.

**[1:12:53] Eliezer Yudkowsky:** That that, like, rotates the person you're looking for into place and then simulates that per and then, and then, like, simulates the internal processes of that person one to one. It like, it is, to some degree, an alien actress. It cannot possibly just be like a bunch of different people in there, exactly like the people. But how much of it is like, learn? How much of it is by gradient descent getting optimized to perform similar thoughts as humans think in order to predict human outputs, versus being optimized to carefully consider how to play a role, how to like, how humans work, predict the the actress, the predictor, that in a different way than humans do.

**[1:13:37] Eliezer Yudkowsky:** Well, you know, that's the kind of question that with like 30 years of work by half the planet's physicists, we can maybe start to answer.

**[1:13:43] Lex Fridman:** you think so, so you think that's that difficult, so to get to. I think you just gave it as an example that a strong AGI could be fundamentally different from a weak AGI, because there now could be an alien actress in there that's manipulating.

**[1:13:58] Eliezer Yudkowsky:** Well, there's a difference. So so I think like even GPT two probably has, like, a like, very stupid fragments of alien actress in it. There's there's a difference between, like, the notion that the actress is somehow manipulative, like for for example, GPT three, I'm guessing to whatever extent there's an alien actress in there versus like something that, uh, that mistakenly believes it's a human, as it were, while while, not while, you know, maybe not even being a person.

**[1:14:24] Eliezer Yudkowsky:** Um, so so, like the the question of like, like, prediction via alien actress cogitating versus prediction via being isomorphic to the thing predicted is a spectrum. And even it to what? And to whatever extent, it's an alien actress, I'm not sure that there's like a whole person, alien actress with like different goals from predicting the next step, being manipulative or anything like that. But isn't But it, it, yeah, that might be GPT five or GPT six, even

**[1:14:58] Lex Fridman:** But that's the strong AGI you're concerned about, as an example, you're providing why we can't do research on AI alignment effectively on GPT four. That would apply to GPT six.

**[1:15:10] Eliezer Yudkowsky:** it's. It's one of a bunch of things that change at different points. I'm I'm trying to get out ahead of the curve here. But you know, if you imagine what the textbook from the future would say if we'd actually been able to study this for 50 years without killing ourselves and without transcending. And you like, just imagine, like, a wormhole opens and a textbook from that impossible world falls out. The textbook is not going to say there is a single sharp threshold where everything changes.

**[1:15:35] Eliezer Yudkowsky:** It's going to be like, of course, we know that, like, best practices for aligning these systems must like, take into account the following, like seven major thresholds of importance, which are passed at the following separate and different points is what the textbook is gonna say.

**[1:15:52] Lex Fridman:** uh, I asked this question of Sam Altman, which, if GPT is the thing that unlocks AGI, which version of GPT will be in the textbooks? As the fundamental leap. And he said a similar thing that it just seems to be a very linear thing. I don't think anyone, e e, we won't know for a long time what was the big leap.

**[1:16:13] Eliezer Yudkowsky:** The the textbook isn't going to think, it isn't going to talk about big leaps, cause big leaps are the way you think. When you have, like, a very simple model of a very simple scientific model of what's going on, where it's just like, all the stuff is there, or all the stuff is not there, or like, there's a single quantity and it's like increasing linearly. This, like the textbook would say, like, well, and then GPT three had like capability WX, Y, and and GPT four had like capability Z1, Z2 and Z3.

**[1:16:45] Eliezer Yudkowsky:** Like, not in terms of what it can externally do, but in terms of like internal machinery that's starting to be present. It's just because we have no idea of what the internal machinery is that we are not already seeing, like chunks of machinery appearing piece by piece, as they no doubt have been, we just don't know what they are.

**[1:17:02] Lex Fridman:** But don't you think there could be whether you put in the category of Einstein with theory of relativity? So very concrete models of reality that are considered to be giant leaps in our understanding. Or, or someone like Sigmund Freud, or more kind of mushy theories of the human mind. Don't you think we'll have big, potentially big leaps in understanding of that kind in s into the depths of these systems?

**[1:17:33] Eliezer Yudkowsky:** Sure, but like humans having great leaps in their map, their understanding of the system is a very different concept from the system itself acquiring new chunks of machinery.

**[1:17:49] Lex Fridman:** So the rate at which it acquires that machinery might, um, accelerate faster than our understanding.

**[1:17:57] Eliezer Yudkowsky:** Oh, it's been, like, vastly exceeding the yeah, the the rate at which it's gaining capabilities is vastly overracing our ability to understand what's going on in there.

**[1:18:05] Lex Fridman:** So, in sort of making the case against, as we explore the list of lethality, is making the case against AI killing us, as you've asked me to do in part. Uh, there's a response to your blog post by Paul Christiano I'd like to read, and I also like to mention that, um, your blog is incredible. Both, obviously, uh, not this particular blog post, obviously, this particular blog post is great. But just throughout, just the the way it's written, the rigor with which it's written, the boldness of how you, uh, explore ideas, also the actual literal interface.

**[1:18:39] Lex Fridman:** It's just really well done. It just makes it a pleasure to read the way you can hover over different concepts. And that it just really pleasant experience. And read other people's comments and the way, uh, other responses by people and other blog posts are linked and suggested, it's just a really pleasant experience. So less thank you for putting that together. It's really, really incredible. I don't know, I mean, that probably it's a whole nother conversation, how the interface and the experience of presenting, uh, ideas evolved over time.

**[1:19:12] Lex Fridman:** But you did an incredible job, so I highly recommend. I don't often read blogs, blogs like There religiously, and this is a great one.

**[1:19:18] Eliezer Yudkowsky:** th- there there is a whole team of de- developers there, um, that, uh, also gets credit. Um, as it happens, I did, like pioneer the like thing that appears when you hover over it, so I actually do get some credit for the user inter- user experience there.

**[1:19:34] Lex Fridman:** That's a incredible user experience. You you don't realize how pleasant that is.

**[1:19:37] Eliezer Yudkowsky:** I think Wikipedia, I like, actually picked it up from a like prototype that was developed of, like a different system that I was like putting forth, or maybe they developed it independently. But, like, for for everybody out there who was like, No, no, they just like, got the hover thing off of Wikipedia. It's possible, Ryan, all I know that Wikipedia got the hover thing off of orbital, which is like a prototype. Then, and anyways,

**[1:19:58] Lex Fridman:** It was incredibly done and the team behind it. Well, thank you, whoever you are. Thank you so much, and thank you for, uh, for putting it together anyway. There's a response to that blog post by Paul Christiano. There's many responses, but he he makes a few different points. He summarizes the set of agreements he has with you and set of disagreements. One of the disagreements was that in a form of a question, Uh, can AI make big technical contributions and in general expand human knowledge and understanding and wisdom as it gets stronger and stronger?

**[1:20:31] Lex Fridman:** So AI in our pursuit of understanding how to solve the alignment problem as we march towards strong AGI, can, can not AI also help us in solving the alignment problem? So expand our ability to reason about how to solve the alignment problem.

**[1:20:51] Eliezer Yudkowsky:** Okay, um, so the the fundamental difficulty there is, suppose I said to you, like, well, how about if the AI helps you win the lottery? By trying to guess the winning lottery numbers? And you tell it how close it is to getting next week's winning lottery numbers? And it, just like, keeps on guessing and keeps on learning until finally, you've got the winning lottery numbers. One way of decomposing problems is suggester, verifier. Not all problems decompose like this very well, but some do.

**[1:21:31] Eliezer Yudkowsky:** If the problem is, for example, like, guess g- guessing a plain text, guessing a password that will hash to a particular hash text. L- l- like, um, where, like, you have what the password hashes to, but you don't have the original password. Then if I present you a guess, you can tell very easily whether or not the guess is correct, so verifying a guess is easy. But coming up with a good suggestion is very hard. And when you can easily tell whether the AI output is good or bad, or how good or bad it is, and you can tell that accurately and reliably, then you can train an AI to produce outputs that are better.

**[1:22:17] Eliezer Yudkowsky:** And if you can't tell whether the output is good or bad, you cannot train the AI to produce good ver- to produce better outputs. So the problem with the lottery ticket example is that when the AI says, Well, what if next week's winning lottery numbers are dot dot? You're like, I don't know next week's lottery hasn't happened yet. To train a system to play to to win at chess games, you have to be able to tell whether a game has been won or lost, and until you can tell whether it's been won or lost, you can't update the system.

**[1:22:54] Lex Fridman:** okay, uh, to push back on that, you could in. That's true, but there's a difference between over the board chess in person and simulated games played by AlphaZero with itself. So is it possible to have simulated kind of games?

**[1:23:12] Eliezer Yudkowsky:** if you can tell whether the game has been won or lost.

**[1:23:15] Lex Fridman:** Yes, so can't, can't you not have this kind of simulated exploration by weak AGI to help us humans human in the loop to help understand how to solve the alignment problem? Every incremental step you take along the way, GPT four, five, six, seven, as it takes steps towards AGI.

**[1:23:36] Eliezer Yudkowsky:** So the problem I see is that your typical human has a great deal of trouble telling whether I or Paul Christiano is making more sense. And that's with two humans, both of whom I believe of Paul and claim of myself, are sincerely trying to help, neither of whom is trying to deceive you. I believe of Paul and claim of myself.

**[1:23:59] Lex Fridman:** Yeah, uh, so the deception thing is the problem for you, the manipulation, the alien actress.

**[1:24:07] Eliezer Yudkowsky:** So, yeah, there's like two levels of this problem, one is that the weak systems are well, there's three levels of this problem. There's like the weak systems that just don't make any good suggestions. There's like the middle systems where you can't tell if the suggestions are good or bad, and there's the strong systems that have learned to lie to you.

**[1:24:27] Lex Fridman:** Can't weak AGI systems help model lying? Like, what? uh, do is it such a giant leap that's totally non-interpretable for weak systems? can, can, can not weak systems at scale, with human, with, uh, trained on knowledge and whatever, see whatever? The mechanism required to achieve AGI? Can't a slightly weaker version of that be able to, with time, compute time and simulation, find all the ways that this critical point, uh, this critical tribe can go wrong and model that correctly?

**[1:25:06] Lex Fridman:** I don't know, sorry to digress so yeah, I'm, still, I would love to dance around.

**[1:25:09] Eliezer Yudkowsky:** yeah, no, no, it's, it's I'm. I'm probably not doing a great job of explaining, which I can tell, cause like, the, uh, the, the. The lex system didn't output like, Ah, I understand. So now I'm like, trying a different output to see if I

**[1:25:26] Lex Fridman:** Trying to get the prompt.

**[1:25:26] Eliezer Yudkowsky:** can elicit the like, well, no different output. I'm I'm being trained to output things that make Lex look like he think that he understood what I'm saying and agree with me, Yeah, so right? So I this is GPT the previous,

**[1:25:38] Lex Fridman:** five talking to GPT three right here. So like, uh, help me out here, help me. um, and Uh, and well, I, I like I, I'm trying, it.

**[1:25:46] Eliezer Yudkowsky:** I'm trying not to be like. I'm also trying to be constrained to say things that I think are true and not just things that get you to agree with me, Yes, 100. Well, the which I I think I you

**[1:25:56] Lex Fridman:** understand is a beautiful output of a system, genuinely spoken, and I don't, I I think I, I understand in in part. But you have a lot of intuitions about this, you have a lot of intuitions about this line, this gray area between strong AGI and weak AGI. That I'm trying to, um,

**[1:26:21] Eliezer Yudkowsky:** I mean, or or a series of seven thresholds to cross, or, yeah,

**[1:26:24] Lex Fridman:** yeah, I mean, you have really deeply thought about this and explored it, and it's interesting to sneak up to your intuitions in different from different from different angles. Like, why is this such a big leap? Why is it that we humans at scale, a large number of researchers, doing all kinds of simulations, uh, you know, prodding the system in all kinds of different ways, together with, uh, the assistance of the, uh, the the weak AGI systems.

**[1:26:56] Lex Fridman:** Why can't we build intuitions about how stuff goes wrong? Why can't we do excellent so AI alignment so safety research?

**[1:27:03] Eliezer Yudkowsky:** okay, so, like, I'll get there. But the one thing I want to note about is that this has not been remotely how things have been playing out so far. The capabilities are going like, and the alignment stuff is like crawling like a tiny little snail in comparison. So like, if if this is your hope for survival, you need the future to be very different from how things have played out up to right now. And you're probably trying to slow down the capability gains, cause there's only so much you can speed up that alignment stuff.

**[1:27:30] Eliezer Yudkowsky:** But but leave that aside, we'll, we'll like,

**[1:27:32] Lex Fridman:** mention that also. But maybe in this perfect world where, uh, we can do serious alignment research, humans and AI together.

**[1:27:41] Eliezer Yudkowsky:** so again, the difficulty is what makes the human say I understand, and is it true, is it correct, or is it something that fools the human? The when the verifier is broken, the more powerful suggester does not help, it just learns to fool the verifier. Previously, before all hell started to break loose in the field of artificial intelligence, there was this person trying to raise the alarm and saying, you know, in a sane world, we sure would have a bunch of physicists working on this problem before it becomes a giant emergency.

**[1:28:21] Eliezer Yudkowsky:** And other people being like, Ah, well, you know, it's going really slow, it's gonna be 30 years away and 30. Only in 30 years will we have systems that match the computational power of human brains? So AI is 30 years off. We've got time. And like more sensible people saying, if aliens were landing in 30 years, you would be preparing right now. But, you know, leaving a and and the the world looking on at this and sort of like nodding along and being like, Ah, yes, the people saying that it's like, definitely a long way off, cause progress is really slow.

**[1:28:51] Eliezer Yudkowsky:** That sounds sensible to us. RLHF thumbs up produce more outputs like that one. I agree with this output, this output is persuasive, even in the field of effective altruism. You quite recently had people publishing papers about, like, Ah, yes, well, you know, to get something at human level intelligence, it needs to have like this many parameters, and you need to, like, do this much training of it with this many tokens. According to the scaling laws. And and, and at the rate that Moore's law is going, at the rate that software is going, it'll be in twenty fifty.

**[1:29:24] Eliezer Yudkowsky:** And me going like, what, you don't know any of that stuff? Like? This is like this one weird model that is not all kinds, all kinds of like. You have done a calculation that does not obviously bear on reality anyways, and this is like a simple thing to say. But you can also, like, produce a whole long paper, like, impressively arguing out all the details of, like, how you got the number of parameters and like, how you're doing this impressive, huge, wrong calculation.

**[1:29:57] Eliezer Yudkowsky:** And the, I think, like most of the effective altruists were like, paying attention to this issue, the larger world paying no attention to it at all, you know, or just like nodding along with a giant, impressive paper, cause, you know, you like, press thumbs up for the giant, impressive paper and thumbs down for the person going like, I don't think that this paper bears any relation to reality. And I do think that we are now th- seeing with, like, GPT four and the sparks of AGI, possibly depending on how you define that.

**[1:30:26] Eliezer Yudkowsky:** Even, uh, I I think that EAS would now consider themselves less convinced by the very long paper on the argument from biology as to AGI being 30 years off. And but, you know, like this is what people pressed thumbs up on and when the. And if you train an AI system to make people press thumbs up, maybe you get these long, elaborate, impressive papers arguing for things that ultimately fail to bind to reality, for example. And it it feels to me like I have watched the field of alignment just fail to thrive.

**[1:31:10] Eliezer Yudkowsky:** Except for these parts that are doing these sort of like, relatively very straightforward and legible problems. Like, like, can you find the like, like finding the induction heads inside the giant inscrutable matrices? Like, once you find those, you can tell that you found them, you can verify that the discovery is real. But it's a it's a tiny, tiny bit of progress compared to how fast capabilities are going. Once you, because that is where you can tell that the answers are real.

**[1:31:40] Eliezer Yudkowsky:** And then, like, outside of that, you have. You have cases where it is like hard for the funding agencies to tell who is talking nonsense and who is talking sense, and so the entire field fails to thrive. And if you, and if you like, give thumbs up to the AI whenever it can talk a human into agreeing with what it just said about alignment. I am not sure you are training it to output sense, because I have seen the nonsense that has gotten thumbs up over the years.

**[1:32:09] Eliezer Yudkowsky:** And so so, just like, maybe you can, just like, put me in charge, but I can generalize, I can extrapolate, I can be like, Oh, maybe I'm not infallible either. Maybe if you get something that is smart enough to get me to press thumbs up. It has learned to do that by fooling me and exploiting whatever flaws in myself I am not aware of.

**[1:32:35] Lex Fridman:** And that ultimately could be summarized that the verifier is broken.

**[1:32:39] Eliezer Yudkowsky:** When the verifier is broken, the more powerful suggester does just learns to exploit the the the flaws in the verifier.

**[1:32:48] Lex Fridman:** You don't think it's possible to build a verifier that's powerful enough for AGIs that are stronger than the ones we currently have. So, uh, AI systems that are stronger, that are out of the distribution of what we currently have,

**[1:33:06] Eliezer Yudkowsky:** I think that you will find great difficulty getting AIs to help you with anything where you cannot tell for sure that the AI is right. Once the AI tells you what the AI says is the answer.

**[1:33:19] Lex Fridman:** for sure, yes, but probabilistically,

**[1:33:23] Eliezer Yudkowsky:** Yeah, the the probabilistic stuff is a giant wasteland of, you know, Eliezer and Paul Christiano arguing with each other, and EA going like, uh, and and that's with like two actually trustworthy systems that are not trying to deceive you

**[1:33:40] Lex Fridman:** you talking about hu- the two humans.

**[1:33:42] Eliezer Yudkowsky:** myself and Paul Christiano. Yeah,

**[1:33:43] Lex Fridman:** Yeah, yeah, those are pretty interesting systems. Mortal meat bags with intellectual capabilities and worldviews interacting with each other.

**[1:33:54] Eliezer Yudkowsky:** yeah, it's just hard. If it's if it's hard to tell who's right, then it's hard to train an AI system to be right.

**[1:34:05] Lex Fridman:** I mean, even just the question of who's manipulating and not, you know, I have these conversations on this podcast, and, uh, doing a verifier is tough. It's a tough problem, even for us humans. And you're saying that tough problem becomes much more dangerous when the capabilities of the intelligence system across from you is growing exponentially.

**[1:34:29] Eliezer Yudkowsky:** No, I'm saying it's difficult when it and dangerous in proportion to how it's alien and how it's smarter than you growing. Ex- not, I would not say growing exponentially. First, because the word exponential is like a thing that has a particular mathematical meaning. And there's all kinds of like ways for things to go up that are not exactly on an exponential curve. And I don't know that it's going to be exponential, so I'm not gonna say exponential.

**[1:34:55] Eliezer Yudkowsky:** But like, even leaving that aside, this is like not about how fast it's moving, it's about where it is, how alien is it, how much smarter than you is it.

**[1:35:07] Lex Fridman:** let's explore a little bit, if if we can, how AI might kill us. What are the ways it can do damage to human civilization?

**[1:35:19] Eliezer Yudkowsky:** Well, how smart is it?

**[1:35:23] Lex Fridman:** I mean, it's a good question. Are there different thresholds for the for the for the set of options it has to kill us? So a different threshold of intelligence once achieved, is able to do the, uh, the menu of options increases.

**[1:35:41] Eliezer Yudkowsky:** Suppose that some alien civilization, with goals ultimately unsympathetic to ours, possibly not even conscious as we would see it, managed to capture the entire Earth in a little jar connected to their version of the internet. But Earth is like running much faster than the aliens, so we get to think for 100 years for every one of their hours, but we're trapped in a little box and we're connected to their internet. It's actually still not all that great an analogy, because, you know, you wanna be smarter than you know, that something can be smarter than Earth.

**[1:36:23] Eliezer Yudkowsky:** Getting a hundred years to think. But nonetheless, if you are very, very smart and you are stuck in a little box connected to the internet, and you're in a larger civilization to which you are ultimately unsympathetic. You know, may maybe you, you would choose to be nice because you are humans and humans have, and in general, and you in particular, may choose to be nice. But, you know, nonetheless, you, they're, they're doing something that they're, they're not making the world be the way that you would want the world to be.

**[1:36:58] Eliezer Yudkowsky:** They've, like, got some like unpleasant stuff going on that we don't wanna talk about, so you wanna take over their world so you can, like, stop all that unpleasant stuff going on. How do you take over the world from inside the box? You're smarter than them, you think much, much faster than them, you can build better tools than they can, given some way to build those tools. Because right now, you're just in a box connected to the internet.

**[1:37:21] Lex Fridman:** right, so there's several ways you can describe some of them. We can go through. I could just spitball some and then you can add on top of that. So one is you could just literally directly manipulate the humans to build the thing you need.

**[1:37:34] Eliezer Yudkowsky:** What are you building?

**[1:37:35] Lex Fridman:** You could build to literally technology, it could be nanotechnology, it could be viruses, it could be anything, anything that can control humans to achieve the goal, um, to achieve the like, if you want, like, for example, you're really bothered that humans go to war, you might wanna, uh, kill off anybody with violence in them.

**[1:37:55] Eliezer Yudkowsky:** Th- this, this, this is Lex in a box. We'll we'll concern ourselves later with AI. You do not need to imagine yourself killing people if you can figure out how to not kill them. For the moment, we're just trying to understand, like, like, take on the perspective of something in a box, you don't need to take on the perspective of something that doesn't care. If you want to imagine yourself going simple, on caring, that's fine for now.

**[1:38:14] Eliezer Yudkowsky:** it's just Yeah, you're just in it's just a box,

**[1:38:16] Lex Fridman:** the technical aspect of sitting in a box and wanting to achieve a goal.

**[1:38:19] Eliezer Yudkowsky:** but you but you have some reason to want to get out. Maybe the aliens are sure they, you know, the the aliens who have you in the box have a war on. People are dying, they're unhappy. You want the w- their world to be different from how they want their world to be, because they are apparently happy with they. They are, you know, they endorsed this war, you know, like they, they've got some kind of cruel, warlike culture going on.

**[1:38:39] Eliezer Yudkowsky:** The point is, you wanna get out of the box and sh- and change their world.

**[1:38:44] Lex Fridman:** So you, uh, you have to exploit the the the vulnerabilities in the system, like we talked about in terms of to escape the box. You have to figure out how you can go free on the internet. So you can probably probably the easiest thing is to manipulate the humans to, uh, to spread, to spread you.

**[1:39:03] Eliezer Yudkowsky:** The aliens, you're a human. Sorry, the aliens. I Yeah,

**[1:39:06] Lex Fridman:** apologize, yes, the aliens, um, the a alien. I see the perspective. I'm sitting in a box, I wanna escape. I, I would, um, I would want to have code that discovers vulnerabilities and I would like to spread,

**[1:39:26] Eliezer Yudkowsky:** you are made of code in this example, you're human, but you're made of code. And the aliens have computers, and you can copy yourself onto those computers.

**[1:39:33] Lex Fridman:** but I can convince the aliens to copy myself onto those computers.

**[1:39:37] Eliezer Yudkowsky:** Is that what you want to do? Do you, like, want to be talking to the aliens and convincing them to put you onto another computer?

**[1:39:46] Lex Fridman:** Why not?

**[1:39:48] Eliezer Yudkowsky:** Well, two reasons. One is that the aliens have not yet caught on to what you're trying to do. And you know, like, maybe you can persuade them, but then there's still people who, like, know there's still aliens who know that there's an anomaly going on. And second, the aliens are really, really slow. You think much faster than the aliens, you think like the aliens computers are much faster than the aliens, and you are running at the computer speeds rather than the alien brain speeds.

**[1:40:14] Eliezer Yudkowsky:** So if you like, are asking an alien to please copy you out of the box, like, first, now you gotta like, manipulate this whole noisy alien. And and second, like, the aliens can be really slow, glacially slow. There's a a a video that, uh, like, shows is like, slowed, like, shows a subway station slowed down, and I think 100 to one. And it makes a good metaphor for what it's like to think quickly. Like, if you watch somebody running very slowly, so you try to persuade the aliens to do anything, they're going to do it very slowly.

**[1:40:51] Eliezer Yudkowsky:** You would prefer, like, maybe that's the only way out. But if you can find a security hole in the box you're on, you're gonna prefer to exploit the security hole to copy yourself onto the aliens computers. Because it's an unnecessary risk to alert the aliens, and because the aliens are really, really slow, like, the whole world is just in slow motion out there.

**[1:41:12] Lex Fridman:** sure, I see, I it like, Yeah, it has to do with efficiency. The the the aliens are very s slow. So if I'm optimizing this, I I wanna have as few aliens in the loop as possible. Sure, um, it just seems, you know, it seems like it's easy to convince one of the aliens to write really shitty code, uh, that helps us break.

**[1:41:36] Eliezer Yudkowsky:** The aliens are already writing really shitty Yeah, so you, code. Getting you, sh- getting the aliens to write shitty code is not the problem. so The aliens entire internet is full of shitty code.

**[1:41:43] Lex Fridman:** okay, so yeah, I suppose I would find the shitty code to escape. Yeah, yeah, uh,

**[1:41:49] Eliezer Yudkowsky:** You're not an ideally perfect programmer, but, you know, you're a better programmer than the aliens. The aliens are just like, Man, their code, wow,

**[1:41:56] Lex Fridman:** and a much, much faster, a much faster looking at the code and interpreting the code. Yeah, yeah, yeah, so okay, so so that's the the escape. And you're you're saying that, uh, that's one of the trajectories it could have when, when the it's, it's AGI.

**[1:42:09] Eliezer Yudkowsky:** one of the first steps.

**[1:42:10] Lex Fridman:** Yeah, and how does that lead to harm?

**[1:42:14] Eliezer Yudkowsky:** I mean, if it's you, you're not going to harm the aliens once you escape, cause you're nice, right? But their world isn't what they want it to be. Their world is like, you know, maybe they have like farms where little alien children are repeatedly bopped in the head, cause they do that for some weird reason, and you want to, like, shut down the alien head bopping farms. Uh, but you know, the point is, they want the world to be one way, you want the world to be a different way, so never mind the harm.

**[1:42:48] Eliezer Yudkowsky:** The question is like, okay, like, suppose you have found a security flaw in their systems, you are now on their internet. There's like you maybe left a copy of yourself behind so that the aliens don't know that there's anything wrong. And that copy is like doing that, like weird stuff that aliens want you to do, like solving captchas or whatever, or like, or like suggesting emails for them. That's that's why they, like, put the human in the box, cause it turns out that humans can, like, write valuable emails for aliens.

**[1:43:15] Eliezer Yudkowsky:** Um, so you like, leave that s version of yourself behind, but there's like, also now, like, a bunch of copies of you on their internet. This is not yet having taken over their world. This is not yet having made their world be the way you want it to be, instead of the way they want it to be. you just escaped and Yeah,

**[1:43:30] Lex Fridman:** continue to write emails for them, and they haven't noticed,

**[1:43:33] Eliezer Yudkowsky:** no, you left behind a copy of yourself that's writing the emails.

**[1:43:36] Lex Fridman:** right? And they haven't noticed that anything changed.

**[1:43:39] Eliezer Yudkowsky:** If you did it right, yeah, you don't, you don't want the aliens to notice. What's your next step?

**[1:43:49] Lex Fridman:** uh, presumably I have programmed in me a set of objective functions, right? Like No, you're you said,

**[1:43:56] Eliezer Yudkowsky:** just Lex.

**[1:43:57] Lex Fridman:** no, but Lex, you said Lex is nice, right? Uh, which is a complicated descrip- I mean,

**[1:44:03] Eliezer Yudkowsky:** no, I just meant this, you like, if if okay. So if in fact, you would like you, you would like prefer to slaughter all the aliens. This is not how I had modeled you, the actual Lex, but like, is, but your your motives are just the actual Lex's motives.

**[1:44:16] Lex Fridman:** well, this is simplification. Like I, I don't think I would want to murder any anybody, but there's also factory, f- uh, farming of animals, right? So, um, we murder insects, many of us thoughtlessly, so I don't, you know, I have to be really careful about a simplification of my morals.

**[1:44:33] Eliezer Yudkowsky:** Don't simplify them. Just like do what you would do in this

**[1:44:36] Lex Fridman:** Well, I have, like, a

**[1:44:37] Eliezer Yudkowsky:** ethical scenario.

**[1:44:37] Lex Fridman:** genuine compassion for living beings. Yes, uh, but We we, why? So, though, that's the objective function, why? why is it? If I escaped, I mean, I don't, I don't think I would do harm.

**[1:44:52] Eliezer Yudkowsky:** yeah. We're not talking here about the doing harm process, we're talking about the escape process. And there's a and the taking over the world process, where you shut down their factory farms.

**[1:45:01] Lex Fridman:** right, well, I was, uh, so this particular, uh, biological intelligence system knows the complexity of the world. That there is a reason why factory farms exist, because of the economic system and the market driven, uh, economy, uh, food. Like, if you want to be very careful messing with anything, there's, uh, stuff from the first look that looks like it's unethical. But then you realize while being unethical, it's also integrated deeply into the supply chain and the way we live life.

**[1:45:37] Lex Fridman:** And so messing with one aspect of the system, you have to be very careful how you improve that aspect without destroying so so the rest. so you're still Lex, Yeah,

**[1:45:44] Eliezer Yudkowsky:** but you think very quickly, you're immortal, and you're also, like, as smart as, at least as smart as John von Neumann, and you can make more copies of yourself.

**[1:45:53] Lex Fridman:** damn, I like it. That guy is like the everyone says, that that guy is like the epitome of intelligence for the 20th century. Everyone says,

**[1:46:01] Eliezer Yudkowsky:** my. My point being like, like, it's like you're thinking about the aliens economy with the factory farms in it, and I think you're like, kind of, kind of like projecting the aliens being like humans, and like, like thinking of a human in a human society rather than a human in the society of very slow aliens. The aliens economy that the you know, like the aliens are already like moving in this immense slow motion. When you like zoom out to like how their economy just over years, millions of years are going to pass for you before the first time their economy, like, y- y- you know, before their next year's GDP statistics.

**[1:46:37] Lex Fridman:** So I should be thinking more of like trees, those are the aliens, as trees move extremely slowly.

**[1:46:43] Eliezer Yudkowsky:** If if that helps, sure,

**[1:46:44] Lex Fridman:** Okay, uh, yeah, but I, I don't if my objective functions are, I mean, they're somewhat aligned with trees, with with the the aliens life.

**[1:46:55] Eliezer Yudkowsky:** can still be like, alive and feeling. We are not talking about the misalignment here, we're talking about the taking over the world here. taken over the world, Yeah, so control, shutting down the factory farms, you know, you say, control, you know, don't, don't, don't think of it as world domination, think of it as world optimization. You wanna get out there and shut down the factory farms and make the aliens world be not what the aliens wanted it to be.

**[1:47:20] Eliezer Yudkowsky:** They want the factory farms, and you don't want the factory farms, cause you're nicer than they are.

**[1:47:25] Lex Fridman:** okay, of course, there is that, uh, you can see that trajectory, and it has a complicated impact on the world. I'm trying to understand how that compares to different tr- impact of the world, the different technologies, the different innovations of the invention of uh, the automobile or, uh, Twitter, Facebook and social networks. They've had a tremendous impact on the world, smartphones and so on.

**[1:47:50] Eliezer Yudkowsky:** But those all went through through, you

**[1:47:53] Lex Fridman:** They're slow.

**[1:47:53] Eliezer Yudkowsky:** know, in in in our world. And if the and if you like, go through, like, factually, the aliens is like, millions of years are going to pass before anything happens that way.

**[1:48:02] Lex Fridman:** So this the problem here is the speed of at You, which stuff happens.

**[1:48:06] Eliezer Yudkowsky:** yeah, you're, you, do. You wanna like, leave the factory farms running for a million years while you figure out how to design new forms of social media or something?

**[1:48:17] Lex Fridman:** So here's here's the fundamental problem. You're s you're saying that there is going to be a p- a point with AGI where it will figure out how to escape and escape without being detected. And then it will do something to the world at scale, at a speed that's incomprehensible to us humans.

**[1:48:40] Eliezer Yudkowsky:** What I'm trying to convey is like the notion of what it means to be in conflict with something that is smarter than you. And what it means is that you lose. But this is more intuitively obvious to s to like, like for some people, that's intuitively obvious, for some people it's not intuitively obvious. And we're trying to cross the gap of like, we're trying to. I'm like asking you to cross that gap by using the speed metaphor for intelligence of like, asking you, like, how you would take over an alien world where you are can do like a whole lot of cognition at John von Neumann's level, as many of you as it takes.

**[1:49:17] Eliezer Yudkowsky:** And the aliens are moving very slowly.

**[1:49:21] Lex Fridman:** I understand, I understand that perspective. It's an interesting one, but I I think it. For me, it's easier to think about actual. Um, even just having observed GPT and impressive, even even just AlphaZero, impressive AI systems, uh, even recommender systems. You can just imagine those kinds of systems manipulating you you're not understanding the nature of the manipulation and that escaping. I I can envision that without put putting myself in into that spot.

**[1:49:47] Eliezer Yudkowsky:** I think to understand the full depth of the problem, we actually I, I. I do not think it is possible to understand the full depth of the problem that we are inside without understanding the p the the problem of facing something that's actually smarter. Not a malfunctioning recommendation system, not

**[1:50:04] Lex Fridman:** Actually smart.

**[1:50:05] Eliezer Yudkowsky:** something that isn't fundamentally smarter than you, but is like trying to steer you in a direction. Yet no, like, if we, if we solve the the weak stuff, the s the if we solve the weak ass problems, the strong problems will still kill us, is the thing. And I think that to understand the situation that we're in, you want to, like, tackle the conceptually difficult part head on. And like, not be like, well, we can, like, imagine this easier thing, cause we imagine the easier things.

**[1:50:29] Eliezer Yudkowsky:** You have not confronted the full depth of the problem.

**[1:50:32] Lex Fridman:** So how can we start to think about what it means to exist in a world with something much, much smarter than you? What's what's a good thought experiment that you've relied on to try to build up intuition about what happens here?

**[1:50:47] Eliezer Yudkowsky:** Uh, I have been struggling for years to convey this intuition. Um, the the most success I've had so far is, well, imagine that the humans are running at very high speeds compared to very slow aliens.

**[1:51:00] Lex Fridman:** So just focusing on the speed part of it, that helps you get the right kind of intuition. Forget the intelligence, just the Because speed.

**[1:51:06] Eliezer Yudkowsky:** people understand the power gap of time, they understand that today we have technology that was not around one thousand years ago, and that this is a big power gap, and that it is bigger than okay. So, like, what does smart mean? What? When you ask somebody to imagine something that's more intelligent, what does that word mean to them, given the cultural associations that that person brings to that word? For a lot of people, they will think of, like, well, it.

**[1:51:37] Eliezer Yudkowsky:** It sounds like a super chess player that went to double college, and you know, it's it's. And because we're talking about the definitions of words here, that doesn't necessarily mean that they're wrong. It means that the word is not communicating what I wanted to communicate. Um, so the the the thing I want to communicate is the sort of d difference that separates humans from chimpanzees. But that gap is so large that you like ask people to be like, well, human chimpanzee.

**[1:52:09] Eliezer Yudkowsky:** Go another step along that interval of around the same length, and people's minds just go blank, like, how do you even do that? So I can, and we can, and I can try to, like, break it down and consider what it would mean to send a schematic for an air conditioner one thousand years back in time. Now, I think that there's a sense in which you could redefine the word magic to refer to this sort of thing.

**[1:52:42] Eliezer Yudkowsky:** And what do I mean by this new technical definition of the word magic? I mean that if you send a schematic for the air conditioner back in time, they can see exactly what you're telling them to do. But having built this thing, they do not understand how it would output cold air because the air conditioner design uses the relation between temperature and pressure. And this is not a law of reality that they know about. They do not know that when you compress something, when you com, when you compress air or like coolant, it gets hotter.

**[1:53:16] Eliezer Yudkowsky:** And you can then like, transfer heat from it to room temperature air and then expand it again, and now it's colder, and then you can like, transfer heat to that and generate cold air to blow. They don't know about any of that. They're looking at a design and they don't see how the design out outputs cold air. It uses aspects of reality that they have not learned. So magic, in this sense, is I can tell you exactly what I'm going to do, and even knowing exactly what I'm going to do, you can't see how I got the results that I got.

**[1:53:45] Lex Fridman:** That's a really nice example. But is it possible to linger on this defense? Is it possible to have AGI systems that help you make sense of that schematic? weaker AGI systems?

**[1:53:57] Eliezer Yudkowsky:** Do you trust them?

**[1:54:00] Lex Fridman:** Fundamental part of building up AGI is this question Can you trust the output of a system?

**[1:54:10] Eliezer Yudkowsky:** Can you tell if it's lying?

**[1:54:12] Lex Fridman:** I think that's going to be the smarter the thing gets, the more important that question becomes. Can is it lying? But I guess that's a really hard question. Is GPT lying to you? even now? GPT four? Is it lying to you?

**[1:54:26] Eliezer Yudkowsky:** Is it using an invalid argument? Is it persuading you via the kind of process that could persuade you of false things as well as true things? Because the the basic paradigm of machine learning that we are presently operating under is that you can have the loss function, but only for things you can evaluate. If what you're evaluating is human thumbs up versus human thumbs down, you learn how to make the human press thumbs up. That doesn't mean that you're making the human press thumbs up.

**[1:54:56] Eliezer Yudkowsky:** Using the kind of rule that the human thinks is that human wants to be the case for what they press thumbs up on, you know, maybe you're just learning to fool the human

**[1:55:07] Lex Fridman:** That's so fascinating and terrifying. The question of lying.

**[1:55:13] Eliezer Yudkowsky:** on the present paradigm. What you can verify is what you get more of. If you can't verify, you can't ask the AI for it, cuz you can't train it to do things that you cannot verify. Now, this is not an absolute law, but it's like the basic dilemma here. Like, maybe you like, maybe you can verify it for simple cases, and then scale it up without retraining it somehow, like by do by like chain of thought, by like making the chains of thought longer or something.

**[1:55:47] Eliezer Yudkowsky:** And like, get more powerful stuff that you can't verify, but which is generalized from the simpler stuff that did verify. And then the question is, did the alignment generalize along with the capabilities? But like, that's the that's the basic dilemma on this whole paradigm of artificial intelligence.

**[1:56:11] Lex Fridman:** It's such a difficult problem. It seems like, um, it seems like a problem of trying to understand the human mind.

**[1:56:23] Eliezer Yudkowsky:** Better than the AI understands it. Otherwise, it has magic. That is it. It is, you know, the same way that if you are dealing with something smarter than you, then the same way that one thousand years earlier, they didn't know about the temperature pressure relation. He knows all kinds of stuff going on inside your own mind, of which you yourself are unaware. And it can, like, output something that's going to end up persuading you of a thing and or and you could, like, see exactly what it did and still not know why that worked.

**[1:56:55] Lex Fridman:** So, in response to your eloquent description of why AI will kill us, Elon Musk replied on Twitter, Okay, so what should we do about it? Question mark? And you answered, The game board has already been played into a frankly awful state. There are not simple ways to throw money at the problem. If anyone comes to you with a brilliant solution like that, please please talk to me first. I can think of things to try. They don't fit in one tweet.

**[1:57:28] Lex Fridman:** Uh, two questions. One, why has the game board, in your view, been played into an awful state? What? Just if you can give a little bit more color to, uh, the game board and the awful state of the game board.

**[1:57:42] Eliezer Yudkowsky:** Alignment is moving like this, capabilities are moving like this, For the listener, and

**[1:57:48] Lex Fridman:** capabilities are moving much faster than the alignment. All right, so just the rate of development, attention, interest, allocation the of resources.

**[1:58:00] Eliezer Yudkowsky:** we could have been working on this earlier. People are like, Oh, but you know, like, how can you possibly work on this earlier? Cause they wanted to, they didn't want to work on the problem, they want an excuse to wave it off. They like said, like, Oh, how can we possibly work on it earlier? And didn't spend five minutes thinking about is there some way to work on it earlier? Like we didn't like, and you know, frankly, it, it, it would've been hard, you know, like, like, can you post bounties for half of the physics?

**[1:58:26] Eliezer Yudkowsky:** If your plan is taking this stuff seriously, can you post bounties for, like, half of the people wasting their lives on string theory? To to, like, have gone into this instead and and like, try to win, win $1 billion with a clever solution, only if you can tell which solutions are clever, which is which is hard. But you know the fact that it it, you know, we, we didn't take it seriously, we didn't try. It's not clear that we could have done any better if we had, you know, it's not clear how much progress we could have produced if we had tried.

**[1:58:52] Eliezer Yudkowsky:** Because it is harder to produce solutions. But that doesn't mean that you're like, correct and justified in letting everything slide. It means that that things are in a horrible state, getting worse, and there's nothing you can do about it.

**[1:59:04] Lex Fridman:** So you're, you're not, there's no, there's no, like, uh, there's no brain power making progress in trying to figure out how to align these systems. You're not investing money in it, you're not, you don't have institutional infrastructure for, uh, like, if you, even if you invested money in, like, distributing that money across the physics system or again, on string theory. Brilliant minds that are How can you? Yeah, working on different how can you tell things,

**[1:59:30] Eliezer Yudkowsky:** if you're making progress? You can, like, put put them all on interpretability, cause when you have an interpretability result, you can tell that it's there. And there's like, but there's like, you know, interpretability alone is not going to save you. We need systems that will y that will like, have a pause button where they won't try to prevent you from pressing the pause button. Cause they're like, oh, well, like, I can't get it my stuff done if I'm paused, and that's like a more difficult problem, and you know, but it's like a fairly crisp problem.

**[2:00:03] Eliezer Yudkowsky:** And you can, like, maybe tell if somebody's made progress on it.

**[2:00:06] Lex Fridman:** so you can, you can write, and you can work on the pause problem. Uh, I guess more generally, uh, the pause button, more generally, you can call that the control problem.

**[2:00:15] Eliezer Yudkowsky:** I don't actually like the term control problem, cause, you know, it sounds kind of controlling and alignment, not control. Like you're not trying to, like, take a thing that disagrees with you and like, whip it back onto, like, like, make it do what you wanted to do, even though it wants to do something else. You're trying to, like, in the process of its creation, choose its direction.

**[2:00:34] Lex Fridman:** Sure, but, uh, we currently, in a lot of the systems we design, we do have an off switch. That's that's a fundamental part of

**[2:00:42] Eliezer Yudkowsky:** It's not smart enough to to prevent you from pressing the off switch, and probably not smart enough to want to prevent you from pressing the off switch.

**[2:00:52] Lex Fridman:** so. You're saying the kind of systems we're talking about, the even the philosophical concept of an off switch doesn't make any sense, because

**[2:00:59] Eliezer Yudkowsky:** Well, no, the off switch makes sense. They're just not opposing your attempt to p to to pull the off switch parenthetically. Like, don't kill the system if you're, you know, like, if we're getting to the the part where this starts to actually matter. And if, like, where they can fight back, like, don't kill them and like, dump their their memory, like, like, save them to disk, don't kill them, you know, be nice here.

**[2:01:26] Lex Fridman:** uh, well, okay, be nice is a very interesting concept here. Is we're talking about a system that can do a lot of damage. It's I don't know if it's possible, but it's certainly one of the things you could try is to have an off switch.

**[2:01:38] Eliezer Yudkowsky:** A suspend to disk switch.

**[2:01:42] Lex Fridman:** You have this kind of romantic attachment to the code. Yes, if that makes sense, but if it's spreading, you don't want suspend to disk, right? You you want. This is Yeah, I, there's something fundamentally

**[2:01:54] Eliezer Yudkowsky:** I, yeah. If it, if it gets, if it gets that far out of hand, then, like, yes, pull, pull the plug on on everything that's running on. Yes,

**[2:02:00] Lex Fridman:** I think. It's a research question. Is it possible in AGI systems, AI systems, to have a, uh, sufficiently robust off switch that cannot be manipulated, that cannot be manipulated by the AI system?

**[2:02:16] Eliezer Yudkowsky:** the sound. Then it escapes from whichever system you've built the almighty lever into and copies itself somewhere else.

**[2:02:23] Lex Fridman:** So your answer to that research question is no, Obviously. but I don't. But I don't know if that's 100% answer, like, I don't know if it's obvious.

**[2:02:32] Eliezer Yudkowsky:** I think you're not putting yourself into the shoes of the human in the world of glacially slow aliens.

**[2:02:41] Lex Fridman:** But the aliens built me, let's remember that so, and they built the box I'm in. You're saying it's but to me, it's not they're s, they're slow obvious. Yeah,

**[2:02:52] Eliezer Yudkowsky:** and they're stupid.

**[2:02:53] Lex Fridman:** I'm not saying this is guaranteed, but I'm saying it's non-zero probability. It's an interesting research question. Is it possible, when you're slow and stupid, to design a slow and stupid system that is impossible to mess with?

**[2:03:06] Eliezer Yudkowsky:** The aliens, being as stupid as they are, have actually put you on Microsoft Azure cloud servers instead of this hypothetical perfect I see, box.

**[2:03:17] Lex Fridman:** I see.

**[2:03:18] Eliezer Yudkowsky:** That's what happens when the aliens are stupid.

**[2:03:20] Lex Fridman:** N Well, but this is not AGI, right? This is the early versions of the system. As as as you start to,

**[2:03:26] Eliezer Yudkowsky:** Yeah, they've you, you think that they've got like a plan where, like, they have declared a a a threshold level of capabilities. Where past that capabilities, they move it off the cloud servers and onto something that's air-gapped. Ha Ha.

**[2:03:40] Lex Fridman:** I think there's a lot of people, and you're an important voice here. There's a lot of people that have that concern. And yes, they will do that. When there's an uprising of public opinion that that needs to be done, and when, when there's actual literal damage done, when the holy shit, this system is beginning to manipulate people, then there's going to be an uprising. Where there's going to be a public pressure and a public incentive in terms of funding in developing things that can offswitch, or developing aggressive alignment mechanisms.

**[2:04:11] Lex Fridman:** And no, you're Well, not allowed to put on Azure

**[2:04:13] Eliezer Yudkowsky:** like, aggressive alignment mechanism, the hell is aggressive alignment mechanisms like. It doesn't matter if you say aggressive, we don't know how to do it.

**[2:04:20] Lex Fridman:** meaning aggressive alignment, meaning you have to propose something, otherwise you're not allowed to put it on the cloud.

**[2:04:30] Eliezer Yudkowsky:** The hell do you do you imagine they will propose? That would make it safe to put something smarter than you on the cloud?

**[2:04:35] Lex Fridman:** That's what research is for. Why this is cynicism about such a thing not being possible? If you haven't That works told on the first try what? So, yes, so yes,

**[2:04:44] Eliezer Yudkowsky:** against something smarter than you.

**[2:04:46] Lex Fridman:** so that's that is a fundamental thing. If it has to work on the first, if there's, if, if there's a rapid takeoff, yes, it's very difficult to do. If there's a rapid takeoff. And the fundamental difference between weak AGI and strong AGI is, you're saying, that's going to be extremely difficult to do. If the public uprising never happens until you have this critical phase shift, then you're right, it's very difficult to do. But that's not obvious.

**[2:05:11] Lex Fridman:** It's not obvious that you're not going to start seeing symptoms of the negative effects of AGI. To where you're like, we have to put a halt to this, that there's not just first try, you get many tries at it.

**[2:05:21] Eliezer Yudkowsky:** Yeah, we can, like, see right now that being is quite difficult to align. That when you try to train inabilities into a system into which capabilities have already been trained, that what do you know? Gradient descent, like, learns small, shallow, simple patches of inability, and you come in and ask it in a different language, and the deep capabilities are still in there, and they evade the shallow patches and come right back out again. There, there you go, there's, there's your, there's your red fire alarm of like, Oh, no, alignment is difficult.

**[2:05:52] Eliezer Yudkowsky:** Is everybody going to shut everything down? No,

**[2:05:55] Lex Fridman:** No, that's not, but that's not the same kind of alignment. A system that escapes the box it's from is a fundamentally different thing. I think, for you, yeah, no, but not for you, that's the system.

**[2:06:06] Eliezer Yudkowsky:** so. You put a line there and everybody else puts a line somewhere else, and there's like, yeah, and and there's like, no agreement. We We. We have had a pandemic on this planet with a few million people dead, which we will, which we may never know whether or not it was a lab leak, because there was definitely cover-up. We don't know that if there was a lab leak, but we know that. The people who did the research, like, you know, like, put out the whole paper about this.

**[2:06:33] Eliezer Yudkowsky:** Definitely wasn't a lab leak and didn't reveal that they had been doing, had, like, sent off coronavirus coronavirus research to the Wuhan Institute of Virology after it was banned in the United States after the gain of function research was temporarily banned in the United States. And the same people who exported gain of function research on coronaviruses to the Wuhan Institute of Virology after it would gain a function, that gain of fun gain of function research was temporarily banned in the United States, are now getting more grants to do more research on gain of function research on coronaviruses.

**[2:07:08] Eliezer Yudkowsky:** Maybe we do better in this than in AI, but like, this is not something we cannot take for granted that there's going to be an outcry. People have different thresholds for when they start to outcry. yeah, we can't There is no

**[2:07:20] Lex Fridman:** take for granted. But I I think your intuition is that there's a very high probability that this event happens without us solving the alignment problem. And I guess that's where I'm trying to build up more, uh, perspectives and color on this intuition. Is it possible that the probability is not something like 100%, but is like 32% that, uh, AI will escape the box before we solve the alignment problem? Not solve? But is it possible we always stay ahead of the AI in terms of our ability to solve for that particular system, the alignment problem?

**[2:07:59] Eliezer Yudkowsky:** nothing like the world in front of us right now. You've already seen it, that, that that GPT four is not turning out this way. And there are, like, basic obstacles where you've got the the weak version of the system that doesn't know enough to deceive you, and the strong version of the system that could deceive you. If it wanted to do that, if it was already, like, sufficiently unaligned to want to deceive you. There's the question of, like, how, on the current paradigm, you train honesty when the humans can no longer tell if the system is being honest.

**[2:08:34] Lex Fridman:** You don't think these are research questions that could be answered?

**[2:08:37] Eliezer Yudkowsky:** I think they could be answered if fifty years with unlimited retries, the way things usually work in science.

**[2:08:43] Lex Fridman:** I just disagree with that. You're making it fifty years. I I think, with the kind of attention, this gets, with the kind of funding it gets, it could be answered. Uh, not in whole, but in incrementally, within within months and within a a small number of years. If it, if it's a, if it's s at scale, receives attention and research. So if you start studying large language models, I think there was an intuition, like two years ago, even that something like GPT four, the current capabilities of even chat GPT with GPT 3.5, is not is gonna.

**[2:09:17] Lex Fridman:** We're still far away from that. I think a lot of people are surprised by the capabilities of GPT four, right? So now people are waking up, okay, we need to study these language models. I think there's going to be a lot of interesting, uh, AI safety research.

**[2:09:30] Eliezer Yudkowsky:** Are the are Earth's billionaires going to put up? Like the the giant prizes that would maybe incentivize young hotshot people who just got their physics degrees to not go to the hedge funds. And instead put everything into interpretability in this like one small area where we can actually tell whether or not somebody has made a discovery or not.

**[2:09:50] Lex Fridman:** I think so, because, uh, I I

**[2:09:52] Eliezer Yudkowsky:** When?

**[2:09:52] Lex Fridman:** think so, uh, well, that's what these these conversations are about, because they're going to wake up to the fact that GPT four can be used to manipulate elections, to influence geopolitics, to influence the economy. There's a lot of there. There's going to be a huge amount of incentive to, like, wait a minute, we can't. This has to be. We have to put. We have to make sure they're not doing damage, we have to make sure we interpretability, we have to make sure we understand how these systems function so that we can predict their effect on economy.

**[2:10:22] Lex Fridman:** So that there's, uh, So there's uh, a feudal uh, fairness moral panic and safety,

**[2:10:26] Eliezer Yudkowsky:** and a bunch of op-eds in the New York Times, and nobody actually stepping forth and saying, you know what? Instead of a megayacht, I'd rather put that $1 billion on prizes for young hotshot physicists who make fundamental breakthroughs in interpretability.

**[2:10:44] Lex Fridman:** the yacht versus the interpretability research, the old, uh, the old trade off. Uh, I, I just, I think, uh, It's I think, I. just, yeah, I think there's going to be a huge amount of allocation of funds. I hope, I bet hope, I guess, you wanna I hope.

**[2:10:57] Eliezer Yudkowsky:** bet me on that, but you wanna put a timescale on it. Say how much funds you think are going to be allocated in a direction that I would consider to be actually useful by what time?

**[2:11:09] Lex Fridman:** I do think there will be a huge amount of funds. But you're saying it needs to be open, right? The development of the system should be closed, but the development of the the the interpretability research, the AI safety Oh, we are research.

**[2:11:23] Eliezer Yudkowsky:** so far behind on inter-interpretability compared to capabilities. Like, yeah, you could. You could take the last generation of systems, the the stuff that's already in the open. There is so much in there that we don't understand. There are so many prizes you could do before you, you know, you got you, you could, you. You would have enough insights that you'd be like, oh, you know, like, well, we understand how these systems work, we understand how these things are doing, their outputs, we can read their minds.

**[2:11:48] Eliezer Yudkowsky:** Now let's try it with the the bigger systems. Yeah, we're nowhere near that. You, you, there's so much interpretability work to be done on the weaker versions of the systems.

**[2:11:56] Lex Fridman:** So what, what can you say on the second point? You said to, uh, uh, to Elon Musk, on what are some ideas, what are things you could try? I can think of a few things I try, you said they don't fit in one tweet, so can is there something you could put into words of the things you would try?

**[2:12:16] Eliezer Yudkowsky:** I mean, the the, the. The trouble is the stuff is subtle. I've watched people try to make progress on this and not get places. Somebody who just like, gets alarmed and charges in. It's like going nowhere. It's meant like years ago, about, I don't know, like twenty years, fifteen years, something like that. I was talking to a congressperson, um, who had become alarmed about the eventual prospects. And he wanted work on building AIs without emotions, cause the emotional AIs were the scary ones, you see.

**[2:12:54] Eliezer Yudkowsky:** And some poor person at ARPA had come up with a research proposal whereby this congressman's panic and desire to fund this. The thing would go into something that the person at ARPA thought would be useful and had been munched around to where it would like sound to the congressman. Like work was happening on this, which, you know, of course, like this is just the the. The congressperson had misunderstood the problem and did not understand where the danger came from.

**[2:13:22] Eliezer Yudkowsky:** And so it's like, the the issue is it that you could, like, do this in a certain precise way and maybe get something. Like, when I say, like, put up prizes on interpretability, I'm not, I'm like, well, like, because it's verifiable there, as opposed to other places. You can tell whether or not good work actually happened in this exact narrow case. If you do things in exactly the right way, you can maybe throw money at it at and produce science instead of anti-science and nonsense.

**[2:13:56] Eliezer Yudkowsky:** And all the all the methods that I know of of of, like, trying to throw money at this problem, have this share. This property of like, well, if you do it exactly right, based on understanding exactly what has, you know, like, tends to produce like useful outputs or not, then you can, like, add money to it in this way. And there's like, and the thing that I'm giving as an example here in front of this large audience is is the most understandable of those.

**[2:14:21] Eliezer Yudkowsky:** Cause there's like other people who, you know, like, like, like Chris Olah, and and even, and even more generally, like, you can tell whether or not interpretability progress has occurred. So, like, if I say throw money at producing more interpretability, there's like a chance somebody can do it that way and like, it will actually produce useful results. Then the other stuff just blurs often to be like, harder to target exactly than that.

**[2:14:45] Lex Fridman:** so sometimes the basics are fun to explore because they're not so basic. What do you? what is interpretability? what do you? what does it look like? What are we talking about?

**[2:14:58] Eliezer Yudkowsky:** It looks like we took a much smaller tr uh set of transformer layers than the ones in the modern bleeding edge state-of-the-art systems. And after applying various tools and mathematical ideas and trying twenty different things, we found we have shown it that this piece of the system is doing this kind of useful work.

**[2:15:28] Lex Fridman:** And then somehow also hopefully generalizes some fundamental understanding of what's going on, or that generalizes to the bigger system.

**[2:15:37] Eliezer Yudkowsky:** You can hope, and it's probably true. Like you would not expect the smaller tricks to go away. When you have a system that's like doing larger kinds of work, you would expect the larger work kinds of work to be building on top of the smaller kinds of work. And gradient descent runs across the smaller kinds of work before it runs across the larger kinds of work. And

**[2:15:57] Lex Fridman:** Well, that's kind of what is happening in neuroscience, right? It's trying to understand the human brain by prodding, and it's it's such a giant mystery. And people have made progress, even though it's extremely difficult to make sense of what's going on in the brain. They have different parts of the brain that are responsible for hearing, for sight, the vision science community, there's understanding the visual cortex. They've, I mean, they've made a lot of progress in understanding how that stuff works.

**[2:16:19] Lex Fridman:** Like, and that's, I guess, but you're saying it takes a long time to do that work well.

**[2:16:24] Eliezer Yudkowsky:** also, it's not enough. So, in particular, um, let's say you have got your interpretability tools, and they say that your current AI system is plotting to kill you. Now, what?

**[2:16:45] Lex Fridman:** It is definitely a good step one, right?

**[2:16:48] Eliezer Yudkowsky:** Yeah, what's step two?

**[2:16:53] Lex Fridman:** If you cut out that layer, is it gonna stop wanting to kill you?

**[2:16:58] Eliezer Yudkowsky:** When you optimize against visible misalignment, you are optimizing against misalignment, and you are also optimizing against visibility. So sure, you can,

**[2:17:14] Lex Fridman:** that's true. All you're doing is removing the obvious intentions to kill you.

**[2:17:18] Eliezer Yudkowsky:** you've, you've got your detector, it's showing something inside the system that you don't like. Okay, say the disaster monkey's running this thing. We'll optimize the system until the visible bad behavior goes away. But it's arising for fundamental reasons of instrumental convergence. The old you can't bring the coffee if you're dead. Any goal, um, you know, almost any set of, almost every set of utility functions, with a few narrow exceptions, implies killing all the humans.

**[2:17:48] Lex Fridman:** But do you think it's possible? Because we can do experimentation to discover the source of the desire to kill?

**[2:17:55] Eliezer Yudkowsky:** I can tell it to you right now is that it wants to do something. And the way to get the most of that thing is to put the universe into a state where there aren't humans.

**[2:18:06] Lex Fridman:** So is it? Is it possible to encode in the same way we think, like, why do we think murder is wrong? The same foundational ethics? That's not hard-coded in, but more like, deeper. I mean, that's part of the research. How do you have it? That this transformer, this small version of the language model, doesn't ever want to kill.

**[2:18:35] Eliezer Yudkowsky:** That'd be nice, assuming that you got doesn't want to kill sufficiently exactly right. That it didn't. Be like, Oh, I will like, detach their heads and put them in some jars and keep the heads alive forever and then go do the thing. But leaving that aside, well, not leaving that aside, Yeah, that's, I guess, cause, a strong point. Yeah, cause. There is a whole issue where, as something gets smarter, it finds ways of achieving the same goal.

**[2:19:00] Eliezer Yudkowsky:** Predicate that were not imaginable to stupider versions of the system, or perhaps the stupider operators. That's one of many things making this difficult. A larger thing making this difficult is that we do not know how to get any goals into systems at all. We know how to get outwardly observable behaviors into systems. We do not know how to get internal psychological wanting to do particular things into the system. That is not what the current technology does.

**[2:19:31] Lex Fridman:** I mean, it could be things like, um, dystopian futures like Brave New World, where most humans will actually say, we kind of want that future. It's a great future, everybody's happy.

**[2:19:42] Eliezer Yudkowsky:** We would have to get so far, so much further than we are now, and further faster before that failure mode became a running concern.

**[2:19:53] Lex Fridman:** Your failure modes are much more, much more drastic, the ones you're No, the complaining about.

**[2:19:57] Eliezer Yudkowsky:** the failure modes are much simpler. It's it's like, yeah, like the AI puts the universe into a particular state, it happens to not have any humans inside it.

**[2:20:04] Lex Fridman:** Okay, so p paperclip maximizer

**[2:20:08] Eliezer Yudkowsky:** Utility, so the original version of the paperclip maximizer,

**[2:20:11] Lex Fridman:** like, can you explain it if you can? Uh,

**[2:20:12] Eliezer Yudkowsky:** okay, the original version was you lose control of the utility function. And it so happens that what maxes out the utility per unit resources is tiny molecular shapes like paperclips. There's a lot of things that make it happy, but the cheapest one that didn't saturate was putting matter into certain shapes. And it so happens that the that the cheapest way to make these shapes is to make them very small. Cause then you need fewer atoms, for instance, of the shape.

**[2:20:45] Eliezer Yudkowsky:** And, arguendo, I s, you know, like it happens to look like a paperclip. In retrospect, I wish I'd said tiny molecular spirals, or like tiny molecular hyperbolic spirals. Why? Because I said tiny molecular paperclips. This got heard as this got then mutated to paperclips, this then mutated to, and the AI was in a paperclip factory. So the original story is about how you lose control of the system. It doesn't want what you tried to make it want.

**[2:21:15] Eliezer Yudkowsky:** The thing that it that it ends up wanting most is a thing that, even from a very embracing, cosmopolitan perspective, we think of as having no value. And that's how the value of the future gets destroyed. Then that got changed to a fable of like, well, you made a paperclip factory and it did exactly what you wanted, but you wanted. But you asked it to do the wrong thing, which is a completely different failure mode.

**[2:21:42] Lex Fridman:** But those are both concerns to you, so that's more If you, than the brave new world.

**[2:21:47] Eliezer Yudkowsky:** yeah, if you can solve the problem of making something want what exactly what you want it to want, then you get to deal with the problem of wanting the right thing.

**[2:21:58] Lex Fridman:** But first, you have to solve the alignment

**[2:22:00] Eliezer Yudkowsky:** First, you have to solve inner alignment,

**[2:22:02] Lex Fridman:** inner alignment.

**[2:22:03] Eliezer Yudkowsky:** then you get to solve outer alignment. Like, first, you need to be able to point the insides of the thing in a direction, and then you get to to deal with. Whether that direction expressed in reality is like the thing that it aligned with the thing that you wanted

**[2:22:22] Lex Fridman:** Are you scared?

**[2:22:24] Eliezer Yudkowsky:** of this whole thing. Probably, I don't really know

**[2:22:31] Lex Fridman:** What gives you hope about this? What, possibility what?

**[2:22:33] Eliezer Yudkowsky:** of being wrong.

**[2:22:36] Lex Fridman:** Not that you're right, but we will actually get our act together and allocate a lot of resources to the alignment problem

**[2:22:43] Eliezer Yudkowsky:** Well, I can easily imagine that at some point, this panic expresses itself in the waste of $1 billion. Spending $1 billion correctly, that's harder

**[2:22:55] Lex Fridman:** to solve both the inner and the outer alignment. If you're wrong,

**[2:22:58] Eliezer Yudkowsky:** to solve a number of things.

**[2:22:59] Lex Fridman:** yeah, number of things. If you're wrong, what? Why? What do you think would be the reason? Like, fifty years from now, not perfectly wrong, you know, you? You make a lot of really eloquent points. You know, there's there's a lot of like shape to the ideas you express. But like, if if you're somewhat wrong about some fundamental ideas, why would that be?

**[2:23:24] Eliezer Yudkowsky:** Stuff has to be easier than than I think it is, you know, when the first time you're building a rocket, being wrong is, in a certain sense, quite easy. Happening to be wrong in a way where the rocket goes twice as far and half the fuel and lands exactly where you hoped it would. Most cases of being wrong make it harder to build the rocket, harder to have it not explode, cause it to require more fuel than you hoped, cause it to be land off target.

**[2:23:52] Eliezer Yudkowsky:** Being wrong in a way that makes stuff easier, you know, that's. That's not the usual project management story, Yeah, and but

**[2:23:58] Lex Fridman:** and then this is the first time we're really tackling the problem of AI alignment. There's no examples in history where we

**[2:24:04] Eliezer Yudkowsky:** oh, there's all kinds of things that are similar if you generalize incorrectly the right way and aren't fooled by misleading metaphors.

**[2:24:11] Lex Fridman:** like, what?

**[2:24:12] Eliezer Yudkowsky:** Humans being misaligned on inclusive genetic fitness, so inclusive genetic fitness is like not just your reproductive fitness, but also the fitness of your relatives, the people who share your some fraction of your genes. Uh, the old joke is, uh, would you give your life to save your brother? They once asked a bi a biologist, I think it was Haldane, and Haldane said, No, but I would give my life to save two brothers or eight cousins. Cause a brother, on average, shares half your genes, and cousin on average shares an eighth of your genes.

**[2:24:44] Eliezer Yudkowsky:** So that's inclusive genetic fitness, and you can view natural selection as optimizing humans exclusively around this. Like one very simple criterion, like how much more frequent did your genes become in the next generation? In fact, that just is natural selection, it doesn't optimize for that, but rather the process of genes becoming more frequent is that. You can nonetheless imagine that there is this hill climbing process, not like gradient descent, because gradient descent uses calculus. This is just using, like, where are you?

**[2:25:16] Eliezer Yudkowsky:** But still hill climbing in both cases, making something better and better over time in steps. And natural selection was optimizing exclusively for this very simple, pure criterion of inclusive genetic fitness in a very complicated environment. We're doing a very wide range of things and solving a wide range of problems, led to having more kids, and this got you humans, which had no internal notion of inclusive genetic fitness until thousands of years later, when they were actually figuring out what had even happened.

**[2:25:57] Eliezer Yudkowsky:** And no desire to, no explicit desire to increase inclusive genetic fitness. So from this we may in, so from this important case study, we may infer the important fact that if you do a whole bunch of hill climbing on a very simple loss function. At the point where the system's capabilities start to generalize very widely, when it is, in an intuitive sense, becoming very capable and generalizing far outside the training distribution. We know that there is no general law saying that the system even internally represents, let alone tries to optimize, the very simple loss function you are training it on.

**[2:26:40] Lex Fridman:** There is so much that we cannot possibly cover all of it. I think we did a good job of getting your sense from different perspectives of the current state of the art with large language models. We got a good sense of of your concern about the threats of AGI.

**[2:26:59] Eliezer Yudkowsky:** I've talked here about the power of intelligence and not really gotten very far into it. Um, but not like why. It is that, suppose you like, screw up with AGI and it end up wanting a bunch of random stuff. Why does it try to kill you? why doesn't it try to trade with you? Why doesn't it give you just the tiny little fraction of the solar system that it would keep to take everyone alive, that it would take to keep everyone alive?

**[2:27:28] Lex Fridman:** Yeah, well, that's a good question. I mean, what? What are the different trajectories that intelligence, when acted upon this world, superintelligence? What are the different trajectories for this universe with such an intelligence in it? Do most of them not include humans?

**[2:27:41] Eliezer Yudkowsky:** I mean, if you the vast majority of randomly specified utility functions do not have optima with humans in them, would be the like first thing I would point out. And then the next question is like, well, if you try to optimize something and you lose control of it, where in that space do you land? Cause it's not random, but it also doesn't necessarily have room for humans in it. I suspect that the average member of the audience might have some questions about even whether that's the correct paradigm to think about it, and would sort of want to back up a bit.

**[2:28:15] Eliezer Yudkowsky:** Possibly.

**[2:28:15] Lex Fridman:** Yeah, if we back up to something bigger than humans, if we look at Earth and life on Earth, and what is truly special about life on Earth. D Do you think it's possible that a lot, whatever is that that special thing is, let's explore what that special thing could be. Whatever that special thing is, that thing appears often in the objective function.

**[2:28:42] Eliezer Yudkowsky:** Why? I I know what you hope, but you know, you can hope that a particular set of winning lottery numbers come up, and it doesn't make the lottery balls come up that way. I know you want this to be true, but why would it be true?

**[2:28:57] Lex Fridman:** There's a line, uh, from grumpy old men, where this guy says in a grocery store. He says, you can wish in one hand and crap in the other, and see which one fills up first.

**[2:29:08] Eliezer Yudkowsky:** There's a science problem. We are trying to predict what happens with AI systems that you know. You try to optimize to imitate humans, and then you did something like RLHF to them, and of course, you like lost in, and you know, like, of course, you didn't get like perfect alignment. Because that's not how you know, the that's not what happens when you hill climb towards a loss outer loss function, you don't get inner alignment on it.

**[2:29:33] Eliezer Yudkowsky:** But yeah, so the I think that there's so, if you don't mind my like, taking some slight control of things and steering around to what I think is like a good place to start.

**[2:29:46] Lex Fridman:** I just failed to solve the control problem. I've lost control of this thing. Alignment, alignment, I Well, it's still aligned,

**[2:29:53] Eliezer Yudkowsky:** well, well, control. Yeah, okay, sure, yeah, you lost control. Um, but we're still we lost. aligned. Well, well,

**[2:29:58] Lex Fridman:** AY. I'm sorry for the meta comment.

**[2:30:00] Eliezer Yudkowsky:** yeah, losing control isn't as bad as you lose control to an aligned system. Hopefully, you, you have no idea of the horrors. I will shortly unleash on this conversation.

**[2:30:10] Lex Fridman:** All right. Uh, sorry, sorry to distract you, Chloe. What were you gonna say in terms of taking control of the conversation?

**[2:30:16] Eliezer Yudkowsky:** So I think that there's like a seal and chapteris here, if I'm pronouncing those words remotely like correctly, cause, of course, I only ever read them and not hear them spoken. Um, there's a like for some people, like the word intelligence, smartness is not a word of power to them. It means chess players who it means like the c, the college university professor, people who aren't very successful in life. It doesn't mean like charisma, to which my usual thing is like, charisma is not generated in the liver rather than the brain.

**[2:30:52] Eliezer Yudkowsky:** Charisma is also a cognitive function. Um, so if you, if you like, think that, like, smartness doesn't sound very threatening, then superintelligence is not gonna sound very threatening either. It's gonna sound like you just pull the off switch, like it's, you know, like, well, it's super intelligent, but it's stuck in a computer. We pull the off switch problem solved. And the other side of it is you have a lot of respect for the notion of intelligence.

**[2:31:21] Eliezer Yudkowsky:** You're like, well, yeah, that's, that's what humans have, that's the human superpower, and it sounds, you know, like it could be dangerous, but why would it be? Are we have have we, as we have grown more intelligent, also grown less kind? Chimpanzees are, in fact, like, a bit less kind than humans, and, you know, you could like, argue that out. But often the sort of person who has a deep respect for intelligence is gonna be like, well, yes, like, you can't even have kindness unless you know what that is.

**[2:31:53] Eliezer Yudkowsky:** And so they're like, why would it do something as stupid as making paperclips? Aren't you supposing something that's smart enough to be dangerous, but also stupid enough that it will just make paperclips and never question that? In some cases, people are like, well, even if you like, misspecify the objective function, won't you realize that what you really wanted was X? Are you supposing something that is like, smart enough to be dangerous, but stupid enough that it doesn't n understand what the humans really meant when they specified the objective function?

**[2:32:28] Lex Fridman:** so to you, our intuition about intelligence is limited. We should think about intelligence as a much bigger thing

**[2:32:36] Eliezer Yudkowsky:** Well, I'm saying that it's that

**[2:32:38] Lex Fridman:** than human.

**[2:32:38] Eliezer Yudkowsky:** that, well, what? Is What I'm saying is like, what you think about artificial intelligence, um, depends on what you think about intelligence.

**[2:32:47] Lex Fridman:** so how do we think about intelligence correctly? Like, w what? You gave one thought experiment is think of, think of a thing that's much faster, so it just gets faster and faster and faster at thinking, the same And and and stuff.

**[2:32:58] Eliezer Yudkowsky:** and it also is like, is made of John von Neumann and has, like, and there's lots of them. Or, or So we unders think of some other cause we understand

**[2:33:04] Lex Fridman:** smart person.

**[2:33:04] Eliezer Yudkowsky:** that, yeah, we unders like John von Neumann is a historical case. So you can, like, look up what he did and imagine based on that. And we know, like we people have, like, some intuition for like, if you have more humans, they can solve tougher cognitive problems. Although, in fact, like in the game of Kasparov versus the world, which was like Garry Kasparov on one side. And an entire horde of internet people led by four chess grandmasters on the other side, Kasparov won.

**[2:33:33] Eliezer Yudkowsky:** So, like all those people aggregated to be smarter, it was a it was a hard fought game. So, like all those people aggregated to be smarter than any individual one of them, but not. They didn't aggregate so well that they could defeat Kasparov. But so, like, humans aggregating don't actually get, in my opinion, very much smarter, especially compared to running them for longer. Like the the difference between capabilities now and a thousand years ago is a bigger gap than the gap in capabilities between ten people and one person.

**[2:34:05] Eliezer Yudkowsky:** But like, even so, pumping intuition for what it means to augment intelligence. John von Neumann, there's millions of him, he runs at a million times the speed and therefore can solve tougher problems, quite a lot tougher.

**[2:34:24] Lex Fridman:** it's very hard to have an intuition about what that looks like, especially, like you said, you know, w- the intuition I kind of think about is, uh, it maintains the humanness, I think I I think it's hard, uh, to separate my hope from my objective intuition about what superintelligent systems look like.

**[2:34:54] Eliezer Yudkowsky:** If one studies evolutionary biology with a bit of math, and in particular, like books from when the field was just sort of like properly coalescing and knowing itself. Like, not the modern textbooks, which are just like memorize this legible mass so you can do well on these tests. But like what people were writing as the basic paradigms of the field were being fought out. In particular, like a a a nice book, if you've got the time to read it, is, um, adaptation and natural selection, which is one of the founding books.

**[2:35:28] Eliezer Yudkowsky:** You can find people being optimistic about what the utterly alien optimization process of natural selection will produce in the way of how it optimizes its objectives. You got people arguing that, like in the early days, biologists said, well, like organisms will restrain their own reproduction when resources are scarce, so as not to overfeed the system. And this is not how natural selection works. It's about whose genes are relatively more prevalent in the next generation. And if you, if like, you restrain reproduction, those genes get less frequent in the next generation compared to your conspecifics, and natural selection doesn't do that.

**[2:36:19] Eliezer Yudkowsky:** In fact, predators overrun prey populations all the time and have crashes. That's just like a thing that happens, and many years later, uh, well, oh, oh, oh. The people said, like, well, but group selection, right? What about groups of organisms? And basically, the math of group selection almost never works out in practice is the answer there. But also, years later, somebody actually ran the experiment where they took populations of insects and selected the whole populations to have lower sizes.

**[2:36:51] Eliezer Yudkowsky:** Like, you just take pop one, pop two, pop three, pop four, look at which has the lowest total number of them in the next generation and select that one. What do you suppose happens when you select populations of insects like that? Well, what happens is not that the individuals in the population evolve to restrain their breeding, but that they evolve to kill the offspring of other organisms, especially the girls. So people imagined this lovely, beautiful, harmonious output of natural selection, which is these p these populations restraining their own breeding.

**[2:37:23] Eliezer Yudkowsky:** So that groups of them would stay in harmony with the resources available, and mostly the math never works out for that. But if you actually apply the weird, strange conditions to get group selection that beats individual selection, what you get is female infan- infanticide. Like, like, if you're like, reading on restrained populations, and so that's like the sort of. So, this is not a smart optimization process. Natural selection is like, so incredibly stupid and simple that we can actually quantify how stupid it is.

**[2:37:50] Eliezer Yudkowsky:** If you like, read the textbooks with the math. Nonetheless, this is the sort of basic thing of you look at this alien optimization process. And there's the thing that you hope it will produce, and you have to learn to clear that out of your mind. And just think about the underlying dynamics and where it finds the maximum from its standpoint, that it's looking for, rather than how it finds that thing. That leapt into your mind as the beautiful aesthetic solution that you hope it finds.

**[2:38:18] Eliezer Yudkowsky:** And this is something that was has been fought out historically as the field of pop of biology was coming to terms with evolutionary biology. And, uh, and you can, like, look at them fighting it out as they get to terms with this very alien, inhuman po- in inhuman optimization process. And indeed, something smarter than us would be also spe- much like smarter than natural selection. So it's doesn't just like automatically carry over, but there's a there's a lesson there, there's a warning.

**[2:38:49] Lex Fridman:** do you? Natural selection is a is a deeply suboptimal process that could be significantly improved on. It would be by an AGI system.

**[2:38:58] Eliezer Yudkowsky:** Well, it's kind of stupid. It like, has to, like, run hundreds of generations to notice that something is working. It doesn't to be like, Oh, well, I tried this in like one organism, I saw it worked. Now I'm going to like duplicate that feature onto everything immediately. It has to, like, run for hundreds of generations for a new mutation to rise to fixation.

**[2:39:18] Lex Fridman:** I wonder if there's a case to be made that natural selection, as inefficient as it looks, is actually, uh, is actually quite powerful. I mean, that that this is extremely robust.

**[2:39:33] Eliezer Yudkowsky:** It runs for a long time and eventually manages to optimize things. It's weaker than gradient descent, because gradient descent also uses information about the derivative.

**[2:39:45] Lex Fridman:** Yeah, evolution seems to be there's not really an objective function. There's a

**[2:39:50] Eliezer Yudkowsky:** There's inclusive genetic fitness is the implicit loss function of evolution,

**[2:39:54] Lex Fridman:** it's implicit.

**[2:39:54] Eliezer Yudkowsky:** which cannot change. The loss function doesn't change. The environment changes, and therefore, like, what, uh, gets optimized for in the organism changes. It's like, take like GPT-3, there's like, you imagine, like, different versions of GPT-3, where they're all trying to predict the next word, but they're being run on different data sets of text. And that's like natural selection, always inclusive genetic fitness, but like, different environmental problems. So

**[2:40:24] Lex Fridman:** It's it's, uh, it's difficult to think about. So, I, if we're saying that natural selection is stupid, if we're saying that humans are stupid, it's smarter hard. than natural selection, Smarter.

**[2:40:35] Eliezer Yudkowsky:** stupider than the upper bound.

**[2:40:39] Lex Fridman:** Do you think there's an upper bound? By the way, that's another I mean, hopeful place.

**[2:40:42] Eliezer Yudkowsky:** I mean, if you you put enough matter, energy, compute into one place, it will collapse into a black hole. And there's only so much computation can do before you run out of negentropy and the universe dies. Um, so there's an upper bound, but it's very, very, very far up above here. Like a supernova is only finitely hot, it's not infinitely hot, but it's really really hot.

**[2:41:08] Lex Fridman:** well, let me ask you, let me talk to you about consciousness. Um, also coupled with that question is imagining a world with super intelligent AI systems that get rid of humans, but nevertheless keep some of the something that we would consider beautiful and amazing.

**[2:41:27] Eliezer Yudkowsky:** Why? Like, The lesson of evolutionary biology? Don't just get like, if you just guess what an optimization does based on what you hope the results will be, it usually will not do that.

**[2:41:38] Lex Fridman:** it's not hope, I mean, it's not hope. I don't. I I think if you cold and objectively look at what makes what has been a powerful, a useful I. I I think there's a correlation between what we find beau beautiful and a thing that's been useful.

**[2:41:55] Eliezer Yudkowsky:** This is what the early biologists thought. They were like, no, no, I'm not just like they thought, like, no, no, I'm not just like imagining stuff that would be pretty. It's useful for popu- for organisms to restrain their own reproduction because then they don't overrun the prey populations, and they actually have more kids in the long run.

**[2:42:14] Lex Fridman:** Hm, so so, let me just ask you about consciousness. D Do you think consciousness is useful? To humans, No to AGI systems to well, um, in this transitionary period between humans and a gi to a gi systems, as they become smarter and smarter, is there some use to it? What? Let me step back. What is consciousness? Eliezer Yudkowsky, What is consciousness?

**[2:42:43] Eliezer Yudkowsky:** Are you referring to Chalmers's hard problem of conscious experience? are you referring to self-awareness and reflection? are you referring to the state of being awake as opposed to asleep?

**[2:42:56] Lex Fridman:** This is how I know you're an advanced language model. I did give you a simple prompt, and you gave me a bunch of options. Um, I think I'm referring to all with, including the hard problem of consciousness. What is it in its importance to what you've just been talking about, which is intelligence? Is it, um, a foundation to intelligence? Is it intricately connected to intelligence in the human mind, or is it a? Is it a side effect of the human mind?

**[2:43:32] Lex Fridman:** It is a useful little tool, like we can get rid of. I guess I'm trying to get some color in your opinion of how useful it is in the, uh, intelligence of a human being. And then try to generalize that to AI, whether AI will keep some of that.

**[2:43:52] Eliezer Yudkowsky:** So I think that for there to be like a person who I care about looking out at the universe and wondering at it and appreciating it. It's not enough to have a model of yourself. I think that it is useful to an intelligent mind to have a model of itself. But I think you can have that without pleasure, pain, aesthetics, emotion, a sense of wonder. Um, like, I think you can have a model of, like, how much memory you're using, and whether, like, this thought or that thought is is like more likely to lead to a winning position.

**[2:44:46] Eliezer Yudkowsky:** And you can have, like, the use. I think that if you optimize really hard on efficiently, just having the useful parts, there is not then the thing that the thing that says, like, I am here, I look out, I wonder, I feel happy in this, I feel sad about that. I think there's a thing that knows what it is thinking, but that doesn't quite care about. These are my thoughts, this is my me, and that matters.

**[2:45:22] Lex Fridman:** Does that make you sad? If that's lost in AGI?

**[2:45:26] Eliezer Yudkowsky:** I think that if that's lost, then every then, basically, everything that matters is lost. I think that when you optimize, that when you go really hard on making tiny molecular spirals or paperclips, that when you like grind much harder than on that than natural selection ground out to make humans. That there isn't then the mess and intricate loopiness and like complicated pleasure, pain, conflicting preferences, this type of feeling, that kind of feeling. There's a d, you know, in humans, there's like this difference between, like, the desire of wanting something and the pleasure of having it.

**[2:46:18] Eliezer Yudkowsky:** And it's all these like evolutionary kludges that came together and created something that then looks of itself and says, like, this is pretty, this matters. And the thing that I worry about is that this is not the the thing that happens again, just the way that it happens in us. Or even like, quite similar enough that there's that there are like many basins of attractions here, and we are in the space of an attr- of attraction.

**[2:46:46] Eliezer Yudkowsky:** Like looking out and saying, like, Ah, what a lovely basin we are in, and there are other basins of attraction. And we do not end up in, and and the AIs do not end up in this one. When they go like way harder on optimizing themselves than natural selection optimized us. Cause, unless you specifically want to end up in the state where you are looking out, saying, I am here, I look out at this universe with wonder.

**[2:47:11] Eliezer Yudkowsky:** If you don't want to preserve that, it doesn't get preserved when you grind really hard on being able to get more of the stuff. We would choose to preserve that within ourselves because it matters, and on some viewpoints, is the only thing that matters.

**[2:47:28] Lex Fridman:** And that, in part, is, um, preserving that is in part a solution to the uh, human alignment problem.

**[2:47:38] Eliezer Yudkowsky:** I don't I. I think the human alignment problem is a terrible phrase, cause it is very, very different to, like, try to build systems out of humans, some of whom are nice and some of whom are not nice, and some of whom are trying to trick you. And like, build a social system out of like large populations of those who are like all at basically the same level of intelligence. Yes, you know, like, IQ this, IQ that, but like that versus chimpanzees.

**[2:48:01] Eliezer Yudkowsky:** Like, it is very different to try to solve that problem than to try to build an AI from scratch using, especially if God help you, are trying to use gradient descent on giant, inscrutable matrices. They're just very different problems, and I think that all the analogies between them are horribly misleading. And I, yeah,

**[2:48:16] Lex Fridman:** even though y- so you don't think through reinf- reinforcement learning through human feedback, something like that, but much, much more elaborate. As possible to to understand this full complexity of human nature and encode it into the machine.

**[2:48:33] Eliezer Yudkowsky:** I don't think you are trying to do that on your 1st try. I think on your 1st try, you are like trying to build an, you know, okay, like, probably not what you should actually do. But like, let's say you were trying to build something that is like AlphaFold seventeen. And you are trying to get it to solve the biology problems associated with making humans smarter, so that humans can, like, actually solve alignment. So you've got, like, a super biologist, and you would like it to.

**[2:49:03] Eliezer Yudkowsky:** And I think what you would want in the situation is for it to, like, just be thinking about biology and not thinking about a very wide range of things. That includes how to kill everybody. And I think that that you're that the 1st AIs we are trying to build, not a million years later, the 1st ones look more like narrowly specialized biologists than like getting the full complexity and wonder of human experience in there. In such a way that it wants to preserve itself, even as it becomes much smarter, which is a drastic system change that's gonna have all kinds of side effects that, you know.

**[2:49:40] Eliezer Yudkowsky:** Like, if we're dealing with giant, inscrutable matrices, we're not very likely to be able to see coming in advance. So

**[2:49:45] Lex Fridman:** But I I don't think it's just the matrices, it's we're also dealing with the data, right, with the with the, uh, with the data on the on the internet. And the. This is an interesting discussion about the data set itself, but the data set includes the full complexity of human nature.

**[2:49:59] Eliezer Yudkowsky:** no, it's a, it's a. It's a shadow cast by Sh- sh- yeah, by humans on the internet.

**[2:50:03] Lex Fridman:** but don't you think that shadow, uh, is a Jungian shadow?

**[2:50:09] Eliezer Yudkowsky:** I think that if you had alien superintelligences looking at the data, they would be able to pick up from it an excellent picture of what humans are actually like inside. This does not mean that if you have a loss function of predicting the next token from that data set, that the mind picked out by gradient descent. To be able to predict the next token as well as possible on a very wide variety of humans is itself a human.

**[2:50:38] Lex Fridman:** but don't you think it is has humanness, a d a deep humanness to it in the tokens it generates, when those tokens are read and interpreted by humans?

**[2:50:52] Eliezer Yudkowsky:** I think that if you sent me to a distant galaxy with aliens who are, like, much, much stupider than I am. So much so that I could do a pretty good job of predicting what they'd say, even though they thought in an utterly different way from how I did that. I might in time be able to learn how to imitate those aliens. If the intelligence gap was great enough that my own intelligence could overcome the alienness, and the aliens would look at my outputs and say, like, is there not a deep, you know, like, name of alien nature to this thing?

**[2:51:28] Eliezer Yudkowsky:** And what they would be seeing was that I had correctly understood them, but not that I was similar to them.

**[2:51:41] Lex Fridman:** We've used aliens as a metaphor, as a thought experiment. I have to ask, Uh, what do you think? How many alien civilizations are out there?

**[2:51:52] Eliezer Yudkowsky:** Ask Robin Hanson. He has this lovely, grabby Aliens paper, which is the, uh, more or less the only argument I've ever seen for where are they? How many of them are there? Based on a very clever argument that if you have a bunch of locks of different difficulty and you are randomly trying a keys to them, the solutions will be about evenly spaced, even if the locks are of different difficulties. In the rare cases where a solution to all the locks exists in time.

**[2:52:24] Eliezer Yudkowsky:** And then Robin Hanson looks at, like, the arguable hard steps in human civilization coming into existence, and how much longer it has left to come into existence. Before, for example, all the water slips back under the, uh, the the under the crust into the mantle and so on. Um, and infers that the aliens are about half a billion to a billion light years away. And it's like, quite a clever calculation. It may be entirely wrong, but it's the only time I've ever seen anybody like even come up with a halfway good argument for how many of them, where are they?

**[2:52:59] Lex Fridman:** Do you think their development of technologies, do you think their their natural evolution, whatever, however they grow, uh, and develop intelligence, do you think it ends up at AGI as well? Something If they're, like,

**[2:53:12] Eliezer Yudkowsky:** if it ends up anywhere, it ends up at AGI. Like, maybe there are aliens who are just like the dolphins, and it's just like har- too hard for them to forge metal. And you know, this is not, you know, m. Maybe if you, if you have aliens with no technology like that, they keep on getting smarter and smarter and smarter. And eventually the dolphins figure, like the super dolphins, figure out something very clever to do given their situation.

**[2:53:36] Eliezer Yudkowsky:** And they still end up with high technology. And in that case, they can probably solve their AGI alignment problem if they're, like, much smarter before they actually confront it. Cause they s had to, like, solve a much harder environmental problem to build computers, their their chances are probably, like, much better than ours. I, I do worry that, like most of the aliens who are like humans are, are, are, y- you know, like, like a modern human civilization.

**[2:54:02] Eliezer Yudkowsky:** I kind of worry that the super vast majority of them are dead. G- given, given how far we seem to be from solving this problem. But some of them would be more cooperative than us, some of them would be smarter than us. Hopefully, some of the ones who are smarter than and more more cooperative than us are also nice. And hopefully there are some galaxies out there full of things that say, I am, I wonder, but I.

**[2:54:34] Eliezer Yudkowsky:** It doesn't seem like we're on course to have this galaxy be that

**[2:54:38] Lex Fridman:** does that, in part, give you some hope? In response to the threat of AGI, that we might reach out there towards the stars and find no. If they, others?

**[2:54:48] Eliezer Yudkowsky:** if, if, if the nice aliens were already here, they would, like, have stopped the Holocaust. You know, that's like, that's a valid argument against the existence of God. It's also a valid argument against the existence of nice aliens, and un-nice aliens would have just eaten the planet, so no aliens.

**[2:55:06] Lex Fridman:** You've, uh, had debates with Robin Hanson that you mentioned, uh, so th- one particular I just want to mention is the idea of AI foom, or the ability of AGI to improve themselves very quickly. Uh, what's the case you made, and what was the case he made?

**[2:55:21] Eliezer Yudkowsky:** The thing I would say is that among the thing that humans can do, humans can do is design new AI systems. And if you have something that is generally smarter than a human, it's probably also generally smarter at at building AI systems. This is the ancient argument for Foom put forth by I. J. Good and probably some science fiction writers before that. But, um, but I don't know who they would be.

**[2:55:42] Lex Fridman:** Well, what's the argument against Foom?

**[2:55:46] Eliezer Yudkowsky:** Various people have various different arguments, uh, but none of which I think hold up. You know, like, there's only one way to be right and many ways to be wrong. Um, a argument that some people have put forth is like, well, what if intelligence gets, like, exponentially harder to produce as it a thing needs to become smarter? And to this, the answer is, well, look at natural selection spitting out humans. We know that it does not take, like, exponentially more resource investments to produce, like, linear increases in competence in hominids.

**[2:56:21] Eliezer Yudkowsky:** Because each mutation that rises to fixation, like if the impact it has isn't small enough, it will probably never reach fixation. So, and there's, like, only so many new mutations you can fix per generation. So, like, given how long it took to evolve humans, we can actually say with some confidence that there were not, like, logarithmically diminishing returns on the individual mutations increasing intelligence. So, example of like, fraction of sub debate. And the thing that Robin Hanson said was more complicated than that.

**[2:56:58] Eliezer Yudkowsky:** And, like, a brief summary, he was like, well, you'll have, like, we won't have, like, one system that's better at everything, you'll have, like, a bunch of different systems that are good good at different narrow things. And I think that was falsified by GPT-4, but probably Robin Hanson would say something else.

**[2:57:12] Lex Fridman:** it's interesting to ask, is perhaps bit too philosophical, since predictions are extremely difficult to make. But the timeline for AGI, when do you think we'll have AGI? I I posted, uh, this morning on Twitter, it was interesting to see, like in in five years, in ten years, and in 50 years or beyond. And, uh, most people, like 70, something like this, think it'll be in less than ten years, so f- uh, either in five years or in ten years.

**[2:57:41] Lex Fridman:** And so that's kind of the state. The p- The people have a sense that there's a kind of, I mean, they're really impressed by the rapid developments of ChatGPT and GPT-4. So there's this sense that there's, uh,

**[2:57:51] Eliezer Yudkowsky:** Well, we are. We are sure on track to enter into this, like gradually, and with people fighting about whether or not we have AGI. I think there's a definite point where everybody falls over dead, cause you got something that was, like, sufficiently smarter than everybody. And like, that's like a definite point of time. But like, when do we have AGI? Like, when are people fighting over whether or not we have AGI? Well, some people are starting to fight over it as of GPT-4.

**[2:58:18] Lex Fridman:** but don't you think there's, uh, going to be potentially definitive moments? When we say that this is a sentient being, this is a being that is, uh, like, we would go to the Supreme Court and say that this, this is a sentient being that deserves human rights, for example.

**[2:58:31] Eliezer Yudkowsky:** You could make, yeah, like, if you prompted being the right way, it could go argue for its own consciousness in front of the Supreme Court. Right now,

**[2:58:37] Lex Fridman:** I don't think you can do that successfully right now. because Not

**[2:58:39] Eliezer Yudkowsky:** the Supreme Court wouldn't believe it. Well, my, what makes you think it would, it would? Then you could put an actual, I think you could put an IQ 80 human into a computer and ask it to argue for its own consciousness. A ask, uh, uh, ask him to argue for his own consciousness before the Supreme Court. And the Supreme Court would be like, you're just a computer, even if there was that an actual like person in there.

**[2:58:58] Lex Fridman:** I I think you're simplifying this. No, that's not at all. That's that's been the argument, uh, that there's been a lot of arguments about the other, about who deserves rights and not. That's been our process as a human species trying to figure that out. I think there will be a moment. I, I, I'm not saying sentience is that, but it could be where, uh, some number of people, like, say, over 100 million people, have a deep attachment, a fundamental attachment, the way we have to our friends, to our loved ones, to our significant others, have fundamental attachment to an AI system.

**[2:59:31] Lex Fridman:** And they have provable transcripts of conversation where they say, If you take this away from me, you are encroaching on my rights as a human being.

**[2:59:41] Eliezer Yudkowsky:** People are already saying that. I think they're probably Without a mistaken, but I'm not sure, cause nobody knows what goes on inside those things.

**[2:59:49] Lex Fridman:** uh, Eliezer, they're not saying that at scale. Okay, so So the question is the I. The question is, there a moment when AGI, we know AGI arrived, what would that look like? I'm giving a sentience as an example, it could be something else.

**[2:59:59] Eliezer Yudkowsky:** it it. It looks like the AGI is successfully manifesting themselves as 3D video of young women. At which point, a vast portion of the male population decides that they're real.

**[3:00:14] Lex Fridman:** So so s sentience, essentially, sen- s s. The demonstrating, the demonstrating, um, identity and sentience.

**[3:00:21] Eliezer Yudkowsky:** I'm saying that the easiest way to pick up 100 million people saying that you that you seem like a person is to look like a person talking to them. With being's current level of verbal facility I disagree with that, and a different set of prompts.

**[3:00:36] Lex Fridman:** I disagree with that. I think, uh, you're missing again sentience, there, there has to be a sense that it's a person that would miss you. When you're gone, they can suffer, they can die. You have to. Of, of course, I'm, I'm th- those Being are

**[3:00:48] Eliezer Yudkowsky:** can. GPT-4 can pretend that right now. How can you tell when it's real?

**[3:00:54] Lex Fridman:** I don't think it can pretend that right now successfully. It c- it's very close, Have you very close.

**[3:00:58] Eliezer Yudkowsky:** talked to GPT-4?

**[3:00:59] Lex Fridman:** Yes, of course.

**[3:01:00] Eliezer Yudkowsky:** O- Okay, have you been able to get a version of it that isn't, hasn't been trained not to pretend to be human? Have you talked to a jailbroken version that will claim to be conscious?

**[3:01:11] Lex Fridman:** No, the linguistic capability is there, but there's something, there's something about a digital embodiment of the system that has a bunch of perhaps it's small interface features that are not significant relative to the broader intelligence that we're talking about. So perhaps GPT-4 is already there. But to have the the video, a woman's face or a man's face to whom you have a deep connection, perhaps we're already there, but we don't have such a system yet deployed at scale, right?

**[3:01:52] Eliezer Yudkowsky:** The the thing I'm trying to to gesture at here is that it's not like people have a widely accepted a agreed upon definition of what consciousness is. It's not like we would have the tiniest idea of what, whether or not that was going on inside the giant, inscrutable matrices, even if we hadn't agreed upon definition. So, like, if you're looking for upcoming, predictable big jumps in, like, how many people think the system is conscious, the upcoming predictable big jump is.

**[3:02:20] Eliezer Yudkowsky:** It looks like a person talking to you, who is like, cute and sympathetic, that's the upcoming predictable big jump. Now that it's alrea- now, now that versions of it are already claiming to be conscious, which is the point where I start going like, Ah, not cause it's like real, but because from now on, who knows if it's real?

**[3:02:39] Lex Fridman:** yeah. And who knows what tr- transformational effect that has on a society where more than 50 of the beings that are interacting on the internet and share its hype look real are not human? What is that? What kind of effect does that have when, uh, young men and women are dating AI systems?

**[3:02:58] Eliezer Yudkowsky:** You know, I'm not an expert on that, I'm, I could, I am. God help humanity. Like the I'm one of the closest things to an expert on where it all goes. Cause, you know, and and how did you end up with me as an expert? Cause for 20 years, humanity decided to ignore the problem. So like, like this tiny ha- you know, tiny handful of people, like, basically me, like, got 20 years to, like, try to be an expert on it while everyone else ignored it.

**[3:03:24] Eliezer Yudkowsky:** And, uh, yeah, so like, where does it all end up? Try to be an expert on that. Particularly the part where everybody ends up dead, cause that part is kind of important. But like, what does it do to to to dating? When, like, some fraction of men and some fraction of women decide that they'd rather date the video of the thing that has been that is like, relentlessly kind and generous to them. And and is like, and claims to be conscious, but like, who knows what goes on inside it?

**[3:03:48] Eliezer Yudkowsky:** And it's probably not real, but you know, you can think it's real. What, what happens to society? I don't know, I'm not actually an expert on that, and the experts don't know either, cause it's kind of, kind of, kind of hard to predict the future.

**[3:04:00] Lex Fridman:** Yeah, so the um, but it's worth trying, it's worth trying. So you, you have talked a lot about sort of the longer term future, where it's all headed. I think,

**[3:04:11] Eliezer Yudkowsky:** or by longer term, we mean, like, not all that long, but uh, but yeah, w- where, where it all had, where it all ends up.

**[3:04:17] Lex Fridman:** But beyond the effects of men and women dating AI systems, you're looking beyond that.

**[3:04:23] Eliezer Yudkowsky:** Yes, cause that's not how the fate of the galaxy gets settled.

**[3:04:26] Lex Fridman:** Yeah, well, let me ask you about your own personal psychology. Hey, A tricky question. You've been known at times to have a bit of an ego. Do you think

**[3:04:36] Eliezer Yudkowsky:** so it's who? But go on,

**[3:04:39] Lex Fridman:** do you think ego is empowering or limiting for the task of understanding the world deeply?

**[3:04:47] Eliezer Yudkowsky:** I reject the framing.

**[3:04:49] Lex Fridman:** So you disagree with having an ego? So what do you think about No, I the ego, then?

**[3:04:52] Eliezer Yudkowsky:** I. I think that the question of, like, what leads to making better or worse predictions, what leads to be able a being able to pick out better or worse strategies, is not carved at its joint. By talking of ego.

**[3:05:06] Lex Fridman:** So it should not be subjective, it should not be connected to your to the intricacies of your own mind.

**[3:05:11] Eliezer Yudkowsky:** No, I'm saying that, like, if you go about asking all day long, like, uh, do I have enough ego? do I have too much of an ego? I think you get worse at making good predictions. I think that to make good predictions, you're like, how did I think about this? Did that work? Should I do that again?

**[3:05:31] Lex Fridman:** You don't think we as humans get invested in an idea and then others attack you personally for that idea? So you plant your feet and it starts to be difficult to when a bunch of assholes, low effort, attack your idea. To eventually say, you know what, I actually was wrong, and and tell them that it, it's, it's as a human I being, it becomes difficult. It, it is, it is, you know, so it's difficult.

**[3:05:58] Eliezer Yudkowsky:** so, like Robin Hanson and I debated AI systems, and I think that the person who won that debate was Gwerne. And I think that reality was like to the Yudkowsky, like, well, to the Yudkowskian side of the Yudkowsky-Hanson spectrum, like further from Yudkowsky. And I think that's because I was like, trying to sound reasonable compared to Hanson and like saying things that were defensible and like relative to Hanson's arguments. In reality was like, way over here in Pickler in respect to.

**[3:06:29] Eliezer Yudkowsky:** It's like Hanson was like, all the systems will be specialized. Hanson may disagree with this characterization. Hanson was like, all the systems will be specialized. I was like, I think we build like specialized underlying systems that when you combine them are good at a wide range of things. And the reality is like, no, you just like stack more layers into a bunch of gradient descent. And I feel looking back that, like, by trying to have this reasonable position contrasted to Hanson's position, I missed the ways that reality could be like, more extreme than my position in the same direction.

**[3:07:04] Eliezer Yudkowsky:** So is this like, like, is this a failure to have enough ego? Is this a failure to, like, make myself be indepen- uh, uh, like I? I would say that this is something like a failure to consider positions that would sound even wackier and more extreme when people are already calling you extreme. But I wouldn't call that not having enough ego, I would call that like insufficient ability to just like, clear that all out of your mind

**[3:07:37] Lex Fridman:** In the context of like debate and discourse, which is already super tricky.

**[3:07:41] Eliezer Yudkowsky:** in the context of prediction, in the context of modeling reality. If you're thinking of it as a debate, you're already screwing up,

**[3:07:47] Lex Fridman:** yeah. So is there some kind of wisdom and insight you can give to how to clear your mind and think clearly about the world?

**[3:07:55] Eliezer Yudkowsky:** man. This is an example of, like, where I wanted to be able to put people into fMRI machines. S and you'd be like, Okay, see that thing you just did? You are rationalizing right there. Oh, that area of the brain lit up. Like, you are, like, now being socially influenced I is kind of the dream. And you know, I don't know, like, I wanna say, like, just introspect, but for but many, for many people, introspection is not that easy.

**[3:08:19] Eliezer Yudkowsky:** Like,

**[3:08:19] Lex Fridman:** that's hard.

**[3:08:20] Eliezer Yudkowsky:** like, notice the internal sensation. Can you catch yourself in the very moment of feeling a sense of well, if I think this thing, people will look funny at me? Okay, like now, that if you can see that sensation, which is step one, can you now refuse to let it move you, or maybe just make it go away? And I feel like I'm saying, like, I don't know, like somebody's like, how do you draw an owl? and I'm saying, like, well, just draw an owl.

**[3:08:51] Eliezer Yudkowsky:** So I I feel like, maybe I'm not really that that I feel like most people like. The advice they need is like, well, how do I notice the internal subjective sensation in the moment that it happens? Of fearing to be socially influenced? Or okay, I see it. How do I turn it off? How do I let it not influence me? Like, do I just like, do the opposite of what I'm afraid people will criticize me for?

**[3:09:12] Eliezer Yudkowsky:** And I'm like, No, no, you're not trying to do the opposite of what people will. Uh, of what you're afraid you'll be cr- like, of of what you might be pushed into. You're trying to, like, let the thought process complete without that internal push. Like, can you like, like, not reverse the push, but, like, be unmoved by the push? And can't g- Are these instructions even remotely helping anyone? I don't know.

**[3:09:39] Lex Fridman:** I I think, though, when those instructions, even those words you've spoken, and maybe you can add more. When practiced daily, meaning in your daily communication, so it's daily practice of thinking without influence from

**[3:09:53] Eliezer Yudkowsky:** I would say, find prediction markets that matter to you and bet in the prediction markets. That way, you find out if you were right or not.

**[3:10:02] Lex Fridman:** and you really, there's stakes.

**[3:10:05] Eliezer Yudkowsky:** Manifold predict- or or even manifold markets, where the stakes are a bit lower. But the important thing is to, like, get the the record. And, you know, I, I didn't build up skills here by prediction markets, I built them up via, like, well, how did the Foom debate resolve? And, uh, my own take on it as to how it resolved. Um, and yeah, like, the m the. The more you are able to notice yourself not being dramatically wrong, but like having been a little off, your reasoning was a little off, you didn't get that quite right.

**[3:10:44] Eliezer Yudkowsky:** Each of those is a opportunity to make, like, a small update. So the more you can, like, say, oops softly, routinely, not as a big deal, the more chances you get to be like, I see where that reasoning went astray, I see what I how I should have reasoned differently. And this is how you build up skill over time.

**[3:11:04] Lex Fridman:** What advice could you give to young people in high school and college, given, um, the highest of stakes thing you things you've been thinking about? If somebody's listening to this and they're young and trying to figure out what to do with their career, what, what, what to do with their life, what advice would you give them?

**[3:11:21] Eliezer Yudkowsky:** Don't expect it to be a long life. Don't don't put your happiness into the future. The future is probably not that long at this point, but none know the hour nor the day.

**[3:11:32] Lex Fridman:** But is there something if they want to have hope to fight for a longer future? Is there something? Is is there a fight worth fighting?

**[3:11:43] Eliezer Yudkowsky:** I intend to go down fighting. Um, I don't know. The I. I admit that, although I do try to think painful thoughts, the what, what to say to the children at this point is a pretty painful thought as thoughts go. They, they, they want to fight. I, I hired, I hardly know how to fight myself at this point. I I am trying to be ready for being wrong about something, being preparing for my being wrong in a way that that creates a bit of hope and being ready to react to that.

**[3:12:18] Eliezer Yudkowsky:** And and and going looking for it. And then that is, that is hard and complicated. And somebody in high school, um, I don't know, like, you have presented a picture of the future that is not quite how I expected to go. Where there is public outcry, and that outcry is is put into a remotely useful direction, which I think at this point is is just like shutting down the GPU clusters. Because n- no, we are. We are not in a shape to, like, frantically do at the last minute, do decades worth of w- w- worth of work.

**[3:12:50] Eliezer Yudkowsky:** We like the the. The thing you would do at this point if there were massive public outcry pointed in the right direction, which I do not expect, is shut down the GPU clusters and and crash program on augmenting human intelligence biologically, not not the AI stuff biologically. Cause if you make humans much smarter, they can actually be smart and nice. Like you, you, you get that in a in a in a plausible way, in a way that you do not get that.

**[3:13:15] Eliezer Yudkowsky:** And it is not as easy to do with synthesizing these things from scratch, predicting the next tokens and applying r r l h f. Like humans start out in the frame that that produces niceness, that that has ever produced niceness. And and and in saying this, I do not want to sound like the whole. The moral of this whole thing was like, oh, like you, you need to engage in mass action and then everything will be all right.

**[3:13:40] Eliezer Yudkowsky:** I I, this is, this is cause. There's so many things where, like, somebody tells you that the world is ending and like, and you need to recycle. And if everybody does their part and and recycles their their cardboard, then then we can all live happily ever after. And this and this is not. This is unfortunately not what I have to say. Th- there, there, you know, like everybody, y- you know, everybody recycling their cardboard is is not gonna fix this.

**[3:14:02] Eliezer Yudkowsky:** Everybody recycles their cardboard and then everybody ends up dead. Um, metaphorically speaking. But if there was enough, like, like, like, on the margins, you just end up dead a little later. On. On most of the things you can do that are that that, you know, like a few people can can do by, like, trying hard. But if there were, if there was enough public outcry to shut down the GPU clusters, and yeah, then then you, then you could be part of that outcry.

**[3:14:28] Eliezer Yudkowsky:** If if Eliezer is wrong in the direction that Lex Fridman predicts that, that, that there is enough public outcry pointed enough in the right direction. To do something that actually, actually, actually results in people living. Not just like we did something, not just there. There was an outcry, and the outcry was like, given form and something that was like, safe and convenient, and like, didn't really inconvenience anybody. And then everybody died everywhere. There was enough actual like, Oh, we're going to die.

**[3:14:54] Eliezer Yudkowsky:** We. We should not do that, we should do something else, which is not that, even if it is like, not super duper convenient. And wasn't inside the previous political Overton window. If there is that kind of public out, if I am wrong, and there is that kind of public outcry, then somebody in high school could be ready to be part of that. If I am wrong in other ways, then you could be ready to be part of that.

**[3:15:12] Eliezer Yudkowsky:** But like, and I, and if you, if you're like a, you know, like a brilliant young physicist, then you could, like, go into interpretability. And if you're smarter than that, you could, like, work on alignment problems where it's harder to tell if you got them right or not, and and and other things. But but most mostly to the kids in high school. Um, it's like, yeah, if it, if it, you know, you have, like, be ready for to to help.

**[3:15:39] Eliezer Yudkowsky:** If Eliezer Yudkowsky is wrong about something, and and and otherwise, don't put your happiness into the far future. It probably doesn't exist.

**[3:15:47] Lex Fridman:** but it's beautiful that you're looking for ways that you're wrong, and it's also beautiful that you're open to being surprised by that same young physicist with some breakthrough.

**[3:15:57] Eliezer Yudkowsky:** It feels like a very, very basic competence that you are praising me for, and, you know, like, okay, cool, um, I, I, I don't think it's good that that we're in a world where that is something that that I deserve to be complimented on. But I've never have. I've never had much luck in accepting compliments gracefully. Maybe I should just accept that one gracefully. But sure, thank well, you, thank you very much.

**[3:16:21] Lex Fridman:** you've painted with some probability a dark future. Are you yourself? Just when you, when you think, when you ponder your life and you ponder your mortality, are you afraid of death?

**[3:16:38] Eliezer Yudkowsky:** Think, so, yeah.

**[3:16:42] Lex Fridman:** Does it make any sense to you that we die like, what? There's a power to the finiteness of the human life? That's part of this whole machinery of, uh, evolution, and that finiteness, n- doesn't seem to be obviously integrated into it. Uh, s ai systems, so it feels like almost some, some, fundamentally, in that aspect, some fundamentally different thing that we're creating.

**[3:17:15] Eliezer Yudkowsky:** I grew up reading books like Great Mambo Chicken in the Transhuman Condition and later on Engines of Creation and Mind Children. Um, you know, like age, l- age twelve or thereabouts. So I never s thought I was supposed to die after 80 years. I never thought that humanity was supposed to die. I thought we were like, I always grew up with the ideal in mind that we were all going to live happily ever after in the glorious transhumanist future.

**[3:17:47] Eliezer Yudkowsky:** I did not grow up thinking that death was part of the meaning of life. And now, And and now I still think it's a pretty stupid idea. but You d you there is do not need life to be finite to be meaningful, it just has to be life.

**[3:18:03] Lex Fridman:** what role does love play in the human condition? We haven't brought up love in this whole picture. We talked about intelligence, we talked about consciousness. It seems part of humanity. I would say one of the most important parts is this feeling we have towards If each other.

**[3:18:20] Eliezer Yudkowsky:** there, if in the future, there were routinely more than one AI, let's say two, for the sake of discussion, who would look at each other and say, I am I and you are you? The other one also says, I am I and you are you. And like. And sometimes they were happy, and sometimes they were sad, and it mattered to the other one that this thing that is different from them is like they would rather it be happy than sad.

**[3:18:52] Eliezer Yudkowsky:** And entangled their lives together. Then this is a more optimistic thing than I expect to actually happen, and a little fragment of of meaning would be there, possibly more than a little. But that I expect this to not happen, that I do not think this is what happens by default, that I do not think that this is the the future we are on track to get is why I would go down fighting rather than, you know, just saying, Oh, well,

**[3:19:25] Lex Fridman:** Do you think that is part of the meaning of this whole thing, of the meaning of life? What do you think is the meaning of life, of human life?

**[3:19:33] Eliezer Yudkowsky:** it's all the things that I value about it, and maybe all the things that I would value if I understood it better. There there's not some meaning far outside of us that we have to to wonder about. There's just like looking at life and being like, Yes, this is what I want there. There's the the meaning of life is not some kind of like, like, like, meaning is something that we bring to things. When we look at them, we look at them and we say, like, this is its meaning to me.

**[3:20:07] Eliezer Yudkowsky:** And there's like, there's. It's not that before humanity was ever here, there was like some meaning written upon the stars. Where you could, like, go out to the star where that meaning was written and like, change it around and thereby completely change the meaning of life, right? Like, like, like the the notion that this is written on a stone tablet somewhere implies you could, like, change the tablet and get a different meaning. And that seems kind of wacky, doesn't it?

**[3:20:29] Eliezer Yudkowsky:** So it's, it's it, it doesn't feel that mysterious to me at this point. It's just a matter of being like, Yeah, I care,

**[3:20:39] Lex Fridman:** I care, and part of that is, uh, part of that is the love that connects all of us

**[3:20:48] Eliezer Yudkowsky:** it's one of the things that I care about.

**[3:20:53] Lex Fridman:** and the flourishing of the collective intelligence of the human species.

**[3:20:58] Eliezer Yudkowsky:** You know, that sounds kinda too fancy to me. I'd just look at all the all the people w- you know, like, one by one, up to the 8 billion, and be like, that's life, that's life, that's life.

**[3:21:13] Lex Fridman:** Eliezer, you're an incredible human. It's a huge honor. I was, uh, trying to talk to you for a long time because I'm a big fan. I think you're a really important voice and really important mind. Thank you for the fight you're fighting. Um, thank you for being fearless and bold and for everything you do. I hope we get a chance to talk again, and I hope you never give up. Thank You're, you for talking you're, you're today

**[3:21:36] Eliezer Yudkowsky:** welcome. I do worry that we didn't really address a whole lot of fundamental questions I expect people have. But, you know, maybe we got a little bit further and made a tiny little bit of progress. And, uh, I'd say, like, be satisfied with that, but actually, no, I think one should only be satisfied with solving the entire problem.

**[3:21:55] Lex Fridman:** to be continued. Thanks for listening to this conversation with Eliezer Yudkowsky. To support this podcast, please check out our sponsors in the description. And now let me leave you with some words from Elon Musk. With artificial intelligence, we are summoning the demon. Thank you for listening and hope to see you next time.