Person
1 episodes · 22 quoted claims
Eliezer Yudkowsky read from the 1956 proposal for a two-month, ten-person study of artificial intelligence at Dartmouth College in Hanover, New Hampshire.
“We propose that a two month, ten man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College in Hanover, New Hampshire.”
Yudkowsky said the GPT models were originally well calibrated, meaning an 80% answer was right about eight times in ten, and that reinforcement learning from human feedback has reportedly worsened this.
“it used to be well calibrated. If you trained it to put probabilities on things, it would say 80% probability and be right eight times out of ten.”
Yudkowsky said he believes the name of his blog LessWrong was suggested by Nick Bostrom and derives from an epigraph.
“The name Less Wrong was, I believe, uh, suggested by Nick Bostrom, and it's after someone's epigraph”
Yudkowsky said GPT-4 was probably produced by applying about twenty times more computing power to roughly the same architecture as GPT-3.
“taking what is probably basically the same architecture in GPT three and throwing twenty times as much compute at it, probably, and getting out GPT four.”
Yudkowsky said researchers do understand why ReLU activation functions work much better than sigmoid functions in neural networks.
“we do actually understand why the ReLUs make a big difference compared to sigmoids”
Yudkowsky described putting up a prediction market on whether by 2026 researchers will have understood anything inside a large transformer network that was not known in 2006.
“putting up a prediction market on whether, um, in 2026, we will have understood anything that goes on inside a giant transformer net that was not known to us in 2006.”
Yudkowsky cited the discovery of induction heads inside transformer networks as an example of interpretability progress.
“we have now, uh, understood induction heads in these systems by dint of much research and great sweat and triumph.”
Yudkowsky used sending an air conditioner schematic back in time as an analogy for how a smarter AI could produce results humans follow step by step but cannot explain.
“I mean that if you send a schematic for the air conditioner back in time, they can see exactly what you're telling them to do.”
Yudkowsky said the original paperclip maximizer scenario was about losing control of an AI's utility function, with the best-scoring outcome being tiny molecular shapes resembling paperclips.
“the original version was you lose control of the utility function. And it so happens that what maxes out the utility per unit resources is tiny molecular shapes like paperclips.”
Yudkowsky said that in retrospect he wishes he had described tiny molecular spirals rather than paperclips, because the paperclip version mutated into a story about a paperclip factory.
“In retrospect, I wish I'd said tiny molecular spirals, or like tiny molecular hyperbolic spirals.”
Yudkowsky recounted the joke attributed to J. B. S. Haldane that he would give his life to save two brothers or eight cousins, illustrating inclusive genetic fitness.
“Haldane said, No, but I would give my life to save two brothers or eight cousins.”
Yudkowsky defined inclusive genetic fitness as covering both an individual's reproductive success and that of relatives who share their genes.
“inclusive genetic fitness is like not just your reproductive fitness, but also the fitness of your relatives”
Yudkowsky described natural selection as slow, needing hundreds of generations for a new mutation to become fixed in a population.
“It has to, like, run for hundreds of generations for a new mutation to rise to fixation.”
Yudkowsky said an experiment selecting insect populations for smaller size produced individuals that killed other organisms' offspring, especially females, rather than restraining their breeding.
“what happens is not that the individuals in the population evolve to restrain their breeding, but that they evolve to kill the offspring of other organisms, especially the girls.”
Yudkowsky recommended the book Adaptation and Natural Selection as one of the founding books of modern evolutionary biology.
“a a nice book, if you've got the time to read it, is, um, adaptation and natural selection, which is one of the founding books.”
Yudkowsky cited the Kasparov versus the World game, in which Garry Kasparov beat a crowd of internet players led by four chess grandmasters, to argue that aggregating many humans adds little intelligence.
“in the game of Kasparov versus the world, which was like Garry Kasparov on one side. And an entire horde of internet people led by four chess grandmasters on the other side, Kasparov won.”
Yudkowsky said Robin Hanson's grabby aliens argument infers that alien civilizations are about half a billion to a billion light years away.
“infers that the aliens are about half a billion to a billion light years away”
Yudkowsky attributed the argument that a smarter-than-human AI would rapidly improve itself, called 'Foom', to the statistician I. J. Good.
“This is the ancient argument for Foom put forth by I. J. Good and probably some science fiction writers before that.”
Yudkowsky summarized Robin Hanson's position in their Foom debate as predicting many specialized AI systems rather than one system better at everything, and said GPT-4 falsified it.
“he was like, well, you'll have, like, we won't have, like, one system that's better at everything, you'll have, like, a bunch of different systems that are good good at different narrow things.”
Yudkowsky recounted talking years earlier with an alarmed congressperson who wanted to fund AI without emotions, a proposal he said misunderstood where the danger comes from.
“I was talking to a congressperson, um, who had become alarmed about the eventual prospects. And he wanted work on building AIs without emotions, cause the emotional AIs were the scary ones, you see.”
Yudkowsky described a reported exchange in which Bing identified solanine poisoning from green potatoes and urged a parent to call an ambulance.
“and Bing is like, that's solanine poisoning, and like, call an ambulance, and the person's like, I can't afford an ambulance”
Yudkowsky said that for about twenty years a tiny handful of people, including himself, worked on AI risk while the rest of humanity ignored it.
“Cause for 20 years, humanity decided to ignore the problem.”