2017
Amodei recalls that GPT-1 was trained on a narrow body of literary text, which generalized poorly, whereas GPT-2 used a broad internet scrape.
“GPT-1 itself was trained on a bunch of, I think it was fan fiction actually.”
Topic
5 facts from 5 episodes, in date order. These are what speakers said on the shows, each linked to the moment it was said. They aren't independently verified.
2017
Amodei recalls that GPT-1 was trained on a narrow body of literary text, which generalized poorly, whereas GPT-2 used a broad internet scrape.
“GPT-1 itself was trained on a bunch of, I think it was fan fiction actually.”
2019
Elad Gil said Microsoft made its first significant investment in OpenAI around the time GPT-2 launched, before GPT-3.
“Microsoft made its first investment, or its sig- its first significant investment in OpenAI right after GPT two launched”
2019
Tri Dao says OpenAI committed to scaling after GPT-2, which had 1.5 billion parameters and showed strong capabilities.
“That was the bet that they made after, I think, GPT-2. So they saw that scaling these things up to, back then was 1.5 billion parameter, seemed to give you amazing capabilities.”
2019
After GPT-2 was released, Gomez called his future co-founders Nick and Ivan and proposed building large language models.
“I called up Nick, I called up Ivan, my co-founders, and I said, You know, maybe we should figure out how to build these things.”
Undated
Chris Van Pelt said GPT-2 seemed cool but toy-like to him, and he did not initially see its practical uses.
“I thought it was cool, but I also thought, like, all right, well, what can you do with this?”
Episodes