Topic
40 facts from 26 episodes, in date order. These are what speakers said on the shows, each linked to the moment it was said. They aren't independently verified.
2020
Lex Fridman says the viral claim that GPT-4 had 100 trillion parameters came from a presentation of his, given when GPT-3 was released, that was taken out of context.
“I said, like, GPT four, like the next as it progresses, what I should have said is GPT n or something.”
2022
Sam Altman says OpenAI finished training GPT-4 in the summer before its release and immediately gave it to outside red teamers to probe for harms.
“we finished last summer. Um, we immediately started giving it to people to, uh, to red team.”
2023
Ethan Mollick expects OpenAI to release GPT-4, a model much larger than the GPT-3.5 models behind ChatGPT.
“there is going to be elements of what's called GPT four release, which is a model that's much larger than the ChatGPT GPT 3.5 models we've been playing with”
2023
With GPT-4, OpenAI introduced the system message, a way for users to steer the model's behavior that is separate from RLHF.
“We launched this thing with GPT four called the system message, um, which is not RLHF, but is a way to let users have a good degree of steerability over what they want.”
2023
Sam Altman says the interview took place six days after GPT-4 launched.
“Today's Monday, we launched the previous Tuesday, so it's been six days.”
2023
Sam Altman says he considers consciousness strange but does not claim GPT-4 is conscious.
“I still think it's something very strange”
2023
Yudkowsky said GPT-4 was probably produced by applying about twenty times more computing power to roughly the same architecture as GPT-3.
“taking what is probably basically the same architecture in GPT three and throwing twenty times as much compute at it, probably, and getting out GPT four.”
2023
Lex Fridman said Sam Altman had asked him how open OpenAI should be about GPT-4 and whether he would open-source the code.
“He turned the question to me of how open should OpenAI be about GPT four? Would you open source the code?”
2023
Kevin Scott said GPT-4's visual input capability was not yet widely available but would reach wide distribution.
“openAI doesn't have GPT V and wide distribution, but like, it'll get to wide distribution at some point in the not too distant future.”
2023
Intercom charges 99 cents per resolution for Fin, and Des Traynor attributes the price to the cost of GPT-4.
“So GPT four is expensive, and as a result, Fin is perceived to be expensive. We charge ninety nine cents a resolution.”
2023
Jerry Liu describes evaluating answers by prompting GPT-4 to score a generated response from 1 to 5 against a ground-truth response.
“you can basically prompt, uh, GPT-4 to output a score between 1 to 5 on how much your predicted response, like the generated response, matches the ground truth response”
2023
CaseText received early access to GPT-4 from OpenAI before its public release.
“we got really early access to GPT four”
2023
Heller says OpenAI was coy about the model it gave CaseText early on and described it as a new version of GPT-3, which he thinks was called Davinci three internally.
“they called it Davinci three internally or something like that.”
2023
Heller says CaseText ran an unpublished bar exam on GPT-4, and it did better than at least 70 to 90 percent of human test takers.
“So we knew it wasn't part of the training data and it like, did better than at least like 70 to 90 of of human test takers”
2023
Snowflake invested in building its own foundation model, Arctic, after it appeared many teams could catch up to GPT-4.
“late last year, it looked very much like many teams could catch up to GPT four. That's when we invested in it”
2023
Wade Foster recalls that GPT-4 came out roughly six months after GPT-3.5.
“So first, it came out roughly six months, I think, after 3.5.”
2024
Ethan Mollick predicted that GPT-5-class models would appear in summer 2024 and significantly improve on GPT-4.
“I would suspect that this summer of 2024, we will see GPT five class models that will represent a significant improvement over GPT four.”
Undated
Yudkowsky said the GPT models were originally well calibrated, meaning an 80% answer was right about eight times in ten, and that reinforcement learning from human feedback has reportedly worsened this.
“it used to be well calibrated. If you trained it to put probabilities on things, it would say 80% probability and be right eight times out of ten.”
Undated
Yudkowsky summarized Robin Hanson's position in their Foom debate as predicting many specialized AI systems rather than one system better at everything, and said GPT-4 falsified it.
“he was like, well, you'll have, like, we won't have, like, one system that's better at everything, you'll have, like, a bunch of different systems that are good good at different narrow things.”
Undated
Huang says one breakthrough of GPT-4 is that it can learn from both language and images at the same time.
“One of the one of the ones is, uh, the fact that it can learn from both language and images at the same time.”
Undated
Stitch Fix rolled out an optional stylist note-writing template built on OpenAI's GPT-4.
“And we've rolled out a new feature with OpenAI's GPT four that enables a stylist to choose from a sort of starting place, a template.”
Undated
Codeium found GPT-4 more steerable than GPT-3.5 for code edits, but its per-token latency was too high, so it trained its own edit model.
“And then what we did instead was we later used GPT four because it it actually is more steerable, but then the latency on a per token basis is too high for users to want to use the product.”
Undated
Heller says CaseText had the model output a relevance score before its reasoning, which they found improved performance.
“you can have it output the number 1st and then provide its reasoning.”
Undated
CaseText used a copy of GPT-4 that had not been fine-tuned for most of its tasks.
“we've been using a, uh, not fine tuned copy of, uh, GPT four for the vast majority of the tasks that we do.”
Undated
Heller says CaseText had not found Meta's Llama 70B model able to outperform GPT-4 on its tasks, though it came close.
“we haven't seen, you know, uh, say, a Llama 70B outperform GPT four in a task, get close, but not outperform GPT four.”
Undated
Bret Taylor says conversational interfaces passed an inflection point of effectiveness, probably with GPT-4.
“Now that conversational interfaces work effectively, and I just think we passed that inflection point, probably with GPT-4, though that's a interesting debate topic on its own.”
Undated
Lütke said that even if GPT-4 were frozen as it was before o1, roughly a trillion dollars of industry value could be extracted over ten years.
“then I think there's probably $10 1 trillion of value for industry that can be pulled out of just that for over the next ten years.”
Undated
Bret Taylor credits the creation of GPT-3, GPT-4 and ChatGPT, built by people including some who have left, as the reason OpenAI exists in its current form.
“I wouldn't be in this chair if not for the creation of GPT three, GPT four, ChatGPT, and it's transformed society.”
Undated
Gomez says a model matching GPT-4 on enterprise-relevant tasks can now be built for roughly $10 to 20 million, far less than GPT-4 cost to develop.
“today you can build a model that's as good as GPT-4 in all the things that enterprises might care about. For 10 million, 20 million, like just orders of magnitude less than what was spent to develop that model.”
Undated
Sarah Friar says GPT-3 could handle roughly a 5-second task and GPT-4 roughly a 5-minute task.
“GPT three could handle maybe a 5 s task, GPT four could probably handle a 5 min task.”
Undated
Ethan Mollick said GPT-4 had become better than any intern a manager could hire.
“Well, what just happened over the last summer now is GPT four is better than any intern you're going to hire, right?”
Undated
A Boston Consulting Group study involving Mollick and academics from Harvard, MIT and Warwick found a 40% quality improvement for people using GPT-4.
“40% improvement in quality for people who used GPT-4 versus not, um, big speed improvements”
Undated
Geoffrey Hinton says he asked GPT-4, without web access, why a compost heap is like an atom bomb, to test whether it could see analogies.
“at the point when G P T four couldn't look on the web, I asked it, Why is a compost heap like an atom bomb?”
Undated
Kenny Song says early ChatGPT models like GPT-3.5 and GPT-4 answered from a single forward pass, whereas newer models search the internet and reason.
“Originally it was just like three, five and GPT four, that, uh, just answer a question based on a forward pass of the model.”
Undated
Harvey gained access to GPT-4 when it first raised money from OpenAI.
“when we 1st raised from OpenAI, we got access to GPT four, and I think GPT three to GPT four was such a big model jump.”
Undated
On the first day of GPT-4 access, Harvey co-founder Winston Weinberg spent 14 hours redoing his associate tasks with the model.
“Winston went in his room for 14 h and just redid a bunch of his associate tasks.”
Undated
Elad Gil recalled that Harvey's demos worked on GPT-4 but not GPT-3.5.
“I remember, are you showing side by side, GPT three point, five versus four? And what you were doing worked on four, but not on 3.5.”
Undated
Reid Hoffman says that in a blind test with Indian friends, the top half of GPT-4 poems were those written in English and then translated into Hindi.
“literally the top half were the one that were written in English and translated to Hindi”
Undated
Bar Kogan says AutoGPT did not work well because models such as GPT-4 were not yet good enough.
“Now it granted, it didn't work that well, it was too early, the models were not good enough, uh, GPT four was not good enough.”
Undated
Ethan Mollick says consultants' claimed AI success stories are often based on GPT-4, a model he calls obsolete.
“if they have successful cases, it's for GPT four, a model that's obsolete and no one would use for anything anymore”
Episodes