2023
Joseph Kim says AI hallucination was a frequent concern in 2022 and 2023 but comes up much less often now.
“the hallucination used to be a big term that used to come up in 2022, 2023, that doesn't come up nearly as often nowadays.”
Topic
Joseph Kim
14 facts from 1 episode, in date order. These are what speakers said on the shows, each linked to the moment it was said. They aren't independently verified.
Undated
Joseph Kim says about 95% of ChatGPT for financial services could have been built by a startup or another company.
“95% of what we built was something that absolutely a startup could do, uh, another company could do”
Undated
Joseph Kim says Codex was the first OpenAI product where the product, model, harness and research were all developed together under one roof.
“it was the first time we developed a product where the product, the model, harness, and the the research was all kind of under one house and on one roof”
Undated
Joseph Kim's team worked from the thesis that Codex gets a product about 80% of the way to market success and that the missing 20% is pivotal.
“the thesis of my team was Codex was Codex will get you about 80% of the way to a product that would be successful in the market, but, you know, um, the 20% is missing.”
Undated
Joseph Kim predicts that a billion-dollar business will be built on top of a ChatGPT plugin.
“I really do think that there's gonna be a billion-dollar business built on top of a, uh, ChatGPT plugin, right?”
Undated
Joseph Kim describes OpenAI's launched finance product as a productivity tool for knowledge workers that assumes a human is always reviewing the work.
“the existing product that we launched is a productivity tool for knowledge workers working in the space.”
Undated
Joseph Kim says that when he worked at Google, engineers needed a certification called readability before changing code.
“when I was at Google, you need, you, you need something called readability.”
Undated
Joseph Kim says investment banking already has a layered review system in which junior analysts' work is reviewed by associates, then VPs, directors and managing directors.
“there's already this system in place for junior analysts doing a lot of the grunt work. Uh, their work gets passed off to associates, associates are reviewing the work”
Undated
Joseph Kim says coding combines verifiable rewards, such as whether code compiles, with human taste, such as clean system design.
“coding is a combination of verifiable rewards and human taste, right?”
Undated
Joseph Kim says financial work such as Excel models has verifiable checks, like whether totals sum and whether there are reference errors.
“I have an Excel model, does it sum up? Are there reference errors? There's a whole bunch of verifiable things that we can check”
Undated
Joseph Kim says OpenAI's GDP eval benchmark is saturated, with its unsaturated parts effectively broken, so OpenAI no longer reports or tracks it.
“we know that the eval is completely saturated, uh, and that it's, uh, there's a, and the pieces that are not saturated are essentially broken, right?”
Undated
Joseph Kim's team is building profession-specific evals, such as an investment banking eval meant to predict whether a model could automate much of a junior analyst's work.
“we'll have something like an investment banking eval that predicts, you know, if you score well on this eval, you could probably automate a lot of the work of a junior analyst, uh, investment banking.”
Undated
Joseph Kim's team at OpenAI is hiring subject matter experts in finance to work with engineers, researchers and product managers.
“the other role that I wanna give a special c shout-out to is we're hiring something called a subject matter expert.”
Undated
Joseph Kim says OpenAI's coding progress benefited from the people training the models knowing what good code looks like, an advantage it lacks in finance.
“the people that were training the models and making them better understood what good code looks like, right?”
Episodes