Joseph Kim says about 95% of ChatGPT for financial services could have been built by a startup or another company.
“95% of what we built was something that absolutely a startup could do, uh, another company could do”
Person
Joseph Kim
1 episodes · 25 quoted claims
OpenAI released ChatGPT for financial services, an enterprise-focused product.
“we just released a new product, ChatGPT for financial services. This is an enterprise-focused product”
Morgan Stanley and Evercore were design partners for OpenAI's ChatGPT for financial services.
“we've been working with our design partners, Morgan Stanley and Evercore, on that, and reception to that's been incredible.”
OpenAI folds its financial-reasoning training data into its main training run, which becomes models such as GPT-6 Astra, so the data reaches developers building on those models.
“we do upstream into what we call the big run, which is the the main kind of run that becomes, you know, something like six Astra or six, one Astra, etcetera.”
Within OpenAI, Codex adoption in finance, legal and sales was near zero and then jumped to about 90%, around the time the Codex desktop app was released in February.
“if you look at finance, finance, legal, and sales, um, you know, it's pretty much at zero, and suddenly it kind of jumps up to 90%.”
Joseph Kim says Codex was the first OpenAI product where the product, model, harness and research were all developed together under one roof.
“it was the first time we developed a product where the product, the model, harness, and the the research was all kind of under one house and on one roof”
OpenAI says some Model Context Protocol (MCP) connectors from major data providers had error rates above 40% in its testing.
“Some, some of the MCPs that we connected, some big names, 40% plus error rate on their MCPs, right?”
OpenAI decided to bundle financial data into its product so it could control the experience end to end, after finding outside connectors unreliable.
“The only way we can go about fixing this at scale is to bring the data in-house, like, just control that experience end to end.”
Most of OpenAI's ChatGPT for financial services was built on the ChatGPT plugin architecture, which is available to outside builders.
“most of the product that we built was built on top of the Codex, um, or I should say ChatGPT now, um, plug-in architecture and plug-in primitives.”
Joseph Kim's team worked from the thesis that Codex gets a product about 80% of the way to market success and that the missing 20% is pivotal.
“the thesis of my team was Codex was Codex will get you about 80% of the way to a product that would be successful in the market, but, you know, um, the 20% is missing.”
Joseph Kim predicts that a billion-dollar business will be built on top of a ChatGPT plugin.
“I really do think that there's gonna be a billion-dollar business built on top of a, uh, ChatGPT plugin, right?”
Joseph Kim describes OpenAI's launched finance product as a productivity tool for knowledge workers that assumes a human is always reviewing the work.
“the existing product that we launched is a productivity tool for knowledge workers working in the space.”
Joseph Kim says that when he worked at Google, engineers needed a certification called readability before changing code.
“when I was at Google, you need, you, you need something called readability.”
Joseph Kim says investment banking already has a layered review system in which junior analysts' work is reviewed by associates, then VPs, directors and managing directors.
“there's already this system in place for junior analysts doing a lot of the grunt work. Uh, their work gets passed off to associates, associates are reviewing the work”
Joseph Kim says AI hallucination was a frequent concern in 2022 and 2023 but comes up much less often now.
“the hallucination used to be a big term that used to come up in 2022, 2023, that doesn't come up nearly as often nowadays.”
Early reinforcement learning from human feedback at OpenAI used pairwise comparisons in which humans chose the better of two options.
“there was reinforcement learning through human feedback, and the human feedback was always a pairwise signal. It was, like, given two options, which one's better?”
OpenAI improved presentation generation by having humans compare pairs of slides and pick the better one.
“we just get a bunch of humans to look at pairs of slides and say which one looks better.”
Joseph Kim says coding combines verifiable rewards, such as whether code compiles, with human taste, such as clean system design.
“coding is a combination of verifiable rewards and human taste, right?”
Joseph Kim says financial work such as Excel models has verifiable checks, like whether totals sum and whether there are reference errors.
“I have an Excel model, does it sum up? Are there reference errors? There's a whole bunch of verifiable things that we can check”
Joseph Kim says OpenAI's GDP eval benchmark is saturated, with its unsaturated parts effectively broken, so OpenAI no longer reports or tracks it.
“we know that the eval is completely saturated, uh, and that it's, uh, there's a, and the pieces that are not saturated are essentially broken, right?”
OpenAI found its internal financial evals were saturated and that remaining wrong answers came from incorrect answer keys, poorly specified prompts or other flaws.
“we found that we had saturated the evals, and the questions that we were getting wrong were driven by incorrect answer keys, um, poorly specified prompts, or just some other flaw with the eval.”
Joseph Kim's team is building profession-specific evals, such as an investment banking eval meant to predict whether a model could automate much of a junior analyst's work.
“we'll have something like an investment banking eval that predicts, you know, if you score well on this eval, you could probably automate a lot of the work of a junior analyst, uh, investment banking.”
ChatGPT for financial services is an enterprise product, and OpenAI is trying to make it available to prosumers on the ChatGPT Pro plan.
“it is an enterprise product, but we're trying very hard to make it available to more, uh, prosumers on the pro plan.”
Joseph Kim's team at OpenAI is hiring subject matter experts in finance to work with engineers, researchers and product managers.
“the other role that I wanna give a special c shout-out to is we're hiring something called a subject matter expert.”
Joseph Kim says OpenAI's coding progress benefited from the people training the models knowing what good code looks like, an advantage it lacks in finance.
“the people that were training the models and making them better understood what good code looks like, right?”