⚡ Building ChatGPT for Finance — Joseph Kim, OpenAI's Head of Financial Services
Latent Space: The AI Engineer Podcast · 31 Oct 2026
Source: https://www.youtube.com/watch?v=v-d1WTqIULs
Transcript: https://podcasts.opennash.com/shows/latent-space/building-chatgpt-for-finance-joseph-kim-openais-head-of-financial-services-2ac7ab/ (via OpenNash Podcasts, machine-generated)

I I really do think that there's gonna be a billion dollar business built on top of a a chat GPT plugin. But 95% of what we built was something that absolutely a startup could do, uh, another company could do at OpenAI, we ran into problems with pre-training, and therefore had difficulty scaling the model. In 2026, we've been able to solve many of these issues.

Today, we have Joseph Kim, one of the heads of ChatGPT Finance from OpenAI, and our co-host, and head of editorial from Latent Space, Richard, in the studio. It's a special episode, we're here in New York at AIE Finance. Joseph just gave a great keynote, um, I think we don't wanna repeat most of the information, so

That makes sense.

honestly, go watch it, but give, give, like a thirty second of what you talk about, and then we'll leave the breadcrumbs, you guys can go watch the keynote.

Sure, yeah, I can give a quick thirty second intro to kind of what we just started about. Um, one, I would say, like, we just released a new product, ChatGPT for financial services. This is an enterprise-focused product that's really trying to bring the pow, mix the power of ChatGPT, kind of the tools and things that you already, you know, like to use in the consumer product, but also add in the enterprise layer, so that's firm-specific templates, um, you know, sources like Deloopa, S and P, etcetera.

Uh, and so really exciting. Uh, we've been working with our design partners, Morgan Stanley and Evercore, on that, and reception to that's been incredible. So, that was kind of the first half of the talk. And then the second half of the talk, really much more, um, forward-looking, so thinking deeply about, um, some of the trend lines that we're seeing in terms of model improvements and drawing those trends, line out to the future, and trying to think a little bit about, okay, like, what are these models gonna be able to do at the end of, you know, mid twenty-seven, end of twenty-seven?

And what does that mean for folks that are building products on top of that model? And how can you kind of start to plan ahead to the intelligence that's coming in, you know, just a matter of months?

I think it's a fun time because you actually come from the research side of things, so you now lead a lot of the product and engineering, but tell us what it takes to make a financial-specific product. You mentioned in the keynote that there was sort of an uptick in users when you launched ChatGPT work, so Codex started to become available to the masses, people started using it for finance, and now you're shipping your first finance product.

What is, like, financial reasoning data like? What goes into finance-specific models?

Yeah, and it's, and it's a great question. I mean, one thing I will say is that, like, you know, um, our all of the training data that we do for financial, um, you know, reasoning analysis, um, we do upstream into what we call the big run, which is the the main kind of run that becomes, you know, something like six Astra or six, one Astra, etcetera. And so even though we're adding a bunch of financial data to the model, um, it's going into the main, main model, and so therefore it's available to folks that want to build on top of it.

And that is something that's very important to us. Um, you know, at OpenAI, we really do want that intelligence to be accessible, not only to our products that we're building in-house and first-party ways, but also for folks that want to build startups, build businesses on top of that, as well as, you know, financial firms that want to integrate that directly into their companies. And so, um, we will make all that intelligence accessible there. Um, I think you touched a little bit on product and research, and I think that's, like, really core to what we do at OpenAI.

Um, internally, we call ourselves a research and deployment company, and so that, and we kind of structure that in order on purpose, like, research comes first, and then deployment comes second. Um, and so, as we think about, you know, at the end of the day, trying to change the way that people do knowledge work, we kind of work in that order, right? We think about the model capabilities, where they're headed, what kind of new things can they enable, so that's one.

And then two, we think very deeply about the product, um, features and surface areas that we should build to make that model intelligence, um, accessible to more people. And I think you alluded to to a slide in my com, my, um, keynote where, you know, we had this chart of Codex adoption within OpenAI. Uh, and it's funny because, uh, you can, you can see the the software engineering adoption curve, and it's just, like, up and to the right, pretty steady, people are picking it up and and incorporating it through their daily routines.

Um, but if you look at finance, finance, legal, and sales, um, you know, it's pretty much at zero, and suddenly it kind of jumps up to 90%. And, and if you, and if you look at that line and try to see what happened around the time where it jumped up, we, we had released Codex desktop app, um, you know, uh, in February, and that was when that adoption jumped up. And so I do think it kind of shows the importance of co-developing those things, and, uh, the the purpose of the product is to make that model intelligence accessible to as many users as possible.

So, in the, um, you know, that uptick of, uh, data for, uh, using Codex for financial, you know, uh, applications, um, what, what sort of things did you learn that you then applied into the post-training of, uh, financial data?

Yeah, I mean, it's, it's a great question. Uh, I would say, like, so, yeah, I could talk a little bit about how we went about building ChatGPT for financial services. It's it was very unique. We, we haven't built a whole lot of, um, first-party products. I would say, like, Codex was really, like, our biggest and main one, and, um, it was the first time we developed a product where the product, the model, harness, and the the research was all kind of under one house and on one roof, and really gives us a lot of flexibility to, um, go about building things.

I think the biggest takeaway that I have from having worked on this for the last six months or so, um, is in enterprise, it's, you know, you have to deal with the reality of the industry that you're in, and go build around that. And I think that piece is, it's easy to ignore. Like, if you just think about it from first-based principles, you can come to a conclusion that makes sense, and you talk about it, and you can defend it with, you know, with logic and reason.

But if you ignore the state of the world that you're in when you start building, and the forces that are at play within the industry to try, you know, it's either competitive forces, people have their own objectives, and if you just don't think about that, you're not gonna build the right product, right? And so we saw that with, um, ChatGPT for financial services, and we thought about that a lot. I'll give you one example. Um, data, data is, like, a huge part of how we thought about, um, building the product.

And, you know, and, and data was one of those things where, on first principles, it was a little bit tough when I was talking about, you know, building, working on this product to defend. Because, you know, you say, hey, like, I wanna go and bring quarter earnings transcripts into the product and bundle it. And the first question that the first thing people are gonna say are, is, look, there's a MCP connector in the store. Like, what is, what is the incremental value that you're providing by, uh, bundling that in?

On paper, it looks bad. It's like, you can, they might already have a license for it, so they might feel like they're paying twice for it, um, and they might have an MCP. And so, like, why are you building this? And the reality is, we went and we tested all these MCPs, and what we found was the error rates were were really, really high. Some, some of the MCPs that we connected, some big names, 40% plus error rate on their MCPs, right?

And so, um, MCPs is one of those things where it, it looks great on paper, but you have to go see, look at the data and see what's actually going on. And so for data, we're like, hey, look, this is really bad. The only way we can go about fixing this at scale is to bring the data in-house, like, just control that experience end to end. And the extra money that you pay on top to bundle that data is worth it, right?

So, that was a product decision that we had to make that was very industry-specific, um, and it's been resonating with customers. Customers love it. The, you know, the number one thing they call out is the data, and so, um, it worked.

So, you had good success in this. Tell us about the technical side of things for people building a harness adaptation for a specific service, right? So, you guys went out, you changed Codex, you did post-training, you did whatever you did to make ChatGPT for financial services. If someone was to the to do the same for another domain, say legal, sales, whatever, uh, what are the best practices there? What did you learn? What goes into just taking off the shelf and actually, you know, making a certain product around it?

Yeah, it's a, uh, yeah, I like the question a lot. Um, it, it, you know, I tried to allude to this in my keynote as well, but the, like, interesting thing is, you know, it's fun to talk about the post-training stuff because it's unique to what we do and, uh, gets people excited. But 95% of what we built was something that absolutely a startup could do, uh, another company could do, and we didn't really have to go leverage our ability to go and cha make changes in the model to go about building fantastic product, right?

And so, I think it's a great call to kind of mention that piece. Um, what I would say is, like, uh, fewfold. One is most of the product that we built was built on top of the Codex, um, or I should say ChatGPT now, um, plug-in architecture and plug-in primitives. And that is something that we've been investing a lot of time and energy in. Um, you know, o, OpenAI is like an interesting company where, you know, we have, we kind of face the same problems internally that a startup would face, uh, if they were to build on top of OpenAI.

So, in very, in a lot of ways, like, my team is kind of like a startup that's building on top of OpenAI technology, right? And so, um, even internally, we need to find scalable ways for teams like myself to stand on the shoulders of giants and build on top of Codex, right? Like, Codex is fantastic. Um, it's got great memory, it's got good skills, like, there's a, you know, it's persistent. Um, there's a lot of things that are wonderful about Codex, but, th, th, the thesis of my team was Codex was Codex will get you about 80% of the way to a product that would be successful in the market, but, you know, um, the 20% is missing.

And so, we need to find some way for my team to go build that 20% to actually go in, because the other half of the thesis was if you just took Codex and launched it as a finished product, um, it would fail. And so, that 20% is actually pivotal. And so, what I would say is we, we went and we built on top of the plug-in architecture. It's using the same primitives that are available to any of the builders that want to go build on top of, um, ChatGPT.

And I, you know, had a line in my, um, presentation, but I, I really do think that there's gonna be a billion-dollar business built on top of a, uh, ChatGPT plugin, right? Like, it's like, you're gonna be able to leverage all of that horizontal functionality, and just add your secret sauce on top. And that secret sauce might be data, it might be UI changes, um, it could be an integration, but, you know, that secret sauce is really valuable, and I think there's there's $1 billion worth of value to be built there.

So, uh, yeah, it's an interesting time for the industry cause we're at a a time now where orchestration is, uh, is becoming a lot more automated, you know, uh, teams of agents going out and doing things. But for finance, obviously, there are a lot of risks, uh, involved in that. So, can you talk a bit about, uh, about the safety, sort of, uh, that you baked into the product?

Yeah, um, safety is super, super important. Um, I would say that, like, and it's only going to get more important, uh, because the, uh, the you know, the agents are already getting more autonomous and more independent, and I think that trend we expect it to continue, uh, into twenty-seven, into twenty-eight. Um, I would say, uh, for the for the product that we built and launched last month, um, we there was a bunch of paradigms that already existed, um, as it pertains to safety, um, and, you know, being thoughtful around auditing the work, making sure that the model's not making a mistake that's gonna impact business outcomes.

And we were able to lean heavily into those. And and what I would say is the existing product that we launched is a productivity tool for knowledge workers working in the space. And so, the idea behind it is there's a s a deep assumption that there's always a human in the loop, right? Like, so the idea there is, you know, uh, y, you're a financial analyst, and you have a a laundry list of things you wanna get done that day.

And and just like how software engineers do with Codex, for every one of those items, the first thing you do is try to build it with ChatGPT, right? And it might get you 30% of the way there, it might get you 80% of the way there, depending on the task. But there's always this presumption of humans in the loop, uh, that are presiding over that work. And, um, that made it very easy for us to to build a product where we could kind of pass off the responsibility of auditing, review, uh, I, into people's hands and into people's responsibilities.

And that worked for us well, and it was easy to communicate. Um, so that was wonderful. I will say, we're thinking very deeply about what comes next, and and what comes next looks different, uh, than than a productivity tool, um, that sits and and is always being monitored by a human. And I think, as we think about what comes next, there's kind of two things we're thinking about. One is how do we kind of shape the security, safety profile of the product into something that people understand intuitively.

And some, I would say something that people understand very intuitively is how to work with other people. Um, and so, our thesis is if we make the product very similar to how you work with your, your coworkers, your interns, people around you, that is something that people can pick up and learn very quickly, very easily. And we, and therefore, we need to shape the security profile of that to look like a person. And so, there's a few things I'll call out there.

One is, there's this notion of read-write, uh, you know, even for people. Like, uh, you're a new software engineer at a company. You oftentimes, you know, when I was at Google, you need, you, you need something called readability. And so, there's these kind of, like, checks and balances and systems in place before you can start monkeying around and touching the system. And so, there's already systems in place to monitor the work of people. So, if, mon, if models behave more like people, then you can leverage those systems.

And that makes it very easy to kind of talk about and work through this, um, with these financial institutions. So, that's something we're leading into pretty heavily coming up. And the very last thing I would say is, at least in the context of banking, which is where we kind of started, um, there's already this system in place for junior analysts doing a lot of the grunt work. Uh, their work gets passed off to associates, associates are reviewing the work, um, then the associates are kind of adding their secret sauce, their magic, their reasoning.

That gets passed off to the VP, VP goes up to the director, managing director. So, there's this kind of, like, a hierarchical system where there's this implicit work checking that's happening that's already been built out to stop, uh, to try to prevent against, uh, mistakes that humans make, right? Like, mistakes that humans make just as many mistakes as models, and so those systems are built there. And so, we're also leaning heavily in terms of those systems as it co as it pertains to building financial products.

I'd say there's this, like, sort of misconception where in the finance world, one mistake really compounds, like, your your whole career is at risk if you submit something wrong. How should people think about getting started and using it? So, if you're an analyst, um, how should you use these tools? And do you have, like, any analogies towards where we're at right now? So, there's, like, levels of autonomy in self-driving. One of your slides, I think, said, like, twenty, twenty-seven.

You see no more PowerPoint. A lot of people will be using AI tools in the loop. So, how should people think about where we are now and where things are going, and how should they adopt these tools if they have that misconception of, like, one mistake, my career's over? Obviously, there's human in the loop, but how should they also think about it going forward as they need to offload more and more, or

Yeah, no, that's that's wonderful. Um, I I have two perspectives on it. And they come, they start with different assumptions. Um, I think, I I think the notion that, like, my career is over if you make one mistake is an incorrect notion. And I have a line of thought that, like, follows that. But let me let's first assume that it's true, right? And and in the case that it's true, um, what I would say is, like, not all mistakes are created equally, and not all model capabilities are created equally.

And so, when you think about the mistake that a model or a human can make, that could be career-ending, um, you know, it's it's usually once you get up to a certain level, like, if you're a, you know, VP, the stakes are higher than when you're a junior analyst. Because there's a presumption of work having been checked up to that point. So, if an MTSMDC's a mistake, that's a huge problem. A lot of the safety nets have been, you know, fa have failed for that mistake to make it up to an MD, right?

Um, but what I would say is, like, you know, the career-ending mistake is something where you get a piece of information factually incorrect, and that piece of factual information changes the conclusion. Like, that is obviously, like, the kind of the biggest type of mistake that you could make. And what I would say is we're seeing a lot of progress in in, uh, attention to detail. And, uh, you know, the hallucination used to be a big term that used to come up in 2022, 2023, that doesn't come up nearly as often nowadays.

I would say that, like, we're starting to see huge improvements in the reliability of models as it pertains to factual mistakes, so the the kind of egregious mistake that, uh, would get you fired, models are getting much better at and continue to get much better. And so, I feel actually quite confident that, uh, a combination of that continual improvement, plus the existing system checks in place are will make it such that, you know, that type of mistake, the chance of that propagating, uh, up to an MD is just very low probability, right?

Um, what, what I would say, uh, the models are lacking today, and they still need to improve, is actually not the career-ending mistake. Um, but it's the, uh, um, it's the, uh, bit that involves human judgment and taste. So, it's the this slide looks ugly, right? And it's, it's not wrong, but it's formatted like the colors look a little bit off, something about it looks sloppy, and therefore, you know, this is not a good slide. And and that's that's not a career-ending mistake in the way that, uh, in in, uh, uh, like, a factual mistake is.

Um, but, um, it still matters. It still matters in terms of how useful that work is. And so, as we think about the progression in going into 2027, what I would say is, I expect the models to get much better at the human taste component. And that's what I was kind of alluding to in my keynote is, I think the model will get much better at human taste on things that are softer objective functions. It's not, is it right or wrong?

It's comparative. I, I is there something that's better than this? And I think models are gonna get much better, uh, along that dimension, moving forward.

How so? How do we solve taste? How do we train taste into the models? Um,

Yeah, uh, so it's funny, like, very early on in the like, early RLF days, the only thing we trained models on were essentially taste, right? Like, the there was reinforcement learning through human feedback, and the human feedback was always a pairwise signal. It was, like, given two options, which one's better? And so, the early days of reinforcement learning were built on the notion of training on human taste. Uh, and then, obviously, things like verifiable rewards and environments, and, you know, we built a lot of machinery to do other types of rewards and other types of RL.

And so, people kind of forgot that a lot the early days of of our reinforcement learning were all around human taste. And so, how do you train on human taste? It's I think the technical answer is quite simple. I think the, like, practical answer obviously gets more complicated. Um, the practical answer is you just train a reward model, right? You just train a reward model, you get a bunch of humans to look at pairs of things and say which one is better.

And this still works. Um, so, like, um, fun fact, like, we did this for slides, right? Like, I mean, how do you, how do you go about making these models better at generating presentations? Well, we just get a bunch of humans to look at pairs of slides and say which one looks better. And humans are actually really, really good at aesthetic, um, judgment, right? Actually, much better than models is something that we discovered is, like, a human can look at two slides and, within seconds, say this one looks better.

And if you have a model, like a grading grading model, try to go about doing that, um, it's gonna spend a whole bunch of reasoning tokens, this pixel is this way, this thing's that, uh, and then maybe come to the wrong conclusion. And so, humans are still the the really valuable in making these models better, and so, we rely, we will rely a lot on, uh, human preference, um, to imbue things like taste. So, that's one.

Um, the last thing I'll kind of add is there's precedent for this, which makes us confident that this strategy will work. And that precedent, I would say, is coding. Um, the models have gotten, I would say coding is not fully solved, but, like, let's say we're 80% of the way there, and we're making progress on the on the last 20%. Um, coding is a combination of verifiable rewards and human taste, right? And so, the verifiable rewards are, you know, does the does the code compile?

Are there syntax errors? Um, is the output of the function expected? Um, and so, there's these verifiable rewards that are in place. But a lot of good software engineering is, um, is this clean system design? Did you build a system that's maintainable over time? Um, is it easy to unit test? Is it following paradigms that the vast majority of the community or the, um, industry thinks is the right paradigms? And if you look at it from that lens, um, financial, uh, a lot of financial work looks very similar.

It's the, you know, I have an Excel model, does it sum up? Are there reference errors? There's a whole bunch of verifiable things that we can check to make sure that the work is passing this, you know, first-pass sanity check. But there's also a lot of, is this the way I would have built this model? Um, something I learned about as I started working in finance is there's this notion of, like, the feel of the model, which is, you know, you, you kind of perturb the inputs and you look at the outputs.

And an experienced financial professional can kind of perturb the inputs, look at the outputs, and kind of understand there's a data flow through this model in a way that feels good, right? And that is a human preference that looks a lot like, does this code look clean? Is it well-written? And I think our success in coding gives me confidence that we can solve a lot of these problems in finance as well.

Yeah, we can shout out, uh, frontier code from cognition. So, it's not only, like, Sweet Bench is very, um, you know, is the answer correct? This is also, would you merge the code? So, you can have multiple answers that are proper PRs that pass, but which one would you merge? And that's why you sometimes see the charts of lower reasoning is better, but it's a taste benchmark as well, right?

Yeah, and, uh, one of your other, uh, 2027 predictions I thought was interesting was that, uh, you know, the models that you're training will become much better at, you know, uh, investment theses and sort of generating ideas for investment and trading. Um, but obviously, you know, uh, that's something that humans, uh, uh, or experienced kind of financial professionals are very good at, and they kind of have an intuition for that kind of thing. But it's often counter to the prevailing wisdom out there.

Otherwise, you know, uh, the there's, uh, less of a chance of it making a a, a great return. So, h how are you tackling that problem in terms of, you know, training the agent to do that kind of work?

Yeah, um, I would say that, like, up until date, the vast majority of applications of AI and finance have been really focused on, I would say, like, the junior analysts work. Um, and I think that's, and I think that's for two reasons. One is you have to build the product based on the like capabilities of the model of today. And so, if you look at the capabilities of the model today, they're really quite good at taking pretty concrete instructions of what to do, uh, and kind of executing, right?

So, build me a slide, I want it to look this way, um, you know, make it look good, use our template, these types of things. The model's quite capable of doing those things, and so, therefore, we've built product experiences around that. Um, and the other thing I would call out is, um, there's a strong preference to make the model good at things that humans are not good at or don't like to do, right? And so, uh, human, you know, most humans I talk to, there's obviously exceptions, but a lot of the humans I talk to, like, um, humans I talk people I talk to, um, uh, you know, they really don't like building Excel models.

It's it's a tedious kind of work, and it's very detail-oriented, it's easy to get wrong, and similarly with slides. Um, you know, you're you're copying logos, you're placing them very precisely on a specific part of the page. That's very tedious work that a lot of people don't like to do, and so that's been part of the impetus for us to go and focus on some of these tasks, right? It's a combination of two those things. There's, uh, like, exi, the existing molecular is pretty good at, uh, certain types of tasks, and also human preferences of what they like to do and not like to do helps drive the direction of the product as well, right?

It's a combination of those two things. Um, I think that, like, we've solved a lot of the low-hanging fruit in those two things, and so, going into 2027, we're gonna start to move up the stack, right? We're gonna go from the tasks that a junior analyst can do to some of the tasks that an associate can do to some of the tasks that an MD would do, and move up that chain. And as you move up the chain, what I would say is the objectives become much fuzzier.

Uh, it goes from this is clearly a good or bad slide, or something that would put in front of a client or not, to, well, I don't know if this investment idea is good. And it's and it would take me, you know, three years of waiting to really be able to give you a cl, a clear, concrete answer of, like, what the market has done. And so, the objectives start to become fuzzier. And so, the the way you go about training the model and the way you think about how to make the model better, those things becomes more difficult.

It becomes more difficult from a research perspective. And so, um, but it's not impossible, is what I would say. And so, what it takes is a lot of research focus, it takes a lot of wonderful data, it takes a lot of feedback from experts in the field. We really wanna get the the, you know, the P 99.9 type of feedback, uh, to make the model, you know, as good as humans, but I don't think it's an insurmountable problem.

And I, and what it takes is continued scaling of the base model and focus, focus from the research labs to think that this is an important problem to go solve. And one of the things that I'm doing, uh, uh, on my team is making this a an area of focus and priority for OpenAI. And and I think because of the combination of those two things, I feel confident we'll make a lot of strides in 2027.

Do you have any comments on any of the current financial benchmarks? So, I think we're still in the state of, you know, uh, does this check out numbers-wise? Uh, how do you think about making your own adage in taste? Um, I think this is also an open call for people working in this space to work on their own benchmarks, and a sort of follow along to that is, you said, build for build systems for models today, but you also give a lot of predictions for the future.

Um, how do you put guardrails, limitations, um, checks on financial services that you build for people today just so they're confident in, you know, this is not like going out off the rails?

Yeah, makes sense. Um, let's, yeah, let's talk a little about evals. Uh, evals is, yeah, something near and dear to my heart, spent a lot of time working on evals. I would say that, um, the like, um, the prior that I would share with people is building a good eval is incredibly hard. Peop, peop, you know, that's not obviously people who haven't worked on eval before. Um, to build an eval that, actually, like, correlates with the thing that you're actually trying to measure, like the skill you're trying to measure, capability you're trying to measure, very difficult.

Um, because whatever you put together is just a proxy for whatever that thing is, right? And it's just one dimension. So, that's one. Two is it's very tedious to get the answer key or the way that you grade this correctly. And if you don't get the way that you get, you grade the answers correctly, it's a eval that's very easy to hack and very easy to reward hack. And therefore, you know, you could score very highly on the eval, um, but actually do really poor on the on the actual underlying skill or objective that you're trying to measure, right?

And so, the the the like, prior that I will share is evals are really hard, um, not just in financial work, but just across the board. And, uh, what I would say is, uh, we don't think that there's a really good eval today, um, for financial, um, work. Um, there's a bunch of existing prior work. I'll, I'll reference some of the work that we did, which was, uh, GDP eval, uh, which is like a little bit more broadly in knowledge work.

Um, if you guys look at the Astra announcement, we don't talk about GDP eval. Uh, we, we know that the eval is completely saturated, uh, and that it's, uh, there's a, and the pieces that are not saturated are essentially broken, right? And so, we actually don't report on that, we don't track it anymore. And so, one of the tricky things for, um, uh, GPT six Astra was, um, we didn't talk a whole lot about financial evals.

We have some internally that we didn't talk too much about and didn't want to share at the time yet. They're coming, um, but it was there was no good way for us to show progress, uh, on financial tasks on these evals. Um, there's a funny like, funny things, like, I think there's this bias in evals where if the numbers look good, the researchers assume that the model's gotten better and they don't investigate it. If the model, uh, the you've made a much more capable model, and the eval numbers look bad or haven't increased in the the scale that you want, then you go investigate, right?

So, there's this kind of, like, bias baked into, uh, investigation these evals. Something like this happened with GPT six, where we, we ran GPT six on a set of, um, financial evals, and the numbers didn't really change much. We're like, oh, like, what's going on here? Let's take a look at this eval. We, we did a deep dive into these different evals, and lots of things were broken. Uh, we like, the basically, we found that we had saturated the evals, and the questions that we were getting wrong were driven by incorrect answer keys, um, poorly specified prompts, or just some other flaw with the eval.

Um, and so, all that to say, like, one of the things that my team is working on currently is to try to build a much better eval that actually predicts the ability to automate the work of that specific profession. And so, yeah, and so the idea is, like, we'll have something like an investment banking eval that predicts, you know, if you score well on this eval, you could probably automate a lot of the work of a junior analyst, uh, investment banking.

We gotta wrap here soon. I wanna ask, uh, any call-outs for people in the finance space, the Latent Space audience? Uh, your team is probably hiring, but last thing to put out there, how how can people be helpful to you?

Yeah, wonderful. Well, I would say, like, first and foremost, like, please check out ChatGPT for financial services. Um, it is an enterprise product, but we're trying very hard to make it available to more, uh, prosumers on the pro plan. And so, hopefully, we'll have some cool announcements there. Um, two, yes, my team is hiring. Uh, we're seeing incredible, uh, customer traction with the product, and, uh, because of that, we're hiring and scaling the team rapidly. It's a very diverse team with lots of different roles, and so I just wanna call out a few of them very quickly.

Um, there's obviously the the standard, uh, front-end, back-end, uh, engineering, so, um, kind of traditional product building, so, that's one. Two, we're very much looking for wonderful product managers. I think in some ways, uh, great product management is the the main bottleneck in trying to go win in this space, and so we're trying to hire wonderful product managers in investment banking, equity research, asset management, uh, you know, across the board, we're tr we're trying to hire product managers.

Uh, and then the other role that I wanna give a special c shout-out to is we're hiring something called a subject matter expert. Um, and, you know, it's really important. Uh, one of the things that we did well in in coding is, you know, the the people that were training the models and making them better understood what good code looks like, right? And that was a huge tailwind for us as we're making the models better at these things.

We don't have that advantage when it comes to making the models models better at finance. And so, we're we're building out and trying to hire a lot of folks that are able to bring that financial expertise and work with the engineers, work with the researchers, work with the product managers to make the model and the product better in your specific field. And so, if any of those resonates with you, like, I highly, uh, you know, recommend that you reach out to me.

I'm available on LinkedIn. We also have these, uh, job listings posted on our careers page. Um, it's pretty easy to find.

Great. Well, you answered my question, and I was gonna ask, when I, when can I have a play with this? But if it's coming Yeah, we're to pretty, pretty working very hard soon.

to make it available to more people.

It's very common open AI stuff, you know, it makes sense you make stuff for enterprise, and then this trickles down to the regular models. It comes to pro plans. So expect for your general usage down the line, it'll get better at financial stuff. Awesome, well, very excited to have you. Check out the keynote. You gave a lot, you gave good predictions. You talked a little bit about some juicy, uh, we fixed pre-training issues, so I'll leave breadcrumbs for you guys to go watch, but thanks for joining.

Yeah, awesome, thank you so much.