Inside OpenAI Enterprise: Forward Deployed Engineering, GPT-5, and More | BG2 Guest Interview

We literally had to bring the weights of the the model, uh, physically into their supercomputer.

In San Francisco, you could take a car from one part of SF to the other fully autonomously. As opposed to the digital world, I can't book a ticket online right now. Physical autonomy is ahead of digital autonomy in 2025.

I think AI agents are, like, really in day one here, like ChatGPT only came out in 2022, and the slope I think is incredibly steep. I actually do think self-driving cars have s a good amount of scaffolding in the world. Um, you have roads, roads exist, they're pretty standardized, you have s stoplights. AI agents are just kind of dropped in the middle of nowhere.

We'll start with long short game.

I'm short on, uh, the entire category of, like, tooling evals products.

Healthcare is probably the industry that will benefit the most from AI. I think I may J-Piled.

You're definitely a J-Piled.

The first one was the realization in 2023 that I would never need to code manually, like, ever, ever again.

Hey folks, I'm Apoorv Agrawal, and today at the OpenAI office, we had a wide-ranging conversation about OpenAI's work in enterprise. I have with me the head of engineering and head of product of the OpenAI platform, Sherwin Wu and Olivier Godement. OpenAI is well known as a creator of ChatGPT, which is a product that billions across the world have come to love and enjoy. But today we dive into the other side of the business, which is OpenAI's work in enterprise.

We go deep into their work with specific customers and how OpenAI is transforming large and important industries like healthcare, telecommunications, and national security research. We also talk about Sherwin and Olivia's outlook on the next what's next in AI, what's next in technology, and their picks, both on the long and short side. This was a lot of fun to do. I hope you really enjoy it. Well, two world-class builders. Two people who make look building easy. Sherwin, uh, my Palantir 2013 classmate, tennis buddy, with two stops at Cora and OpenDoor through the IPO before joining OpenAI.

Before ChatGPT, you've now been here for three years and lead engineering for all OpenAI platform. Olivier, former entrepreneur, winner of the Golden Llama at Stripe, where you were for just under a decade, and now lead all of the product at, uh, OpenAI platform.

That's right.

Thanks for doing it.

Thank you.

Thanks for having us.

You know, as a shareholder, as a thought partner, kicking ideas back and forth, I always learn a lot from you guys. And so it's a treat. It's a real treat to be do this for everybody. You know, I'll open with, people know OpenAI as the firm that built ChatGPT, the product that they have in their pocket that comes with them every day to work, to personal lives. But the focus for today is OpenAI for enterprise. You guys lead OpenAI platform.

Tell us about it. What's underneath the OpenAI platform for B2B for enterprise?

Yeah, so this is actually a really interesting question, too, cause I, like I said, when I joined OpenAI, uh, around three years ago to work on the API, it was actually the only product that we had. So I think a lot of people actually forget this, where the original product for, uh, from OpenAI actually was not ChatGPT, it was a B2B product. It was, uh, the API we were catering towards developers. Um, and so I've actually seen, you know, the launch of ChatGPT and and all of you know, everything downstream from that.

Um, but at its core, I actually think the the reason why we have a platform and why we started with an API is uh, kind of comes back to the OpenAI mission. So our our mission obviously is to build AGI, uh, which is pretty hard, uh, in and of itself, but also to distribute the benefits of it to everyone, uh, in the world, uh, to all of humanity. Um, and, you know, it's pretty clear right now to see ChatGPT doing that, because, you know, my mom, you know, maybe even your parents are using ChatGPT.

But, um, we actually view our platform and especially our API and how we work with our customers, our enterprise customers, as our way of getting the benefits of AGI, of of AI, to as many people as possible, to everyone in every corner of the world. Uh, ChatGPT obviously is really, really, really big now. It's, I think, like the fifth largest website in the world, um, but we actually, by working through developers, using our API, we're actually able to reach even more, uh, people and, you know, every corner of the world in every different use case that you might have.

Uh, and especially with some of our enterprise customers, we're able to reach even, uh, use cases within businesses and reach, uh, end users of those businesses as well. And so we actually view the the platform as kind of our way of fully expressing our our mission, uh, of getting the the benefits of AGI to to to everyone. And so, um, concretely, though, what the platform actually includes today, um, the biggest product that we have is is obviously, um, our developer platform, which is our our API.

Um, you know, many developers, the, you know, the majority of of the startup ecosystem builds on top of this, as well as, uh, a lot of star uh, digital natives, uh, Fortune 500 enterprises at this point. Um, we also have a product that we sell to governments, um, as well in the public sector, so that's all part of this as well. Um, and also an emerging product line for us in the platform is our, uh, enterprise products, so what we actually might sell directly to, uh, enterprises, uh, beyond just a a core API offering.

Fascinating.

And maybe to to to double down, like, um, I think B2B is actually quite core to the OpenAI mission. What we mean by distributing AGI benefits is, you know, I wanna live in a world where, you know, there are, like, ten x more medicines, like, you know, going out every year. I wanna live in a world where, you know, education, like, you know, public service, civil service, you know, are, like, you know, increasingly, like, optimized, you know, to everyone.

And, you know, there are, like, a large, like, category of use cases that only go through B2B, frankly, unless you enable, like, the enterprises. And, you know, we talked to Palantir. I think that's probably the same thesis at Palantir. Is like, hey, like, those are the businesses who are actually, like, you know, making stuff happen in the real world. And so if you do enable them, if you do accelerate them, like, that's how essentially you benefit, you know, distribute AGI.

Yeah. Well, maybe we can double-click into that, Olivia. You know, the reach for chat is obviously wide billions of users, but for enterprise, it's maybe tell us about it. Maybe we go deep into a customer example or two, and what is an organization that we've helped, uh, transform maybe, uh, and at what layers?

So if I were to step back, like, we started our B2B efforts with the API, like, a few years ago. Initially, the customers were startups, developers, indie hackers, extremely technically sophisticated people, like, you know, who are building, like, you know, cool new stuff, essentially, and taking massive, like, you know, market-attacking risk. Um, so, we still have a bunch of customers in that category, and we love them, um, and we keep building with them. On top of that, you know, over the past couple of years, we've been working one more with traditional, like, enterprises, and also, like, digital natives.

Um, essentially, I think, basically, everyone woke up, like, with ChatGPT on, like, um, those models are working. There is a ton of value, and they could see essentially many use cases in the enterprise. Um, couple of examples which I like the most, um, one which is very both fresh and, you know, is quite cool. Um, we've been working a lot with T-Mobile. T-Mobile. So T-Mobile leading, like, U.S. uh, telco operator. Um, T-Mobile has, like, you know, a massive customer support load.

Like, you know, people asking, like, you know, hey, I was charged, like, that amount of money. What's going on? Or, you know, my my cell phone, like, isn't working anymore. Um, a massive, like, you know, um, share of that load is, like, you know, voice calls, like, people wanna talk to someone. Um, and so for them, like, you know, to be able to essentially automate, like, more and more, and, you know, to help, like, people, like, self-serve in a way, like, you know, um, debug their their subscription was pretty big.

And so we've been working with T-Mobile pretty much for the past year at that point to basically automate, like, not only, like, text support, but also voice support. And so today, like, you know, there are, like, features, like, in the T-Mobile app that if you call, actually handled by OpenAI, you know, models behind the scenes. And, you know, it does sound, like, super natural, like, you know, human-sounding latency, quality-wise. So that one was, um, really fun.

A second one, uh, which is

Can I just on that? Can I can I ask you a follow-up question? So we've got text models, we've got voice models, maybe even video models someday, that are deployed at T-Mobile. But what above the models or or or adjacent to the models might we have helped T-Mobile with, for example?

Yeah. Yeah, there is a tone we're doing. Um, the first one is, you know, you have to put yourself in the shoes of an enterprise buyer. Like, their goal is to automate, you know, or reduce, like, you know, optimize customer support. And, you know, going from, like, a model, like tokens in, tokens out to, the use case, it's hard. And so, you know, first, like, there's a lot of design, like, you know, system design. Um, we do have actually now four deployed engineers who are helping us quite a bit.

Essentially,

Right, right, right. Yeah. Yeah, yeah. Yeah, I mean Forward deployed engineers. Yeah, it's familiar to the

Far the term from Palantir. Um, yeah, it's a great term.

were you F D Es at, uh, Palantir?

I was not an F D E. I was on, I think they called it the dev side, right? Just, like, software engineering. I was also also only an intern at Palantir. Um, but yeah, it's a great term. I think it a accurately describes what we're asking folks to do, which is, like, embed very deeply with customers and and honestly, like, build things specific to their systems that are deployed, uh, onto these customers. Um, but yeah, we we are obviously growing and hiring that team quite a bit cause they've been very effective, like, at T-Mobile.

Yeah.

Yeah. Four years of my life. Yeah. Forward deployed. But but go ahead. So forward deployed engineering on

Yeah. Four deployed engineers. And the sort of, like, systems and, like, integrations they're doing is, you know, first, like, you know, you have to orchestrate those models. Like, those models are not just, you know, uh, those models, like, know nothing about, like, you know, the CRM, like, you know, and, like, what's going on. And so you have to plug the model to, like, many, many different external tools. Many of those, like, tools, like in the enterprise, do not even have, like, APIs or like clean interfaces, right?

It's the first time they're being exposed, like, you know, to a third-party, uh, system. And so there's a lot of, you know, standing up, like, you know, API gateways, like, tools, connecting. Um, then you have to essentially, like, define what good looks like, you know? Uh, again, like it's a pretty new exercise for everyone, like, you know, defining, like, a golden set of evals is, you know, easier than it sounds. Um, harder than it sounds. Um, and so we've been spending, like, a bunch of time with them.

Evals are important.

Evals are super important.

Especially, like, audio evals. I know audio evals are, like, extra hard to grade and and get right. And, but, like, the bulk of the use case here is actually audio. And we have, like, I don't know, five minute, like, call transcript. How do you actually know that the right thing happened? Uh, it's a pretty tough problem.

Yeah, it's pretty tough. And then, you know, actually nailing down, like, the quality of the customer experience, like, you know, until it feels natural. And here, latency and interruptions, they're really, like, you know, important part. We shipped, um, in GA, um, an API, real-time API, I think it was last week. Uh, yeah. Um, which is, like, a beautiful work of engineering. Uh, you know, there was a really cracked team behind the scenes. Um, which basically allows us, like, to get, like, the most, like, natural sounding, like, you know, voice experience without having, like, these weird interruptions or no lag, where you can feel that essentially the thing is off.

Um, so, yeah. Cobbling all that together, you know, and you get, like, you know, a really good experience.

Yeah. Yeah. That's a lot more than just models.

Yeah, I was gonna say, um, uh, one actually really great thing that I think we've gotten from the T-Mobile experience is actually working with them to improve our models themselves. So, for example, the last, uh, real-time, uh, the real-time GA last last week, we actually released a new snapshot of the GA snapshot. Uh, and a lot of the improvements that we actually got into the model came out of, you know, the learnings that we had from T-Mobile.

It brings in a lot of other change from other customers, but because we were so deeply embedded into T-Mobile and and we were able to understand what good looks like for them, we were able to bring that to some of our models.

That makes sense. So this is a large customer with tens of millions of users, if not hundreds of millions. And the before and after is on the support side, both tech support internally and then their customer support. Makes sense. Is there another one that you guys can share?

I like a lot, uh, Amgen.

Amgen, the healthcare business.

Amgen, yeah. So we are working quite a bit with, uh, healthcare companies. Amgen is one of the leading, like, healthcare companies. We specialized into drugs for, um, uh, cancer or, like, you know, inflammatory diseases. Uh, they're based out of LA. Um, and we've been working essentially with Apent to essentially speed up, like, the drug, like, development and like, acceleration process. So, you know, the the sort of the the north star is, like, pretty bold. Um, and it's really interesting, like, when you similarly, like, you know, we embedded, like, pretty deeply with Amgen to understand what are their needs.

And it's really interesting, like, when I look at those healthcare companies, I feel like there are two big buckets of needs. One is, like, pure R and D. It's like, you know, you're seeing, like, a massive amount of data, and, like, you have super smart scientists who are trying to, you know, combine, test out things, you know. So that's one bucket. A second bucket is, like, you know, much more, like, you know, common across other industries.

It's, like, pure, like, you know, admin, document authoring, documents reviewing work, which is, you know, by the time, like, your R and D team has essentially locked the recipe of a medication, getting that medication to market is a ton of work. Like, you have to submit to, like, various regulatory bodies, get a ton of reviews. And, you know, when we looked at essentially those problems, what we knew, what models were capable of, we saw, like, you know, a ton of benefits, a ton of, like, opportunities to automate and, you know, augment essentially the work of those teams.

Um, and so, yeah, Amgen has been, like, a top customer of GPT-5, for instance.

Wow. I mean, this could be hundreds of millions of lives if a new drug is developed faster. Huge impact.

So that's, you know, that's, I think, one good example of, like, a kind of impact on which you need to enable enterprises, like, to do it, you know? Uh, and so I think we're going to do more and more of those. Uh, and yeah, frankly, like, you know, on a personal level, like, it's a delight, you know. If I can play, like, you know, a tiny role, essentially in, like, doubling, like, you know, the kind of, um, medication that people, you know, get in the real world, that feels like, you know, a pretty good, like, you know, achievement.

Huge. Huge, huge. I know you had one as well.

Yeah, so one of my favorite deployments that we've done, uh, more recently, actually, is with the, uh, Los Alamos National Labs. So this is the the the, like, government national research lab that, uh, the U.S. government is running in Los Alamos, New Mexico. It's also where, you know, the Manhattan Project happened, uh, back in the forties and fifties, um, back when it was a secret secret project. So, you know, after that, they ended up formalizing it as a city and a program, and then now it's a pretty sizable national, uh, laboratory.

Um, this one is is very interesting because one, just the depth of impact here is is, like, unimaginable. For me, it's like on the scale of of of Amgen and and some of these other, uh, uh, larger companies. But, you know, obviously, they're doing a lot of actual n- new research there, so a lot of new science. They're doing a lot of stuff with, uh, our, uh, defense department, uh, and defense use cases as well. So very intense, you know, very intense stuff.

Um, but the other thing that's actually actually very interesting about this one was that it's it's a it's also a story of a very, like, bespoke and, like, new type of, uh, deployment that we've done. Um, so because they are so their government lab, they're so, you know, uh, restrictive and high security and high clearance with a lot of their things, uh, we couldn't just do a normal deployment with them. They couldn't, you know, you can't have people doing national security research just hitting our, uh, our APIs.

And so, um, we actually did a custom on-prem deployment with them onto one of their supercomputers called, uh, Venado. Um, and so this actually involves a bunch of, you know, uh, very bespoke work with some FDEs, uh, also with a lot of our developer team, um, to actually bring one of our reasoning models, o three, um, into their laboratory, into an air-gapped, you know, supercomputer Venado, and actually deploy it, uh, and get it installed to work on their hardware, on their networking stack.

Um, uh, and, uh, and actually run it in this in this particular environment. And so it was it was actually very interesting because we literally had to bring the weights of the the model, uh, physically into their, uh, into their supercomputer. Um, in an environment, by the way, where you're not allowed to have, you know, it's very it's very locked down for a good reason. Um, you're not allowed to have, like, cell phones or, like, any electronics with you, um, as well.

And so I think that was a very, very unique challenge. Um, and then, uh, the other interesting thing about this deployment is just how it's being used, right? So, um, the interesting thing is because it's so, uh, locked down and on-prem, we actually do not have much visibility into exactly what they're what they're doing with it. But we do have, you know, they give us feedback. Yeah, yeah. Um, they actually do have some telemetry, but it's, you know, within their own their own systems.

But we do know that it's it's it's, you know, being used for, uh, a bunch of different things. It's being used for, um, aiding them in in terms of speeding up their experiments. They have a lot of data analysis use cases, a lot of notebooks that they're running with reams of data that they're trying to process. Um, they're actually using it as a thought partner, which is something that's pretty interesting to me. Um, o three is, like, pretty smart, uh, as a as a model, and a lot of these people are tackling really tough, you know, novel research problems.

And a lot of times they're kind of using o three, um, and, uh, going back and forth with it on their experiment design, on, like, what they actually should be using it for. Um, uh, which is, you know, something that we couldn't really say about our our our older models. Um, and so, yeah, it's it's just being used by by, um, for a lot of different use cases for the for the national lab. And, um, the other cool thing is it's actually being shared, uh, between Los Alamos and some of the other labs, uh, Lawrence Livermore, uh, Sandia, uh, as well, uh, because it it's the supercomputer setup where they can all kind of, uh, connect with it remotely.

Fascinating. I mean, we've just gone through three pretty large-scale enterprise deployments, right, which might touch tens, if not hundreds of millions of people. But there's this, on the other side of this, is the MIT report that came out a couple of weeks ago. 95% of AI deployments don't work. A bunch of, you know, scary headlines that even shook the markets for a couple of days. Like, you know, put this in perspective. Like, like, for every deployment that works, there's presumably a bunch that don't work.

So maybe we can, you know, may maybe talk about that. Like, what does it take to build a successful enterprise deployment, a successful customer deployment? Um, and and and the counterfactual, uh, based on all your experience serving all these large enterprises. I think at that point, I may have worked with, like, a couple of hundreds, I think, enterprises. So, okay, I'm going to pedal on that. Um, what I've seen being, like, clear leading indicator of success.

Uh, number one is, like, the the interesting combination of, like, top-down, like, buy-in and, like, enabling, like, you know, very clear group of, like, a tiger team, essentially. Like, you know, at the enterprise, which is sometimes a mix of, like, OpenAI, like, you know, enterprise employee. So, you know, typically, like, you know, you take, like, T-Mobile, like, the top leadership was, like, extremely bought in, like, you know, it's a priority. But then letting the team, like, you know, organize and be like, okay, if you wanna start small, start small, you know, and then can scale it up, essentially.

So that would be part number one.

Couple of hundreds. Yeah. Right. Right, right. Right. Right. So top-down buy-in and a bottom call it tiger team.

Tiger team, you know, of people, like, you know, a mix of, like, technical skills and, like, people who just have, like, the organizational knowledge, like, institutional knowledge, you know? It's really funny, like, in the enterprise, like customer support, a good example, like, what we found is that the vast majority of the knowledge is in people's heads. Right? Which is probably, like, a thing that, you know, we have these, like, in general, but, like, you know, you take a customer support, you would think that, you know, everything is, like, perfectly documented, like, in a Jira, et cetera.

The reality is, like, the the standard, like, operating procedures, like, SOPs, are largely in people's heads. And so, unless you have that tiger team, like, mix of, like, technical and, like, you know, subject matter expert, really hard, like, to get something off the ground. That would be one. Um, two would be evals first. Like, whenever we define a good evals, like, that gives, like, a clear, clear common goal for people to hit whenever, like, you know, the customer, like, fails to come up with good evals.

It's a moving target. You never know if essentially, you know, if you've made it or not. Um, and, you know, evals are much harder than what it looks, uh, to get done.

And evals also oftentimes need to come up bottom-up, right? Because all of these things are kind of in people's heads, in the actual operators' heads. Like, it's actually very hard to have a top-down mandate of, like, you got like, this is how the evals should look. A lot of it needs to be bottoms-up adoption.

Yeah, yeah, yeah. And so we've been building quite a bit of tooling on evals. We have, like, an evals product, and, you know, we're working on more to essentially solve, like, you know, that problem, or, you know, make it as easy as we can. Um, the last thing is, you know, you wanna hill climb, essentially. You have your evals, the goal is to get to 99%. You start at, like, you know, forty-six. Uh, you know, how do you get there?

And here, frankly, I think, often time, like, you know, a mix of, like, like, I will say, like, almost wisdom from people who've done it before. Like, you know, a lot of that is, like, you know, like, art sometimes more than science. Or, like, you know, knowing, like, the quirks of the model, the behavior. Sometimes we even need to fine-tune ourselves, the models, you know, when there are some clear limitation. Uh, and, you know, being patient, getting your way, you know, up there, and then, you know, ship.

Can we go under the hood a little bit? You know, one of the things that we think about a lot is autonomy more broadly, right? What is the makeup of autonomy? On one side, you know, in San Francisco, you could take a car from one part of SF to the other fully autonomously. No humans involved. Nope, you press a button. Right? They've done billions of rides. I think it was, like, what, 3.5 billion rides.

Um, on on the Tesla, this is on the Tesla FSD. I think Waymo's done, like, million, tens of millions of rides. That's a lot of autonomy. In the physical world, as opposed to the digital world, I can't book a ticket online right now. It's there's all sorts of problems that happen if I have my operator try to book a ticket. And it's very counterintuitive because the bar for physical safety is so much higher. The bar for physical safety is higher than the human's capability because lives are at stake.

The bar for digital safety, not that high, because all you're gonna lose is money. Nobody's life is at stake. But yet, physical autonomy is ahead of digital autonomy in 2025. What seems counterintuitive, like, why is that the case at, you know, at a technical level? Why is it that what should sound easier is actually a lot harder?

Yeah, so I I think there are there are kind of two things at play here, and I really like the analogy with self-driving cars cause, you know, they they've been they've actually been, like, one of the best applications of AI, I think, that I've I've used recently. Um, but I think there are two things in play. One of them is, honestly, just the timelines. Like, we've been working on self-driving cars for so long. Like, I remember when I, you know, back in, like, 2014, it was kinda like the the advent of this, and everyone was like, oh, it's happening in, like, five years.

It turns out it took, like, I don't know, ten, fifteen years or so for this time. So there's been a long time for the technology to to really mature. And I think there's probably, like, dark ages, you know, back in, like, 2015 or 2018 or something, where it felt like it wasn't gonna happen.

That's right. The trough of disillusionment.

Yes, yes, yeah. And then now we're, you know, um, I'm finally seeing it get deployed, um, which is really exciting. But it has been, like, I don't know, ten years, maybe even twenty years from the very beginning, uh, of the research. Uh, whereas, uh, I think AI agents are, like, really in day one here, like ChatGPT only came out in 2022, uh, so, like, around three year like, less than three years ago. Um, but I actually think the, uh, what we think about with, uh, AI agents and all that really, I think started, uh, with the reasoning paradigm that, uh, when we released the the o one preview model back in, uh, la late last year, I think.

Uh, and so I actually think this whole reasoning paradigm with AI agents and the robustness that those bring has only really unfolded for, like, a year, uh, less than a year, really. And so, um, I I know you had a chart in your blog post, which I really like, which, you know, the the slope is very meaningfully different now. Like, self-driving started very, very early. Slope seems to be a little bit slower, but now it's reaching the promised land.

But man, like, we we started super recently with AI agents, and the slope, I think, is incredibly steep, and we'll probably see it cross over at some point. Um, but, uh, uh, uh, we really have only had, like, a year, really, to explore these, uh,

Do you think we haven't crossed over already when you look at, like, the coding work in particular?

Yeah, it's it's a it's a good point. It's like, you know, um, your chart actually shows, uh, AI agents as below self-driving, but, like, you know, it's like, what is the Y-axis? Like, by some measures, like, I would not be surprised actually if, you know, AI products or AI agent products are making more revenue than Waymo at this point. Like, Waymo's making a lot, but, like, just look at all the startups coming up, look at, you know, ChatGPT and how many subscriptions are happening there and all of that.

And so maybe we have actually crossed, and, you know, a couple years from now, it's gonna look very, very different.

That's a good point. Yeah. Yeah, yeah, yeah. Yeah, the Y-axis is tangible felt autonomy.

Don't pick the objective.

Yeah, yeah. How do I feel about, yeah, AIs?

Right Right. Exactly. Vibes more than, uh, revenue. But revenue is a good one. We should probably redo that with revenue.

The, um, there's a second thing I wanted to, uh, mention on on this as well, which is, uh, the scaffolding and the environment in which these things operate in. So, um, I actually remember in the early days of self-driving, a lot of the, like, researchers around self-driving were saying that the roads themselves will have to, like, change to accommodate self-driving, right? There might be, like, sensors everywhere so that the self-driving cars can interact with it. Um, which I think is, like, you know, retrospect overkill.

Um, but I actually do think self-driving cars have s a good amount of scaffolding in the world for them to operate in. It's, like, not completely, like, unlimited. Um, you have roads, roads exist, they're pretty standardized, you have s stoplights. People generally, uh, operate in, like, pretty normal ways, um, and there are all these traffic laws that that you can learn. Um, whereas, um, uh, AI agents are just kind of dropped in the middle of nowhere, and they kind of have to feel feel around for them.

And I actually think, you know, going off of what Olivier just said, too, my hunch is some of the, um, uh, enterprise deployments that don't actually work out, likely don't have the scaffolding or infrastructure for these agents to interact with as well. Um, a lot of the, like, really successful deployments that we've made, a lot of what our FDEs end up doing with some of these customers is to create, um, almost like a platform or some type of scaffolding connectors, organizing the data so that the models have something that they can interact with in a more standardized way.

And so my sense is self-driving cars actually have had this, uh, in in in some degree with with roads over the last, uh, uh, you know, over the course of their deployment. But I actually think it's still very early in the AI agents, uh, space. And I would not be surprised if a lot of these, uh uh, a lot of enterprises, a lot of companies, just don't really have the scaffolding ready. So if you drop an AI agent in there, it kinda doesn't really know what to do.

Uh, and and its impact will be limited. And so I think once this scaffolding gets built out, uh, across some of these companies, I think the deployment also speed up. Um, but again, to our point earlier, I think there's no slowdown. Uh, there's no, you know, things are still moving very fast.

That's great. Well, you know, uh, I I've thought about autonomy as a three-part structure. You've got perception, you've got the the reasoning, the brain, and then you've got the call it the scaffolding, the last mile of making things work. Uh, maybe we can dive into the second part, which is the reasoning, which is the the the juice that you guys are building with GPT five most recently. Huge endeavor. Congrats. The first time you guys have launched a full system, not a model or a set of models, but a full system.

Talk about that, I mean, the full arc of that development, what was your focus? I mean, honestly, the benchmarks all seem so saturated. Like, clearly, it was more than just benchmarks that you were focused on. And so what was the North Star? Like, tell us about GPT five. Soup to nuts.

It's been the work of love of many people for a long time. Um, and to your point, I think GPT five is amazingly intelligent. You look at the benchmark, like, you know, the sweet bench and the likes, you know, it is going pretty high. But I think to me, equally important and impactful was, I would say, the craft, like the style, the tone, the behavior of the model. So, you know, capabilities, intelligence, and, you know, behavior of the model.

Um, on the behavior of the model, I think it's the first model, like, large model release, for which we have worked so closely with a bunch of customers for, like, month and month, essentially, to better understand, like, what are the concrete, like, locks, like, what are the concrete blockers of the model? And often, like, you know, it's not about, like, you know, um, like, having a model which is way more intelligent, a model which is faster, a model that better, like, follows instruction, a model that is more likely to say no, you know, when, you know, it doesn't know about something.

Um, and so that, like, super close, like, you know, customer feedback loop on GPT five was pretty impressive to, uh, to to see. And I think, like, all the love that GPT five has been getting, like, you know, in the past, like, couple of weeks, I think people are starting to feel that essentially, the builders. And once you see it, like, it's really hard, essentially, to come back to a model, which is, like, extremely intelligent, but more, you know, an exclusively, like, academic, essentially way.

Yeah.

Yeah, yeah, yeah. Are there trade-offs that you made as you were going through it? Like, maybe what is the hardest trade-offs you made as you were building GPT five?

I actually think a very clear trade-off, which I think I honestly think we are still iterating on, is, uh, the trade-off between the reasoning tokens and how long it thinks versus, uh, performance. Um, because, and I honestly, this is something that I think we've been, um, working on with our customers since the launch of the reasoning models, which is these models are so, so smart. Uh, uh, especially if you give it all this, like, thinking time.

Um, I I I think the the feedback I've been seeing around GPT five Pro has been pretty crazy, too. It's just like, like, you know, these unsolved Yeah, yeah, I saw that Sam retweeted it. Um, uh, but, like, these, like, unsolved problems that none of the other models could handle, you throw it to GPT five Pro, and, uh, it just, like, one-shots it, is is pretty crazy. But the trade-off here is you're you're you're waiting for ten minutes.

Uh, it's quite a long time. And so, like, these things just get, like, so smart with with, um, uh, with more inference time. But on the, like, product builder, on the API side, for some of these, like, business use cases, I think it's pretty tough to, like, you know, manage that trade-off. And for us, it's been difficult to figure out where to where we wanna fall on that spectrum. So we've had to make some trade-offs on, like, how much of the model think versus, like, how intelligent it get.

Because as a product builder, there's a latency there's a real latency trade-off that you have to deal with where, you know, your user might not be happy waiting ten minutes for, like, the best answer in the world. They might be more okay with a substandard answer and, like, no wait at all.

Yeah. Andrea had a great tweet last night. Yeah. Yeah, yeah.

Do you think we haven't crossed over already when you look at, like, the coding work in particular?

Yeah, it's it's a it's a good point. It's like, you know, um, your chart actually shows, uh, AI agents as below self-driving, but, like, you know, it's like, what is the Y-axis? Like, by some measures, like, I would not be surprised actually if, you know, AI products or AI agent products are making more revenue than Waymo at this point. Like, Waymo's making a lot, but, like, just look at all the startups coming up, look at, you know, ChatGPT and how many subscriptions are happening there and all of that.

And so maybe we have actually crossed, and, you know, a couple years from now, it's gonna look very, very different.

Yeah. Yeah, yeah, yeah. Yeah, the Y-axis is tangible felt autonomy.

Don't pick the objective.

Yeah, yeah. How do I feel about, yeah, AIs?

Right Right. Exactly. Vibes more than, uh, revenue. But revenue is a good one. We should probably redo that with revenue.

The, um, there's a second thing I wanted to, uh, mention on on this as well, which is, uh, the scaffolding and the environment in which these things operate in. So, um, I actually remember in the early days of self-driving, a lot of the, like, researchers around self-driving were saying that the roads themselves will have to, like, change to accommodate self-driving, right? There might be, like, sensors everywhere so that the self-driving cars can interact with it. Um, which I think is, like, you know, retrospect overkill.

Um, but I actually do think self-driving cars have s a good amount of scaffolding in the world for them to operate in. It's, like, not completely, like, unlimited. Um, you have roads, roads exist, they're pretty standardized, you have s stoplights. People generally, uh, operate in, like, pretty normal ways, um, and there are all these traffic laws that that you can learn. Um, whereas, um, uh, AI agents are just kind of dropped in the middle of nowhere, and they kind of have to feel feel around for them.

And I actually think, you know, going off of what Olivier just said, too, my hunch is some of the, um, uh, enterprise deployments that don't actually work out, likely don't have the scaffolding or infrastructure for these agents to interact with as well. Um, a lot of the, like, really successful deployments that we've made, a lot of what our FDEs end up doing with some of these customers is to create, um, almost like a platform or some type of scaffolding connectors, organizing the data so that the models have something that they can interact with in a more standardized way.

And so my sense is self-driving cars actually have had this, uh, in in in some degree with with roads over the last, uh, uh, you know, over the course of their deployment. But I actually think it's still very early in the AI agents, uh, space. And I would not be surprised if a lot of these, uh uh, a lot of enterprises, a lot of companies, just don't really have the scaffolding ready. So if you drop an AI agent in there, it kinda doesn't really know what to do.

Uh, and and its impact will be limited. And so I think once this scaffolding gets built out, uh, across some of these companies, I think the deployment also speed up. Um, but again, to our point earlier, I think there's no slowdown. Uh, there's no, you know, things are still moving very fast.

That's one interesting trade-off, actually. Cause, you know, when you ask people developers, like, what do you want? Like, you want the model to follow instructions, of course, you know. But once you have a model which w- who is, like, that is, like, you know, extremely literal, essentially, then essentially it forces you to express extremely clearly what you want. Otherwise, the model, you know, may go sideways. And so that one was interesting feedback, yeah.

Yeah. It's almost like the monkey paw, where it's like developers and and platform customers ask for better instruction following. So we're like, yes, we'll give you really good instruction following, but it was like, you know, follows it almost to a to a T. And so it's obviously something that the the the team is actually working working through. I think a good example of this, by the way, is, um, some customers would have these prompts. I remember when we were testing GPT five, one of the, uh, negative feedback that we got was the model is too concise.

We were like, what's going on? Why is the model so concise? And then we realized it was because they were reusing their old prompts from other models. And with the other models, they have to, like, you have to, like, really beg the model to be precise uh, concise. So they're like ten lines of, like, be concise, really be concise, also keep your answer short. Um, and it turned out when you give that to GPT five, it's like, oh my gosh, this person really wants it to be concise.

And so the response would be, like, one sentence, which is too terse. And so just by removing the extra prompts, uh, around being concise, the model behaved in a much better way, uh, and much closer to what they actually wanted.

Yeah. Turns out writing the right prompt is still important.

Yes, yes, yeah. Prompt engineering is still very, very important. Yeah. Um, on constructive feedback for GPT five, um, there's actually been a good amount as well, which, uh, we're all we're all working through. Uh, one of them that I think is, uh, I'm I'm really excited, um, to for for the next, uh, you know, snapshot to come out to to fix some of this is code quality and, like, small, like, code, uh, like, paradigms or, like, uh, uh, idioms that they might use.

I think there are, like, feedback around, um, the types of code and and and the patterns in which it was using, which I think we're working through, uh, as well. Um, and then, uh, the the other bit of feedback, which, uh, I think we've already made good progress on internally, uh, is around the trade-off of the reasoning tokens and thinking and latency around intelligence. I think, especially for these simpler problems, you don't usually need a lot of thinking.

The thinking should ideally be a little bit more dynamic. Um, and, of course, we're always trying to squeeze as much reasoning and performance into as little reasoning tokens as possible. So I'd imagine that that code going down as well.

Yeah. Well, huge congrats. I mean, it's been, I know it's a work in motion for a bunch of our companies. They've had incredible outcomes with GPT five. One of them is expo, uh, cybersecurity business. Just like a huge, huge, huge upgrade from whatever they were using prior to that. And, um

Yeah, I saw the charts from that. It was pretty crazy. Yeah.

right. They're gonna need a new eval. It's all about evals. On on the multi-modality side of it, uh, obviously, you guys announced the real-time API last week. Uh, I saw T-Mobile was one of the featured customers on there. Uh, talk about that, like, how obviously the text models are are leading the pack, but then we've got audio and we've got video. Talk about the progress on the multi-modal, uh, models. Uh, when should we expect to have, like, the next big unlock, and what would that look like?

Uh, it's a good question. Um, the teams have been making amazing progress on multimodality, on voice, image, video. Frankly, the last generation of models have been unlocking, like, quite a few cool use cases. Um, one of the feedback that we've received is, you know, because, like, text was so much leading the pack on intelligence, like, people felt, like, in particular, on voice, that the model was somewhat a little less intelligent. And, you know, until you actually see it, like, it does feel weird, like, you know, to have, like, you know, to have a better answer, like, on text versus voice.

And so that's pretty much a focus that we have at the moment. Um, I think we, like, filled, like, part of that gap, but not the full gap, for sure. So I think, you know, catching up, I would say, you know, with the text, like, you know, would be one. Um, a second one, you know, which is absolutely fascinating, is, uh, the model is, like, excellent at the moment on, like, you know, easy, like, casual conversation, like, talk to your coach, your therapist.

And we basically had to teach the model, like, to, um, speak essentially better, like, in actual, like, you know, work, economically valuable setups. Give an example. Like, the model has to be able to understand what an SSN is and, you know, what it means to spell the SSN. And if one digit is actually, like, you know, fuzzy, it should actually has to repeat versus, you know, guess. You know, there are lots of, like, you know, intuitions like that that someone, you know, of course, has, you know, of our voice that we are currently, like, teaching the model.

And that's, like, an ongoing work, actually, with our customers. You know, until we actually confront the model to, like, actual, like, customer support calls, actual sales call, it's really hard, like, to get a feel, you know, for those gaps. So that's, um, a top, like, you know, priority as well.

Right. This is, uh, completely off script, but an interesting question that comes up in voice models, particularly the real-time API, is, you know, previously people were doing they were taking the speech input, convert that to text, then have some layer of intelligence. Then you would have a text-to-speech model that would sort of play it back. And this would be it would be a stitch of these three parts. But the real-time API, you guys have integrated all of that.

And h- you know, how how does it happen? Because a lot of the logic is written in in text. A lot of the Boolean logic or any call it any function calling is written in text. How how does it work with the real-time API? Is that

That's an excellent question. Um, so the reason why we ship the real-time API is that we saw that for the stitch model

Hm. The stitch model. Oh, the realtime. Yeah Yeah.

Yeah. The stitch, the stitch, yeah, model.

Oh, they're, like, a stitch together, right? Yeah, yeah, yeah.

Oh, stitch. Yeah. Like speech-to-text, thinking, text-to-speech. Like, we saw essentially a couple of issues. One, slowness, like, you know, hops, essentially. Two, loss of signal, like, you know, across each model. Like, the speech-to-text model is less intelligent, essentially. Exactly. Exactly.

I see. Yeah. Yeah. Yeah, yeah, yeah.

Yeah. Things like that.

Right. Buses. Yeah.

And, you know, when you when you are doing, like, actual voice, like, phone, like, calls, essentially, like, the those signals are, like, so important, like, you know, for this entity to understand. Um, one of the challenges that we have is what you mentioned, which is, you know, it means, like, a slightly different architecture, essentially, for text versus voice. And so that's something that we are actively working on. Uh, but I think it was the right call to start essentially with let's make, like, the voice experience, like, natural sounding to a point where, essentially, you're feel comfortable, like, putting it in production and then working backward, like, to unify, like, the sort of, um, the orchestration logic, essentially, across modalities.

And to be clear, like, a lot of customers still stitch these together. It's, like, kind of what worked in the last generation. But what we're increasingly seeing is more and more customers moving towards the real-time approach because of how natural it sounds, how much lower latency it is, especially as we up-level the intelligence of the model. Uh, but also even, like, taking a step back, I will say it's, like, pretty mind-blowing to me that it works.

Like, the fact that, like, I think it's mind-blowing that these LLMs work at all, where you just train it on a bunch of text, and it's just, you know, auto-regressively coming up with the next token, and it sounds super intelligent. That's, like, mind-blowing in and of itself. But I think it's actually even more mind-blowing that this speech-to-speech setup actually works correctly, cause you're literally taking the audio bits, uh, from it from someone speaking, streaming or putting it into the the model, and then it's generating audio bits back.

And so, uh, to me, it's actually crazy that it works at all, uh, let alone the fact that it can understand accents and tone and pauses and things like that, and then also be intelligent enough to handle a support call or something like that.

Yeah. Of course. Of course.

Yeah. Yeah.

So I I've been noticing, like, the same pattern, which is the the teams that build, like, breakout, like, startups in AI are extremely pragmatic. Like, they're not super, like, you know, intellectual, but, like, the perfect world, et cetera. And it's funny, cause I feel like, you know, our generation has basically started in tech in a very, like, stable, like, you know, moment where, you know, technology had been building up, like, you know, for years and years with, like, SaaS, like, cloud, et cetera.

And so we were, in a way, like, raised, like, you know, in that very stable moment where it makes sense at that point to, you know, design, like, very, like, you know, good, like, abstractions and toolings. Cause, you know, you have a sense of where it's going. But it's so different today. Like, no one quite knows what's gonna happen next year or two. So it's almost impossible, like, to define, like, the perfect tooling platform.

Right. Right, right, right. Well, that's, uh, there's a lot of that going around right now. Um, spicy. A lot of homework there. Uh, Olivier, over to you, sir. Long short.

Long short. I've been thinking a lot about education for the past month, uh, in the context of kids. Um, I'm pretty short on an education, which basically emphasizes, uh, human memorization at that point. And I say that having mostly been through the education myself. But, you know, like, I learned so much on, like, you know, history facts, like, you know, legal things that are, um, you know, some of it, like, does shape your way of thinking.

A lot of it, frankly, is just, like, you know, knowledge tokens, essentially. And those knowledge tokens, you know, it turns out, like, you know, LLMs are pretty good at it.

Yeah, yeah, yeah. Yeah, yeah, yeah.

Um, so I'm I'm quite short on that. Uh

That's right. You don't need memory once ChatGPT is bionic. You can just think about it.

there we go. Exactly, exactly. What am I long at? Um

Straight into your head, yeah.

frankly, I think healthcare is probably the industry that will benefit the most from AI in the next, like, year or two.

Interesting. I would say more.

I think, like, all the ingredients are here for a perfect storm. Um, a huge amount of, like, structure and unstructured data. You know, it's basically the heart of, you know, like, the the pharma companies. Uh, the moas are excellent at digesting, processing that kind of data. Uh, a huge amount of, like, admin, like, you know, heavy, like, documents heavy, like, you know, culture. But at the same time, like companies which are very technical, very R and D friendly.

Like, you know, companies, like, you know, who's, like, sort of technology in a way is at the heart of what they do. And so, yeah, I'm pretty bullish on the healthcare.

This is like life sciences. So you mean life sciences research organizations that are predicting drugs. Gotcha. Gotcha.

Yeah, it's it's almost like, you know, over the last twenty, thirty years, these these, like, pharma or, like, biotech companies have have basically, if you look at the work that they're doing, like, only a small amount of it is is actual research. And so much of it ends up being admin and, like, you know, documents and and things like that. And that area is just so ripe for, you know, something to happen with AI. And I think that's what we're seeing with Amgen and some of these other customers.

And it's also, like, not what they want to do. It's I think it's good that we have some regulations there, obviously, but, like, just means that they have, like, reams and reams of things to kind of go through. And so, you know, w- like, when you have a technology that's able to really help, like, bring down the the cost of something like that, I think it'll just, you know, tear right through it.

And I think once governments and, like, you know, institutions are going to realize that, like, if you step back, like, it is probably one of the biggest bottlenecks to, like, human progress, right? You step back in the past decade, like, you know, how many, like, true, like, breakthrough drugs have there been? Like, you know, not that many. Like, you know, how life would be different, like, if you double that rate, essentially. So once you realize what is at stake, yeah, my hunch is that we're going to see quite a bit of momentum in that space.

Wow. All right. Lots of homework there as well. Next one. Favorite underrated AI tool, other than ChatGPT, maybe?

I love granola.

Oh, man, you stole my you stole my answer.

Granola, I love so much granola. Like, um,

Two votes for granola.

there is something like, um,

Hey, what about ChatGPT record?

I like ChatGPT record as well, but, um, there are some features of granola which I think are really done well. Like the whole, like, you know, integration with your Google Calendar is excellent. And just, you know, the quality of, like, the transcription and, like, the summary is pretty good.

Do you just have it on? Cause I know your calendar is back to back. You just have granola on.

So the funny thing is that I don't use granola internally. Uh, I use granola for my personal life, mostly. Yeah. Yeah.

Oh, God bless. I see. I see. On dates. I'm joking.

Uh, I was gonna say, yeah, granola is actually gonna be mine. So, uh, two votes for granola. Um, I was gonna say the easy answer for me is Codex, uh, as a software engineer. It's just like so it's gone so good recently. Codex CLI, especially with GPT five. Uh, especially for me, I tend to be less time sensitive about, like, uh, you know, the iteration loop with with coding. And so, uh, leaning into GPT five on Codex, I think, has been really, really

Interesting. H- what about Codex has changed? Cause, you know, Codex has also been through a journey. Codex has been around for a bit. I remember, like, it's been launched for, like, a year more than over a year ago. So, like, what's what's changed about Codex?

I. Yeah, I was actually gonna say it's it's

Codex CLI has been around for a bit.

uh, so I feel like it's been less than a year for Codex. The time dilation is so crazy in this field, it feels like it's been around for real.

A few months, I would say. Like, a year ago, with GPT four-Oh, like, you remember that demo? Like, that feels like ages ago at that point.

Yeah, yeah, yeah. Oh, one hadn't even come out yet. Twelve days of Christmas hasn't happened yet.

The voice demo is there.

Maybe it was a naming thing, okay, but anyway, yeah.

Oh, there was a Codex model.

That's what I'm thinking about.

There's there was a Codex model.

We are. We are. Yeah. We are. You're not you're not to blame for that confusion.

We had Also, I think the the GitHub thing was called Codex for that as well. Yes, yes, yes.

We are.

That's right. That's right.

Uh, but I'm talking about our, uh, coding product within ChatGPT, which is the Codex Cloud offering, and then also Codex CLI. So actually, maybe if I were to narrow my answer a little bit more to Codex CLI, which I've I've really, really liked. I like the local environment setup. Um, the thing that's actually made it really useful in the last, I'd say, like, month or so is, uh, one, I think the team has done a really good job of just, like, getting rid of all the paper cuts, like the small product polish and, like, paper cut things.

It just is a it kind of feels like a joy to use now. I feel more reactive. Uh, and then, uh, the second thing, honestly, is is GPT five. Like, I I just think GPT five really allows the product to shine. Um, it's, you know, at the end of the day, this is kind of a this is a product that really is dependent on the model underlying model. And when you have to, you know, like, iterate and go back and forth with the model, like, four or five times to get it right, to get it to, like, you know, do the change that you want, versus, uh, having it think a little bit longer, and it just, like, one-shots and

does exactly what you want it to do. You get this, like, weird, like, bionic feeling where you're, like, I feel so mind melded with the model right now, and, like, perfectly understands what I'm doing. And so getting, like, that kind of, like, dopamine hit and, like, feedback loop constantly with Codex has made it, uh, uh, kind of, like, an indispensable thing that I really, really like. Um, and the other thing I'd say Codex is just really good for me is, um, so I use it in my, uh, for, like, personal projects.

Um, I also use it to, like, help me understand codebases, like, as a, like, as a, um, uh, engineering manager. Now I'm not as in the weeds on, uh, uh, on the actual code. And so you're actually able to use Codex to really understand what's happening with the code base, um, have it, like, an ask questions and have it answer about, uh, things and really catch up to speed on things as well. So, like, even the non-coding use cases are really useful with Codex CLI.

Fascinating. Sam had this tweet about Codex, um, usage, uh, ripping. I think, like, like, yesterday. So I wonder, uh, what's going on there? But you're, uh, you're not alone.

Yeah, I I think I think I'm not alone. Just judging from the Twitter feedback, I think people are really realizing how great of a combination Codex CLI and GPT five are. Um, yeah, I know that team is undergoing a lot of scaling challenges. But I mean, it the system hasn't gone down for me, so props to them. Um, but we are in a GPU crunch, so we'll see how, you know, how long that goes.

Awesome, awesome. All right. The next one, um, will there be more software engineers in ten years or less? There's about, what, forty fifty million

Full-time professional soft software engineers. That's what you mean, like, full-time, like, day-to-day jobs.

Yeah. Full-time, yeah. Yeah.

Yeah, cause it's a hard one, cause, like, I think without a doubt, there's gonna be a lot more software engineering going on. Um, there was actually

of course.

a really great post that was shared, I think in our internal Slack. It was like a Reddit post recently. I actually think that highlights this. Um, it was a really touching story. It was a Reddit post about someone who has a brother who's non-verbal, and I actually don't know if you saw this, it was just posted. Um, it's a person on Reddit posted, they have a non-verbal brother who they have to take care of. The brother, like, they tried all these types of things to help the brother interact with the world, use computers, but, like, vision tracking didn't work, cause I think his vision wasn't good.

Um, all the tools didn't work. And then, uh, this brother, um, uh, ended up using ChatGPT. I don't think he used Codex, but he used ChatGPT and basically taught himself how to create a set of tools that were tailor-made to his non-verbal brother. Basically, a custom software application just for them. And because of that, he he now has this, like, custom setup that was written by his brother, uh, and allows him to, like, browse the internet.

I think the video was, like, him watching The Simpsons or something like that, which is really touching. But I think that's actually what we'll see a lot more of. Like, this guy's not a professional software engineer, his title is not software engineer, but he, like, did a lot of software engineering, probably pretty good, um, good enough, definitely, for his brother to use. So the amount of code, the amount of, like, building that'll that'll happen, I think, is just gonna go through an incredible transformation.

I'm not sure what that means for software engineers like myself. Maybe there's, you know, equivalent, or maybe there's Of course, more even

shall we.

Yeah, more of me, more of me to go around.

We we will.

That's right. Yeah. But definitely a lot more software engineering and a lot of coding.

I buy that completely, like, I buy completely the thesis that there is a massive software shortage like in the world, like, we've been sort of, uh, accepting it, you know, for the past twenty years. But, like, the goal of software was never to be that super rigid, super hard to build, you know, uh, artifact. It was to be, like, you know, customized, like, malleable, uh, and so I expect that we'll see, like, way more a sort of a a reconfiguration of, like, people's, like, you know, job and skill set where way more people code, like, you know, I expect that product managers are going to code, like, more and more, for instance.

Um,

Yeah, you made you made your PM's code recently if you remember.

oh, yeah, we did that, that was very fun. Uh, we started, like, essentially not doing, like, PRDs, like product, uh, requirements documents, you know, classic P M thing, you write like five pages, like my product does that, etcetera. And, you know, P Ms have been basically, like, coding prototypes. And one, it's pretty fast with GPT vibes and, like, Codex,

Yeah, just a couple hours, I think.

like, freaking fast. And second, like, it sort of conveys, like, so much more information than a document. Like, you get a feel, essentially, for the feature, like, is it to write or not? So, yeah, I expect that sort of, you know, behavior we're going to see more and more.

Yeah, instead of writing English, you can actually now write the actual thing you want. Yeah, yeah. And, yeah, that's amazing. Advice for high school students who are just starting out their career.

My advice is I don't know,

Go for it.

maybe it's evergreen, like prioritize critical thinking above anything else. If you go in a field which requires, like, extremely high critical thinking, like, you know, skills, I don't know, math, physics, or, you know, maybe philosophy is in that bucket, you will be fine regardless. If you go in a field that sort of turns down that thing. And, again, gets back to, like, memorization, like, you know, pattern matching, I think you will probably be less future proof as a, you know, uh

What's a good way to sharpen critical thinking?

Use ChatGPT and have it test you.

that's actually, having, like, you know, world-class tutor who essentially knows how to put the bar, like, 20% above what you can do all the time, you know, uh, is actually probably, like, a really good way to do it. Yeah.

Nice. Anything from you, sir?

Um, mine is I I think it's just I think we're actually in such a interesting, like, unique time period where, um, the like, younger so, like, maybe this is a more general advice for not just, like, high school students, but just, like, the younger generation, maybe even, like, college students. It's like, uh, I I think the advice would be don't underestimate how much of an advantage you have relative to the rest of the world right now because of how AI native you might be or how like, you know, in in the weeds of the tools you are.

My hunch is, like, high schoolers, college students, when they come into the workplace, they're gonna have actually a huge leg up on, uh, how to use AI tools, how to actually transform the workplace. And my push for, like, some of the younger, I guess, high school students is, like, one, like, just really immerse yourself in this thing. And then two, just, like, really take advantage of the fact that you're you're in a unique time where, like, no one else in the workforce really understands these tools as deeply probably as you do.

Um, a good example of this is actually we had our first intern class, uh, uh, recently at at at OpenAI, um, a lot of, uh, software interns. Uh, and some of them were just, like, the most incredible cursor power users I've, like, ever seen. They were so productive. I was Yeah.

I was shocked either way.

Yep, yep, yep. I was like, Yeah, I know I know we can get good interns, but, like, I don't know they'd be, like, this good. Um, and I think part of it is just, like, they've grown up, um, using these tools for better or worse in college. Um, but I think the meta-level point is they they're they're so, like, AI native. And even, like, I don't know, me and Olivier, we're, like, kind of AI native. We work with OpenAI, but, like, we haven't, like, been steeped in this, uh, and kinda grown up in this.

And so, uh, the advice here would just be, like, yeah, leverage that. Like, you know, don't be afraid to, kinda, like, go in and spread this knowledge and take advantage of it in the workplace because it is a pretty big advantage for them.

Yeah, I can't remember who said this to us at Bouncer, but every intern class was just getting faster, smarter, like laptops, like smarter every generation.

You sure didn't peak in 2013, you know, when I was an intern. Yeah, that summer Two guys, of 2013. like Yeah. in

That's right.

Throughout the years. two.

That's right. That's right. That's right.

Well, lots happened, you know, lots happened since you guys joined OpenAI, right, what, three years and almost three years. Um, in your OpenAI journey, what has been the the rose moment, your favorite moment, the bud moment where you're, like, most excited about something, but but still opportunity ahead. And the thorn, toughest moment of your of your of your, uh, three-year journey.

The thorn is easy for me, what we call the blip, which is, you know, the crew of the board. Like, that was a really tough moment. It's funny cause, you know, after the fact, it's actually reunited quite a bit the company. Like, there was a feeling OpenAI had a pretty strong culture before, but, you know, there was a feeling of, like, camaraderie, essentially, that was even stronger. But, you know, sure, like, tough on the day off.

It's very rare to see that anti-fragility. Most orgs, after something like that, break break apart, but I feel like OpenAI got stronger. OpenAI came back.

It's a good point. I feel it made OpenAI stronger for real now, essentially, when I look at other fact when I look at, you know, other, like, you know, news, like departures or, you know, whatever, like, you know, bad news, essentially, I feel the company has built, like, you know, a thicker skin and, you know, an ability to, like, recover, like, way quicker.

I actually think part I think it's definitely right. Uh, part of it too, I think, is also just the culture. I also think this is why it was such a, you know, low point for a lot of people. So many people just at OpenAI care so deeply about what we're doing, which is why they work so hard. Um, you just care a lot about the work. It almost feels like your life's work, like, it's a very audacious mission and thing that we're doing, which is why I think the blip was, like, so tough on a lot of people, but also is what I think helped bring people back together.

And why we were able to hold together and and get that that thick skin as well. Yeah. I have I have a separate, uh, worse moment, uh, which was, uh, the big outage that we had in, uh, December, uh, last year, if you remem- yeah, you remember.

I do.

Uh, it was like a multi-hour, uh, outage. Um, really highlights to to, uh, to us how, uh, essential of almost, like, a utility the API was. Uh, so so the background is I think we had, like, a, like, a three, four-hour outage, uh, sometime in, uh, November or December of last year. Um, really brutal. PureSF zero. No one could hit ChatGPT. No one could hit the APIs. It was it was really, really rough. Um, that was just really tough just from a, like, you know, customer trust perspective.

I remember we we, like, talked to a lot of our customers to kinda, like, um, uh, t postmortem on what happened and kind of our plan moving forward. Thankfully, we haven't had anything close to that, uh, since then, and I've I've been actually really happy with, um, all the investments we've made in reliability over the last, uh, six months. Um, uh, but, uh, in that moment, I think it was really it was really tough.

On the happy side, like, on the roses, um, I think I have two of them. The first one would be GPT four was really good, like, the sprint up to GPT four, I think, really, like, you know, showed, like, the best of OpenAI, like, you know, having, like, cutting-edge, like, science research, like, you know, extreme, like, customer focus, like, you know, extreme, like, you know, infrastructure and inference, like, you know, talent. Uh, and the fact that we were able, like, to ship, like, such a big model and, like, scale it to, like, you know, many many tokens, you know, per minute, like, almost, like, immediately, I think speaks to it.

So that one, I really

With no with no outages.

with no outages.

Yeah, really good reliability, yep.

Like, I remember when we shipped, like, GPT four turbo, like, a year ago, a year and a half ago, we were terrified by, you know, like, the Yeah, the the influx of, uh, scale. traffic. And I feel like we've really gotten, like, much better at, you know, shipping, like, those, you know, massive updates. Um, the second, uh, rose, like, you know, happy moment for me would be the first dev day was really fun. It felt like a coming of age, like OpenAI, like, you know, we are embracing that, you know, we have, like, a huge community of developers, you know, we are going to ship models, new products.

And I remember basically seeing, like, you know, all my favorite people, OpenAI or not, you know, like, you know, home, essentially nerding out on, you know, what are you building? Like, you know, what's coming up next? It felt really, like, you know, a special moment, uh, in time.

No, that was actually gonna be mine as well. So I'll just piggyback off of that, which is the very first Dev Day, 2023, uh, November. Uh, I actually I remember it. So, I mean, obviously, a lot of good things have happened since then. There was just a very I don't know why, for for me, it was a very memorable moment, which was, uh, one, uh, it was actually quite a rush up to Dev Day.

We were we shipped a lot. So our team was just really, really sprinting. So it was, like, this high, you know, uh, uh, high-stress environment kinda going up. Uh, to add to that, um, you know, of course, because we're OpenAI, we did a live demo on Sam's, uh, and Sam's keynote of all the stuff that we shipped. And I just remember being in the back of the audience, sitting with, like, the team and, like, waiting for the demo to happen.

And once it finished happening, we all just, like, let out a huge sigh of relief. We were like, Oh my God, thank you. Um, uh, and so there's I think there's just, like, a lot of, like, you know, build-up to it. For me, the most memorable thing is I remember right after Dev Day, all the demos worked well, all the talks worked well. We had the after-party, and then I was just in a way mode driving home at night with the music playing, and it was just, like, such a great end to the Dev Day.

That was that was what I remember. That was my rose for the last, uh,

Love it. last year. Yeah. That's Love it.

awesome. I assume you guys are, but please tell me if you're AGI-pilled, yes or no. And if so, what was the moment that got you there? What was your aha moment? When did you feel the AGI?

I think I'm AGI-pilled. I think I'm AGI-pilled.

You're definitely AGI-pilled.

I am. Okay. Uh, I've had a couple of them. The first one was the realization in 2023 that I would never need to code manually, like, ever, ever again. Like, I'm not I'm not the best coder, frankly, you know, I chose my job, like, you know, for a reason. Um, but realizing that, you know, what I thought was a given that we humans would have, like, to write, like, basically machine language, like, forever is actually not a given.

And that, you know, the pace of price is huge. Feeling the AGI, um, the second, like, feel the AGI moment for me was, uh, maybe the progress on voice and multimodality. Like, you know, text, like, at some point, you get used to it. Like, okay, you know, the machine can write pretty good text.

Yeah, voice makes it real.

But once you start actually talking, like, you know, to something that actually understands your tone, like, you know, understand my accent, like, in French, um, it felt like, sort of a, uh, pretty new moment, like, okay, machines are going beyond, like, cold, mechanical, deterministic, like, you know, like, logic to something, like, much more, like, emotional and, like, you know, tangible.

Yeah. That's a great one.

Yeah, mine mine are, uh, uh, so I do think I I am a G-I-pilled. I probably gradually became A G-I-pilled over the Definitely. last couple years. I think there are two. And and for me, um, yeah, I think I I I actually get more get more shocked from the text models. I know the multimodal ones are are really great as well. For me, I think they actually line up with with two, like, general breakthroughs. So, um, the first one was right when I joined the company in September 2022.

We it was pre ChatGPT, Yeah, two months ago, um, but at the time, GPT four already existed internally, and I think we were trying to figure out how to deploy. I think Nick Turley's talked about this a lot, um, early days of ChatGPT. But it was the first time I talked to GPT four, and it was it was, like, going from nothing to GPT four was just the most mind-blowing experience for me. I think for the rest of the world, maybe going from nothing to GPT 3.5 in chat was maybe the big one, and then going from 3.5 to four.

But for me, and I think for a lot of maybe some other people who joined around that time, going from nothing to or or not nothing, but, like, what was publicly available at the time, going from that to GPT four was just incredible. Like, I just remember asking throwing so many things at it. I was like, There's no way this thing is gonna be able to give an intelligible answer. And it just, like, knocks it out of the park.

It was it was absolutely incredible.

GPT four was insane. I remember GPT four came out when I was interviewing with OpenAI, and I was still, like, on the fence, should I, Josh, should I join? And so that thing, I was like, Okay. I'm in. I'm in. There's no way I can walk on anything else at that point.

Yep, yep, yep. Yeah, so GPT yeah, GPT four was just crazy. Um, and then, uh, the other one was, uh, like, is the other breakthrough, which is, like, the reasoning paradigm. I actually think the the purest represent representation of that for for me was deep research, uh, and throwing, like like, asking it to, like, really look up things that I didn't think it would be able to know, and seeing it, like, think through all of it, like, be really persistent with the search, uh, get really detailed with the write-up and all of that.

That was pretty pretty crazy. I don't remember the exact query that I threw it, but I just remember, like, I feel like the field AGI moments for me are, like, I'll throw something at the model that I was, like, There's no way this thing will be able to get, and then it just, like, knocks it out of the park. Like, that is kind of the field AGI moment, and I definitely had that with deep research with some of the things I was asking it.

Yeah. Well, this has been great. Thank you so much, folks. You guys are building the future. You guys are inspiring us every day. And appreciate the conversation. Yeah. Thank you so Thank you. much. Thanks for having Thank you. me. Thank you.

As a reminder to everybody, just our opinions, not investment advice.