19 Sep 2024 · 48m · Bret Taylor · 21 quoted facts
Bret Taylor argues that the first AI agents worth building are 'company agents': branded, customer-facing agents that do what a company's website and call center do today. Sierra's pitch is that conversations get about ten times cheaper, and that companies can steer these agents with goals and guardrails rather than rigid rules. He also argues the AI market will resemble cloud computing, with a few model builders, tool vendors and many solution companies.
The main tension is control versus creativity. Companies want an AI that acts safely inside regulated, multi-system processes, yet too many restrictions make it robotic and too few invite hallucination and brand risk. Taylor says the answer is a platform that lets customer-experience teams set where the AI may improvise. He ties this to outcome-based pricing, which makes software vendors answerable for results, and ends on a hope that agents free people from screens.
Taylor says an average phone call costs about $13 once labor and technology are counted, while AI can handle it for well under a dollar. He expects companies to start by digitizing existing work and then to hold far more conversations because they are cheap.
Taylor says retrieval-based chatbots, which answer from a company's documents, are 'woefully insufficient' because real customer requests need actions. Those actions include upgrades, returns and claims, which often touch ten or more business systems and are bound by compliance rules.
Instead of a decision tree, a company states what it wants to achieve and where the AI may be creative. Too much control makes the agent robotic, while too much freedom risks hallucinations, broken policies and a poor brand image.
Taylor expects a few capital-heavy companies to build the foundation models, a layer of tools companies, and many solution companies like Sierra and Harvey. He says a startup is vulnerable if every new model release erodes its value.
Sierra charges for the job done, such as a resolved case, rather than for tokens or seats. Taylor compares this to the move from impression-based ads to cost-per-click ads, where buyers pay more for results that can be directly measured.
Bret Taylor describes a company agent as a branded AI that customers talk to for everything they could do on a company's website, such as buying, getting support or filing a claim. Personal agents work for an individual, and he thinks those are earlier-stage and will mostly come from big consumer firms like Apple, Google and OpenAI.
Taylor says a typical phone call costs about $13 to handle, while AI can bring that to well below a dollar. He calls that a tenfold drop in the cost of a conversation.
Taylor says that technique, called retrieval augmented generation, only lets the model summarize content a company supplies. Most real customer needs, like cancelling a subscription or filing an insurance claim, require taking actions across many internal systems under business and legal rules.
It means the vendor is paid when the job is done, such as a resolved customer case, instead of paying for usage like model tokens. Taylor says this aligns the vendor with the customer and makes the value easy to measure.
Taylor thinks not, comparing it to cloud computing, where Amazon, Microsoft and Google rent out infrastructure but software-as-a-service companies still thrived. He says startups survive by solving a whole business problem, and he warns that those who only add a thin layer on top of a model are at risk.
Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.
Retrieval augmented generation combines a large language model with a database of content so that answers are grounded in a company's own information rather than the model's pre-trained knowledge.
“you take a large language model, and rather than using the model and its innate knowledge from the pre-training process to, um, emit answers, you combine that model with a database of content”
Sonos customers who buy a speaker or have technical problems chat with a Sonos AI agent powered by Sierra for onboarding and troubleshooting.
“if you buy a new Sonos speaker or you're having technical issues with your speaker, you get the dreaded flashing orange light, you'll now chat with the Sonos AI, which is powered by Sierra to help you onboard”
SiriusXM's customer-facing AI agent, named Harmony, handles tasks such as upgrading and downgrading subscription levels.
“If you're a Sirius XM subscriber, their AI agent is named Harmony, which I think is a delightful name, and, uh, it's everything from upgrading and downgrading your subscription level”
Bret Taylor says a typical phone call to a call center costs about $13 to service, and that AI can reduce this to well below a dollar.
“for most phone calls, it's call it $13, you know, to to service that phone call. Now, with AI, you can bring down that cost to well below a dollar, you know?”
Cost per contact is a key service-team metric measuring the all-in labor and technology cost of fulfilling a customer contact such as a phone call.
“one of the key metrics for most service teams is their cost per contact, which is like, what is the all-in cost of the labor and the technology to fulfill that phone call?”
Bret Taylor expects personal AI agents to be built mainly by very large consumer companies such as Apple, Google and OpenAI.
“the domain of personal agents is probably of the very large consumer companies like Apple and Google and OpenAI and others that have big consumer brands.”
Bret Taylor cites Harvey as an example of a persona-based AI agent company that serves a legal function.
“there's companies like Harvey that, you know, serve a legal function.”
Bret Taylor compares company AI agents to the 1995 website, saying that in 1995 existing digitally meant having a website and a Yahoo directory listing.
“if it were 1995, you know, if you existed digitally, it meant having a website and being in Yahoo directory, right?”
Bret Taylor says most software for the past two decades has been rules engines, implemented as source code or low-code platforms, and that AI moves business software toward goals and guardrails.
“broadly speaking, most software systems for the past two decades have been rules engines that execute really quickly, whether the rules are implemented as source code or perhaps in a low-code platform.”
Bret Taylor says cloud computing consolidated into a small handful of infrastructure-as-a-service providers, such as Amazon Web Services, Azure and Google Cloud, that account for most cloud capital spending.
“you ended up with a small handful of infrastructure as a service providers that represent the vast majority of the capex investment in cloud”
Bret Taylor predicts only a relatively small number of companies will do pre-training, the capital-intensive part of building foundation models.
“we'll end up with a relatively small number of companies, um, doing pre-training, uh, you know, which is the really capital-intensive part of of model building.”
Sierra's platform is called Agent OS, and customers use it to define the goals and guardrails of their customer experience.
“our customers, you know, in our platform, which we call Agent OS, are essentially defining the goals and the guardrails of their customer experience.”
Sierra uses outcome-based pricing, charging customers for the job done rather than for model tokens.
“At Sierra, we're really focused on what we call outcome-based pricing, you know, charging for the job done.”
Bret Taylor compares outcome-based pricing for AI software to the advertising shift from impression-based ads to cost-per-click ads.
“reminds me a little bit of going to from impression-based ads to cost-per-click-based ads.”
Elad Gil says Klarna publicly reported AI-driven customer support gains and reduced its team by about 700 representatives.
“At the same time, they also reduced the size of the team by, I think, 700 representatives or people”
Sierra uses supervisor models, in which one AI model oversees another, to enforce limits such as avoiding medical or financial advice.
“And we do a lot of what we call supervisor models, so we have models supervising other models.”
Bret Taylor explains that instruction tuning, which makes large language models conversational, lets them reflect back the sentiment and tone of the person they are talking to.
“because of instruction tuning, which is the mechanism of making these large language models conversational, they'll naturally sort of reflect back the sentiment and tone that you have”
Bret Taylor says the number one reason for bad customer experiences is waiting, and that instant AI responses address this.
“The number 1 reason people have bad customer experiences is they had to wait.”
Bret Taylor says an AI agent can be updated on a new product at the push of a button, while retraining 5,000 call center agents would take far longer.
“think of how long it would take to retrain, you know, 5,000 agents in a call center about that new product. Well, you can do that with a push of a button with AI.”
Bret Taylor argues that over roughly the past 15 years the smartphone has absorbed most adjacent technologies and made it hard for other consumer devices to reach scale.
“the smartphone has come to basically consume all, you know, adjacent technologies.”
Bret Taylor says conversational interfaces passed an inflection point of effectiveness, probably with GPT-4.
“Now that conversational interfaces work effectively, and I just think we passed that inflection point, probably with GPT-4, though that's a interesting debate topic on its own.”
Taylor splits agents into personal, persona-based and company agents, and explains why company agents are ready to build now.
“there's probably 3 categories of agent that I think are or or on”
Examples like Sonos and SiriusXM show what a branded customer-facing agent does in practice.
“we help companies build branded customer-facing agents.”
The economics of a $13 call versus under $1 explain why consumer companies gain most and why more conversations become possible.
“you can bring down that cost to well below a dollar”
Taylor explains why answering questions is not enough and how companies must balance AI creativity against policy and brand risk.
“And now we're moving to a world of goals and guardrails.”
He maps AI onto cloud's layers of infrastructure, tools and solutions, and says what that means for startups.
“the AI market will rhyme with the cloud market”
Charging for results changes the vendor relationship, much as cost-per-click changed advertising.
“charging for the job done”
Taylor names analyst-style synthesis work as a promising area and notes where models struggle.
“the Iron Man suit, you know, for analysts”
Agents can mirror a customer's tone and language, and supervisor models help enforce brand limits.
“your agent should be a brand ambassador”
Taylor asks whether conversational AI will loosen the smartphone's grip and let technology recede.
“what is the primary form factor by which we work with computers”
Hi, listeners, welcome back to No Priors. Today, we have Bret Taylor, whose legendary career spans from creating Google Maps to serving as the CTO of Facebook and co-CEO of Salesforce, founding 2 companies along the way, as well as chairing the board of Twitter and now OpenAI. He and Clay Bavor have started Sierra, which is creating company agents for the next generation of customer experience. I'm thrilled to have such an amazing technologist and leader at all scales with us today and longtime friend, welcome, Bret.
Well, thanks so much for joining us today, Bret.
My pleasure. Thanks for having me.
Let's get right into it. Um, do agents work today?
How do you define agents? Or do you want me to define agents?
You define agent, you're the expert.
Agents mean something different in academia than I think they mean in industry right now. Um, I think both definitions are important. Just starting, what I view is sort of the classic academic definition is a an agentic system is one where software can reason and take action autonomously, and it comes from the word agency. And as a consequence of such a broad academic definition, I think it becomes sort of the proverbial inkblot test for people using the word.
In industry right now, there's probably 3 categories of agent that I think are or or on the cusp of working. The 1st, I think, which a lot of people online talk a lot about, is personal agents, and I think that's probably the earliest of the 3 categories that I see, but but maybe one of the more exciting ones, you know, and this is the agent that will triage your inbox, schedule a vacation, um, help you prep for a meeting, manage your calendar, all of that.
And the reason why I think that's earliest, I think it's really interesting to make some demos, but I think the human-computer interaction and even how the agents interact with all the systems we depend on as people is quite complex. You can think of sort of the surface area of both reasoning and systems integrations is almost infinite. And so, as a consequence, I think it probably a prerequisite for a great personal agent probably demands more technology than is currently available, though there's lots of interesting startups in this space, and you could imagine some interesting companies carving out meaningful niche use cases that expand as the technology improves.
The 2nd category of agent, and I think this one does exist in some categories, is what I call persona-based agents. So, they're agents that do a job, a very specific job. Um, you know, there's companies like Harvey that, you know, serve a legal function. Um, there's all the coding agents, I think there's some fairly effective ones right now that serve the job of a, yeah, a computer programmer. Um, I think this is really exciting because I think when you narrow, I call those cases narrow but deep.
If you're just trying Both task to
scope and, uh, perhaps integration scope.
that's right, the tools you access and even how you evaluate the effectiveness is, you know, if you're building a coding agent, there's actually really good benchmarks already. Similarly, compilers emit error messages, and you might have integration tests. You end up with this scaffolding that actually, practically speaking, limits the scope of, sort of, the true research that you have to do to accomplish it. I think, broadly speaking, with the advent of foundation models, a lot of effective AI right now is where you've taken areas of research and you made them areas of engineering, and I think you can engineer very effective, uh, persona-based agents for certain domains where the technology applies, like the law, um, like, uh, uh, areas of of
software engineering and things like that. My take is the domain of personal agents is probably of the very large consumer companies like Apple and Google and OpenAI and others that have big consumer brands. For the persona-based agents, I think there's probably meaningful companies in each of those spaces because I think to do those effectively, it involves sort of the confluence of AI expertise and expertise in that domain. The other category, which is the area that my company, Sierra, works in, is what I call company agents, um, and it's really, uh, less simply about automation or autonomy, but in this world of conversational AI, how does your company exist digitally?
I always use the metaphor, if it were 1995, you know, if you existed digitally, it meant having a website and being in Yahoo directory, right? In 2025, existing digitally will probably mean having a branded AI agent that your customers can interact with to do everything that they can do on your website, um, whether it's, you know, asking about your products and services, doing commerce, doing customer service. Um, that domain, I think, is shovel-ready right now with current technology because, again, like, the persona-based agents, it's not boiling the proverbial ocean, technically.
You know, you have well-defined processes for your customer experience, well-defined systems that are your systems of record, and it's really about saying, in this world where we've gone from websites to apps to now conversational experiences, what is the conversational experience you want around your brand? And it doesn't mean it's perfect or it's easy, otherwise, we wouldn't have started a company around it, but it's at least well-defined. Um, and I think that right now in AI, if you're working on artificial general intelligence, your version of agent probably means something different, and that's okay.
Um, that's just a different problem to be solved. But I think, you know, particularly in the the areas that that Sierra works and a lot of the companies that you all have invested in, is this saying, you know, are there some shovel-ready opportunities right now with existing technology? And I absolutely think there are.
Can you describe the, um, like, shoveling cycle of building a company agent? Like, what is the gap between research and reality? Like, how do you, um what do you invest in as an engineering team? Like, how do you understand the scope of different customer environments? Just like, what are the sort of vectors of investment here?
And and maybe it's hard to unpack,
Yeah, of course.
but as a starting point, it may even be worth also defining, like, what are the products that Sierra provides today for its customers, and then, where do you want that to go? And then maybe we can feed that back into, like, what are the components of that? Cause, I think, o- obviously, you folks are really emerging as a leader in your vertical, but it'd be great just for our broader audience to understand what you focus on.
Yeah, sure, I'll just give a couple examples to make it concrete. So, if you buy a new Sonos speaker or you're having technical issues with your speaker, you get the dreaded flashing orange light, you'll now chat with the Sonos AI, which is powered by Sierra to help you onboard, help you debug, whether it's a hardware issue, a Wi-Fi issue, um, things like that. If you're a Sirius XM subscriber, their AI agent is named Harmony, which I think is a delightful name, and, uh, it's everything from upgrading and downgrading your subscription level to if you get a trial and you purchase a new vehicle, speaking to you about that.
Broadly speaking, I would say we help companies build branded customer-facing agents. Um, and branded is an important part of it. It's it's part of your brand, it's part of your brand experience, and I think that's really interesting and compelling because I think just like, you know, when I go back to the proverbial 1995, you know, your website was on your business card. It was the 1st time you had sort of this digital presence, and I think the same novelty and probably will look back at the agents today with the same sense of, Oh, that was quaint.
Uh, you know, I remember if you go back to the Wayback Machine, you look at early websites, it was either someone's phone number and that's it, or it looked like a DVD intro screen with, like, lots of graphics. You know, a lot of the agents that customers start with are often around areas of customer service, which is a really great use case. But I do truly believe if you fast forward 3 or 4 years, your agent will encompass all that your your company does.
I've used this example before, but I like it. But just imagine an insurance company, all that you can do when you engage with them. Maybe you're filing a claim, maybe you're comparing plans. Uh, we were talking about our kids earlier, maybe you're adding your child to your insurance premium when they get old enough to have a driver's license. All of the above, you know, all of the above will be be done by your agent. So that's what we're helping companies build.
And Sierra's initially focused on facing, like, um, consumer-facing companies.
Yeah, we're the vast, vast majority of our customers are consumer companies. Technically speaking, there's not a huge difference between sort of a B2B company and a consumer company except for the volume of customers that you have. And I always like to think from 1st principles about what does this technology enable that was impossible before? And if you think about the typical cost of a conversation, so, if you call into a call center today, uh, one of the key metrics for most service teams is their cost per contact, which is like, what is the all-in cost of the labor and the technology to fulfill that phone call?
You know, for most phone calls, it's call it $13, you know, to to service that phone call. Now, with AI, you can bring down that cost to well below a dollar, you know? And so all of a sudden, you've literally decreased the cost of a conversation by an order of magnitude. And so, if you're just doing the math o- on that, like, what companies would benefit most from that cost? And, um, you know, I'm not sure, depending on the math equation, see the the numerator, the denominator, but if you're measuring in in millions of consumers, obviously, the value is is really different for consumer companies.
For a lot of consumer brands, because having conversations is a really expensive thing to do, you don't necessarily make it easy, you know? Um, there's entire websites devoted to finding companies' phone numbers because often, in some ways, are really consumer-friendly. You push these towards these digital, self-service experiences. I'm really excited about now that having conversations with your customers is an order of magnitude cheaper, maybe you can do it an order of magnitude more. You know, what does that actually mean?
So with these technology trends, I think you often start with just digitizing what you currently do. But I actually think the 2nd order effect will be, gosh, now that having a conversation isn't a formidable cost center, how do I actually wanna incorporate having a conversation as a key part of my customer experience? So going back to your question, I think that's a much more meaningfully different conversation with a large-scale consumer company than it is with, say, a B2B company with 100 customers.
It doesn't mean it's not valuable. I just say the level of impact and the difference in the decisions you make are are quite different.
Can you describe some of the key challenges in, like, taking, you know, the capabilities of foundation models today and then making them work in, like, the company agent context?
One of the techniques, and I think you all have probably talked about on your your podcast, that's very common today is it's what's called retrieval augmented generation. Um, and essentially what that means is, uh, you take a large language model, and rather than using the model and its innate knowledge from the pre-training process to, um, emit answers, you combine that model with a database of content, and you say, use the content as a source of truth, and you ask the model to summarize, uh, selected content from that that database.
And that's kind of a roundabout way of saying you can ground the agent in knowledge that you provide it, but also you can take off-the-shelf models and and integrate it with proprietary business data. So, it's a really popular technique right now. I would say that's a really exciting area, but what we've found in practice is that broad category of technology investment is woefully, uh, insufficient for almost any meaningful customer experience. If you think about, you know, all of the interactions you've had with brands that you care about, what percentage of those conversations were asking questions?
Probably none of them. It's all about taking action, right? It's upgrading or downgrading a subscription, it's returning an order, it's a warranty exchange, it's a, you know, filing a claim with an insurance company. All of those are not only not simply answering questions, but also taking action against probably 10 plus systems of record. It's probably a very complex process. Often, that process has both business goals, you know, how do we, you know, prevent you from canceling or convince you not to?
There's probably compliance goals. If you imagine being a, you know, HIPAA compliant, you know, a healthcare adjacent firm, you have a lot of restrictions on what you can and can't do. You might be in a truly regulated industry, and all of that means that, you know, this idea of sort of building agents that can be grounded in content is a great demo, but actually not necessarily an impactful product. That's the area of technology we've really tried to solve.
Um, we are really trying to create a platform where you can orchestrate a process of arbitrary complexity, not simply have agency in the AI, but also have guardrails as well. Um, you know, broadly speaking, most software systems for the past two decades have been rules engines that execute really quickly, whether the rules are implemented as source code or perhaps in a low-code platform. And now we're moving to a world of goals and guardrails. And so people, businesses now have the opportunity to express a business process not simply as a set of rules and a decision tree, but saying, What are you trying to achieve?
Where do you want the AI to have agency? I e, where do you want it to have creativity? And where do you not want it to have creativity? Um, and it's a remarkably interesting technical problem. It's also remarkably interesting, I would say, social and business problem. A lot of companies will start out, um, saying, I wanna control precisely what the AI does, which is a fine goal, and actually, our platform does support it. But if you do that, it can be fairly robotic, and you're actually removing a lot of the magic that people feel when they engage with things like ChatGPT, which is fundamentally the creativity and agency innate in, uh, some of these models.
On the other hand, if you turn that knob up to, you know, this is Spinal Tap 11 on agency, you know, you could get hallucinations, uh, it could violate your policies, or more subtly, it could just not be a great brand ambassador, you know, for what your brand does. So I would just say that I I think there's a really deeper thing that we're trying to build, which is how do you program against nondeterministic creative software?
What are the abstractions that we need to build to express goals and guardrails so that you don't remove the creativity and the agency that I think make these experiences delightful? It's why ChatGPT got to 100,000, 000 users faster than any service in history. But also, you can represent to your board, your CEO, your customers, that there's the right guardrails in place. And then there's, like, where you actually are comfortable, where are you comfortable with this AI having agency?
So it's a really fun technical problem. I think it's also really a new design problem, almost a a philosophical question about where you want to seed certain amounts of creativity to software in a way that just wasn't a conversation one could have more than a couple years ago.
What, uh, how much do you think those different, uh, aspects you mentioned, the guardrails or in some cases, I've seen people working on agents build their own reasoning engines and other things, you know, their own modules that go on top of the core foundation models or LLMs. How much of that do you think, as a company, you need to keep doing yourselves versus will eventually get integrated into the core model companies, like OpenAI or Anthropic or people like that?
If you don't mind, I'll zoom way out for a second to give you my view of the marketplace, and then I'll I'll jump into that question. There's a Mark Twain quote, History doesn't repeat itself, but it rhymes. I think the AI market will rhyme with the cloud market of the past 15 years. And if you look at how that played out, broadly speaking, you ended up with a small handful of infrastructure as a service providers that represent the vast majority of the capex investment in cloud, uh, ie, most software as a service companies pay rent to one of those infrastructure providers like Amazon Web Services or Azure or Google Cloud.
And again, the because there's economies of scale in data center development, it didn't make sense for a startup to either build their own data center or for a startup to actually build, uh, infrastructure as a service business, just the capital expenditures required and that positive, uh, feedback loop on capex just didn't work out. I think that will probably play out with the frontier models. Um, we'll end up with a relatively small number of companies, um, doing pre-training, uh, you know, which is the really capital-intensive part of of model building.
Uh, not because, you know, there's not they're the only places with good researchers, but again, if you look at the capex requirements to actually make a return on that capex, it really you wanna lease it out to a large number of people. And then for a lot of companies that, especially startups who have done pre-training, they're finding, like, the making a return on that is is, uh, questionable mathematically, uh, at least for Do do you all of us think in the long on the
run that just ends up being the main cloud providers or cloud providers plus one or two other players? Because fundamentally, to your point, there's a capex and ability to afford it side. The second piece of it is if you're actually running all your application, all the data, everything on some on these cloud providers, pinging out to a third-party service just adds latency. So you add the round trip, you add a second sort of buying behavior around, um, approval, budget, uh, security, et cetera.
So do you think it's just gonna roughly consolidate around the clouds plus or minus?
I do think it will roughly end up, the cloud providers in partnership with the big research labs, which is is roughly the current, uh, you know, landscape. I'm not sure I completely agree on the security and latency front. It's possibly true. It was interesting, I think that, you know, most companies, most large enterprises now use multiple cloud providers. Uh, most of them use software as a service and don't necessarily care where it's hosted, as long as the security and reliability requirements are met.
And and there's obviously some exceptions to this, but I think thanks to 20 years of software as a service, people have sort of evolved their expectations to not ask, you know, Where do you get your power? and just say, What is your, you know, SLA for this service? And I think that's probably a positive trend. So I do think there's probably meaningful latency and security issues to overcome, but I that all being in the same substrate, I'm not sure I make that leap.
I might be wrong. I just, you know, I I view the evolution of software as a service having evolved that. But going back to my history rhyming point, I think you'll have a relatively small number of foundation model builders doing pre-training. I think there will be a market of tools companies. Um, you know, well, great one in AI might be Scale AI. Um, you know, Snowflake was a great example in cloud that might also be an example in AI.
And I think all those tools companies, and it's the proverbial pickaxes in the gold rush. If you're trying to transition to the cloud, what software do you need? If you're trying to transition to using AI in your business, what what are the tools and software that you need? And then the final category would be solutions. Just like in the cloud era, you you can take, you know, the services from Amazon Web Services or Azure or GCP and build almost anything, but most companies don't want to.
Most companies want to solve a problem, and the total cost of ownership of building your own CRM or ERP system is nonsensical. Um, and I think it, you know, took a long time for companies to realize that, but certainly they have now. I think the same will largely be true of AI. You know, if you want to, uh, you know, automate customer service, working with Sierra is much easier and lower cost than building it yourself. If you wanna, you know, automate parts of your legal process, talking to Harvey is probably a much more logical path than trying to roll your own.
For all the same reasons it was true of software as a service. So broadly speaking, going back to your question, you know, how do you build technology? And what will the foundation model providers do? I think the high order bit is what is the value you're providing, and how do you decouple, and are you adding enough value on top of models to be a real company? And I the answer is, if every time there's a new release of an AI model, somehow it decreases your value, it probably indicates you're you're not actually a solution, you're maybe it might be a slight value add on top of the models.
I think there's a number of startups that, unfortunately, sort of s smell like that. You know, it's not not necessarily a lot of value, but
What happens for Sierra when models improve?
if we're doing our job right, our platform gets better. You know? So, you know, I think that our customers, you know, in our platform, which we call Agent OS, are essentially defining the goals and the guardrails of their customer experience. And every time we have new technology available to make that work more effectively, we plug it in and, you know, you get better case resolution, better customer satisfaction, you know, fewer negative experiences, and that's just great.
In the same way, you know, when any web service that you provide from a software as a service company just gets better when the technology gets better, um, that's that's effectively what we wanna provide. But what our customers are hiring us to do is not related to the models, it's related to their customer experience. So, fundamentally, that's the way we think about it. And as a, uh, an entrepreneur, I would, you know, I think there's danger if you don't fit into one of those lanes, at least that's my opinion, because there's a real question of, you know, when a model improvement comes out, you know, if that was 50 of the value you provided, you you you're in this sort
of uncanny valley of of value. But I do think the idea that all use cases will come from foundation models is probably wrong. I mean, it's I'm it's hard to predict the future right now. But I think that would be the equivalent of saying, you know, 15 years ago, Gosh, there's not gonna be a single software as a service company. Everyone's just gonna build their own, or from the Lego bricks provided by
There were enterprises that said that. Yeah. Yeah, and I actually For a long time, think, uh, I yeah.
actually think it perhaps the opposite came true there. And, you know, most businesses, like, I always like to know, like, Where do you wanna innovate? You know? Like, with the relatively few engineers you have, if you're a, uh, large retailer, you don't have the resources to implement everything yourself. Like, where do you wanna stand out? Where do you wanna stand apart? And for most companies, you know, they benefit from the rising tide lifting all boats of investing in a software as a service platform.
I see the same thing happening here. So, I'm very bullish on, uh, the, but, like, going back to our definition of agents, you know, all the companies creating persona-based agents, they'll obviously compete with each other, but I think there's meaningful, you know, companies in that space, and I would probably work with them over, assuming it's coming from the foundation model providers, because they're solving all the unique problems. This will take a coding agent of developer workflows, of security, of different programming languages, all these things.
I actually think there's a ton of value there, and I also think there's probably second-order effects of relying on coding agents, how they incorporate into your team, governance, code reviews, all these things that I, uh, I'm I'm not necessarily thoughtful enough to enumerate right now, but that's why there's a company in this space, you know? And and I'm very bullish on that company existing for the long term, not even knowing half the half the I think to your names.
point, the analog with SaaS is a really telling one because people always talk about wrappers on foundation models and how those companies will go away, and you could argue that a lot of SaaS is like a wrapper on a SQL database, right? It's kinda like the same thing in some Yeah, I think sense.
the same was true, probably, said of Shopify, Salesforce, ServiceNow, and those are all great companies, you know? So
It's interesting. They ended up being very, like, let's say, the database vendor to Salesforce ended up being a very important vendor for a long time, and, like, it did actually get yanked out eventually.
for what it's worth, like, what's really interesting about the cloud market, if you buy my analogy, um, and analogies are always sometimes dangerous too, and, well, let's say the exceptions to the analogy can sometimes be hidden. But, you know, the foundation model providers will all benefit from this investment, you know? These, uh, I really do think these foundation models have a ton of innate value, and, you know, so any solution built on top, the foundation model providers will collect a tax on on all those amazing use cases, and that's really great for everyone involved.
Um, as you sort of alluded to, you know, different application companies can decide to use different models at different times, so it creates a lot of, probably healthy competition, um, there as well. And the most important thing for, say, Sierra customers is we're future-proofing our customers from, you know, that both future-proofing so you don't end up in the, you know, dreaded situation of something breaking when new technology comes out. Um, but in a more meaningful way, when there is great new technology coming out, can I just turn it on?
You know? Can I benefit from it? And I think for a lot of the solutions and applications companies in this space, that will end up over time one of the main values they provide. You know, for anyone who's experimented with prompt engineering or prompt engineering with tool use, and kind of, I would say, the low level of these models, it's not like your prompts just work with future. I mean, the model could be better, but it's not strictly better.
It's actually, you know, there's a very tight fit between the tokens and the model and all these things. And, you know, while there's lots of interesting tools there, I'm not sure that's, like, the layer that most companies should or will want to be operating. Just like, you know, your company doesn't want to know that you're doing a database migration. It's boring, but important. And, you know, what software as a service provides is you don't need to care about that, and you There's no can.
downtime. Yeah.
Yeah. The other thing that's interesting is business model. At Sierra, we're really focused on what we call outcome-based pricing, you know, charging for the job done. I see a lot, you know, startups in this space doing it. That's another really powerful part of software as a service in the era of AI is I think you can, you know, I think the best AI companies are aligning their business model with their customers business models, uh, charging for the outcome, you know?
And I think that's a really powerful new, uh, business model, um, maybe as powerful as the idea of subscription-based software in the era of software as a service. The again, providing an out-of-the-box solution and aligning, you know, the actual business model of your company with the outcome is very meaningful, and it's very meaningfully different than than paying for tokens, you know? And that and I think actually building that alignment with your customers is valuable, um, as well.
Also, commitment to that suggests, like, a lot of confidence and ambition in, like, how valuable these solutions can be as well. It's it's not trivial, but one of the most exciting things about, M solution companies, application companies in the landscape is, like, you do see, like, magnitudes of, like, value improvement versus the existing solutions.
Yeah, I think for if you talk to economists, they'll talk about software as drivers of productivity and sometimes in a very abstract way. And certainly, if you, you know, sometimes it's really obvious, like, I I can't remember it, but the pre- and post-Microsoft Excel in finance departments, it has to have driven just first reasoning, like, a slide rule versus Excel or a calculator versus Excel, like, of course, it drove productivity. But for the past 20 years, it's been quite indirect, you know?
And and everyone who's Or incremental listening gains. or incremental, but every every person who's listening who's been, like, in an enterprise sales cycle has presented some slide on return on investment, ROI, and there's all these, like, ROI calculations, and you spend all this time trying to, like, you know, if you get every person gets 5 more of this, and I don't wanna say it's BS, but it's like, you know, I think a lot of procurement and IT folks Indirect.
have seen, like, 100 of those presentations, and you're like, Did we actually decrease the number of people in the department? Did we actually measure those things? I actually think in the age of AI, because these systems can autonomously take action with the appropriate guardrails, we're closer to actually software actually doing a job that's quite measurable. If there's an analogy, I sort of reminds me a little bit of going to from impression-based ads to cost-per-click-based ads. It doesn't mean you're totally towards the transaction, but you're getting closer, and I in that transition, which Elad and I have sort of lived through at Google, customers are just willing to pay disproportionately more for the click because even if you could sort of
halfway measure some of the impression
Direct attribution.
stuff, the closer you are, the direct attributions is worth a ton, and it's a great thing for companies right now that, you know, you should be holding your software, you know, providers to a higher standard, you know, and, you know, and and not, and you should get closer to the value. And I actually think that's a great trend. And, you know, going back to our analogies of legal and coding and service, like, you can actually see the value it produces this function, you know?
It it actually analyzed this contract, you know, it did this thing, and you're like, that I actually know how to value that. Like, I we've been valuing that in our employees for a long time. You know how much you'd have to pay a consultant to do X, Y, or Z, you know, your cost per contact in your call center, and that's really remarkable. I think that's gonna really change the relationship between software vendors and companies. I think it will really make software vendors true partners to the companies they work with, if done appropriately, because you're actually delivering valuable, it's actually measurable.
I think it's an incredibly positive change. Because you talk to any CIO and you ask them, Are you getting the value you hope from all the software you purchase? You'll see, like, the blood drain from their face. They'd, you know, they'll have horror stories, right, of, you know, the difference between the
S dejection.
Yeah, the dejection, and it's complicated. I think this is a really positive trend.
Maybe a very high-profile example of that, um, was Klarna, where they publicly talked about how implementing effectively customer support workflows, uh, for their own business, I think, ended up with dramatically higher net promoter scores, higher customer satisfaction, less time per customer. They basically automated a bunch of workflows. At the same time, they also reduced the size of the team by, I think, 700 representatives or people, and so it had a huge impact in terms of the how their business functioned and how they were able to deal with customers and the languages they could support people in and all the rest of it.
So, it seems like there are these very sort of prominent examples now emerging in terms of this massive impact that you're talking about.
I think the the impact is here, and that's why I'm really excited about many of the companies sort of in the application space, 'cause I think they're closer to the tangible value right now, um, as opposed to, like, broadly,
What do you think are some of those other key application areas? You mentioned what I view as sort of the 3 most popular ones right now in terms of adoption by enterprise, which is basically coding, uh, customer success. I think there's a lot of effort right now on, uh, ongoing in sort of sales productivity or sales and marketing productivity. Are there other areas that you think are c- uh, you mentioned legal? Are there other areas that you view as sort of the most near-term next wave of these areas where, you know, it's very clear that these things will will be very impactful?
I'm not sure this is one job, but I'm really excited for automating the role of an analyst, uh, especially back office analysis, and not necessarily replacing, but sort of the Iron Man suit, you know, for analysts. If you think about the very superficially high-level role of an analyst, it's to synthesize complex data to provide insights to stakeholders. And, you know, if you think just first principles about what large language models are good at, which is, uh, summaries, synthesis, reasoning, I think there's a really some interesting applications there.
It does seem complex, you know? Language models aren't necessarily good at numerical or tabular data without a lot of work. Um, domain-specific data might have, uh, you know, uh, connotations or complexities that aren't necessarily present in foundation models on their own. So it strikes me as one of those areas, like coding, like the legal, where actually there's benefits to fine-tuning, there's benefits to domain-specific expertise. As I said, I'm not sure analyst is a role. I think there's probably different departments have different analysts, but I I if you look at a company and a larger firm, you know how many people's job it is to take data, make a presentation, you know, all these Do some things and do some transforms.
transforms, and again, I think that whether it's, you know, replacing, I'm not sure, but certainly augmenting and making that tremendously more effective, more real-time, I think that's that's really exciting as well.
Can we go back to, um, you you said goals and guardrails for a minute? Um, like, um, as you describe, we're going away from, you know, complex rules engines as business software. That's, like, a pretty big mindset shift for your customers to ingest. How do you work with them on, um, or I guess, evaluation of, like, how well Sierra agents work and get people comfortable with that?
Yeah, so a couple things. I'll describe technically, and then talk a little bit more operationally as well. So, technically, we work a lot with our customers to actually formalize and define their processes, you know? And, um, sometimes our customers come in with really well-defined processes, sometimes they don't. We like to say an agent's made up of not only the factual knowledge but the procedural knowledge, you know, what the process follows in addition to the the the integrations with systems.
And we spend a lot of time talking about, Where do you want guardrails? Where do you want creativity? And where do you want agency? And then we do a lot of experimentation, you know, uh, in a proof of concept, have it live, and actually through sort of, uh, this technology, encountering the cold, hard reality of actual people, you know, did this actually meet the expectations you thought you had? And then with that, we've developed a lot of tools for customer experience teams.
Uh, so we think that AI should not be the domain of technology teams exclusively. Um, you know, the team that owns your customer experience at your company, maybe it's in the office of the chief digital officer, maybe it's a formal customer experience team. They should be the ones with their hands on the steering wheel of these experiences. So we built a a lot of tools and platforms where those teams can audit and improve the agent and actually have their hand on the steering wheel for, you know, what their agent does over time, and it's not something that's ever done.
Going back to sort of the deeper question on making people comfortable, these are very organic systems. So if you just imagine you're a retailer and you go to a retail website, um, there's probably a menu somewhere on it that goes over all the categories you have: men, women, shoes, pants, whatever it might be, and you click on them and it filters the listings, and there's sort of a standard retail template at this point. I'm not sure it's the best, but this is, like, the world that we live in.
If you imagine having a conversational AI agent, it's a freeform text box. So it is completely freeform. So it's my going back to my bad analogies, it's a little bit like going from Yahoo directory to Google search. You know, you have a taxonomy of everything you can do to you have a freeform text box saying, What do you want us to do? And as a consequence, it tends to be a lot broader than I think people originally contemplate.
Uh, I think it tends to there tend to be, uh, sort of a long tail of customer experiences that not only did we not design the agent for, but our customers did not anticipate either. And I think that's a really interesting, deeper question.
We talked about, like, a crazy, like, fraud return case where nobody knew what was going on.
Yeah, exactly. I mean, there's just it's a voice of your customer quite literally. Um, so I think that's a really exciting dynamic. Uh, there's a book called The Long Tail, and I I associate it a lot with Google. I think maybe Eric Schmidt wrote the foreword to it if I'm remembering correctly, but I do think as the world of the internet transitioned from directories to search and the number of web pages increased, you ended up with not only big popular sites, but this long tail of of blogs, and, you know, it was a really it's is and was a really remarkable part of the web.
I think we're kind of moving it towards that in customer experience where you curate the few screens available to your customers, and if you move to a world of an AI agent, um, you can just say, Sorry, I can't help you with that, but probably what you will do is treat it more like paint-by-number. You know, Wow, here's the things our customers want to talk to us about, how do we fulfill that that desire and that need?
So it's a really interesting combination of customer insight, but also, I think, a very new way of developing customer experiences that is much more organic.
So it's gonna be adaptive as people learn.
Quite adaptive. It's an always-on
The water.
system, and it's not just, like, running an A/B test. It's a little bit less controlled, like, it's a it's a system and or organism that you're constantly so a lot of our platform is, how do you empower customer experience teams to manage that? Um, and I The new think it's super
edge case, the emergent customer behavior, not model behavior.
that's exactly right. Like, what what new thing is happening in the wild today? External events, controversies, products that were popular that got changed, you know? And how do you not only just get insights from that, but how do you actually constantly evolve this agent in a way that doesn't remove the the, uh, agency of the customer experience team whose job it is to define this, but also embraces the natural, organic, emergent behaviors of AI?
If you, um, scroll forward, I don't know what it is, 6 months, 12 months, like, say, we have the text box, right? Um, voice mode is coming, video avatars exist now. Is what we should expect that the Sierra avatar is, like, you or Clay or something, like, more personified and richer? Like, does fidelity matter, like, that?
It does, and actually, it's really fun to go to some of the Sierra agents in the wild and just see the radically different personalities, um, in each of those agents. I think that your agent should be a brand ambassador. What's so remarkable about large language models is their ability to observe the sentiment of the person talking to it, you know, because of instruction tuning, which is the mechanism of making these large language models conversational, they'll naturally sort of reflect back the sentiment and tone that you have, but you can also control it and modify it.
So for your brand, if you want an irreverent brand, you can have that. Um, if you want a more austere brand, you're, like, a luxury brand, you can have that.
Do you do that as part of the prompts, or do you do post-training, or how do you actually implement that into your product?
A combination of all of the above. You know, there's some parts of tone and brand that are adequate for prompts depending on the models. There's some parts of brand that are more sensitive. You know, like, you don't want your AI agent giving medical advice or giving financial advice, and that's sort of tone, that sort of substance. And we do a lot of what we call supervisor models, so we have models supervising other models. Um, it's turtles all the way down.
Our joke in our office is the solution to every problem in AI is more AI, um, and which is really exciting. And I think it is the fun part of our platform is, you know, we have a lot of tools, uh, at our disposal to solve these meaningful problems. I think it is really exciting. In the same way, I always think of Apple when I think of brand experiences. If you go to their office in Cupertino or you walk into an Apple store, you unbox their product, it's kinda got the same vibe, you know?
And you see that made in California, designed in California, and you're like, This is an Apple, uh, experience I'm getting. I think you should, you know, uh, think about your agent as a part of your brand experience, and and because it can have personality, you know, it could change per person. That's really different. You know, it's sort of the difference, like, Black Friday maybe 15 years ago, everyone got the same campaign. My guess is this Black Friday, most people's incoming emails will be personalized.
So we've kind of moved, you know, towards more personalized experiences where agents start off with one personality, and then, you know, maybe a few years from now, people have the confidence of saying, Let's actually reflect back the the personality or demographic of the person talking. The short answer is, you know, we're not there to prescribe that for our customers, but the fact that it's possible is really cool. I mean, that's just awesome. And you talk about language, uh, what a remarkably empathetic thing to be able to reflect back the language of the person speaking, not the language of the people you've staffed your call center with.
And it enables something that would have been previously cost prohibitive to do something that's remarkably empathetic. Um, and and the other thing, you know, there's the delight and personality of of chat, voice, video. Video avatars will be mind-blowing, you know? And that's that's the face time-ing with, uh, a brand. It's just like a pretty cool idea. I also think, you know, your point on you were talking about, I think, the Klarna use case, I think we can't underestimate just how impactful this is for consumers.
The number 1 reason people have bad customer experiences is they had to wait. Um, particularly in contexts of things like customer service, like, you know, for most inbound interactions, something is not right, you know? You have a need that needs to be fulfilled no matter how effective the person is on the other side of that email or chat or phone number, you're not gonna be connected or resolved instantly. And I think this is the opportunity of AI, and it's why, you know, customer satisfaction, MPS, can really be driven.
And it's not an indictment of the people that were doing it previously. Those people are inherently disadvantaged by being behind a your number 10 in line, you're on hold, right? And so
It's a scale mismatch.
by the time you're off hold, you're already not that happy, and it and and, you know, the great people on the other side can maybe turn that around, but this opportunity AI is is instant. And I think that's remarkable. I mean, that's it's all I also try to remind our customers, like, don't overthink this. Like, instant gratification is actually one of the main values of these systems, the rest is gravy, you know?
Yeah, even just how you think about staffing, because you can suddenly support multiple languages with a single agent versus, you know, with a human, it's hard to know 30 languages. So even those sorts of things, to your point, really impact the queue and the customer.
And slang, and jargon, and idioms, you know, I think that it's completely unreasonable to some expect someone to speak 10 languages, know a term, know this. Um, you also, there's a lot of really subtle things, so let's say your company introduces a new product. Um, think of how long it would take to retrain, you know, 5,000 agents in a call center about that new product. Well, you can do that with a push of a button with AI.
So there's just so many interesting, you know, second-order effects of this technology that is incredibly beneficial for every consumer.
So you have this amazing vantage point on the industry. You know, you were, um, CTO of Facebook quite early on, you were COO of Salesforce, you're on the board of OpenAI, you're running Sierra. We've talked a lot about languages and applications you mentioned, sort of, briefly, video avatars and things like that. Are there other big technology trends that are being impacted by AI or other modalities that you're very excited about outside of the core sort of enterprise language use cases?
Or what what do you think are some of these big trends that are coming?
The trend that I would be really interested in is, what is the primary form factor by which we work with computers and software in the future? My narrative around the last not quite 20 years, but 15 years has been the smartphone has come to basically consume all, you know, adjacent technologies. Um, I don't know if it's possible to measure, but what percentage of human-computer interaction is through a touchscreen on a smartphone right now? 90%, 99%? I don't actually know, and and it depends how you measure it and all that.
It doesn't mean keyboard and mice went away, you know, it's it's for professional tasks as opposed to everyday interactions. And I think that's really interesting, and it's been, uh, almost impossible even for large consumer companies to create more consumer devices that can actually reach scale because the strength of the smartphone at being pretty good at a lot of things has essentially removed the market for everything else. Now that conversational interfaces work effectively, and I just think we passed that inflection point, probably with GPT-4, though that's a interesting debate topic on its own.
You can speak to software and it works now. So, just like multi-touch meant that people could give up their BlackBerrys, does the exist emergence of multimodal voice-to-voice models, chat, which has already, I think, reached that point, and obviously video in the future, does that mean we'll we'll see a shift in in the proverbial consumer device in our pocket in a in a more meaningful way?
Do you have a hypothesis on form factor?
Because of the perseverance of the smartphone, probably if I had to pick, I think the smartphone will remain, but coupled with things like AirPods and, uh, CarPlay and others, you'll interact with it more through different modalities. Um, but the anchor supercomputer in your pocket probably won't go away. But I don't say like I'm hoping for it. You know, I brought it up because how many consumer device companies have tried to build something on the side of a smartphone that was a, you know, perfect fit?
I, you know, I think smartwatches maybe could qualify as a success, but still it's not nearly the market of a smartphone. I think that's really interesting, and I also wonder there was a I don't know what years it was, um, but, you know, when everyone got Alexas on their counters, it was what, a decade ago?
Yeah, I wanna say, like, 2015, 2016, maybe. 2016. Will Yeah.
those make a comeback? You know, will all of a sudden those become effective computers again? Will it make, you know, smart, uh, headphones, uh, trendy again? And then the other thing that I'm really interested in and I'm, um, I don't know whether it'll be optimistic or pessimistic about, which will be, will we spend less time staring at screens? Clearly, the ability of conversational AI to both speak to us, you know, through language and voice, um, and our ability to engage with computers through through language and voice, certainly, theoretically, it means you don't need to have the screen in front of your face the whole time.
Will it mean that technology recedes more into the background, or will it mean, you know, it'll just add on to everything else? I'm hopeful that product designers can take advantage of that. So a lot of the things that require us staring at our screens, the, you know, sort of, like, huge bag of push notifications that suck us back in, can an AI agent help us synthesize some of that so we don't end up with the reflexive response to pick it up?
That might be naively optimistic, but I'm just hopeful. Like, now that we have these new ways of engaging with computers that aren't simply through this one device and one screen, even if it is mediated by that device, technically, I'm excited for that just because I do think we've sort of reached at least a local maxima of, like, what that experience is. And now, you know, just imagine
It's not plugged into your brain yet.
Yeah, I know. That could be interesting
Global namespace.
human brain interfaces. I'm very excited about the I I brought this example recently, but if you remember the 1st apps in the App Store, they were, like, flashlight and things like that, skeuomorphic literal interpretations of what is this hardware capable of? And then future generations said, Okay, what are the confluence of a GPS and a screen and the this and the internet, and you got really meaningful things like WhatsApp and and Uber and and Instacart and DoorDash.
My sense is now that speaking to software has reached a, you know, sort of event horizon of effectiveness, will there be, like, meaningful parts of the computing experience that we depend on that are, like, conversational first, and will that mean that you can use it in completely different ways?
You know, you mentioned, uh, uh, the device may be going away more, but there's 2 ways of going away. There's it's not front and center, and you're not constantly staring at it, so the way you interact with it changes. The 2nd part of it is if things become very conversational or personality-driven or or whatever it may be, what proportion of your social interactions or your day-to-day interactions shift to a computer interface versus a human? So if instead of instead of chatting with a customer support rep, you're chatting with a agent, there may be other applications like that.
I I'm sort of curious how you think about what proportion of human time will go to interact with other humans versus interact with digital, uh, agents or other things over time as well. Do you view that as a trend in one direction or the other? I
I'll I'll give you my what I want the world to be answer and and we can dive into cynicism if you want to. Um, I'm hopeful in this world of AI, agents will become a meaningful part of our experience, uh, in our personal life, in our business life. With these agents, with the appropriate guardrails and safety, software can take action on our behalf, and by doing that, it enables us to not have to do those things and be present in the world that we live.
And, you know, whether let's just take a, you know, a Sierra agent that represents a company, maybe your personal agent's chatting with it. You know, maybe, you know, when you're trying to figure out, you know, this problem, your agent's acting on your behalf, and you can just live your life. You know, I think the purpose of technology, um, is to, you know, solve a problem for us, and and, you know, hopefully in the world of AI and, you know, the agency afforded by AI, technology can melt away and recede in the background.
There's obviously examples of people, you know, speaking with avatars. There's the, you know, uh, things like the the metaverse and all of that, and I think those are meaningful, and and certainly AI will change the landscape of how deep and substantive those spaces are, but I'm hopeful for most people, that's an evolution of w- how we think of video games and things like that, they're a meaningful form of entertainment. But, you know, you can put your phone down.
You can take off the VR goggles and have a conversation and have to spend less time poking buttons on a computer to get it to do things and have your agent do it on your behalf.
That's a great note to, um, end on. Thanks for the conversation, Bret.
My pleasure. Thank you.
Find us on Twitter at NoPriors Pod. Subscribe to our YouTube channel if you wanna see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And send us for emails or find transcripts for every episode at no-dash-priors.com.