Transforming Data into Business Solutions with Salesforce AI CEO, Clara Shih

You're listening to Gradient Dissent, a show about making machine learning work in the real world, and I'm your host, Lucas Bewald. Clara, she is the current CEO of Salesforce AI and the founder of Hearsay Systems. She's also the host of the Ask More of AI podcast. I've known Clara for a long time, and I've always admired her work, and this is an awesome chance to dig into her whole career and how AI is related to that.

I think the place to start is, Y, you have this really interesting job as CEO of Salesforce AI, and I'd love to hear, you know, what that's like, like, what do, what do you work on day to day, what do you think about?

Yeah, it's kind of a. It's a. I feel like I have the the most fun job at the company. Um, I mean, Salesforce has been working on AI for almost ten years. We have this amazing research group, a lot of, uh, former academics who are just, I mean, some of the smartest AI researchers that I I've ever met. And we also have had a lot of applied AI in our products on the predictive side. And so this new role was created last year, um, to really bring everything together and make sure that we weren't building redundant stacks.

You know, that we had one model gateway, that we were building one Einstein trust layer to handle, you know, citations and data masking and grounding, that we were building one data cloud, one vector database, um, so that all of our applications and also our ISVs could take advantage of it.

And you know, it's funny. I've watched, like, um, you know, companies for the past twenty years kind of centralize their data team and then disperse their data team, and then centralize their AI team and disperse it. Always seems like, you know, the challenge with, like, centralizing a team is that you can get like further from the applications. Like, how do you think about that?

Yeah, we've we've had periods of that happen at Salesforce too. I think that's one of the reasons why, uh, I mean, I was told when I I was asked to do the job that for me, coming from the application side was really important. You know, I'd worked on our platform before, you know, fifteen years ago, but after that I started an applications company, I started hearsay systems. You know, we were using NLP and predictive AI to help financial advisors post the right content, send the right content to the right client.

And then when it c- come back to Salesforce in 2020, I was leading the service cloud applications business. And so I think I always try to keep keep myself in mind, you know, as the as the customer. As we build out the platform and these shared services, whether it's our our prompt builder or it's our trust layer and gateway, or it's our new copilot studio.

Totally, and I guess, um, maybe we we should go back a little bit. I didn't actually realize that hearsay was so, um, NLP oriented, but I guess that makes sense. Do you wanna say a little bit about what that did and how, um, AI fit fit in there?

Yeah, so the the here part of hearsay is mining signals, mining, um, tweets and social media posts, and now text messages and web forms for interesting insights, and also actually for compliance infractions, since we focus on selling to regulated industries. And then the say part of hearsay has always been OK, based on what your network is interested in, based on the life events and trigger events that are happening in aggregate in your target market. Here's the type of content, here's a predictive AI engine on the types of content you should share to be a thought leader, to create, to generate demand, um, to to establish credibility, and really address on those hot button issues.

And and I guess, like, um, were were you using NLP, like, right from the beginning? Was that always part of it? I mean, you started, I think, back in, I'm trying to remember, like 2009 or or ten. I mean, it was must have been a really different environment.

It was such a different environment. Um, I mean, our version of NLP in 2009 was we were regular expressions. And we were looking for people who, um, posted something something with pounds and ounces. We knew that they would probably had just had a baby and we would flag that. Uh, we would search for people who, um, you know, there were a lot of congratulations, uh, you know, surge of congratulations in the comments. We knew that there was something meaningful and we started to to pick apart what that would be, and we would build regular expressions around them.

And we kind of gradually got what we thought was more sophisticated. Now, of course, in hindsight, in 2024, looking back, the techniques that we used, um, even just a few years ago, and I think this is true for any company. It just seems, you know, so so elementary compared to what you can do now.

And then I guess when you, when you took over service cloud, what, what were the big AI applications there? Was it similar stuff?

Well, it's such a fascinating time. So, I mean, service cloud is different in the sense that most of the customer support teams that use service cloud, they don't have a one to one relationship the way that hearsay's customers do right. Hearsay's customers are insurance agents and financial advisors and commercial bankers, and they have actual client relationships. They have a book of business of clients and they ha- they do active prospecting in the customer service world. It's usually, you know, except for the very high end, it's it's first come, first serve, right?

It's you call in, or you email in, or you chat in, and whoever's available in the queue that you get assigned to, that's who services you. And joining s salesforce and and taking over that business in 2020, it was a fascinating time because it was right in the midst of the pandemic, the early pandemic. I I came back to the company in late November, early December 2020. This was before there was a vaccine. Um, people were still sheltering in place and working from home, and that's very unusual for a contact center.

You know, customer support environment where they're used to being together and helping each other troubleshoot and being able to, you know, wave down a a supervisor when there, there's an issue. And so there, ha- there was some major capacity issues, um, in terms of the supply side of of labor, of people who are able to answer calls, respond to emails. On the other hand, though, there was a surge in customer demand in a whole bunch of different categories.

So you can imagine, like the airlines and hotel companies and cruise companies, like a lot of people calling in to cancel or ask about, you know, credits and refunds. Um, uh, in financial services, you know, are the stock market tanked? So you had tons of people, you know, calling in wanting to talk to, um, their their broker or their contact center about their portfolio. Um, unfortunately, on, you know, in the healthcare side and in the life insurance side, there were a lot of deaths in the early pandemic.

Again, this is before there were there was a vaccine and before there was Paxlovid, et cetera, and so there was a lot of of, there's a surge in in demand there. So you had this like perfect storm, where you have a huge spike in customer support volume at the same time that your available pool of workers, like a lot of people at that point are either exiting the workforce or not able to work their full num- num amount of hours.

And that created a real need for AI. And so even before then, right there, we had we had Einstein bots as our chat bot solution, we'd had AI agent assist, but I'd say it was like more niche, right? It was only like the most sophistic- sophisticated customers that were using AI. What the pandemic did was it created this need, where there was a surge, and we we grew Einstein AI 700% in in adoption in those twelve months, right from from December 2020 to December 2021.

Um, as more and more companies realize, OK, this needs to be part of my standard operating procedure. I've been putting off, um, setting up a chat bot for years. This is the time to do it. I have to get back to my customers. It's simply unacceptable for them to call in and to be put on hold for two hours, which was happening in a lot of those categories that I was mentioning, and it's unacceptable for them to email and not hear back from us for a week.

So Einstein AI, this is actually clarifying. I'm I'm a longtime Salesforce customer and and fan of Salesforce, but I've I've never quite understood exactly what Einstein AI, um, is. It's actually a service that makes a chat bot for a customer for external use.

That's right. Um, so Einstein AI, um, was initially two things. It was, uh, on the service side, it was a chat bot. It is a chat bot, and it's it's AI assist for a human service rep. And then during the time I was, I was leading service cloud, we expanded our charter from, you know, those two use cases to actually doing kind of more, um, data mining to optimize the overall system of customer service. And what do I mean that by that?

I mean taking all of the conversation transcripts, be they back and forth emails between a support rep and a customer, or chat logs, or, um, voice transcripts. And we started to mine them for things like, what are the top reasons why customers are calling, what are the top resolutions? So that we could continuously find opportunities to create either deflection with a chat bot and self-service, or we would identify bottlenecks where we needed to do a better job routing to the right agent.

And or we found areas where agents, either in aggregate or specific agents, needed more personalized training in a particular subject matter that maybe they weren't doing a good job handling the first time. So we we kind of expanded our charter into those three areas of bots, agent, assist, and then overall systems, um, insights and optimization.

Interesting. And and what was the most popular strategy or or people use all of them? Like, I, you know, I feel like I haven't encountered a lot of customer service where it feels like they've completely flipped over to a bot. And I'm kind of surprised, like, I feel like where I experience chatbots still today, it's often a frus- it's a more frustrating experience than I would expect. Given, you know, what I see in the state of the art in in LLMs.

Yeah, I mean, certainly no one, no one enjoys customer service service. First of all, customer service is one of those things where you don't get any credit, and no one writes a positive review when you get it right, right? It's like, it's just kind of like, assumed, and you move on with your day as it should be. Um, you only hear about the bad cases, and it's sometimes it's really hard to get right. And so, uh, I'd say the most popular, you know, even during during that that pandemic surge I was talking about, was doing the mining and understanding what the the top issues were.

And, um, doing two things. The first thing wa- is opening self-service, not just with bots, but you can have a self. You can have self-service on your website, in your mobile app, right? It doesn't have to be a bot experience. And having kind of more of an explicit navigation menu sometimes, or having search on your website or mobile app, sometimes that's a less frustrating experience than with a bot. Certainly pre LLM era, because you can have more control over exactly what you're trying to get done.

And then the other thing we figured out that was really neat was that, especially given the the Black Swan event of the pandemic, that a lot of times customer support issues, actually, there might be a surge of them, but they're actually many of them are related to each other. They cluster into these these topics, and sometimes it's due to some sort of crisis, some sort of major incident. There's a power outage, and so everyone's calling PG.

E. There's, you know, a natural disaster, so then everybody is calling the public agency and the city government and the insurance company. And so we also came up with this, uh, we expanded our data model to capture what we call incidents, which are, you know, it's like a parent incident with many child cases, where we could address them all at once. Instead of feeling like overwhelmed and having to address each one on its own. Um, so that's we.

We just learned a ton about, but that now certainly LLMs make bots a lot better. We're completely remaking Einstein bots to take advantage of our Einstein generative AI, and I'm sure, you know, every bots company is doing the same thing. So I, I, I hope and expect that it'll get better in the coming months.

Interesting. So, um, so you're currently running off of your own foundation model, or your own kind of language model, and then you're gonna move that to, like, a third party is. Am I understanding that that right?

It's it's a mix. So we've, uh, you know, Salesforce has has this open architecture, we're like this with our, with our data centers, as well with our hyperforce architecture. So at the model layer, same thing. We think that it's too early to call, you know, the winners, and I don't think there's gonna be one winner. I think there's gonna be different models that are right for different tasks, based on, you know, specificity and size of model and need for cost, latency, and performance.

And so what we've done is we've initially launched with our own Salesforce research models. It's called code gen, and it's primarily used for our developer GPT product. So it's the code snippet and test case generation for for Apex code and also for flow, uh, within Salesforce. And then we're actually using, um, we're all actually offering partner models on the foundation model for the other use cases. So we're we're partnered with with OpenAI, with Anthropic, Google, Cohere, and Amazon.

Interesting. Why do you? Why do you partner with so many do? Do you offer that to your customers to pick which one they wanna use, or do you think different ones are better for different use cases?

Both. We have some customers, especially in in the Fortune one hundred, Fortune 500. They're very specific about which models they want to use for what use cases. They may have even fine tuned, um, versions of these models, and they can, they're able to bring their own API keys to the Salesforce Einstein platform. I think most customers, you know, to your other point, they they just want us to choose for them. And so we're continuously evaluating for CRM specific use cases.

When does it make sense to use, you know, GPT 3.5 versus four versus Anthropic versus a different model?

Interesting. And and I guess what are the CRM specific use cases? I mean, now you've actually, you know, you've broadened your scope beyond, um, service cloud. What? What are customers asking for?

Yeah, so we're, um, it's pretty neat, we're. We've approached this in terms of like, uh, phases. So what we have out in the market today in GA are are our applications, basically role specific, out of the box use cases. So you can think of these as, you know, hard coded prompts that Salesforce product managers have optimized for very common bottlenecks and very common automation opportunities. So, for example, in the sales cloud realm, we've got account summarization and opportunity summarization.

You know, you're a salesperson, you're about to walk into a meeting. We're able to generate the latest account information, including external news about the customer, as well as all of the interactions that, you know, your customer success team and your marketing team, et cetera, have had with this customer. So that you're fully up to date and informed as you walk into the meeting. Um, on the customer service side, you Wait, sorry, know,

can I ask? Wait, wait, sorry. Can I use that? Like, what? why? uh, why is it not prompting me to turn that on or something? I would I'd love to use that.

you should use it. It's called sales GPT.

How do I use it? Where do I? I have to turn it on, or how? Like, what, how do I, um, Yeah, you, you enable it?

have to buy the product. It's it's called sales GPT, and I would love for you to start using it.

Okay, awesome. Oh, all right, go on, go on, sorry to interrupt.

yeah, so that that's just one example in sales cloud. Um, there's in in marketing cloud, there's subject line generation for email campaigns, um, in in in segment generation in service cloud, which is the world that I came from. There is reply recommendations for emails and chats that come in that are grounded on your knowledge articles. There's case summarization, so at the end of an issue, as you resolve it, instead of having to manually remember every troubleshooting step, we we use large language models to draft that for you.

So these we have a bunch of these out of the block, out of the box templates that are out there that are being used today by customers. And then what we're g a ing this week, which we're really excited about, is our prompt builder, so it's part of our platform. And what the prompt builder does, it comes with our our prompt library. Is that for any of these out-of-the-box use cases, whether it's by role or by industry, customers can go in and they can customize it.

So almost every Salesforce customer has customized their Salesforce instance, you know, usually with a custom field or custom object of some sort. And so, for example, you could take one of these prompt templates for reply recommendations, and you could ask it. You could have it reference a specific custom field in your Salesforce org. Now, all of a sudden, the grounding is much better. It's much more tailored to your business, just as an example. And then, of course, within this prompt builder, our customers are able to to create their own prompts and kind of play around that with that as well.

I think that's really important, yeah, especially for customers to to build trust with generative AI. They kind of want to understand what they're getting before they they deploy an open-ended gen AI experience. And then also in other use cases, there's, you know, frequently accessed prompts like a reply recommendation, where you wouldn't want everybody in your contact center having to prompt a co-pilot every time. So that's gong. We're also releasing, um, in beta, our copilot studio, which is that natural language, open-ended interface, um, for customers to ask any question of their data across.

Not just their Salesforce data, but data cloud, which federates all of their their data lakes and data warehouses across the enterprise. And then to take action on, um, across again, their Salesforce universe, as well as their MuleSoft universe of, I mean, all those APIs and functions and and workflows across the the company.

Well, so super cool. Um, and so I imagine you have some kind of like rag setup, where it's it's querying your database and then coming back, and then and then looking at the results that's right. So and coming back we, um, to kind of summarize we, we built them.

rag, um, within our VPC, that's very important for our customers in terms of of the the trust boundary, um, and so we have our whole like rag pipeline, and um, we, we, we have. We're building, you know, hybrid search with re-ranking across both structured and unstructured data

And I guess, do you, uh, I for that? For that application, is it similar? Where I can sort of pick the model that I want, or do you have, like, a a point of view on w- which model I should be using?

for embeddings.

Sorry for this, um, querying all of my data.

Oh yeah, so that's an example of of where customers can have model choice, and we're also developing a point of view on on the best models at various, uh, sizes and and performance.

Do you have a point of view that you'd be willing to share at this point? I mean, I'd be curious, like, what y- y- you think customers should be doing?

You know, I think customers need to just get up and running with curating their data sources for RAG. I think so many customers, too many customers last year, were over focused on picking the right model when they didn't. They hadn't figured out their RAG pipeline. And so it's But wait, if like,

I'm, if I'm using your solution here, you're kind of doing that part for me, right?

well, there's still work on the customer's part to connect in their data sources into data cloud, right? And to think through, um, where they might need data that they don't have currently, right on the hallucination front. And so there's plenty to do in terms of making sure that this works at the prompt level and with rag. And then, of course, once you nail that, then you can think about, you know, model optimization, you can think about fine-tuning, et cetera.

Wait, let me make sure I understand what you're offering is. So, like, I was imagining I can query my Salesforce and like, ask questions about my customers, for example, is that? is that what you're anticipating? So, like, do kit kit, you have that kind of already hooked up for me? Like, why do I have to do my own, um, you know, like, thinking and data and stuff? What? Uh, I think I would think that would be kind of a magical experience out of the box, potentially,

it is a magical experience out of the box if you have everything in Salesforce. But most companies, Salesforce isn't the only system that they use. So they've got, you know, they might have, like, they might use Jira, right for, um, to track, like, you know, bugs and issues they they're using. They have, like, a an intranet of like, you know, like a wiki of some sort. They've got multiple siloed knowledge bases, they have a data lake, they're capturing their customer engagement, um, somewhere else, right?

And so that's where it really gets better. So it's it's really stepwise like, so if I were to say, like, step one, it's to understand it's to have prompts, and, um, to to use our prompt templates as a starting point. And that's what a lot of customers have done. They've they've gotten value to your point, comp like, right out of the box. You know, one of our customers, Heathrow Airport, they got up and running with service reply recommendations in ten minutes.

That's pretty unheard of in enterprise software, um, and that has just been phenomenal. And then once customers feel comfortable with prompts and playing around with grounding with with their different custom field and custom objects within Salesforce, inevitably, there's always a question of, well, how do I expand the the knowledge base that my rag can a can access? And so then they're bringing in other data sources, and it is fairly easy to do that, but it is.

It requires some thought and some planning to think about which use cases require what data. And then only then, after that step, do they does it make sense to kind of think about model optimization.

Hm, I mean, even at the prompt engineering step, like, how, how do customers think about improving these these pipelines? Like, do they set up some kind of like ground truth evaluation set? Or, um, does the CEO like, try a few things and get mad and yell at somebody? Or what's the, um, what's the approach?

There's, you know, it's, it's some, it's the the top thing that customers talk to our professional services team about. I think there's not a single answer right now, you know, we're developing CRM and industry specific evals for them to use, and we're using these internally ourselves. And then ultimately, it's about, you know, does this get the job done? You know, is there user acceptance in, uh, uh, for example, in a questioning, question answering, or reply recommendation generation?

Does the service rep actually use the answer that was responded, that was suggested, or how much of it gets used? And then, uh, the other really neat thing about what we have at Salesforce is we're not just a database of customer data, we're actually database of customer outcomes. So in in a very objective way, we can structure the objective function, the reward function, to value. Not whether not just human feedback, but did this reply result in these types of cases being resolved in a shorter amount of time and increasing customer satisfaction scores?

Did these types of sales outreach reduce in a shorter sales cycle? Did it result in the customer opening the prospecting email more often and agreeing more often to meet with the cut with the sales rep? And so over a large data points, right? We can see now intermediate steps and intermediate goals. Did it, did it achieve what was desired?

Although I would, it it seems like, you know, if you're, if you're a customer and you're like, you know, modifying prompts. It's a pretty long feedback cycle, right to, um, you know, to kind of wait to see, like, if a deal closes, like, you know, you push it into production, and then, you know, y- y- I don't know. I is that is that really like realistic? Like, I would think that would might take like a week or two before you kind of know if you, you've actually made an improvement.

Certainly for deal closing, I mean, depending on the type of business, it could take a year for. So it's usually not those like end goals that are driving the most feedback. It's the intermediary goal, right? Cause the next step is you're just trying to get the prospect to meet with you, or just to, like, respond to your email. So those are the signal that we we pick up and and put into our feedback pipelines.

And do you have, like, a sense of best practices around prompts, like, who do you feel like does this well? Like, what does it mean to to do it well?

That's like, uh, there's such. There's so much research being done on this right now. I mean, manually, yes, right. It's like you want context, but you don't want it to be too long. You want it to be the right context. And I mean, this is like a whole, the whole art and science of prompt engineering. Personally, I think where all of this is going is that AI will be able to to take our ill worded intent and and poorly worded intent and be able to create much better prompts.

And we're doing a bunch of research in this area right now.

Interesting. Well, one of the things my my team was wondering about, actually, is your thoughts on kind of open source models versus closed source. I mean, you mentioned kind of a a mix of that. Like, could a customer bring their own, um, kind of open source model to your platform? Would that? Is there a world where that, where that happens?

Definitely, yeah, customers can bring any model to our platform, and we we're seeing a surge of interest in open source, especially, you know, Mistral, of course, and and Llama two. And we ourselves are are running experiments, kind of fine-tuning open source models for speci- for domain specific use cases, and there's a lot of great stuff out there. I think the the the mixture of experts model, like, it's really novel. It's like very efficient way of of getting a smaller model to behave like a much higher parameter model.

And these are all like, again, it's so early, it's hard to to say who the winner is. And I think the key is to have a good systematic way of continuously evaluating models. And as as best state of the art ones emerge, incorporating them into the stack.

I mean, you, you've been in this world a long time, though, and you've seen a lot of real world use cases. Like, do you have any kind of predictions on on where this is going?

I have a lot of predictions. I mean, one is I what I just talked about, which is models are gonna be, they're both gonna become a commodity, and they're gonna continuously change. And so while it's important to have, you know, good prompts and good data curation and cleansing and and hygiene, right? I think being really good at continuously evaluating and continuously swapping out models will be important. I think that's one, um, and we're already starting to see that, I think, too, is, you know, people have been talking about, like, is it gonna be large models?

is it gonna be small models that run on the edge? It's probably gonna be both. And again, being prepared to, like, being very specific about which use cases am I okay running on the edge with a smaller model where I'm gonna have higher latency, but that's OK, because it can run as a batch process, or it can run while I'm working on other stuff. But it can basically get that compute for free, and that's, I think, the only way that a lot of this of these generative AI use cases are gonna scale.

Um, three is, I think we're gonna get much more sophisticated in our RAG pipelines, like so far, like everyone's been talking about fine-tuning the large language models themselves. But there's a lot of like, if you think about how RAG and chunking and embeddings work, like, there's a lot to to to play with there, right? The way that we we chunk and vectorize different types of of documents, the way that we need to pre-process different types of unstructured data is really important.

I'll give you an example. Like people talk about unstructured data, like it's one thing, but think about unstructured data. That's transcripts, where there's a lot of noise, like the the noise to signal ratio is extremely high. You wanna like mine out most of the ums and ahs and just a lot of the conversation and just like, focus on what the insights are. You wanna, you might be able to mine out structured data from those conversations, right?

If a customer is talking to a retailer and tells a retailer that their shoe sizes is is a nine and their son's shoe size is a four and a half, and their favorite color is turquoise. Like, that's all structured data that you can mine out that you will want to pre-process. And then there's also gonna be a lot of junk in that transcript versus unstructured data. That's a knowledge article that was written for for every line in that knowledge article to be important, right?

And so there's just, um, there's a lot of different considerations. I think there's gonna be a lot more work there, and we're gonna see the accuracy of AI improve exponentially as we turn our focus toward both the data pre-processing. And then starting to think about optimizing and fine-tuning embeddings.

Interesting. So if you think the accuracy is gonna improve a lot, w- what kinds of applications do you think that opens up? Like, w- w- where do you see the sort of next offerings from, um, Salesforce coming?

I think it just means that there's a greater number of workflows that will be able to be handled, you know, autonomously versus requiring a human in the loop. And that, you know, increasingly, right, humans will be used for exception handling and creating, you know, new types of, um, of flows.

Have you found, um, I mean, I guess you could think of a RAG pipeline as a kind of agent based workflow, but have you has your team kind of pushed on that further? Like, are you seeing more complicated agent structures looking like they're useful for for customers?

Yeah, well, that's a big part of what our our copilot release about is about is this use of agents to to help with an increasing number of of tasks, right? Going beyond question and answering, beyond content and image generation, to doing reasoning abilities and breaking down goals into plans, and then get executed.

If you think that, um, like, you know, people should be more focused on, you know, kind of the prompts versus the model itself, which, you know, by the way, I completely agree with. Why did you decide to support such a wide variety of models? Is that because customers want it, and and you you think they're sort of overemphasizing it?

Uh, yeah, I I mean, I don't know if they're over. Maybe in the long run, they're not overemphasizing it, and I think customers there. There's a lot to be said for having, you know, trusted vendor relationships. And some of our customers, for example, if you're a big query customer, like, you're most likely using one of the vertex models, and you know, Salesforce, like, we're, we have models, but we're more than anything we're. We're really focused on being the platform that supports an open ecosystem, and so that's why we we structured it that way.

So you think customers are essentially coming with the different models because of some relationship that they have with with that model company or some cloud vendor attached to that that model?

Yeah, the the customers that come to us with models, that is their reason. But like I said, I mean, there's, there's plenty of customers that don't right who that that they're. They're model agnostic, and they would rather we just deal with it and and optimize for them.

Why do customers come to you with open source models?

If they fine tune them right for for broader use in their company, then they might want to use them. Some customers come to us and they say they want to fine tune an open source model using the the data that they have in Salesforce and in Data Cloud. And we allow them to do that too through an offering we have called Einstein Studio.

I mean, I think like most of the closed source models, you could fine tune them now. Do you think that it's, do you think these customers are being like rational in wanting to to fine tune open source models?

Maybe from a cost perspective, um, that, I mean, as you think about scaling out, I I think it's entirely rational, but we we see, first of all, I'd say very few customers want to fine tune these models among the ones that do right now. It's it's a mix between proprietary and and open source, but just like in the grand scheme of things, right? Cause remember, Salesforce serves everybody. We we work with the Fortune 500, but we also have a, I mean, t.

We have a third large enterprise, a third mid market, and we have a third SMBs. And I mean, especially in the SMB and mid market, people are not, they don't even know what, what fine tuning is.

Totally. Do you think that, um, fine tuning will get more popular if it gets easier? Do you do you think it might, you know, kind of go away? Because who needs it if the models are really good? What's your take there?

I think it'll become more important because both for cost reasons, even though the overall costs are coming down. But just the fact that fine tuning allows you to get the same performance at a much, you know, smaller amount model. But then also because it allows customers to use proprietary data sets in the model weights. And as as models become, as foundation models become commoditized, you know, companies are gonna look for those areas of differentiation that they can, they can get through their data.

Are you surprised by how many foundation model startups there are out there? It sort of seems like more than the the the sort of like levels of differentiation that you might expect. Do do you sort of expect that number to go down over time or or grow up over time?

I expect it to go down and for there to be consolidation. I I think it maybe made more sense before before Meta really lit a fire in the industry by open sourcing such a such a good set of models. But I think now that there are open source model alternatives, it pressure, it really adds a lot of pressure to to the to close, um, close model startups, right? Because, like, it's just so expensive to train and to run inference.

And many of these companies, I mean, unless you're, you're Google, um, or Amazon, like, you don't own the compute. And so you can either keep continually raise it crazy valuations, where all of your money goes back to the compute company that invested in you, or you have to find a different way.

But isn't it also kind of incredible that, um, Mistral, um, you know, came with such a small team out of nowhere with such a fantastic model? Like, I think that might, I don't know that that does that make you, and that might make me, you know, say, hey, it's great that lots of people are trying to do this. Maybe someone will find some other, um, you know, new way we haven't thought of.

Yeah, that's true, that's true. Maybe there's a lot more kind of clever architectures and tech techniques. I mean, the IT in hardware like that's we haven't. There's so much, just like we were talking about, like people focused on tuning the LLMs, like, and then there's opportunity with RAG. There's opportunity really at every layer of the stack that's been largely unexplored.

We've been, um, you know, surprised, I think, at at wits and biases by the number of customers that, um, you know, try to try to do, build their own, um, foundation models. Have you considered that at Salesforce, do you would? Is there a world where you would ever, um, take that? I mean, you obviously have a fantastic, um, AI team, I'm sure, you know, somebody would be excited about

We do.

doing that.

We we have foundation models, we've, we've had

Oh, you do.

them for many years. Yeah, we've, we've had them since 2018, and they're, they're really good, they're. They don't perform as well in in evals as some of the ones that are out there. And so we don't have a religion around this, right? We just want to give our customers the best models that will power the the best use cases across CRM. And so, you know, that's why this continuous testing is important, right? As our, as our models, you know, and it's gonna con.

There's never gonna be one leader that lasts very long, whatever leaderboard you look at, as ours surpass benchmarks in certain areas, we'll swap them out, and we just continuously wanna to be offering the best.

Okay, what do you think is like the makeup of a like in in one of your customers of a successful team that gets these applications working? Like, what do you like to see? Is it sort of like, are you interfacing with, like, software engineers, or, um, ML engineers, or product managers? Like, how does that? How does that work in your in your kind of best case examples?

You know, there's no one formula right now, but some of the recurring themes that are starting to emerge, I'd say, like in in large enterprises, almost always it involves the CEO, the CIO, and and the COO. Because it's there's so many different use cases for AI across the company that, you know, these three roles, like, they're in in the hot seat right now, right? They're reporting to their board every board meeting. What their generative AI, not just strategy is, but what they, what experiments they're running, what they're doing.

And so when I think about like work that we've done with amazing customers like triple A or with ADP, we're we're really starting at the top there and aligning with their strategy of, you know, where do they want to see greater productivity, where they, where do they want to see greater efficiency, where do they want to improve their customer experience? And then we're starting there with those use cases, and then we're aligning, of course, all the way through to to the individuals you mentioned, right?

The machine learning and the data science teams, and I mean, the lines of business, right? Working with the sales team, for example, and the customer service teams at ADP to look at, okay, what are the use cases that we want to start testing and validating? At the prompt layer, at the data curation layer and knowledge layer, at the model and fine tuning layer, every step of the way,

You know, it's interesting. I I think we, you know, we tend to start a little bit more lower in an organization than than the CIO, obviously, especially, you know, in big customers. And one of the things that we sometimes hear is, you know, hey, the the CEO just wants to do like a gen AI, um, application here, just like somewhere, you know, just like looking for that. And and that always makes me like, super nervous, like, you know, is this gonna last?

is this gonna get funding? Like, it seems like a pretty bad approach of like, you know, uh, um, it's hard for me to imagine a CEO really saying, hey, I just have this technology and I'm, like, looking for a problem to fit it in. But that is definitely the perspective of a lot of the teams that we talk to. Do do you ever experience that kind of, um, interaction?

I do, and I, I don't think it's it's bad. I think that if you only do that, then it's it's hard to to get it right because you need people involved across the company who are closest to the problems and the opportunities. I, I kind of think I like to think back to, like, the internet, right? This is like, you know, you and I were just, we're just starting our careers, or we were in school, and I imagine like, what would have happened was similar.

Like CEOs are probably under the gun to to deploy something on the internet. And the first ideas were bad. The first websites were people just scanning a brochure and putting it on their website, and they were like, didn't take advantage of anything unique about the internet, like hyperlinks and forms, et cetera. It was just that, but that was the first foray, and so, and then you had individuals at these companies starting to play around with, you know, AOL and chat and AOL instant messenger and and email.

And eventually these things converged. And I think the same thing is happening now, where successful AI adoption and maturity at companies, it's gonna require both top down and bottom up, right? Top down is important because you don't want to be one of these companies that blocks chat, GPT and blocks generative AI of any kind and tells your employees that they can't use it. That's just not. First of all, that's not realistic because people are are already using it in their work.

Second of all, it's you're actually holding yourself back from from learning and from taking, you know, calculated risks in this. In this era. Nor do you want, though, everything to be dictated from the top. You need to empower people with a secure environment, um, like, you know, Einstein or enterprise chat, GPT, to be able to to play around with this. And some of the best ideas will be accidentally stumbled upon by individuals that are just like trying out different things.

And then that creates a groundswell of storytelling and, you know, peer inspiration that ignites this like change management across the company.

That kind of reminds me of another question I wanted to ask you, which is, you know, I saw an interview with you about sort of AI, its effect on the jobs market, and you were talking about, you know, kind of 2024 as like a time when there's like a big change in like skills required. Do you want to maybe articulate that? And and, uh, you know, add any any thoughts there?

Yeah, I mean, it's I always say, like, last year was our 1997 moment, so maybe this year is 1998. Imagine, like, not knowing how to do Google search or use email in in 1998. You probably still could get away with it, but by the late nineties and early 2000s, like, I mean, you'd like good luck getting a job. And so I think it's the same is true for both AI and data fluency.

And so learning, like understanding how prompts work, even if if a person isn't the one who's responsible for always engineering the prompts, like just understanding how it works and the different knobs and dials and levers, and, um, what, what the gotchas are to that result in, you know, hallucinations or or bad responses, and how to avoid them. And just like, really thinking in that way, I think will give each individual worker tremendous leverage. And I also think that this is a year, what, as PR, you know, related to that, that every one of us needs to rewrite our job description and figure out a new way of spending our time.

Like, if we used to spend, it sales. Typical salesperson spends 70% of their time on admin tasks. So these are non selling activities where they're not with the customer actively engaging. So they're, you know, generating a quote, they're waiting to hear back from marketing about what local events are happening in that city that they can invite the prospect to. They're doing research on the prospect. So, like, imagine if we could have that from 70% to 35%.

All of a sudden, you get hours back every week to actually engage with customers. And it just makes people have deeper relationships, be better listeners, be able to s to handle a bigger book of business and and sell to more customers. Um, that's, that's just one type of role, but every role is gonna be changed in that way, and I think this is the year it's gonna happen. And then,

Can you tell me about that for you personally? Like, I, you know, I I think I probably relate to your role as like, you know, kind of senior management. I. I certainly spend quite a lot of time on, um, administrative tasks, like, how have you thought about that for for yourself?

yeah, I mean, a big part of our role as leaders, especially in this like rapidly evolving time, is keeping up with all of the changes that are happening in the market. And making sure that we're reading every big research paper that comes out and playing around firsthand with some of these technologies ourselves. Uh, like, I downloaded Mixtral seven B to my laptop last week. It was like, super fun, right? But I needed time to do that.

So I think, you know, one which is not gen AI related, but it's just to carve out time and be very disciplined about honoring that time. Just like you would, you know, dinner with your family on Friday nights or working out, whatever it is. And then I I use gen generative AI in terms of like, there's so many podcasts out there that I I love, like this one, and I just can't listen to them all. And so I I have a workflow that I I actually summarize, like a bun, uh, there's like twenty podcasts that I want to listen to, so I get twenty summaries every week.

And then I I do like, a, you know, I. I then scan through the summaries, and if there's certain ones that I'm really interested in, then I'll look at the full transcript or even just watch it or listen to it.

No way, really. Wow, do you do you like, just paste it into GPT or something? How do you?

Yeah, my my EA does.

No way. Um, wow, interesting, do you? Um, I mean, I know we both have small kids, like, do you like, do you think about, like, what skills you want your kids to learn in this context?

All the time. Um, I mean, one is, I think that, you know, learning how to learn is really important. If everything is gonna change and what was valued one day no longer is the next day, you kind of have to always be learning and and getting used to that motion is really important versus like, becoming an expert in one thing and feeling really comfortable, uh, and complacent. I think that's one. Two is I. I do think it's important more than ever to learn how to code and learn kind of like the fundamentals of how data science works and and how LLMs work, right, so that it's not just this magical black box.

And my son, my older son, he's only eight, but like, he's like, he's very curious, and so we have a lot of conversations about this and we listen to some of these podcasts together. And, uh, I think the third one is is actually how to experience joy and have leisure. Some one of the things I worry about most in younger generations is like they're so focused, like so much pressure and so hard to get into, you know, the so so-called top schools now, or to compete for these jobs and internships.

That, like, people feel like lost. And there's a huge mental health crisis in every generation, but especially among younger people. And so I think especially as AI, like ups the bar for that competitiveness, cause you're also now competing against AI. I just think people need to, like, learn how to be happy with who they are and like, learn how to experience joy. Both because that is the meaning of life, and also because when it comes time to work, if we're not refreshed and we're not in a mentally strong place, we can't do our best work.

Look, okay, we're getting slightly off topic here, but how do you teach your kids that? I would love to give my children that, um, that skill,

I mean, it's exposing them to a lot of different things, and they naturally, just like we naturally, there are things that we do that that give us joy. Like in the, you know, in the middle of practicing piano a few nights ago, my son like, discovered on our keyboard. There's there's, like, pre-recorded tracks and dance beats. And at first I was like, I was so focused on getting the task done, I just wanted to, like, wanted him to hurry up and finish.

But I was like, Oh my gosh, he's ex- he's so joyful right now, and he's having a dance party by himself and I'm gonna join him. And this is like, this is really, this is amazing, and this is exactly how it should be. It's not really about becoming a concert pianist, he's not gonna become one. It's about appreciating music and learning how to persist through something hard so that we can I improve our muscle memory of learning to learn.

you know, it's so funny. I, I totally, um, relate to that. I, I was thinking, and I think there is kind of an AI component too, where, you know, I feel like AI is getting so much better at doing, you know, stuff that we can do. It just seems like, you know, silly to to push my daughter to become a concert pianist unless she really wants to. I'm just like, you know what? Like, when we do piano lessons together, it's just like, all this is is just like, I want you to like, enjoy music.

And, um, you know, like, the learning a skill seems really only important to the extent that, like, learning a skill, um, is fun. I mean, I just, and especially with music, I feel it's become such a complicated interaction with machine to make it. If you, you know, if you want to, you don't really need to, like, you know, have skills to like playing a particular instrument to make interesting music, but I also love to play instruments.

So it's, um, I, I, I think about that a lot.

Yeah, and there are moments when, you know, it's like the week before his recital and he doesn't wanna practice and he says, I hate this. And I say, okay, well, you can quit, but you have to pick something else. I don't care what you choose to learn, you just have to pick something that's hard so that you can learn how to persist through those moments.

Totally. Well, I, you know, I have this memory of you, um, like, right after college, and you, I think you did this really brilliant thing of like, writing a book about kind of how to apply, you know, social networks, which were new at the time, um, to business. And I I think it was such a, um, such a smart, proactive thing to do. I sort of always imagine it's kind of like your superpower is like seeing new trends and and like, you know, like jumping on them.

Do you think there's like an equivalent for some, you know, kid in college today? Or like, I don't know what, what about like people who are kind of just coming out of school? Do would you have any career advice or or life advice from them as it relates to AI?

I I think it's such a gift to not have baggage of how things were done in the past, and I think that's why so many amazing companies are started by young people. Is because they don't have that baggage. And, you know, as someone who is no longer right out of college, like, I really strive to, like, not become set in my ways, because especially in the technology world, like, it's gonna ch- it's gonna continuously change. And we have to unlearn and let go of assumptions that got us to where we are now, continuously.

So I guess my, maybe it's more encouragement than advice, is like, it can feel so intimidating, graduating, uh, or not graduating, like, you know who? Questionable, like, what value college provides anymore, whatever it is, it can feel intimidating to kind of enter this workforce or this startup ecosystem. Um, but it's actually a gift to not be saddled with with the weight and the and the presumption and the assumptions of of how things are currently done. Because we're entering a time now where every layer of the tech stack is getting blown up and it to come with that true beginner's mind and a a truly open mind, I think is is an amazing gift.

Totally. Do you have any, um, asking for a friend here? Do you have any management tricks on sort of encouraging that mentality in the people that you work with? I I often feel worried that, um, you know, we're not like incentivizing learning enough in a world where things are changing so fast. And it's actually such a huge time commitment to stay on top of, um, all the, all the new things that are happening.

It's such a tough one. I mean, I think some people are wired that way, some people would do that if they weren't totally over capacity of keeping the lights on and doing their day job. So really hard to do, especially in a big company. But just being very mindful about the number of meetings that we put on people's calendars so that they have time and space to think. And then I think it's about rewarding and promoting people who to do those behaviors, because that's the strongest signal that everybody else sees about what is valued in the company.

When you look inside of Salesforce, not like for the customer, but in, like, Salesforce's own processes, have you seen places where AI has, like, made a big difference and and improved them? And and if so, where, where are those?

Yeah, the really neat thing about being at a tech company and particularly Salesforce is how we use our own technologies. And a lot of the time, it's using our own Salesforce products that we offer more broadly. But some of the time, our, our Salesforce research group, for example, they've developed a bunch of AI models to to do early detection and kind of self healing of our, our systems, like our, our first party data centers. And um, it's just really neat, cause it's like through, it's like pervasive throughout the company's operations.

So it kind of knows if, like, something's going haywire in the hardware. Wow, cool, that's as as really good one example, thing to try.

right? And and we, we we have it throughout, like we our, our customer support team, they are customer zero. We call them of service cloud, so they're using our self-service, they're using our bots, they're using our AI agent, assist, they're using service GPT reply recommendations for customers.

okay. Well, that that leads me to one of my, um, one of my last questions, which is, um, you know, if you had a blank sheet of paper and you weren't working at Salesforce, what would you be working on? And thinking about?

I'd be working on AI of some sort. I think it it's the most exciting, not just technology, but I think it's it's just like the most exciting opportunity of our lifetimes. And there's so much to be figured out, from the technology standpoint, from the product standpoint, from a customer standpoint, from like a s industry structure and education and policy standpoint. I mean it, there's a lot of work to be done.

Okay, but you are working on AI today, like, are there are there things that you would like to sort of like, dig into that you don't get a chance to in your in your day to day life?

I mean, a lot of things. I have a lot of of interests, like I love to travel, and I've, uh, always been really interested in in, like, skincare products. And that was my initial dream in in college was to start my own skincare company. Um, but I,

What?

I, I would, I would still rank AI above that.

But I guess, sorry, within AI, are there? P- are there, um, things that you're interested in?

Yeah, I like, I was saying earlier, I'm really interested in the policy implications, and if

I see?

if I were more optimistic that I could have an impact I I think I would I would really like to to do something there

Okay then the last question is a little um broader but I feel like you have like deep experience in this like when you when you think about like taking these taking products from you know kind of like on a sheet of paper to actually deployed I mean like AI um products at you know at hearsay um at Salesforce what have been the biggest challenges in actually getting them out to market

Well hearsay was totally different I mean our our issues with getting product out to market it wasn't because of the AI it was because we're this tiny startup working with fortune 500 banks and insurance companies and they had to to learn to trust us and be able to deploy our systems at scale uh but I do think that trust is the recurring theme and even now like when customers are are looking at deploying AI whether through us or through others like that's the number one question that that we hear is how do we trust this how do we know that our data is being protected how do we trust um the the outputs and the way that it's being used

how do we audit that and I think that's why there's been such a big push and so much interest in what we call our Einstein trust layer which is a bunch of different features from data masking to citations to toxicity filters to citation um to to keeping an audit trail et cetera

Um and I think we'll be you know putting this out like right after some product announcements do do you wanna talk about those at all Yeah so Some things that you have just announced or is coming

Yeah this is really exciting I mean depending on when this this podcast gets released but we just um released our prompt builder and our copilot and copilot studio and so this is what this means is that customers now can take any of this the generative AI use cases across Salesforce whether it's service rep recommendations or sales GPT account summaries they can take those prompt templates and they can customize them with you know their own instructions they can reference specific data fields unique to their Salesforce instances and really make it their own they can also very easily use our no code flow builders within prompt studio to create their own prompts and um so that's exciting and then copilot of course

is the natural language assistant that any Salesforce user can use to now interact with their data across Salesforce and it's just made better by Data Cloud which allows them to expand the universe of data beyond Salesforce as well as by MuleSoft which allows them to expand the universe of where they can take action to every API um across their company and then um we've got I mean this is this has just been a really exciting twelve months like going from from vision to to pilot to to shipping these products and we've been working with our our early customers on this and the feedback has been really exciting so far

Well fantastic I can't wait to try it as a as a Salesforce heavy Salesforce user myself and AI fan um I definitely wanna to hook this up and see what I can do

Yeah send me your feedback

Totally do I have to do I have to turn it on somewhere is how do I uh how do I get started

You do um you have to you have to to buy the the AI product

Alright well I buy the AI product

Yeah thanks for your for your business

Awesome yeah thank you so much Great to talk thanks so much for listening to this episode of grading descent please stay tuned for future episodes