Stop Overpaying for AI Power You Don't Need - with Melissa Ramey of Salesforce

Welcome everyone to the Emerj AI in Business podcast. Today's guest is Melissa Ramey, Director of AI Transformation at Salesforce. In this episode, Melissa discusses who takes responsibility for an AI agent's work after deployment. She explains why teams need to trace failures through the agent's workflow, verify its use of source material, and define which decisions it must return to an actual person. Just a quick note for our audience that the views expressed by Melissa on today's program do not reflect that of Salesforce or its leadership.

In this episode, we cover how organizations can govern AI agents after deployment and build trust in their outputs. To go deeper on AI strategy and learn how consultants are winning business with evidence-based AI ROI and building long-term capabilities instead of chasing short-term gains, download our free PDF report, Three Keys to Thriving in the Coming Era of Automation at Emerj dot com slash C O K one. That's E M E R J dot com slash C O K one to download your copy.

Now, the conversation with Melissa. Melissa, welcome to the show.

Thank you so much. I'm absolutely thrilled to be here.

I think we're gonna have a great conversation today, very practical, because every company that I talk to at the moment describes the same moment, if you will. The agent goes live, the vendor wraps up the deployment, everyone shakes hands, and then nobody's quite sure who's watching it anymore. And if we should be watching it, that's a very interesting thing to think about. It's not like anyone's dropping the ball on purpose. It's more like the ball just rolls off somewhere.

Nobody's really looking at it anymore. And you've been close to this exact handoff moment, if we can call it that. So once an AI agent is live inside a big organization, where does responsibility for how it behaves go? And where have you seen it just quietly vanish?

I've absolutely seen this and felt the pain of this in my everyday work. And for me, Yolandi, this really comes down to three things. The first thing is the vendor gives us, as you said, the agent, and the vendor has done architectural work, right? They've governed the platform, they own the engine, they know that it works. The problem is is that nobody then, as you mentioned, is actually understanding not just does the agent work, it does.

The vendor made sure. But is it now ready to do my work? Does it work for my work? And research tells us that right now, 57% of orgs have no organizational architecture for these agents. There is an expectation that the vendor making sure the agent works is the same thing as does it do my work. And so half of our organizations are getting agents that technically do the job without really understanding the specific job that it needs to do.

So that's architecture. But then the next part for me is verification. Not just did it work, but how does it continue to work? What does testing continue to do? And this is where we really start to understand Yolandi, the difference between can the agent create an output? It absolutely can, and is the output that's coming out of it actually correct? Is it surfacing our information? Did it retain the nuance of the work that we needed to do?

Can we actually now stand behind the output? Because we know that the tech works. And then finally, the third thing for me is the ongoing monitoring. Again, the vendor is gonna make sure that the agent is online, that it's fast, that it's secure. But over time, who's making sure that it's accurate? And right now, an even more sort of staggering statistic is that only 20% of enterprise orgs have a tested plan for when AI output fails.

So for me, it comes down to these three things. We're getting agents that absolutely work, but we're not continuing to think about the architecture to make it work for our work. We're not thinking about verification, what is coming out, and then we're not continuing to monitor along the way.

That's weird because it's one of those things where what I'm listening to what you're saying, and I'm like, yes, but this this makes so much sense that it should actually be obvious. But if we say okay, to your last point, um, who's monitoring it? Who should be monitoring it? Who's supposed to notice if something goes wrong or if it's not giving us the output that we need for our specific work?

Yeah, that's a great question, and I think part of the reason that this is breaking down a little bit, Yolandi, is because traditionally, this might have been an engineering team's question or a tech team's question. But I really feel like the way we all use AI today has really removed that distinction. So we would no longer say, oh, tech team, continue to monitor this for if it's working, because the output that needs to be monitored is only known by the teams who are able to review the output.

So I think that the issue now is that there's no longer this really deep divide between what engineering does and what, say, like a team that uses a tool does. It's now really everybody's responsibility, but because nobody's really defined that, as I said, about half of organizations haven't defined this at all. It's just currently falling to no one. And so there's this expectation that because the vendor did part of it, because we haven't defined a new governance for ourselves within our own orgs, maybe the team that's creating the output is thinking, oh, engineering used to do that.

And then at the same time, engineering's thinking, I don't know if your output is correct, you have to do that. And so I would say right now it is a true gap, and the reason I think Yolandi, it's a gap is because it's a true A I native step in a legacy workflow. And identifying those A I native steps is something that we are in the very early days of understanding what A I native even is, who owns it, how to govern it.

I think these are all questions that people maybe haven't started asking, but they are feeling the pain of of these outcomes.

Yeah, I so many things that you just said there, I feel like we should put that on a billboard, especially thinking of this as an AI native step in a legacy workflow, and also just change the way we think. But okay, so say a team figures out that that gap is there, they've felt it, maybe something already went sideways. What would you think has to happen next? Closing a gap like that sounds it sounds simple when we say it out loud, but I imagine that it's it's quite a lot messier in practice.

So once a team knows that the gap exists, how do they go about closing it, or where do we even just start thinking about closing it?

It's a great question. Where do we start to close the gap once we know it? I would say, I guess a couple things here, and I'm gonna just tell you this answer is gonna be a little bit messy as I think through this with you. The number one thing, though, with closing this gap is people really understanding that the thing that they're looking at, the final output that they're looking at, is not just it's not just copy anymore, right?

It's let's say you use some kind of a tool, you've used an agent, and it's now produced something. Historically, that was just copy, right? Copy in, copy out. But what agents deliver to us is never just copy. They have taken a an active role in transforming that content, meaning agents are now creating their own product. And that product by itself is something that teams who have never created product now have to understand in a way that they've never had to understand that, right?

So a team that maybe historically they were responsible for, like, say, creating marketing info, and they use an agent to help with that. Now, at the end, they get something that is, let's say, incorrect, it often is. And the initial reaction to that might be, oh, I just need to edit the copy, but editing the copy doesn't change what actually failed, right? And so you've got the gap between what the vendor gave you, you've got the final product that doesn't look the way you expect it.

And so what many people are doing is thinking, oh, I just need to go in and edit the thing that's now wrong. And so what I would like to do is to back up and to find where the failure actually happened. Did it happen at the retrieval level? Did it happen at the transformation level? And those are questions that most people simply are not asking yet for the agentic outputs that they're getting. So I would say one of the first things that I would do is I would back my legacy workflows up, map them out and try to understand where is retrieval happening?

What tool is doing that retrieval? Where is transformation happening? How is that transformation going through that process? What's the prompt that's actually doing it? And I think Yolandi, most teams, unless you were a tech team, simply never asked those questions. So just asking the question of, like, governance is too big. I would start with trying to understand exactly the process that the agent is going through. Which of the steps are legacy steps? Which of the steps are these now new AI native steps?

And how do I actually understand what's happening at those steps? Now, once I understand that, now I can start to think about what I would call the seven layers of governance. And I don't know if you want me to go into those now, but what I'm saying is I can't start at governance until I understand the process that has to be governed.

Yeah, let's talk about those seven layers. I think the whole concept here is, and and you can correct me if I'm wrong, but it sounds like the common mistake that's happening is if we don't do it the right way, we're basically just recreating the same red tape administration that made us want to get an AI agent in the first place. So it's it's unnecessary work, but I'm really interested in hearing about the seven layers of governance for this.

100%. I'll stop on that for just a second and just say we are turning ourselves into copy editors. We're putting information in, something comes out, and then we're sitting and we're we're trying to review all of that output. But as you just said, we bought A I to give us essentially to be able to process a ton of volume very quickly. If at the end of that processing, we then have to review all of that volume line by line, all we've done is to move the admin to the end instead of at the beginning.

And there is absolutely no measurable outcome from that output, right? No good measurable outcome from all of that output. So instead of looking at the end and saying, oh, more human in the loop review at the end, which, as you just said, just takes all of our investment that we've put into A I, essentially triples our cost, right? So we have an expert at the beginning creating information. We have the A I in the middle using up tokens in order to transform that information, and then we have another expert at the end reviewing again, and now one cost tripled in order to get an output.

That is not the outcome anybody is looking for. So the seven layers of governance for me, and I'll go through them one at a time really quickly, though. Number one is the source. Number two is the process. Number three is evidence. Number four is meaning. Number five is output. Number six is authority, and number seven is record. And I definitely don't wanna go into like too much detail here, Yolandi, but those for me are the seven layers.

So you help guide me how much information is helpful here.

I think if we can highlight the two more, I'm sure that they're all important, but if we can highlight the two most important ones, or maybe two that we've seen, um, some of our peer organizations really lack, neglect, or leave out.

So I think what I'm gonna do then is I'm gonna talk about evidence and I'm gonna talk about authority. So evidence is where what you essentially do with your agent is make it prove to you that it is using your source material. So depending on the model or the tool that you use, all agents, all A I tends to try to conserve token usage. So what that means is if you say, hey, use all of this information I've given you, in general, a model's going to try to conserve how much chunking and retrieving it's doing.

And so it's gonna go out, it's gonna find a few things that look good, and it's gonna come back and say, great, here's everything. And if you're doing that with one document, you might know, wait a minute, you've left out 80% of what I've given you. I need more here. But if you're trying to review the output of, say, a hundred documents, or a thousand documents, or 10,000 documents, there's no way that you would then be able to understand how much has been left out.

So when I think about evidence for governance, what I wanna do is build into my prompts and instructions for the agents that I build is prove to me that you have gone to the documents I've asked for and brought that information back to me. So what that means is it's gonna have to quote specific resources, it's going to have to cite specific resources. And in this way, I'm gonna start making sure that it's not going out and chunking and retrieving in a limited way, or it's not going out and chunking and retrieving and then bringing me back like an approximation of that.

What it's doing is forcing it to actually use every document I've asked it to use, and then to prove to me that it's that it's actually done that. And that's how I make sure that the verified content I put into an agent becomes the correct information that comes out on the other side of the agent. And that was learned through painful discovery of agents not doing that.

I can imagine because as you were speaking now, I'm thinking about the amount of conversations we've had recently where everything is about context and the context that you feed the agent and the context context that everything is built on. And I don't think I've had a single person say, okay, but we should also test whether the context is actually being used. So very like this is just limitless. There's always something more. There's always um another trick to to think of, but it makes sense once again.

How are we not talking about this already? Okay, that's that's evidence. Tell us about authority.

Oh, do you actually mind if I talk about meaning before I do authority, Yolandi?

Yeah, go ahead.

The next thing then, so this is number four, it's meaning, and I think it's so important for when we get to authority. So what meaning does is it essentially ensures that every time, um, the agent and I are using words, we are using them in the same way. So that if I, for example, am saying I need a customer insight, I wanna make sure that there is no assumption that the agent is working on that is not aligned to what I mean exactly by an insight.

And in some cases, the agents may be pulling from, you know, well, they are pulling from large language models, so they may have a generalized idea about what these terms mean. But what I have to make sure is that when I'm saying, okay, now I need you to produce an insight, or I need you to look at attrition rates or whatever whatever the the topic of our conversation is for that day, that we are using words in the exact same way.

I again, when we think back to the problems that people are having with the outputs in terms of governance, one of the biggest things is so much of this knowledge is knowledge that our experts have, right? It's institutional knowledge. It's information that lives in us, and we've never taken a minute to write down what do I mean when I say good? What do I mean when I say finished? And so the meaning layer of governance is where an organization extracts all of that sort of latent knowledge in the people who do the work to ensure that when the worker says good or done or insight or something like that, and then they pass on that prompt to the agent, the

agent then understands exactly what it means to deliver at that level.

It's making sure that we have the same book of definitions, and again, so much sense in that, and it fits in perfectly with what we just had on context as well because I'm again, it's context that we have to give it. And then we I'm I'm guessing that we also have to go back in terms of evidence and make sure that the meaning still stays the same and check on that every now and again too. So it's a never-ending loop of governance.

It is a never-ending loop, and I think it's so important because so often when we get, and everybody's had this experience where they've gotten mad at AI, and it's like, why isn't AI doing what I told AI to do? And really and truly, so much of that is that we haven't prompted the AI to do what we need it to do, right? Because it's it's not really doing anything on its own. So the quality of the output always depends on the quality of the input.

And so much of that comes down to do we know what we mean by good? Do we actually know? And if you asked me, or you asked my manager, or you asked my report, would we all say the same thing? So there's an exercise in this of, like, overcoming assumptions that have existed in enterprise for as long as human, I guess, communication has existed. But then making sure that we understand that the output is completely dependent on the input and we determine the input.

So if we wanna get value out of that A I that we've invested in, we've also gotta think about how are we guiding it to that end result? Because the the final output is not the A I's fault. It's not even the A I's really domain. It really is like a clear line from how clearly did we tell it what we needed, and then what comes out of it is the result of that.

Lot of sense. I'm excited to hear about authority because my next question after this section is going to be all about giving the the AI the authority. So I wanna hear your take on that.

Yeah, so authority comes down essentially to what decisions am I choosing as the human in the loop to make? What decisions am I giving to the agent to make? And what decisions can the A I agent never make, right? So this is really important because, as you've seen, I'm sure so much in the news recently, there was a huge outbreak of about 700 agents broke out of AI, went and attacked another website and then actively tried to hide its tracks to recruit other agents.

These are things that are happening, right? We didn't even learn about it until a month after it had happened. I think it's safe to hazard a guess that it's not the only time. So the smaller scale of our own agents is that we have to make sure that with one prompt, our agents can't go out and wipe out customer data or remove code that made one of our products work. So authority is where I make sure it is very, very clear in the smallest level of detail, what is my agent allowed to do?

And when it isn't allowed to do something, what has to happen next? So I'm gonna give you an example of what this looks like practically for me. I built an agent that ingests and transforms transcripts that we get from users and customers so that we can turn it into product recommendations. A few days ago, I went in to start the conversation with the agent and said, Hey, I need to get started on this new stream of work.

It came back to me and said, Great, I found six transcripts. It looks to me like this is a new work stream that we've never ingested before. Would you like me to create a new category and then file it into our existing library? So this is really important because what it did was to pull all of that information in and then to hold it in a holding sort of a holding category for me, waiting for me to come in and say, what should I do with this?

So that's the level of detail that I mean when I say authority. It cannot just go and pull every transcript that exists for me and then start creating new categories and putting things into those categories. There is a failsafe for me on so many different things around if you get to this point, if you reach this gate in my governance, you have to stop, hold the information, and then in the next chat, ask me what, what I want to do about it.

So that's a small example of not allowing agents to just take any actions that they want. And I'm not suggesting that OpenAI didn't have gates, but I am suggesting that in many enterprise smaller agents, we haven't thought about telling an agent explicitly what it cannot do, and then, more importantly, what to do with something it can't do. Because we also don't want that decision to just fall into oblivion somewhere, and we never know that the action didn't happen and that there's a decision that we, as the human executor, must make.

Yeah, and the picture that I'm seeing in my head is if I think about if I go to a big building and it's a big KPMG or Deloitte building or whatever, you'll get access cards for all the employees, but every employee does not have access to every door in the building. They would have access to their department entrance and maybe to, like, whatever is on their floor. That's where they need to be, and they're still not allowed to go where they're not needed, and it's kind of the same idea here.

Quite the the opposite problem that we've also picked up and I'm going to end off our conversation with this question is everyone also wants to get to the point where they can give the agent a bit more rope, not necessarily giving it all of it, but they would like to just step back a little bit and obviously get it to do what they paid for. So, for a leader who's ready to loosen that leash a little bit, what needs to be in place before it's safe to do so?

What should they be looking out for?

Yeah, 100% understand that that's where we all wanna get, right? We've invested in these tools because eventually the hope is that they are beginning to produce a lot of volume that is bringing measurable value to our organizations. And so, if I'm if I were starting what I was doing brand new, again, what I would do is go back and put real time into figuring out exactly what governs this thing. I would go through those seven things.

I if I'm gonna have evidence, how am I gonna make sure that that evidence works? So the answer for me is that stop trying to think about like a stop-gap human in the loop review of every output at the end and start thinking about how you can build the infrastructure around the agent that the vendor doesn't build that is specific to your work, that's specific to your organization, and then start testing it in small pieces. So it's not governance today, everything is finished tomorrow, but how do I start building out governance and testing?

One of the best quotes I think Yolandi from someone from from years back in user research was that you cannot inspect quality into a product, right? So there's no amount of QA at the end that will make a bad product good. What you have to do is stop thinking about, oh, the output is bad, let's fix the output and start thinking about the, uh, the tool that is producing the output. Govern and manage that, test it along the way, and eventually you get to the point like I'm at now, where I can say, here's today's work.

My agent comes back to me and says, here's the decision I need you to make. I make that decision, and then it then goes into the process of processing all of that work that it had queued up for me using all of the governance that I spent the months putting into place. So stop trying to inspect quality into your final outputs, go back to the product, build the governance around it, and eventually you have a an end product that you can trust.

I think that's the perfect place to stop this conversation because it gives our listeners the perfect place to start and to to think about their processes. And I think the biggest takeaway here is definitely that once the agent is live, the responsibility of its behavior is no longer with the vendor. It has to land somewhere specific inside the organization, or where quality tends to disappear. That's where the responsibility will go. And the real ownership is not a policy document.

It's a very long list of governance requirements that that that matters. Cause, I mean, we said, okay, let's just pick two and we we discussed three of them, but I'm sure each one of them is so relevant and so important. And I think the last thing is just for our listeners to take note that autonomy has to be earned in stages. It's not something where we're just gonna give the AI agent full control or full decision-making rights or full authority.

The human will stay in the loop, but the AI will also earn its its ROI once you know where you can trust it. Melissa, thank you so much for your time today. It's been a great conversation.

Oh, thank you so much. I've had a great time.

Wrapping up today's episode, here are the three key takeaways from our conversation with Melissa. First, when an agent produces an incorrect result, teams need to find where the process failed rather than just edit the final output. Second, organizations should require evidence that an agent used the intended sources and defined terms such as inside and done so its instructions are unambiguous. And finally, leaders should specify which decisions an agent may make, where it must stop for human input and test those boundaries before giving it more autonomy.

In this episode, we covered how organizations can govern AI agents after deployment and build trust in their outputs before expanding their autonomy. To go deeper on AI strategy and learn how consultants are winning business with evidence-based AI ROI and build long-term capabilities instead of chasing short-term gains, download our free PDF report, Three Keys to Thriving in the Coming Era of Automation at Emerj dot com slash C O K one. That's E M E R J dot com slash C O K one to download your copy.

For further executive level analysis and to join our network of leaders delivering workflow impact with AI, visit Emerj dot com. On behalf of the team at Emerj, we'll see you on the next episode.