What Shared Intelligence Really Fixes in Enterprise AI - with John McMullan of Observe.ai

Welcome everyone to the Emerge AI in Business podcast. Today's guest is John McMullan, Head of Product Marketing and Strategy at Observe.AI. Observe.AI is a C X native A I agents platform that enables enterprises to deploy specialized agents that understand context, reason, and take action across the customer experience lifecycle. The platform includes orchestration, integrations, and governance features that support consistent and reliable automation across C X operations. In today's episode, John unpacks why so many enterprise A I investments in customer experience fail to deliver measurable returns.

He explains why rollouts without operational discipline leave frontline teams rechecking the tool's work, and how a shared layer of interaction data lets every A I agent and human pick up where the last one left off. Today's episode is sponsored by Observe A I. In this episode, we cover how A I investments in customer experience fail to deliver measurable returns. To go deeper on this topic and learn how to identify real A I trends by tracking where venture funding is flowing, and by listening to how leading C E Os describe risk and competitive strategy, download our free P D F report, Three Ways to Discover A I Trends in Any Sector at emerge dot com slash A I T one.

That's E M E R J dot com slash A I T one to download your copy. Now, the conversation with John. John, welcome to the Emerge A I in Business podcast.

Thank you. I'm excited to be here and, uh, look forward to our conversation.

Yeah, I think this is a conversation that really Needs to happen because we have gone past that part of what was it, two, three months ago, where we were still talking about, oh, we need A I, and why do we need A I? And I think everyone now has some sort of A I built into their platforms. And I like to think of C X as the first part where we really saw A I popping up even before what it is now.

But something that we keep on hearing these days is that there's a lot of A I interest, there's a lot of budget, but the results still aren't there. So I want to get into what is that bottleneck, what's been preventing enterprises from getting real, measurable value out of their A I in for the customer, uh, front line and the their back office workflows.

The noise is feverish, right? And the the fun part, I've been in conversational A I somehow, some way, fifteen plus years. A lot of these promises aren't new, but with obviously the advent of L L M and these frontier models that are coming out, there's a lot of hype and a lot of F O M O driving a lot of these decisions. So companies have made investments, jumped in feet first to say we're gonna, you know, adopt A I.

Some have even gone as far as saying we're gonna be A I native, but a lot of that R O I has not come to fruition. And I think that really stems from interest without discipline. And what I mean is, it's not that A I is not prevalent and available, and there's certainly no shortage of problems to solve with A I. But this isn't the old world where you'd buy like a SaaS application and say, oh, well, we need a we need a new billing system or a new C R M and grab it off the shelf and implement it or cut over from your old one and it just works and goes.

A I is the continuous learning cycle, a continuous investment where the system is going to uncover new edge opportunities, need constant tuning and tweaking. So there's been a lot of interest. I think companies have scrambled to bolt on A I or put it into workflows, but they don't fully understand what it means to adopt A I and what goes behind the scenes to make it work. And yeah, we we can get into some of that more, but I think it's not an uncommon problem, right?

Everybody's moving fast, there's a lot of noise, there's a lot of hype. But really, it comes down to what are the tangible next steps for a business? And that comes with, you know, thoughtful experience design, which we can talk about here.

Yeah, I want us to get into that some more. And actually, I think what you just mentioned is very relevant, that A I F O M O was so real. And thinking about, you know, the platforms that we were used to back in the day, it had the functions that the box said, and and that's it, and that's what we expected of it. With A I, it's kind of like introducing new employees to your organization and they need constant learning and they need to professional development as part of their annual training and things like that.

So we should be thinking about it. Is that the only friction points or the only reasons that we see our, our C X environment really flourishing and getting real R O I from A I?

Complicating question, right? Everyone's unique, so there's definitely different challenges, different headwinds, you know, how it comes to manifest in everybody's individual organization. But a prevalent problem I see, with the privilege I have of sitting down with, you know, dozens of customers every week and understanding what they're working on, number one is right people around the table. And this isn't necessarily like a council of A I or your A I steering committee. Those are super valuable, but they are not your implementers, right?

Those are the people that step back and say, what are the ethical use cases? How do we want to think about that? But really, that tiger team or red team that can jump in, and it's not always the people you think it is to get into the mix. And that, frankly, is number one that we miss of like, OK, we want A I to deal with customer conversations, or we need A I to analyze, you know, trends within our customer base.

And A I Council thinks about it globally, but your business analysts, your telephony engineers, your best service agents that know what customers think, those are your tactical people you put on the red team that actually show up and have the insights of. Here is what the best case golden data set should look like. Here's how we build the system. Here's what it needs to know and doesn't need to know. That's usually a misstep is thinking almost too philosophically about the problem rather than tactically what needs to go in it.

So that's usually the the primary issue I bump into every week is getting the right people around the table and really thinking about it. The second quick addendum there is fear of failure, or maybe, you know, scared to start is another way to frame that. One phrase I always say is momentum brings momentum, right? I mentioned A I is a continuing iterative process. A lot of people show up and you know, oh, we could do this or we could do this.

And they try to like over engineer all the things a L L M could do and how this new agentic future can take shape. When really you need to build those battle scars, like you need to pick something small and you need to commit to it with the understanding that it's going to more than likely fail, more than likely, you know, be challenging. But what you're gonna do is figure out again who needs to be involved, figure out where it failed and why it failed, and then build back stronger the next time.

That momentum helps you move faster forward. And it really is that hockey stick effect of companies that get going and with bumps and bruises along the way, and all of a sudden they understand what it takes to implement A I. And you see it s take off and skyrocket across the organization and across the C X stack.

Yeah, and that's exactly what we want. I think one thing, and I'm I'm gonna just ask one more question on this before we move on to the fun part of the conversation, where we can look at what solves for this. But one thing that I'm wondering about is one of the problem areas that we've been hearing about from our audience is that even though they have all of these different A I features built onto their system, it still feels like the humans are doing double the work, right?

So to do a full Q A, they don't trust the systems, or the system is not capable of giving them something that they can trust. So they still have five to eight people evaluating transcripts, um, doing manual Q A. What is it that causes our organizations or our enterprises to feel that they still need five to eight humans to double check the L L M's work or to double check the A I's work?

Yeah, I mean, trust is earned, not given. So anytime you introduce a new system, you know your humans that are doing this day in and day out need to verify. And that's just a natural cause of, you know, implementing a new process in a new system. That challenge, again, is is not unique, but does rear its head in many different ways. And what it ultimately stems from is fragmented systems, fragmented processes, and not thinking, I mentioned earlier, but that end to end design.

So I'll give you an example, you know, some of the solutions I work on and we have provides kind of after call wrap up work. So when a call concludes between a customer and a frontline rep, it fills out the disposition. You know, what are the key notes? What are the actions that need to be taken? It can trigger workflows, all the things a human do when you hang up the phone. Went on site and sat with some agents where this was rolled out, and those agents were actually taking the call transcript, copy and pasting it into a free instance of chat G P T and then measuring those notes back against their notes.

So one, they're taking sensitive company info out of the environment, and two, they're doing double the work that's already done for them. But they just didn't trust it because it wasn't necessarily rolled out and introduced the right way. And come to find out interviewing these reps, it was a non mandatory training. They didn't understand why this new widget showed up on their desktop. And it goes back to that process design of you have to work with your teams, you have to introduce them to why this is helpful, how it helps, and ultimately what's behind the scenes that's verifying this information.

So you build that trust really through that thoughtful design process and pull together some of those fragmented workflows to streamline. And once that's built and shared understanding is had, then you can really, you know, take advantage of it. But again, getting to that point is only found out after you build some of those battle scars.

Yeah, so I I'm guessing that for the leaders thinking about, OK, we have A I, but it's not giving me what I want. It's not giving us the R O I that we expected. This could also be part of that. It was never deployed correctly, and now they're too afraid to think about it from end to end because the little bits and pieces that they have already makes them feel nervous. But let's get into the fun part of this conversation and talk about the A I capabilities.

Setting aside the failures for a second and thinking about what's actually working, what kinds of A I capabilities are proving most effective at improving reliability and cutting down human dependency across all the different workflows.

There's obviously a lot of noise. The concept of a chatbot is nothing new. Concept of like smarter knowledge for the frontline teams not new, but certainly more powerful these days. I actually kind of challenge that statement when I go out into the wild and think about this of in the old world or product based world, you'd show up and be like, here's what my product solves for. In this new agentic world where you can kind of build what you want and shape the experience, it's more about where are those friction points, right?

What what is the customer experience today? And having that real world understanding of, wait a minute, why am I waiting on hold for 18 minutes? Or why am I pressing three for billing or stating in two short words the problem I have? Why is a frontline, you know, agent getting a coaching report every two to three weeks that's cobbled together? Starting with the problem and the friction points from the real world helps you understand where you can improve the process.

So instead of trying to, you know, fit a product into a solution, start with the problem and work backwards. That ultimately, I think, lends to more successful adoption because you get a better view of what can be solved and really where the value will come from.

Great, that's a good starting point. And I'm thinking, um, if we're thinking end to end, we've built the trust, we started with the real friction point, and you said earlier, failure is kind of inevitable. What happens when the A I fails us in that end to end process? If it happens early, if the fail happens early, does it get carried through to the end of the process? How do we then go back and gain that trust back?

How, what do we do to in those situations to also minimize it afterwards? What, what are we thinking here?

No, I think it's an important note, and I do want to circle back. Cause when we say failure, you know, we picture the spectacular failure that ends up on the front page of news articles. Or, you know, the the evening storylines. I'm talking about much smaller failure, right? Like the internal test. Think of it, you know, back to doing homework where you spent a lot of time and you got a B instead of an A. You know, it maybe didn't change your trajectory in life, but like at that moment in time, it felt like a failure.

And a lot of these failures are unseen, unheard. They're standard work that goes on in the development cycle, uh, in the building process. You know, we want to make sure, and there's a lot of capabilities and recommendations I may have, we would get to. But like, to be clear, these failures are not the throw it out into the wild and hope it holds together, you know, spaceship falling apart level failure. This is like, oh, dang it, our test on Tuesday didn't work.

Let's pick it up Wednesday, let's pick it up Thursday. It's those little things that compound. So when it gets to how do you build and how do you prevent, right? There's a lot of tools and technology that are coming out right now. I think one of the the hottest companies coming out of Y Combinator is AI agent evaluation. And the premise of that is if everybody listening went to your favorite L L M and we all asked it the same question, we may all get different responses.

How do you prevent that as a brand when you have a thousand customers coming to ask you? You can't have a thousand different answers out there. You need to make sure it's adhering to policy. So that's some of the capabilities built in for testing to make sure you can simulate all of those conversations, those edge cases, put them through and ensure that you're having consistency on every interaction or on every coaching scorecard or real time recommendation to your frontline team.

And once you get through those and build it and test it and simulate, those failures start to turn into wins. Again, these unseen small failures, and ultimately it leads to a more durable product that you can trust in the hands of customers, frontline or operations teams.

Yeah, okay, so, and then taking it back to thinking about not just deploying it in one specific part or one specific part of the customer journey, end to end being the main goal here. How do we know it's ready? Is it is that based on our foundation? Is the foundation built on the context from our data, our current data? Is fragmentation gonna play a role in there? How do we make sure that we minimize the failure from the start?

You know, how do you make sure it's ready is always a a tough one, but ultimately, right, you are building this based on your business processes and the best existing data set. So starting with data readiness, right? If your data bad data in is bad data out. Making sure that your core systems are accurate and accessible for the A I to use. That's things like clean knowledge bases, making sure all that information is indexed appropriately and ready for A I consumption.

Making sure that your back end systems, your C R M, your E R P, your billing systems, your scheduling systems, all are wired in. Should the that be in scope for your A I agent and running those tests against it, making sure your best humans again, I mentioned getting the right team together, but making sure your frontline reps who already handle hundreds of these conversations every week go and actually test it. They know what good looks like, they also know the edge cases and points of failure.

So once you get through that, the final step is really simulation and testing, making sure you maximize the load on these systems to really uncover. And while it may be some synthetic testing, it is based on real conversations. So that is where ultimately how you build some of that readiness to get out there.

Yeah, uh, and if we're thinking about data readiness, do you think unified data is going to make a unified platform m more of a reality?

Yeah, I I mean, that's gonna be the next, that is the next horizon frontier we are chasing. You already see it with some of these frontier models that are out there, you know, whether it's Open A I or Anthropic, it is the long term thinking and planning for multi-step procedural operations. Think of like loan applications, right? Or going from start to finish on an insurance claim for a car accident or a healthcare booking. That all stems from a unified shared data layer.

And what is important for anybody deploying A I is to understand what that means and what that looks like. So going back to like bolting on A I and the initial R O I question, you know, oh, we threw out a chatbot, are we getting R O I on this? It's like, well, you know, zoom in or zoom out, like, cool, some people talk to it, maybe it saved you a few calls, but long term, humans will still need to talk to humans.

There are some conversations, some instances where a human is needed, and you need to make sure that that data is on a shared layer. So thinking from an A I first perspective, it's not just about building an A I widget or capability that's bolted onto an existing process or system. It's what that A I is pulling from and where it pushes to so that the next system can pick it up. And whether that next system is a human or a different A I agent workflow, that's ultimately where this is leading to.

And I'll give you an example, right? I ordered a package. This is also probably real, but, uh, you know, I think universally fits for everybody. You know, you order a package, it was supposed to be here Tuesday, it's Wednesday, it doesn't show up. I in the middle of a workday fire off a note to the online chat bot and it's like, oh, looking into that. I don't like the answer. I chat again later in the day, I send an email, I finally pick up the phone by Thursday to be like, where the heck is this package I ordered?

And before I get to a human, I have to restate my name, my order number, tell them I've already opened a claim, tell them I already supported it. That's what I mean by unified experience. In this world, that web chat should enrich the data, my email should have created the claim number, and the human should pick up the phone right away saying, John, I know that your package is supposed to be there Tuesday. I'm looking at it now, it is on the truck and it'll be there at four o'clock.

But instead, I have to wait on hold, I have to resubmit my info. So that is where the vision has to come from, is can we build that shared intelligence layer that no matter which A I agent or workflow, human or not, has the same operating data, um, as the preceding one.

As a very frequent online shopper, I am voting for that. Where can I put in my ballot that that needs to happen? But also, so I read this article and I'm not gonna mention the source at, uh, right now, because it was it, it was a bit pessem- pessimistic. Where this writer said, uh, it basically was about a concept very similar to this. But this writer said that connected A I systems sharing context is just a nicer way of describing more integrations.

And from what I've seen based on my preparation for this conversation, is that's not necessarily the case. So can you give us the difference between the two? What is the difference between what we're describing as the connected intelligence versus just more integrations?

Yeah, I mean, and that I will say it varies, right? Because if you are viewing your solution or your approach as one offs or bolt ons, as I mentioned of, oh, we're gonna deploy this thing and we're gonna deploy this thing. You know, it's also been said, maybe by a different author, but like, I T is going to become the H R of A I. Lot of acronyms in there, but essentially, I T is gonna have to manage all these different systems.

And as you can imagine, it's gonna be wildly complex with different policies and rules and chaos is there. The way to get to your optimistic view, right, is to build that interaction intelligence layer. And what that system is and does is take in every touch point that is currently happening and sits in between your core systems and your A I agents. That is the interaction intelligence layer. If you are trusting your C R M to go out there and have customer conversations, then your ticketing system or your billing system is not gonna have visibility.

And frankly, I don't know if these C R M applications or E R P applications that are building A I workflows have ever thought about, you know, integrating with a contact center application or a workforce management application. So it's almost thinking of it as a new layer to your stack. What is that interaction intelligence that can build the integration once into those core systems that are still very much needed, but also can hold that context for a long time, hold that memory and information so that when you do deploy A I agents in any phase of your business, it can go in and pull that information right away.

Again, comes down to thoughtful design and the way you want to approach it, but given it's an iterative process, that's frequently where I see or the pessimistic side play out of, oh, we launched something. And at some point, it's going to dead end or just be stuck in that silo until you step back and think about the interaction intelligence design.

Yeah, so it's it's not just about being optimistic or not, it's about thinking about it intelligently as well. Uh, it's it's been great diving into this with you. We're gonna have, I'm gonna put in one last question before we wrap it up, and that is our leadership question. We know that we have listeners listening to this thinking, yes, we had that A I F O M O. We bolted on everything that we could think of, um, at that moment, everything that our competitors were doing, we were trying to, you know, do better.

And now we don't see the actual R O I because nothing is connected, nothing is speaking to each other. So for a leader listening and willing to admit the problem, let's look at where do they start at fixing it. What are the concrete first steps that our leaders should take right now to start closing that gap and getting real R O I?

Yeah, well, the good news if you have done that, you know, my best advice is start now. Again, momentum brings momentum. You have something to learn from, you've at least tried it and seen what works, or maybe already have some, you know, positive progress going. So if you haven't started, or if you have that F O M O that's driving you, hands on keyboard, right? Figure out what is possible, go play with these systems, even the open models, Anthropic, Claude, Grok, whatever your favorite is, just go push the limits and see what's possible.

That gives you so much familiarity. Other piece of advice is your existing processes and experts are in your organization. You know, you are probably already have tens of thousands of great customer calls where you know it went right, you know the knowledge articles that were used, you can surface that information. Using that to help you understand the friction points and you know what to build, how to build it, where it's gonna make an impact, exist, and it's not like a needle in a haystack search.

Go talk to your frontline reps, go talk to your supervisors, your managers. They can tell you their pain points and where they can see some automation improving things. They'll give you that set. So, yeah, that's ultimately where I think if you go look inside your organization, you can find a lot of those points. But as long as you keep building, keep learning and pushing forward, you're gonna uncover a lot. And then it comes down to, again, thinking through the process and continuing to build in an iterative fashion.

Yeah, and I guess also updating the process as you go along and finding new and better ways to do things. And the one thing with A I is it highlights the faults that we've had in the process all this time, but no one really picked up on it. So being involved and constantly involved, I think that's also gonna make a big difference. So for our listeners, I think our big takeaways from what John has shared with us today is the first thing is the reason why your A I is not paying off is not necessarily because of the technology.

It's launching without having operational discipline. I think I really like the way that John framed that for us. And then the A I capability that will move the needle is having that intelligent layer, having the the connection between all the different platforms, if we wanna call it that. And starting with a single workflow. It worked in the beginning, but

One more thing to chime in.

yes, done.

Obviously, it's gonna come from my perspective inside as a vendor, but having a trusted partner that goes through this, I, you know, the battle scars, the learning, building things out. It's not that your team can't do it. For we meet with a lot of customers and people every week that have teams focused on this. A lot of this is possible in this new future where the A I can almost build itself, you're bringing things out. Having a trusted partner is someone that helps you see around corners.

They've already built the battle scars, they've already gone through it, and it's that ongoing management and maintenance, Yolandi, that you mentioned. You know, we deployed this system, and now it's, you know, I moved ten people from this process that we've automated, but now I got twelve people working on maintaining the A I. Having a trusted partner can help you streamline that process, uncover things, and is the owner of that program. You still have the technology, you get the outcomes, but somebody that has been there and done that is hugely immense.

So you don't have to go through all the peaks and valleys of A I development. So that's one more thing I wanted to throw out there as a best practice is find your trusted partners who can help you see around corners that have been there and done that.

Yeah, absolutely. I think sometimes we forget that our internal teams, even though they they could be brilliant, they're very boxed into a certain environment. Where if you partner up with the right partner, they have experience across different fields, across different organizations, and they've seen so many things come and go, and they've seen what works and what doesn't. And they might be able to borrow some creativity from a previous project and apply it to your project, which is something that your internal team might not be able to do.

So very much agreeing on on that point, and I think it's important to make sure that you pick the right partner as well. John, it's been lovely having you on the show, great conversation. I'm very excited to see where this is going and check in with you in a few months time and see how, uh, this looks and how it's changed so many organizations, workflows, and the way that they see and experience their A I R O I.

Absolutely, greatly appreciate the time and the conversation. This is a lot of fun.

Wrapping up, let's look at the three key takeaways from the conversation with John. First, trust comes from the rollout, not just the model. Teams need to understand why a tool is in their workflow and how its output is validated. Second, build with the people closest to the customer, the analysts, engineers, and top performing reps, not only the leaders setting policy. And finally, start small, expect some failures in testing, learn from them, and use those lessons to accelerate the next deployment.

In this episode, we covered how A I investments in customer experience fail to deliver measurable returns. To go deeper on this topic and learn how to identify real A I trends by tracking where venture funding is flowing, and by listening to how leading C E Os describe risk and competitive strategy, download our free P D F report, Three Ways to Discover A I Trends in Any Sector at emerge dot com slash A I T one. That's E M E R J dot com slash A I T one to download your copy.

On behalf of the team at Emerge, we'll see you in the next episode.