Rebooting Enterprise AI with MCP and Kubernetes

Welcome to the Practical AI podcast, where we break down the real world applications of artificial intelligence and how it's shaping the way we live, work and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn X or Bluesky to stay up to date with episode drops, behind the scenes content, and AI insights.

You can learn more at practicalai. fm. Now onto the show.

Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO of Prediction Guard, and I'm joined as always by my co-host Chris Benson, who is a principal AI and autonomy research engineer. How you doing, Chris?

Uh, doing great today, Daniel, how's it going?

It's it's going, it's going really good. I I feel like I'm going to, uh, you know, just add a bunch of tools to my to my toolkit today. Uh, via via MCP, um, cause today we have with us Craig McLuckie, who is CEO of Stacklok, uh, which we'll learn a little bit more about. And of course, talk a a good bit about MCP and other things. Welcome Craig, it's great to have you.

Hey, thanks for having me on the show.

Yeah, well, I I kind of, I mean, and it people can look, uh, look up the the projects, uh, that you're working on now on Stacklok. A lot of that has to do with MCP and AI on top of Kubernetes. Do you wanna just give us a a little bit of a of a kind of setup of why you are spending your time thinking about this intersection of AI, MCP, Kubernetes?

Yeah, I mean, I think for me, it's it's sort of interesting. And and my, you know, maybe I can just introduce my background a little bit. Um, you know, I'm an infrastructure guy. I've been, uh, building infrastructure technology for pretty much the totality of my career. You know, I built a lot of infrastructure tech when I was at Microsoft and then at Google. I was, uh, happy to meet my friend Joe and we built out, uh, Google Compute Engine, which is, you know, virtual machine infrastructure technology.

We started the Kubernetes project, did a few other things, you know, inside there, we, you know, subsequently, we've built a lot of other, uh, technologies together. Uh, you know, either as in the startup context or or as part of, uh, larger companies. You know, I think for me, there's this sort of a lot of interesting parallels. I think, you know, history doesn't repeat itself, but it often rhymes. And, you know, one of the things that, you know, that got me really kind of captivated by this whole MCP thing was, you know, taking me back to my earlier career.

I remember the first time I saw Docker and so Docker is a technology. If if if the audience isn't familiar with it, that enables people to package up an application and all of its dependencies in a single container that's then highly portable and can be deployed everywhere. And when I saw Docker, I saw two things that were kind of, you know, occupying the same space. It was one of these wonderful technologies where, you know, it solved an obvious problem that developers have, which is how do you package up an application so it can run pretty much anywhere with all of its dependencies?

That's fantastic, but you could also peer through it, and you could see Kubernetes on the other side of it. Meaning an orchestration system that enabled you to build more complex applications that could be run the way that cloud native organizations like Google ran their applications. And it did double duty. And and when I saw MCP, I had that same kind of moment where you could see this technology occupying two spaces at the same time, which is is very rare and it's very wonderful.

And, you know, the the one part of it was, um, it hints at what the future of a end-tier AI native application might look like. Where you start to think of the the LLM as the presentation layer and view model for an application. It's starting to describe how you would formalize interfaces, what that middle tier of the application looks like. If you if you think about existing databases and and systems being the kind of the the sort of persistence tier for the for these these modern applications.

Um, but it also, you know, pointed to a set of capabilities that I think are gonna be extremely exciting to not just exciting, um, extremely necessary to, you know, large organizations. Because as you're bringing agentic systems into your environment, they are stochastic, there's new rules, like they're very, um, you know, they can be quite difficult to integrate s sustainably and securely. And MCP really represents this, this small, sharp protocol that you can use to start reconciling the behavior of systems that are accessing the the real world and also setting up guardrails and controls.

And so I I saw this and I was really captivated by it, and it started, you know, you know, causing me to ask questions like, Hey, I think I can see, you know, what the future might look like. And I think that we need to be able to operationalize this layer for organizations. And that really motivated me to start working on on the work that we've been doing here at Stacklok.

And just so people, I I mean, we've talked about MCP on the show before. Some audience members may remember that, some that maybe that's this is their 1st episode. Could you just give us a kind of high level of, um, maybe maybe a little bit of the why of MCP, and then the what of of what it is?

Yeah, so I mean, the the why of MCP, um, like you have this new technology called a transformer, right? Like a a large language model, generative AI. Um, and one of the most remarkable and wonderful things about it is that it interacts with you in natural language. It's very good at interacting with you in natural language. Uh, is very good at dragging semantic, uh, meaning out of large quantities of information. And And it's it's about, you know, taking, um, data, turning data into knowledge, taking knowledge, turning knowledge into, um, into a a decision, and then, you know, turning a decision into an action, right?

Um, you know, it's this new new capability. But the trick here is that there are some things that it's good at, and there's some things that it's bad at. It, it, it is really programmed to work in a kind of natural language format. So asking it to interact with relatively traditional APIs can be very fiddly. Um, it's not necessarily set up to deal with the authentication authorization process, you know, it sometimes isn't necessary to be completely trusted.

And so by introducing model context protocol, effectively, what Anthropic did was start to describe the outside world in relatively simplistic natural language terms. Uh, with a JSON schema backing it. That would enable large language models to start reasoning about what tools exist and being able to start invoking those tools in a deterministic way. And so the way I think about it is effectively this selectively permeable membrane that an organization can wrap around its existing systems. That allows value to flow through in both directions, but enables you to start asserting the controls that you need to be able to enable AI systems to actually do real work in the real world.

Yeah, that that's amazing. Ca- a And could you give maybe just some concrete examples of some of those kinds of systems within an organization that you might want to tie in this way? I know there's, like, innumerable, and people could imagine anything, but just so people have some concrete, concrete examples.

Yeah, I mean, so let's let's just imagine the the workflow of a recruiter, right? Like a recruiter is gonna be doing their work on a day to day basis. And they're gonna be interacting with email, they're gonna be interacting with LinkedIn, they're gonna be actoring, interacting with a CMS system, they're gonna be interacting with a calendaring system, et cetera, right? Now, the way that they would do their work today is they would do a lot of jumping between different SaaS systems.

So I might go to Gmail and send a note to a candidate. I might go to the Google Calendar app to, you know, calendar something. Or I might have my my content, my my candidate management system that has some natural integration to support these things. But I'm constantly jumping around. And what I'd ideally like to be able to do is have an AI. A system that is able to access my email, is able to access my calendar, is able to access, uh, my candidate management system, um, and but is able to do that in a way that I have some level of control over, right?

Like, I don't want this thing just going rampant and sending things. And so what? What MCP enables you to do is to start taking all those systems and describing nouns and verbs like, what is a candidate? you know, what is a, what is a, what is a, um, what is a calendar invitation? You know, what are the actions I want to perform, schedule an interview, do this type thing and start to describe those as discrete resources that, uh, the AI model can go and acquire and and tools that the AI can actually invoke to do to do work in the real world.

And by, you know, bubbling it up to a level where these things are described in relatively simplistic terms and then presented to the the LLM. It can now discover tools that are available to it, where it's like, Oh, you want to do this? Well, let me see if I can access your calendar. Oh, I can, because this calendar is now available through, um, through this MCP server. And what it also enables you to do is start dealing with the authentication authorization problem, right?

That That recruiter has, uh, an identity, you know, that could be, uh, like in Okta, or it be in Entra, or one of these IDPs. Um, being able to now set it up to say, like, hey, an agent that's working on behalf of this recruiter, uh, should be able to access these systems as the recruiter is able to. But also with certain levels of controls in place, because you don't necessarily just want to give these things unfettered access.

And so MCP is really the gateway to value for enterprise, uh, systems. And um, and and this is why I think people are so excited.

And yeah, just as a a to to, I guess, add, add to that. Even, uh, literally earlier today, like 3 hours ago, I was I was sitting at my desk in our our open office. And I heard one of our go to market folks go over to one of our engineers and say, Hey, I'm, I need to set up these. I forget how many it was, four or five MCP servers in the agent platform that that we use and I in our platform.

And the engineer was saying, like, why? I I mean, I think his mind was in, like, cloud code world. It's like, I use my agent and all my MCP servers to my technical stuff. But this goes beyond that. Uh, certainly it applies to developer tools, because obviously agentic things power developers now, but it it certainly goes beyond that. And I think the ones he was pointing to, I think it was HubSpot and instantly and LinkedIn, or I I forget which ones, but yeah, and that that level of things.

and it's it's really interesting because I think, you know, what we're seeing, you know, look, let's be clear, LLMs are really great at writing code, right? Like it's, it's one of the things that they do tremendously well, but they're getting increasingly good at a lot of other things as well. And, you know, like, when I think about, you know, my own personal workflow, um, you know, how I tend to do work, you know, you know, we've been partnering with Anthropic for a while.

Uh, they introduced a tunnel, so we now have all of the the Stacklok internal knowledge management systems are available through that tunnel to to to Claude. So it's not necessarily tethered to my desktop. I can use this on my phone, I can ask questions of my phone about things that are in my email. You know, it's it's. It becomes a very a a very powerful way to kind of open up the system, uh, in a way that's that's sort of sustainable, and it just fundamentally redefines how people work.

But I think the key thing here is that, you know, what most people are experiencing today, and they assume is that this is really tied to the developers machine. If you look at the way that cloud code is is, uh, structured, that, you know, code work is structured. It's using the developer's desktop as the aggregation point where all of the state is coming in and all of the outbound connections are being terminated. Um, and now, you know, what I think we're gonna start to see is this need to be able to move that off the developer's desktop.

So you actually have this controlled entry point into an environment where an organization can think more holistically about what services you want to provide. But can also do it for Claude and OpenAI and Gemini and and all of the other technology. So it's this, this great democratizer, it's democratizing data access while preserving control, and it's doing it in a way that's not tied to a specific provider. So you go through the exercise of of exposing your data once, setting up the policies once, and now you have the ability to, you know, use a a pretty broad cross section of models.

It occurs to me as we're as we're talking about this. And I think for anyone that's listening or watching that has been, uh, kind of in it, that this is very, very helpful. But I'm also a little bit worried about people joining. Could you? C is we? Could you kind of take a 2nd and talk about what the stack looks like? You know, now you have in MCP in there, you've you've kind of talked about some of the virtues of that and some of the problems.

But could we? Could we actually take a 2nd and look at what the whole stack looks like? Cause we mentioned a whole bunch of different technologies over the last few minutes. And because I think one of the challenges I keep hearing this year is people trying to kind of keep up with all the new things that are in infrastructure. Um, as you roll this world, we're all living in this world, um, but there's a lot of questions and people are hearing things.

You know, we've, we've talked about other technologies that we haven't brought in, things like like claw, Claude, claw and open claw and all these other things. And people are so confused, uh, in conversations that I'm having out outside the podcast. Can you take just a little bit of moment before we move forward into some of the specific things that you guys are addressing? And just kind of talk about the stack and how does it all fit together, and how people should think about that a little bit?

Just a level set before we dive a little bit deeper, because we've already kind of gone into some pretty cool stuff. And I don't want to leave people behind.

yeah, yeah, yeah, no, no, I think it's, and I totally respect that. You know, like everyone's at a different point in the journey, right? Like, a lot of us are geeking about, like, you know, how do we wire in this specific tool and optimize tool calling and deal with tool pollution? And a lot of folks are being like, what is a tool and why would I use one, right? So I totally respect that as a pretty broad cross section, I mean, it's, uh, you know, asking me to describe the stack, it's a very open system and there's a lot of different interpretations of what the stack is.

But let me let me describe, you know, what, what problem that we might be solving, and let me describe the set of technologies that might be used to solve that problem, right? So I think, you know, right now, what most people are experiencing and what works really well for a lot of organizations is just go buy Anthropic, right? Like, like, like, you know, like, it's like, it's almost this world where it's like back in the days of the mainframe, just go buy.

No one got fired for for buying blue, you know, like the blue, like, you know, like, it's the. It's the warm blue blanket that covers you. A and Anthropic is definitely kind of, you know, starting to look and feel a lot like, you know, IBM did around the dawn of the of the PC. And and for good reasons, they're doing amazing work. Like, um, you know, like, we are, you know, rabid fans of their technology. And so for a lot of organizations, the starting point might be, Hey, let me go get Claude Code, or let me go get XYZ.

And the starting point for them is like, Okay, I've got this system. And you know, it has access to my local file system. And so I can, you know, if I want to, you know, develop code, I can go grab some code and copy it locally, and then I can have it futz around with that code. Now I'll get to a point where I need more than the ability to just deliver, um, you know, these this sort of code.

I want to integrate this into, um, you know, a variety of different other systems, like I want to be able to integrate this into, uh, technology like, um, GitHub, um. Or I want to be able to integrate this into Slack, or I want to be able to integrate this into to whatever. And the starting point usually is, okay, well, you know, Claude will offer its own kind of native integration system. So they've gone and partnered with Slack, and they've gone and partnered with Google, and they've partnered with a variety of people.

And you get these basic integrations in place and it works pretty damn well. You know, you, you, you get it, you, you, you authorize it, native integration works. But at some point, you're going to start asking questions about, like, Well, what about all the other systems? You know that I use to do my work? How do I start to expose those? It starts to raise questions around, like, what do I need to build a bridge between this AI system that I use to do my work and this big, wide world of other technologies out there?

And so for a lot of organizations, the answer is, um, I, I need an MCP, uh, platform. And I think there's kind of, uh, what I think of as being four pieces that go into that. The 1st piece that goes into that is I need a runtime, I need somewhere where I can run this thing so that it's it's it's hosted. In the case of, you know, most development technologies, people go npx run. They basically just pull a package down from the internet and it runs locally.

God help you if that happens to have been exploited by a hacker or something like that. And so, you know, having a a secure runtime environment for this kind of makes sense. The 2nd thing you need is a registry, like, what service do I want to use? Like, how do I know whether they're good or bad? Um, you know, can I actually, you know, provide a list of service to my organization that I might use? So the registry's the next important piece.

Um, the the following piece of that is a a gateway. So hey, okay, here's these service like, I want to run them in a in an environment. I want to have a single endpoint that exposes them to something like Claude or to Codex or to any other system. So you need that kind of gateway technology. And then the final piece is what I think of as a control plane. You know, as you go from one to ten to 100 to 1000 servers, as you're starting to reason about mapping service to specific user groups, you need to be able to do that.

And so that kind of starts building up this this kind of what we think of as the the sort of MCP gateway system, like the the the MCP, uh, kind of platform system. But that's not necessarily all that people need. The other thing that, you know, people start to look at is also, well, what about the LLM gateway? Like, maybe I want to start building my own agents, maybe I want to start reasoning about, um, using a variety of different models, maybe I want to be able to institute my own tracking and policy management around, you know, who can talk to what?

So an LLM gateway is is another complement, and I think of those as being kind of two bookends to any kind of agentic platform that that a real world, you know, organization will want to use. You need an LLM gateway so you can start to direct traffic to a variety of different models and and and assert controls, and then you need this kind of MCP gateway so that you can start to connect real world systems. And then between those two bookends, you know it, it gets really fun and interesting.

You know, we can talk a lot about harnesses, we can talk a lot about, like, memory management systems, session management. Gets, you know, there's a lot of like moving boxes. We can talk about agentic frameworks like n8n and how they fit into that, or, you know, or crew or LangGraph, or, you know, there's this. It gets more and more and more detailed. But what I tend to think about is, and the guidance I tend to give most enterprises is like, look, you know, start with a vertically integrated system and then see how far you can get.

Uh, then start to assert, you know, assess an appetite and ask questions like, as for our developers on Claude Code, so for our knowledge workers, what does it look like to get there? And you will inevitably, you know, realize that you really do need these two integration bookends like you need an MCP gateway and you need an LLM gateway. Those two things, you know, typically you want to kind of deploy together. And then there's going to be a lot of other constituent pieces that you might pull into that.

To start creating really great experiences for your knowledge workers that that kind of decouple you from, uh, your, uh, you know, your sort of vertically integrated AI platforms. I don't know if that's helpful, but um, that's just how I think about the space.

No, it was a great expl- a really good explanation there. I appreciate that.

If you've been listening to the show over the past few months, you realize just how transformative agentic AI is. Whether that's Claude Code or Hermes Agent, or custom built software that you're deploying for operational efficiencies or as new products to your customers, regardless of your maturity. Now, this is the world that we're headed towards, this agentic AI world. And there's a lot of security and governance teams that aren't letting these agents go into production because of risks related to agency and autonomy.

And how do you take care of things like prompt injections or insecure tool usage? There's a lot to take care of, and that's why I'm personally spending my time outside of the show working with an amazing team of AI engineers to build prediction guard. Prediction Guard is an AI control plane that you run in your own infrastructure behind your firewall. Developers can build on top of this control plane using everything that they want to use. OpenAI and Anthropic compatible APIs, MCP servers, frameworks like LangChain.

But all of this is plugged into a built-in governance harness that enforces your organization's AI policies. And all of that telemetry goes back to your monitoring and alerting systems. I'd encourage you to check out what we're doing at Prediction Guard dot com slash practical AI. You can schedule a demo with me and the team, and I'd love to get your feedback on what we're doing. So visit us at Prediction Guard dot com slash practical AI. That's Prediction Guard dot com slash Practical AI.

Well, Craig, I, I have a bunch of I, I. I don't know how many I'll get to fit in, but I have a bunch of selfish questions. Just as a practical developer of of of some of these things. I I think one of the things that is some or or, or maybe you could help help people understand, is you have this. Let's say it's an MCP server for, uh, let's say Salesforce, or, you know, HubSpot or whatever.

That is running somewhere at a run i in a runtime, like you said, it's it's hosted somewhere. Then there's this like identity, uh, authentication piece that I think is is often very confusing for people. Or maybe, uh, a a lot of times, if they're building their own MCP server, they say, oh, well, here's this API to X system. And I have an API key for that API. So I'll set that as an environment variable and just all of my traffic will go to that API.

But then you lose that identity piece for who who's using that. Do they have access to the data they should or shouldn't have access to? Could you help us understand, like how that piece fits in the identity of the user, the uh uh, authentication with the MCP server? What are the kind of best practices around that and some of the things that people could think about?

Yeah, I mean, I think there's, there's, uh, there's a lot to unpack here, right? And I think, you know, there's the the world that is, and there's the world that we hope to move into together. You know, I think this is probably, you know, it's funny. My my buddy Joe, who I've worked with for years, you know, we built compute engine together and Kubernetes and HepTio and Tansu. And like, now he's my CTO here at, Um, at Stacklok.

He wrote the Spiffy paper, I don't know if you've heard of Spiffy. It's it's an identity system that's kind of, uh, a sort of, the sort of zero trust, you know, kind of identity framework. And, uh, he wrote that paper about ten years ago. I think we're finally now at a point where AI is the thing that's going to kick us over the over the line to actually move past. You know, relatively traditional kind of OIDC based, uh, systems to something like that.

But let me kind of, you know, te tease this apart into pieces. Like, so, 1st and foremost, MCP as a specification was really grounded in, like O, O, O, two workflows. So the idea being that, you know, the way that sort of Anthropic certainly looks at the world is you have a user, they're using Claude, uh, they have, uh, an OIDC token. That token can then get pushed on to, uh, an MCP service. It's basically identifying the user to the server.

And then what happens on the back end of that is broadly an exercise to the reader, you know? So basically, you know, like, what, whatever you want to do. And I think there's really two problems that have to be answered. The 1st is an authentication problem, and the 2nd is an authorization problem, right? So, um, you know, and and and you know, obviously, w the set of resources that that you're accessing are going to be, you know, sort of varied.

The only thing that we really have right now that's that that works for most organizations is the existing OIDC, uh, kind of tokens. And so I think we just have to accept that's where we are. But over time, as we start to build agents, agents are going to have to have their own identity. And it cannot be, you know, as simplistic as as the way that we've structured identity today. Because it's really gonna be this kind of three legged stool.

There's what I think of as a service account identity that's effectively identifying the agentic endpoint. It's like you're speaking to this specific agent. Um, there's a set of claims that are basically provided or presented to that endpoint based on the role that the owner of the agent has provided. And then there's a set of on behalf of claims that are going to be inherited from the user who's accessing that agent. And that could then get chained through a variety of things.

So there's a lot of really interesting work being done both in the MCP upstream specification. Uh, you can start looking at things like, um, uh, you know, the um, transaction tokens and th this, you know, there's. There's a lot of innovation happening in the IDP space around this, but that's that's only gonna help us tomorrow, it's not gonna help us today. You know, we actually have to get through the the definition and implementation of these these systems.

And so for for most people today, I think, you know, what? What tends to work is you, you. You 1st need to institute some kind of token exchange. So typically, you don't want to be in a situation where you you receive a user credential and then you pass it on to another system. You, you, you want to make sure that you're des scoping the claims to the minimum set of of claims necessary to perform a task.

And so typically, what we tend to do when we work with organizations is institute some kind of token exchange. So it could be, you know, and there's four or five different patterns here that that that might make sense. Um, you have, you know, straight pass through to, you know, where the API receives, uh, an OIDC token. Uh, you could have federated trust where you have to basically exchange the token, uh, to another federated trust domain. Um, you have the the, you know, the the the situation you you talked about, where you basically have to exchange it for, uh, an API key.

And you need to be able to make sure that you know that that action is actually pulled out of the the the agent's kind of purview and is handled individually. And so that's that's what we think of as being one of the primary roles of a technology like ToolHive, is that it it starts to formalize that. So that the the MCP tool developer doesn't have to deal with all of these icky mechanics of of token exchange.

That all hand is handled in the proxy layer, uh, you know, for the user. And then you just have to start setting up, you know, and and reasoning about how you want this to be handled. A very common pattern that, you know, we tend to work with people to do is like, um, I want to use the AWS MCP server and I want to use in read only mode, right? Uh, because God help me, like, I'm.

I have an agent that's running on my desktop. I don't want to keep watching it and having to scrutinize every time it interacts with the system. But I certainly don't want it deleting my RDS instance, you know, just because on a whim to to clear up an issue. Um, so how do you configure that to support read only mode? And one way you can do that is actually just implementing a token exchange where you take your Okta token or whatever, map it to an AWS token, deskop the claims and then hand that to the MCP server or to the to the backend API.

To actually, you know, pass through. And that's the kind of pattern that I think a lot of people, you know, want to be able to institute, but it's fiddly and it requires a fair bit of work. So you really need a platform team that's willing to do this work on behalf of users and and recognize these four or five, these kind of patterns. The other thing that's that I think is really important is is the authorization side of the house.

You know, I think most authorization schemes today, schemas today are really grounded in the idea that, um, you know, it's you. You have deterministic systems that are accessing it, you know, having an unsupervised system that's starting to access resources means you really want to start pulling out a lot more policy and and start put a lot more scrutiny on on on tool calls. And so one of the patterns that we see, you know, being very helpful is, you know, relying on the existing auth, auth C systems, you know, to to decide whether the agent should have access to it because the agent's acting on behalf of a user.

But then start to describe additional agent only policy as code capabilities that you apply to all your MCP servers. So you can start to describe those in a in a technology like Cedar or Rego or what have you. And then if you've got a common proxy, you know, system, you can start to apply that to to every, every tool call that you're making. Um, and so I know, I know, that's maybe, like, a little bit too, you know, specific, but, um, you know, I think you do need to separate out those two things.

And, um, unfortunately, there's no easy answer. When you start, you know, kind of having to deal with with uh, things like token exchange or, uh, or credential mapping, the the the one piece of hope I could, I can give teams is that if you have a platform team that's willing to take this work on. It is relatively easy to get to a point where you can just start to have relatively vanilla servers that rely on platform delivered, um, auth n or C, uh, capabilities, and you can just kind of snap them in and use them.

And yeah, you you mentioned ToolHive, which I think is is super fascinating. And and I I I wanna, uh, I I wanna make sure, um, our our listeners kind of understand also this this proxy layer. I, uh, m, maybe a way to frame this question is I I could perfectly well in some of the AI APIs, whether that's OpenAI, Anthropic, et cetera, sort of on the fly insert information about what MCP server I want to call and just handle that at my application layer, right?

Why? why is, uh, a proxy layer something that is is helpful for people in terms of proxying those MCP connections, rather than kind of integrating that at the application li uh level?

there's, there's several different, you know, reasons why you want to institute a proxy. I mean, the 1st is is basically, uh, visibility and governance, right? So let's imagine you're building a system where, you know, you have your recruiter. And they want to schedule an interview, and that interview is going to touch three different systems and something's going wrong and you need to debug it. Um, like it, it's, it's, you know, meetings are showing up on your calendar, but they're not showing up on the candidate's calendar or something else.

And you don't like, you know, like, if you have a if you have a proxy, which is basically, and you have a tool which is now describing a simple system like schedule interview. And it's it's, it's, it's, it's kind of amalgamating those pieces. You can start to, you know, you know, see a trace through the whole system. So, you know, when you have these workflows that are relatively complex and touch multiple systems by having that single kind of proxy layer, you can start to generate observability.

You can start to apply policy, uh, and so it's just from a general hygiene perspective, it makes a ton of sense. A 2nd reason why you may want to have that kind of that proxy or that that kind of gateway technology is, um, is optimization, right? So one of the things that, and I don't know if this is too deep for, you know, general folks. But one of the things that you hear a lot about is tool pollution, right?

So, uh, an MCP server has a tool description associated with it, so, you know, one well, one MCP server will have multiple tools and resources. Each of those tools and resources has a description associated with it. When you want to make those tools available, those resources and descriptions are in the the context window, uh, you know, all the time. And that might, you know, over time, if you pull in three or four different MCP servers, you may have 150 tools.

You may be burning 20 30k tokens every every interaction. Just just saying, Hey, by the by the way, here's the tools. Input token caching helps somewhat, but only to a certain point. And so being able to start, you know, kind of, you know, basically, uh, amalgamating that and basically saying, Hey, here's two endpoints, find tool and invoke tool. Um, yes, it's gonna be more chatty, meaning the LLM is gonna go, Okay, I need to use a tool.

What tools are available, you know? Find tool with a description of what you're trying to accomplish. And then, you know, provide back a list of tools that actually meet that description. It's a it. It reduces input token consumption by eighty to 90% when you have these these things, and so that's a very big deal. Versus, you know, just allowing the models to access the UM tool selection, you know, particularly, you know, when you s like, when you, when you, when you're working with Opus 47, it's just so damn good, it really doesn't matter, it's gonna figure stuff out.

But the minute you start dropping down to Sonnet or Haiku, or one of the smaller, uh, systems. Or if you're trying to build an autonomous agent, one of the hardest problems is just making sure the thing calls the damn tool when it's supposed to call the tool. And, uh, smaller LLMs are notoriously bad at tool invocation. And if you start putting 2030 tools in there, forget about it, it's just not gonna happen. Um, but if you, if you replace that with a single endpoint that ha, you know, can provide much more, you know, fine grained guidance and distill it down to just the set of actions that it, uh, uh, a system wants.

You can get back up to the sort of ninety five, ninety seven threshold that actually makes the system useful. So it, it, it drives behavior there, and then finally, less clutter in the context window generates better results. Context optimization is another big, you know, point of it. And then the final piece of it is just, you know, like, you know, project or user based views. You know, sometimes you want to construct a set of tools that are specific to a task.

Let me give you an example. If you're working on a GIS system as as a developer, um, that's a like, you know, kind of mapping thing. Feature means something very specific, right? It's a collection of vectors that describes something on the terrain, right? If you're interacting with a GitHub MCP server, feature means something completely different. And if that developer is talking about features in Claude code, it's gonna get that thing completely confused, right? So being able to start, you know, formalizing the nomenclature of like, instead of just describing this as a feature, but like describing this as a GIS feature or something like that, versus a product feature.

Uh, and being able to kind of, you know, sort of augment the tools with something that's semantically more relevant to the task at hand, it enables you to improve the behavior. So the other the other reason to kind of institute this type of abstraction is that you can also start to create much more fine grained tuned views for specific agents, user groups, et cetera. That takes a vanilla tool and makes it far more intrinsically useful.

Such a great explanation there, I really appreciate that. Um, I I think, uh, you know, one of the things that we were were kind of m mentioned by name a moment ago was ToolHive. Um, and as we are kind of taking the concepts that you're sharing with us and and diving into how you guys are approaching, you know, the the proxy issues and stuff. Could you? Could you, uh, for those who haven't, uh, had any exposure to ToolHive?

Could you? Could you take us into what that is as a as a solution? Um, and kind of define how it fits in with some of the the context that you just now addressed, that'd be fantastic.

And let's be clear, uh, ToolHive is an open source project, it's Apache 2 licensed. Uh, you know, my background, hey, I did Kubernetes, it was a great open source project, uh, bootstrapped CNCF. I love open source, I love communities. This is an invitation for people to party on, you know, with us in the open on this technology. Uh, there's nothing, there's no greater compliment than discovering someone is using it. And and you know, you know, reaching out later, fork it, I don't really care, just it's open, that's what it's there for, right?

Um, and so what we built with ToolHive, the the, the philosophy of ToolHive was really this, which is, look, Anthropic, OpenAI, Google are describing the Emerald City, you know, they're telling us about this beautiful place in the future. Someone needs to build the yellow brick road, someone needs to build the the basic procedural things that enable you to actually get to that that destination. And so we started looking at, you know, a technology like MCP, and we were like, Oh gosh, this has to be done right, it just has to be done, you know, to enterprise standards.

Uh, it's such an important thing. So we started asking questions like, well, look, we don't have to reinvent the wheel. Like, there's a lot of great technology that came out of the cloud native ecosystem, which is something that I was very intimately, you know, kind of a participant in, in sort of in, in, in shepherding into existence. Can we take a lot of the learnings and a lot of the technologies out of the cloud native ecosystem and just repurpose them so they work really well in the AI native world?

And so starting point for us was like, Hey, that Linux application container, it's the foundation for Kubernetes, let's just put our MCP servers in a Linux application container. You know what that means? Well, for an enterprise, that means that, uh, it's an OCI image. And they know how to reason about and harden and scan and validate that that image, um, we can complement that. And so what we've done with ToolHive is not just, you know, hey, it, it runs through your full SDLC the way any other piece of technology that you're deploying does.

We also do a lot of like MCP specific scanning and and reasoning. But, you know, so, so we basically provided a pipeline that you can basically generate a container and then deploy it in a runtime environment. The 2nd thing that, you know, we started looking at was like, well, you know, there's a lot of servers out there that are really useful, like, uh, the fetch server is probably one of the most commonly used server. Hey, I have an agent, I want to be able to access something off the internet, I want to use the fetch server.

Um, I might just have given that agent access to the totality of my intranet if it's running behind my firewall, right? Like, how do I constrain its view to, like, I just want it to fetch documentation? So how do I turn the fetch server into my fetch documentation server? And the way to do that would be to constrict which network endpoints it can talk to. Turns out, containers are really great at doing that. So by wrapping it up in a container, you can start to say, Hey, I'm running this thing.

I don't want it to access my personal photos, you know? So I can describe which portions of the file system it can access, I can describe what network endpoints it can access. So it becomes a secure environment to run these MCP servers that you can then, you know, control, uh, and you can, you know, turn them up locally on a developer's desktop, you can turn up in the in the cloud with with Kubernetes, and you can run one ten, a hundred of these things.

The next thing that people tend to encounter is this idea of like, well, um, I want my developers to be able to find and use MCP servers, but I want them to find servers that are vetted, trusted, et cetera. And so the registry becomes a very natural part of that. So basically being able to describe to, uh, a client that speaks the registry protocol, saying, Hey, here's the MCP service for your organization, here's where you can find them.

And you know, you know, whether they're being downloaded and run locally, or whether they're just being accessed via a proxy at a a sort of hosted endpoint, the registry is a very important part. So we've built out an MCP registry, you know, we provide, um, tools and capabilities that allow you to harden the images to your taste. Uh, you know, we'll, we'll, we'll provide a pre populated set of of images that we've scrutinized, we've scanned. They're, they're, they're coming in out of the community, but we we stand behind them, you know, we we hold to a certain standard, uh.

But we can also enable people to start layering in their own attribution, like, hey, I want additional scrutiny on these things. And the registry becomes that, that critical control point. And it becomes the place where you start to describe the policy that follows that server down into the destination where it's running and enables clients to discover those servers. And then the f, the, you know, the the other piece I talked about is this kind of what we think of as the BMCP gateway, the virtual MCP server.

The ability to say, for this set of users, I want to expose this set of tools, and I want them described this way. And some of those tools might be composite. You know, I might, instead of, you know, saying to the to the the the agent that my recruiter's using, Hey, here's Google Calendar, here's whatever, here's whatever. Maybe I want to build an MCP server, which is C, which has a single endpoint, which is schedule interview.

And then there's a s a t a sort of declarative workflow behind the scene that actually goes from system to system and and binds that whole thing in a transactional context. So that that either passes or fails atomically. So you don't have, you know, you know, some calendar invitation showing up here if they're not showing up there. So you can start to, you know, build out those capabilities, uh, where you can take basic MCP services, building blocks, and create this, this virtual view on them.

That that's that's really tailored to specific user cohort, et cetera. And so that's another part of the platform that we've built. And then the final piece is just, you know, what we, you know. One of the things we've observed is when we built the system, most people were running these servers locally. But we're seeing, you know, fifty month over month growth in the Kubernetes use of this technology, you know, like it's it's. It's astonishing how quickly we're seeing people actually adopt the ability to run MCP servers in a in a Kubernetes destination.

And so, you know, we're getting millions and millions and millions of of tool invocations from the the Kubernetes side of the house. Uh, and so that that that Kubernetes control plane is the the final piece of it.

And y you, you mentioned some of the, yeah, the the Kubernetes side of things, the declarative nature of some of that, I think, uh, working with Kubernetes at at certain points, I'm certainly no expert, but I. One of the things that's always, of course, uh, a great feeling is to sort of have that, uh, declarative workflow. Where I say, I want, I want this to be the the state, and it sort of happens on the on the back end, right?

I'm I'm wondering how you see that infrastructure side of things developing? Because now, uh, because the interface that we have as developers, infrastructure, DevOps people, is a lot of times now n uh, in natural language, sort of declarative in its in its own sense. It seems like that Kubernetes, uh, control plane, and, you know, maybe, uh, the the downstream. The things like the ToolHive and and other things that would be declared that way would be very natural to, uh, to to manage and configure, uh, via via natural language.

Is that? Is that something that? Yeah, yeah, I I guess. How do you see that developing and and how do you see that fitting into kind of this cause? These systems, like you say, maybe it's all of a sudden I have 600 agents, or I have 900 agents. Everyone's on a different maturity path here. And maybe some people are listening to this, they have one agent right now. But I think in the future, there's a future where they'll have many, many agents running in their in their environment.

And that that can be very scary infrastructure wise as well.

Yeah, I think, you know, one of the things that was beautiful about Kubernetes, you know, like it's, and this is a testament to, you know, like Joe and Brendan, and you know, some of the earlier people that worked on it. And and then, you know, also a lot of the the sort of hardcore Google engineers that had been sweating the details on this the systems. But this idea of kind of reconciliation driven infrastructure, like the idea where you basically can chew off something, describe how you want it to be, and then have a system that is, you know, solely responsible for making that true, right?

And so I think there's there's a lot of different directions we can go with this. You know, one is, um, you know, those reconciliators today are, in principle, um, t like deterministic systems, I mean, no system's really deterministic. Anytime you're, you're dealing with the real world, entropy has a way of creeping in to anything that you're building, right? Just by virtue of the fact that, you know, life is chaotic, the the world is chaotic. But now we're introducing th there's the, there's certainly the the the possibility that we can start to have, um, stochastic systems driving reconciliation loops, right?

And that's that's a direction we can get, you know, kind of start l like leaning into, you know, where, uh, you know, we can start to, um, you know, describe what we want to have happen. And then, uh, when something goes out of conformance, invoke a stochastic system to reason about why it's out of conformance and then start driving it back into conformance. So I think, you know, one of the things that we will certainly see over the next little while is the ability to, um, have self annealing, self healing, self optimizing systems.

So, you know, you'll be able to, you know, describe what you want to have happen, you know. It'll basically generate the YAML manifest, hand it off to Kubernetes, and then you'll just have very smart systems that are are watching it. And when something goes out of conformance, it can, you know, potentially pull in, um, you know, you know. Obviously, it'll try to reconcile it, and if it gets to a point where the reconciliation's not working, like, Hey, this pod's in crash loop back off.

I'm at the at the boundaries of what a reconciler can currently do. That's when you will have the opportunity to start pulling in stochastic systems to drive it. And I think that's going to be a very interesting direction for us. As, uh, you know, as we, you know, just get even further out of the infrastructure, we just give it to the infrastructure and let the infrastructure run it. Now, in terms of, like, what's necessary to run agents?

I mean, there's a lot to unpack there, you know? I think that we will certainly see Kubernetes-esque patterns. I think we do need to start reasoning about, like, what is the packaging, you know, definition for an agent look like? How do we, you know, maybe is it, is it something as an OCI, you know, entity? How do we make other systems available? And you know, like, um, like, hey, I want this thing to be able to, you know, generate and run code as part of its behavior, but it needs to be isolated.

Um, and so I think there are going to be a number of agent specific platform systems that have to be added. That can then be, you know, fit into that control loop system and then described, you know, as as either tools or other, you know, abstractions to to agents that are running. And then I think the the the harder question, and this one I don't have an answer to. And actually, if there's anyone on this podcast that knows the answer to this and and has a really strong theory around this is, you know, like, actually tracking agent behavior and and what a reconciliation loop looks like when you want to start bounding agent behavior.

Um, you know, certainly evals are pointing us in one direction, like kind of human evaluated, um, you know, kind of human in the loop style systems. Um, you know, being able to signal and s, you know, sample, aggregate signal on on certain patterns, uh, you know, having other, you know, sort of agents watching agents, where you can start to, you know, have a a sort of a a watching agent. Start to reason about the state of another system or behavior system.

There's a lot that has to be done there. I don't know exactly what that pattern looks like yet, you know, we're certainly playing with ideas, you know, we've, we've, we've penciled out a few things, we've built a few things ourselves. Um, but um, I think we we still are just learning as a community of of, you know, whether, you know, stochastic reconciliation, you know, outside of, you know, you know, performing remedial action is is, you know, like, what that looks like.

And we haven't, we haven't yet got there. And I think that's something that we're gonna have to think about as a as a community.

Well, y, you've already kind of started going the direction that that we we usually end up the show on with with a guest, which is kind of looking forward to to to what's next. You know, you mentioned some of those challenges that are yet to be yet to be addressed in the community. What, what are, maybe just as we wrap up here? What are some of those things that you're excited about? That are maybe coming within the ecosystem?

And you see developing that you think would be transformative, or or things that, uh, maybe it is other things that you're kind of what? When you're laying in bed at night, you're, you're thinking about these problems, what? What's at the top of your mind? Oh, what Kind of going

do I dream about?

into this, yeah, w what? I got tired What's at the

of tattoos.

top of your mind? Going into into this next, uh, season of MCP and agents?

I. I think the thing that I'm most excited about is like, you know, as for developers, so for knowledge workers, I'm meaning, you know, what Claude has done. And like, when you s when you look at an individual, like, I look at my team of developers, right? And like, I look at our performance. And you know, we we are very deliberate about instrumenting our our, our code. You know, like we, we have, they have, like, deployed, you know, all of the developers have, you know, hooks.

We know exactly which agents they're using, how they're using, we can, we can, you know, correlate the the the behavior we we treat as a performance. Um, it's almost like a performance sport, like my developers are now performance athletes. And they're, you know, they kind of wired up and they're, they're, they're, they're, they're. And and we can see what's driving productivity and the thing that's driving the most dramatic productivity for my developers is what I think of as agentic concurrency.

Um, so being able to have a system that they set up where they'll have 15 different agents, each with a slightly different configuration recipe role, uh, performing a set of tasks with access to tools that are highly controlled so that they're basically running in YOLO mode. Um, you know, it's it's sort of on on the path to that kind of dog factory story, but there's still a a human operator, you know, spinning plates, like, you know, having somewhere between five and and 15 agents concurrently running.

Um, and the productivity is dramatic, right? Like, it's look, it's costing us a lot of money, we spend burning a lot of tokens, but it's more than making up for that in terms of productivity. I mean, I like, I'll I, I track our weekly productivity. This last week, our person like our, our engineering team's throughput went up 60 in a week. Just because as the team is starting to get better at at sort of systematic, you know, concurrency, our ability to deal with community issues, you know, we're finally over the threshold where we're actually able to burn down issues faster than they're coming in.

Like, everything is changing. What does that look like for knowledge workers? And that's the thing that I'm most excited about. Because I'll tell you now, like, there are things that are the same. Meaning, you know, we will get knowledge workers to that point where they're able to spin plates and imagine themselves as orchestrating a lot of things. But there's a lot of things that are different. The developer's desktop, the the the desktop, just cannot be the aggregation point.

Their threshold for pain is a lot lower. They cannot be trusted to kind of build and run MCP servers, right? Like that has to be provided to them. They really need to be served by a platform team. But I think we can give people superpowers. I think, you know, the the. The productivity gains we're seeing on the development side will translate to every other function. If we can just start to learn from what's really working well.

And, you know, you know, I, I, I love what Anthropic's doing like it. They really write letters from the future. And if you just sit down and bother to read them and then think about what this looks like through the lens of other domains, there's a lot to be, there's a lot to be gained there.

That's awesome. Well, I I appreciate you taking time today and also thank you from from the community for the great work that you and the team are doing on on uh, ToolHive and and other things. And uh, we'll look forward to having you back on the show to to talk about it in the future. Thanks, Craig.

Hey, thanks for having me on.

All right, that's our show for this week. If you haven't checked out our website, head to practicalai. fm and be sure to connect with us on LinkedIn X or Bluesky. You'll see us posting insights related to the latest AI developments, and we would love for you to join the conversation. Thanks to our partner Prediction Guard for providing operational support for the show. Check them out at predictionguard. com. Also, thanks to Breakmaster Cylinder for the beats and to you for listening.

That's all for now, but you'll hear from us again next week.