Governing AI Agents at Scale: Identity, Scope, and Observability (with Glean and Cvent) | CXOTalk #914
CXOTalk · 25 Mar 2026
Source: https://podcasters.spotify.com/pod/show/cxotalk/episodes/Governing-AI-Agents-at-Scale-Identity--Scope--and-Observability-with-Glean-and-Cvent--CXOTalk-914-e3gvpoj
Transcript: https://podcasts.opennash.com/shows/cxotalk/governing-ai-agents-at-scale-identity-scope-and-observability-with-glean-and-cve-e9d03e/ (via OpenNash Podcasts, machine-generated)

AI agents are making decisions in the enterprise, but who's governing, managing them? Glean's Work AI Institute developed the AWARE framework to tackle agent security, governance, and compliance. Pradeep Mannakkara, CIO of Cvent, and Ben Mayrides, their CISO, explain.

Cvent is a leading SaaS hospitality and event software company, about 30,000 customers, 5, 500 employees, over 6,000 agents, and since about 2023, we're part of the Blackstone portfolio.

Ben, when we look at agent governance, agent security, this is a proliferating bundle of issues. What are some of the core challenges from your perspective as a CISO?

Number 1 is education. The space is moving so rapidly, it's evolving so rapidly, and understanding not only what these tools can do, uh, for us, but also understanding what they can do to us, I think, is incredibly important. Um, so education, awareness, uh, from the board level on down, I think is is is a big challenge. Uh, and, again, it's just the rapid pace and the eagerness with which organizations, um, mine included, uh, are adopting these technologies.

So, uh, that's, I think, challenge number 1, but m maybe to get a little bit more technical, um, I think that, uh, when it comes to some of the technical controls, the security architectures that we're all used to are really focused on deterministic systems, right? And as we know, AI is all about non-determinism. Um, so the architectures are broken. So, when you think about agent identity, for example, agents are planning, they're reasoning, they're they're they're, uh, and and they're delegating actions, and our existing IAM controls, um, aren't necessarily architected and built for for that right now.

Um, same thing for observability, for example, uh, you know, a lot of agent reasoning is opaque, um, and it's difficult to sort of reconstruct those those elements as well, and we need that. We need that for compliance reasons with things like the EU AI Act. Um, uh, but also in the case of a security incident, we need to be able to restr- uh, uh, you know, restructure and understand what actions an agent took, and so on.

But, the existing security controls that we have for observability aren't yet attuned, and they don't really integrate very well in an agentic architecture, right? So, we need that's where we need purp- uh, purpose-built, you know, governance, we need education, um, and we need these technical controls to mature.

Pradeep, Ben made a really interesting comment. He said, Agents are valuable for what they can do for us, but also to us. Maybe from a CIO perspective, dig into that for us and to us.

Whatever we implement, we need to make sure there's adequate safeguards in place to manage risk. And I rely on Ben, our legal team, our privacy team, to kinda help sort of shape that aspect of it, but at the same time, move fast. Now, your question around what it can do for us, I look at in terms of the value that it can bring us, right? Whether it's taking action, whether it's finding information, whether it's suggesting all those things that Ben highlighted.

I I I think we've all seen where agents can do really well, and we actually have demonstrated ROI proof of that. The things that you can sort of take to us, and if I necessarily consider it negative, it just means we need to make sure we've got some mitigating controls in place, and that can happen in a variety of ways. Uh, one of the challenges go back to your last question we have is there's a lot of vendors, partners, solutions that are out there.

You multiply that times the number of employees we have, and now we have a a lot of, uh, ideas, amazing, great ideas coming to us, and what we've gotta figure out is how do we review these things? How do we look at the value of each of these things? And as we try to do this in a meaningful manner, one of the things that we also look for is to make sure that there's financial safeguards as well in place.

Um, and then the the 2 things is, you know, I rely on Ben and other teams to make sure we're protecting ourselves. That's how I look at the 2 part.

You spoke about the various ways of looking at governance, looking at protections. The Work AI Institute's AWARE framework covers 5 governance challenges for agents. Intent, context, guardrails, risk scoring and blocking, and ecosystem observability. How do these principles map onto your decision-making as you're deciding what to work on, what not to work on, what to hold back, and so forth?

It's very relevant, of course, uh, to any sort of agentic architecture that we have. So, you know, as we actually have been a adopting the platform, it's been taking a look at, uh, alright, what is what is an agent actually? Uh, what is the identity that it might be assuming or that might be delegated, uh, to it? Um, and it's the it's the who and and what is taking that that sort of action. Um, we need to actually identify that, eh, in advance, like, what, in addition to, what is the scope that that agent is actually gonna run in, what are the data connections, uh, or the data, uh, sources that it's going to run in, uh, and the context

is incredibly important because, like, say an agent that is, um, maybe going to Salesforce and reading a a customer record for the sake of summarizing that record for a salesperson versus an agent that is accessing that record to maybe update it, um, modify it, or maybe write, uh, you know, uh, summarize that, and maybe write some of the sensitive data to another resource. That's a completely different scope, completely different risk profile, so, stepping through the AWARE framework, we're able to actually break that down and then make sure that we have the right kinds of controls in place, right?

So, it's essentially a comprehensive set of controls and sets of issues you need to look at.

In my view, it's a technical structure, it's a technical control structure because, you know, the frameworks that are out there are predominantly organizational governance frameworks. So, when you think about the EU AI Act or the NIST, uh, risk management framework, again, they're great, and they have their purpose, um, but they don't go deep enough, and they don't actually, uh, tie into, sort of, the technical controls and the the technical considerations that you need to build in to an agentic security architecture.

And I agree with Ben, so, which is why, sort of what we look for is try to figure out what's a good pave road of technologies that we can provide to our business. I'm gonna separate our call it our CTO organization, our product development organization, cause there we have a, I'm gonna use word, a mature framework in place with actual appointed people and their roles, and they work within this framework that we, uh, put in place.

On the business side, we see a lot of desire to move fast, right? I see some heads nodding in the room. And what we look for is, there's a lot of demand that's coming to us to say, get us something quickly. So, one of the things that where we look to look for is technologies or platforms that have this sort of inherent within that framework or that platform or tool that allows us to then deploy something quickly, cause we are here with Glean, one of the things that we realized with Glean as we kinda did that pilot initially and then rolled it out was that actually bought us probably about 9 months of time, candidly speaking, which satisfied a lot

of our business users' needs, meaning they didn't wanna rely on IT to do things, they wanted to kind of go and experiment and do things on their own, and we, as IT and security, didn't have a facility to easily enable that within a corporate, uh, ecosystem meaning, can I get access to my email, my Slack messages, my Box files, Salesforce with that fine-grained security controls and things like that. So, by us having the number of platforms that allow our users to achieve that, and then training, which you brought up earlier, uh, which then also security has sort of blessed the sort of the walls around it saying, Hey, if you do it within this space, you don't have to come

to us. We know the controls exist, off you go. Um, actually gave our users a ton of capability, uh, to the point where we had a AI council with numerous requests that we would have to, uh, somewhat custom do, those all sort of went off on the wayside. Now, there's sort of the next evolution of that that's coming, which is, I want workflow. But

Ben, you mentioned the term training. Pradeep, you said, surprisingly to me, that culture is part of your role as CIO. How do you make the framework come alive, which seems to me to be a a cultural aspect, and it's more than just training, right? It has to be more deeply ing- alright, so so talk about that. How do you bring it alive?

We're very bullish on adopting responsibly using AI, you know, with with all those other things that you talked about from a governance point of view. And as we're going through that, I'm like, we're very bullish on this, however, we haven't baselined our employees. We've maybe we have 15%, maybe I'm being generous, I'll say 20% that, for whatever reason, have taken training. They've learnt it on their own, they're using it in their lifestyle, but my belief is, you know, you might have a third, a third, a third, third that are interested, a third that's like, I don't believe in this, a third is like, let's see where this thing goes.

But one of the decisions we made was, we're gonna give mandatory training to everybody in the company, the foundations and a literacy, and in there, security has a piece, legal has a piece, everyone has a piece around the education that kinda talks about some of the pitfalls, uh, with AI as well. So, again, it's not a one and done, but it sets the sort of guidelines. Actually, our CEO was in the 1st session. Right? Just to kinda tell you how important this was.

So, Ben, it sounds like this is fundamentally a different kind of training than we're rolling out a new tool, and here's how, you know, here's how to use the tool. This sounds different.

It is a big mindset shift, I think, as, uh, many would probably already, uh, understand and and and have experienced themselves. Um, but I think in addition to what Pradeep was just saying in terms of that education and that cultural shift, you also have to have a structured ri- risk process, right? So, as ideas, um, you know, this mindset shift happens and ideas, new ideas are coming up and new proposals are coming up from all over the business, we have to have a structured risk process to be able to take those ideas, evaluate them, yes, across a number of different dimensions, including security, um, uh, and we have to have clear success criteria, right, for evaluating and building into those

projects, building into those deployments, and building into the operation of those those AI systems.

Also, the respect that, sort of, the this CISO organization earns, like, it has to be earned, that trust. And I think we had a recent example, right? Claude Co-work is out, right, still in sort of the it it's not GA, but it's like, Hey, it came out, people hear about it, they're like, We wanna test it. And

And Claude gives you messages when you open it up. Hey, Claude is here. You should try this now.

1,000% and they also tell you, Don't use this in a regulated environment, and so on. However, at the Yeah. Read end of the day,

the fine print.

it's a fine print.

Oh, and by the way, we're not responsible.

A now, at the same time, this is where the CISO, CIO role, I'm like, We should m have a framework to evaluate that, to understand capability, but not at the detriment to the company. Ben and team kinda looked at this and said, Risk is too high. And and and not just risk is too high as a generic statement, but here are the reasons what what decisions

Well, and it's risk is too high for now, right? Because, um, you know, there were s there, I think, a a a challenge that, um, security folks run into is that sometimes, um, maybe we're not looking at, uh, the the business, let's say, is not looking at all of the risk factors. Maybe the, you know, traditionally or up to now, um, it's been very common for the business to look at model security, right? Um, where it's about what what data is that model trained with?

What is the way to access that model? Is that model provider actually gonna retain my data, for example, and train and use it to train their their model? Uh, in an agentic architecture, it's completely different, right? It is about what that agent can do, what it can access, what decisions it's making, what what actions it's delegating, uh, what, you know, what users it is impersonating. It's all of those things, and so, whoa, let's kinda it's it's so it's not no, it's let's take a look at this and let's be deliberate about it.

Let's be thoughtful, and, um, again, you know, kinda coming back to the, uh, aware framework, what I like about it is that, you know, it sets that c- clear success criteria, um, for, you know, me and and him in particular, but also other business leaders that are really keen to adopt these tools, we can say, Well, look, if we can just answer these, let's say, 5 questions related to the aware framework, um, if we can agree on and we can get to the same answers, um, it eliminates a lot of the, uh, c- this guy gives me a hard time all the time about, Well, w- why do you not like it?

And I'm just like, Ah, just, you know, i I it's a gut thing, and that's not an acceptable answer, right? I mean, I would figure, um, e- That's a therapist, You know

what I tell my about that messaging. It's it's how you message that same response can Yeah, but be interpreted.

those those answers get to a certain level of specificity, and we've agreed on the r- the the rules. He he hates it when I s I say rules, but, Guidelines. um guidelines. Frameworks. Requirements, um, uh, perhaps,

Helpful hints.

right? Um, but if we can agree on what the answers of those questions ought to be in advance, gets us to a level of specificity, removes the ambiguity of the, you know, s and and and that, from my perspective, gets me a little bit closer from and out of the the the realm of being a gate and maybe just being a part of a business conversation and a business decision.

If I can paraphrase or simplify your roles, correct me if I'm wrong. Pradeep's job role as CIO is to move fast. Your job

Like I said, is to make sure he doesn't trip while he's got the scissors in hand, so we can okay right? So, so we so we yeah.

can agree. How does a framework such as Aware help manage this really fundamental tension? I I am sure you must have users coming to you, and they say, We gotta do this, you know? We got now, we gotta do it. How how do you manage the tension?

I think it's along the lines of what I I I was just trying to get at, um, which is that it gives us a I I mentioned a little bit earlier, having a structured risk process and and clear success criteria. It gives us a set of questions that if we answer those questions, then we can, uh, you know, when it comes to being able to greenlight a project, um, in particular, everybody knows what those what those, uh, criteria are in advance.

They can build it into the design and to the deployment and the operation of those things, and we can get to yes a lot faster.

Pradeep, when your users come to you, and they say, this, we need this now, what do you tell them?

Similar to the sort of aware framework, we've also actually partnered with our PMO office and actually created an AI sort of flow, which thanks to our partnership with our finance organization, says, Hey, there you know, we we if you wanna go and demo something, by all means, go talk with the salesperson, go see a demo, understand what the capabilities are, love it, actually, and inform us, right? I don't have to come up with the idea to implement something.

We are actually we'll learn from each other. Now, you wanna take it a step further means that, ooh, there's value, and and and I love value. So that means if we're gonna help add to the top line or take away from the bottom line, however it is that's gonna drive value, I'm in.

So, we can do whatever we want, in other words?

With our partnership, 1,000%. So so, what where that comes into is great. Now, partnership with the CFO organization, you've gotta have some sort of ROI if you're talking about some new investment, and if you believe there's value, we're gonna listen, and I've actually devoted resources specific to some of these AI, uh, initiatives, so that we don't have those backlog with the rest of what's happening in the business, right, in order to move fast. So, that doesn't have to do with security, that's just reorganizing ourselves so that we can be much more nimble to get to these different gates, which is just to understand, Hey, is there an ROI?

Is there value? Is this thing real, cause there's a ton of vendors and solutions that are out there. So, that funnel within the business teams will naturally filter out before it even has to get to Ben or some of our teams. Stage 2 of this is, I love to use the word test and learn. It doesn't mean full-blown production, it doesn't mean production data, it doesn't mean integrating it with the rest of our ecosystem, which has higher risk, and and and, you know, when you look at the checking the boxes on that framework.

So, we're like, Hey, if it's a sandbox, let's create a a workspace, maybe that's not connected to something to test that out. Let's make sure we have proper t criteria. So, not to kinda drag on, but what we've done is created different buckets based on the type of proof of concept or technology that you want, and once you get past that, that gives security time, legal time, privacy time to really vet these things based on those frameworks around risk that Ben and team have.

Ben, does the shared language that emerges from a framework like Aware make it easier to work with your counterpart?

For sure, right? So, again, it sets And and that criteria.

to ease that tension?

It well, I don't I don't know. I'm not sure I'll go we'll go that first. Right, uh,

I'll go first. far.

when when we're both under pressure or, uh, you know, to to deliver and to to help the company turn transform and adopt and and sort of reimagine itself, uh, leveraging AI, that that that's a lot of pressure, I think, for anybody. Um, but, yeah, it it again gives us a that that sh- I think I like the the how you phrase that in the question. Gives us that shared language where, you know, a user comes to us and says, Yes, we've gotta adopt this tool now.

And then I can say, Well, look, yeah, so, on-device inference, that's awesome. Uh, maybe, um, from the standpoint of it's going to maybe slightly reduce the risk of data exfiltration, um, but there it opens up other cans of worms, and I can actually refer to the Aware framework and say, Well, you know, w- problem is is that I'm gonna lose potentially some observability, the ability to monitor, the ability to actually sort of reconstruct what might be taking place with an agent's reasoning, uh, for example, if all that stuff is going on, like, on an an end user's laptop, we've gotta figure out, uh, also, not only if we can monitor that and, uh, say, if there is an action that

it that agent might be planning to take and we don't want it to take and we wanna intervene before it actually executes anything, um, you know, we need that observability to be able to enable that that kind of thing. So, again, it it helps me to just have a sort of a logical way to step through and, um, you know, along with, uh, Pradeep and other technology partners in particular, be able to talk through these issues, um, and I I like sort of the independence of it, cause it's not just Ben thinks, it's there are a bunch of other smart people out in the industry, uh, have given th- this some thought, and, you know, that, I think, as a

security practitioner, if there is any in the room or or, uh, you know, online here, um, sometimes you have to sort of, say, Well, look, this is the way that other companies do it, um, and this is the way the industry does it. That that does have, uh, you know, it it is a little bit of a credibility, uh, boost.

I'm half joking, but if Ben's bot would skip that answer, it would actually be like, Oh, okay, the bot AI bot told me it was too risky versus versus Ben, I I tease, Well, I'm but there's

on I'm on camera, so, you the know, I gotta

the there there is, uh, little bit of, you know I I call it half joking, but people do look to the responses that AI will generate, and, actually and Ben's team, among other teams, have actually created AI bots to kinda do some of that risk assessment, which I think is one of the components of that framework as well.

We've just spoken about how you overcome the fights. What are some of the real challenges when you're working together to make a go no-go decision about a model, a tool, what have you?

when we're evaluating a go no-go decision and and to greenlight something, um, is a tendency to look at, um, only part of the problem, um, and you wanna take a as holic- holistic an approach as you can, right? So, I mentioned, like, some people, they they only know model security, they don't know the agentic, uh, security, uh, and and the implications of that. Um, the other thing about go no-go decisions is that I think especially given how fast the, uh, AI space is moving, um, how the the volume at which we and other companies are going to be adopting and building agents and integrating them together into, you know, massive workflows, a and all that stuff.

The the technology is gonna evolve, the use cases are gonna evolve, the agents are gonna change, there're gonna be new use cases for them, there're gonna be, you know, uh uh, you know, new delegation chains that are gonna pop up. So, the go no-go decision is not a one and done, is my point.

It's iterative.

It's iterative, and it has to be streamlined. Um, and, you know, so he and others that are the builders and that are really focused on delivering, they need to understand that they need to y- they need to come back to this risk process, and this is where he's like, Oh, man. But that's That was but my job good. That was something is to

to say.

make make sure that I can I can do that risk process at speed and at scale. So we have

So, you so you have to And with clear to. So, you criteria. heard what he said, I mean, that you have to go back actually, and

we've got a better path, which is we're gonna basically have a catalog that he can

there you go.

query. Right? So, this is how do you simplify to Ben's point, Ben and Legal and Privacy have a need to know, Hey, what is going on in terms of your organization, your ecosystem with regards to AI? So, there is a documentation step that is an important piece if you ha- especially if you're regulated and things like that, that's going to become more of a requirement, especially if it's customer-facing, what type of classification it is. So, a lot of those elements of that aware framework are actually going in.

So, one of the things I'm pushing, or actually requiring, is not at a software level. We already have a software catalog if it's AI or not or whatnot, but a task level, activity level catalog of what is an agent doing, which the business or tech or whoever can essentially document, which then enables privacy, legal, security to query or have agent go against it, and then, based on risk, do what they need to. And if it changes

Ben, what's your advice for CIOs, CISOs to put a framework like Aware into practice?

Consider the the bigger frameworks, the the more monolithic organizational frameworks that I mentioned, um, and synthesize a a a framework like Aware with them, um, inventory your agents, um, instrument them for observability and identity. If if if you can only do, uh and focus on any part of the aware framework, it would be focus on those 2 things first, and then go from there.

This is great. Uh, Pradeep, how do we put this into practice? What should we do?

people need to partner together, but you also have to simplify the messaging. Like, the business folks wanna know, Yes, no, can I do it? Can I not? Why not? How quickly? So, you've gotta Yeah, we business have

people can't understand a whole lot more than that.

and to some extent, nor should they. Right, so so, that's the middle ground I think that's important for CIOs and CISOs, and as we work together, how do we help translate, simplify that message, give a framework that says, here's what you can do, here's what we need to work together to enable.

How is this agent governance landscape going to evolve over the next year or two?

So, from a compliance ban uh, perspective, I can see us seeing a control criteria in SOC 2 within, like, say, the next year and a half, 2 years, uh, we need to be ready for that.

I would love to see a uh uh vendors playing in this governance space, so, I don't have to work with Ben and team to configure that every single time we have different frameworks. I want an overlay, kinda like you do with security that can work across different enterprise grade players, so, enterprise grade is gonna be important.

Great. Do we have time for questions at all? Or When you started, you said how many agents? Twenty, 600?

Yeah, we have over 6,000.

6,000, okay. So, when you scaled them starting from 10, 200, what were your scaling challenges as you scaled to 6,000?

A lot of these now, we have different platforms where agents exist, which, you know, Ben's gonna make sure they're all, uh, good to run, but the a lot of these agents, uh, the sprawl, so to speak, happened because of the Glean platform, and it enabled them. Now, the Glean platform is one of the the all the different data sources and things that we have, because of the fine-grained security controls, we trust it, and the users are trained to run.

Now, one of the things should we talked a little bit about this is when we s when that started, we knew we were gonna have more agents, now, even though we have 6,000, how many are actively used? It's probably about 1, 300, candidly speaking, but what we wanted to do was we need to change that culture. We wanted to get people's behavior to understand the art of the possible. So, we encourage people to go ahead and create agents, figure out how your life could be different.

So, they went out and started creating agents, started tweaking agents, learning from them. It was purposely done. Now, we knew that this was going to be a bit of a mess, but the purpose was different. The purpose was get people interacting. Now, meanwhile, we also worked with Glean, said, Hey, we need a way to moderate this stuff, cause it's going to be a hot mess. And I call it within a 3, 4 month time, the capability came in.

We can also see metrics and what's being used, what's not. Now the moderation is in play, right? So, part of this is not to say, How do we stay one step ahead? But it's what's the goal at this point in time? It's a journey. Part of that journey was get people to create them and understand what's okay and get them excited. Now, which we may be a long-winded answer here, but, uh, what we're getting now asked for is people wanna create workflows.

They're like, Hey, great, I get the output. Now I want it to take action, and they're very hungry to go to that next stage.

So, you have agents, and then they have kids, and they move out to the suburbs, and then, before you know it, you have agentic sprawl.

Yep, great analogy. You know, the space is so dynamic right now. Again, you have to have a a good development process, a structured and scalable, uh, you know, tiered, um, uh, risk process as well, because I think if you're applying, you know, governance to, uh, the same level of governance and control to every agent or every AI system out there, your your governance, uh, capabilities are just they're gonna crumble, right? They're not gonna scale.

Okay, and with that, Ben Mayrides and Pradeep Mannakkara. Thank you, Bill.