16 May 2026 · 42m · Anthony Scriffignano, Tim Crawford, Michael Krigsman · 21 quoted facts
Enterprise AI agents are spreading faster than CIOs can govern them. Scriffignano and Crawford argue that employees vibe-coding tools and handing agents their credentials widens the threat far beyond the company perimeter. Old disciplines such as governance, provenance and permissible use still apply, but they must be extended to non-human actors and to systems that change constantly.
The central tension is between control and speed. Both guests say shadow AI cannot be stopped, so leaders should build sandboxes, monitoring and agent-run red, blue and green teams. Yet machine speed makes human review hard to sustain. Their answer is to choose deliberately where humans sit in the loop, to start small, and to train people while keeping business depth at the centre of the CIO role.
Testing relies on changing one thing at a time, but foundation models, training data and environments now shift together. Scriffignano suggests rerunning scripted tests whenever providers announce a model change, then adding non-deterministic tests for misuse and misadventure.
Scriffignano compares stopping it to stopping oxygen. He proposes sandboxes with trusted tools and data, token counting, no-PII guardrails, and agents that watch for valuable data leaving without value returning.
Crawford says agents differ in scale, speed and complexity, and super-agents can call sub-agents through an orchestration layer. Each step needs its own governance, and agents also fail through imperfection, not only bad intent.
Scriffignano says telling a red team to 'see if you can break it' creates a disruptive food fight. Naming about five specific worries, such as data exfiltration, lets defenders know what they are guarding against and makes the contest fair.
Crawford describes user role-based access, an agent governance layer, and a knowledge governance layer. Each is complicated further because data sits in separate systems like ServiceNow, Salesforce and SAP, each with its own rules.
Shadow AI is employees using AI tools, or vibe-coding their own applications, without IT oversight. Crawford says agents can hold employee credentials outside the company's visibility, and may reach more than the user realizes.
Scriffignano says no, because it is not stoppable and trying is a whack-a-mole game. He recommends sandboxes, training, and monitoring controls such as token counting and PII guardrails.
It is when someone who is not a developer, such as a line-of-business user, creates working code by prompting an AI. Crawford notes that this code can then be deployed or used inside the enterprise.
Crawford explains that the red team simulates attacks, the blue team builds defenses against them, and the green team works out how to fix the vulnerabilities found. He says these will increasingly be groups of agents running almost continuously.
Scriffignano says it is often required for critical uses but cannot keep pace with machine speed, and sampling audits are not sustainable. His advice is to place humans deliberately and keep re-evaluating as the environment changes.
Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.
Tim Crawford says users can give AI agents their own credentials so the agents act on their behalf, possibly outside the enterprise's visibility.
“Such that now we can give agents, and this kind of gets to Anthony's point about agency, we can give agents our credentials and have these agents do work on our behalf.”
Tim Crawford says many enterprise AI projects are kept small in scope to limit the blast radius of potential harm.
“Because that's the way that they protect themselves against exposing the blast radius and and potentially opening themselves up to, um, undue risk.”
Anthony Scriffignano says regression testing, which proves previously working functions still work after a change, was a core software tenet about ten years ago.
“One of the tenets of, let's say, ten years ago, uh, the little thing called regression testing.”
Anthony Scriffignano proposes using AI agents inside the enterprise environment to watch for risks such as personal data leaving the organization.
“Nope, I'm gonna have controls in that environment, including AI agents, by the way, that watch what's going on”
Anthony Scriffignano says organizations can count AI tokens to see how much data is going in and out and identify heavy users.
“The good news is we can count tokens, we can look at things to see who the sort of mega users are.”
Anthony Scriffignano gives the example that counting rows in a CRM list overcounts customers because a customer can have more than one row.
“they're gonna grossly overcount because there's more than 1 row per customer, right?”
Tim Crawford cites Moderna as the poster child of the debate over applying HR policies to AI agents.
“Moderna's the poster child of this, you know, should we be taking HR policies and applying it to agents”
Tim Crawford says super-agents can call sub-agents through an orchestration layer, requiring governance at each step.
“Now you have this concept of super agents that can go through an orchestration layer and call sub agents”
Tim Crawford names IBM's Granite, Amazon Web Services' Nova and Google's Gemini as models whose makers are building in safety measures.
“whether it's granite from IBM, Nova from AWS, Gemini from, uh, Google, for example, they are putting safety measures in the mix.”
Anthony Scriffignano says major model providers announce when a model is changing but not what has changed, which is a cue to rerun tests.
“the major models at least do tell you when they're changing. They don't tell you what they're changing, but they tell you that you're they're changing.”
Anthony Scriffignano uses 'misadventure' for cases where an AI system speaks for an organization in unintended ways, which is a dimension to test alongside misuse.
“Another one is misadventure. You can't say hallucination anymore because everybody just gets triggered.”
Tim Crawford defines a red team as the attackers who simulate attacks, with blue teams building defenses and green teams working out fixes.
“a red team is the attackers, right? We're simulating those attacks and trying to understand kind of what's happening.”
Tim Crawford poses a thought experiment in which forced automatic patching would remove the ability to test patches before deployment.
“What happens if you have to take patches automatically? You no longer can do testing before you deploy it”
Anthony Scriffignano says a recent major global outage was caused by releasing an empty file, which he uses to show how simple unforeseen failures can be.
“One of the most recent major oopses that hit a big part of the world happened by releasing an empty file”
Tim Crawford says sophisticated agents are breaking APIs and websites that were not designed for their scale and speed.
“They're breaking APIs, they're breaking websites because they just weren't, they were not never designed for that kind of scale and that kind of speed.”
Anthony Scriffignano says AI increasingly consumes content produced by other AI, which multiplies errors and makes mistakes more chaotic as well as faster.
“a lot of the content that AI is consuming was produced by other AI. So there's a multiplicative effect of this problem.”
Anthony Scriffignano says regulation generally requires human oversight more strictly as the use of an AI system becomes more critical.
“the more critical the use, the more mandatory it is that you have a human in the loop.”
Tim Crawford says business processes span systems such as ServiceNow, Salesforce and SAP, each with its own governance layers that must be reconciled for agents.
“we might have some data in ServiceNow, we might have some data in Salesforce, we might have some data in SAP, and so each of those systems have their own governance layers.”
Tim Crawford defines value as opportunity minus cost when discussing how to measure the true value of AI.
“Value is an equation that is value equals opportunity minus cost, right?”
Tim Crawford says FinOps is extending its FOCUS framework to account for the true cost of AI, and that the TBM Council is also working on understanding IT value.
“Uh, FinOps is is working on versions of focus and their framework to be able to accommodate understanding the true AI cost.”
The CXOTalk host announced that the next episode, two weeks later, would feature the chief technology officer of Mozilla.
“Two weeks from today, we're talking with the chief technology officer of Mozilla.”
The guests frame how vibe coding and agent tools widen the threat surface and make old testing assumptions fail.
“We're changing thousands of things at once right now.”
Scriffignano argues governance, provenance and permissible use remain valuable, while agency and non-human authentication are new.
“We don't have to throw away the experience that caused us to all lose hair”
Sandboxes, training, token counting and PII controls are offered as practical ways to walk alongside uncontrollable adoption.
“The good news is we can count tokens, we can look at things”
Crawford explains that scale, speed and orchestration across sub-agents require governance at every step.
“you have this concept of super agents that can go through an orchestration layer”
Scripted tests still help, but generative systems need non-deterministic testing for misuse and misadventure.
“Note that I didn't say scripts, because this will not be linear.”
The guests discuss disciplined red teaming and automated agent teams, including forced automation of patching.
“It has to be a group of agents that are automatically working on your behalf”
Humans cannot match machine speed, so leaders should choose their position deliberately and revisit it.
“Be where you are on purpose and constantly re-evaluate that”
Crawford lays out layered access governance across systems, and the guests discuss context engineers and AI cost frameworks.
“the agent governance layer sits above that in terms of what the agent itself has access to”
One of the fundamental tenets of testing is don't change 2 things at once. Well, we're changing thousands of things at once right now.
AI agents are running wild inside the enterprise, and CIOs are on the hook. Anthony Scriffignano is a phD data scientist, and Tim Crawford advises CIOs at the world's largest companies.
The way that we used to think about application development, the way we used to think about testing, the way we used to think about data stewardship and ingestion, the way we used to think about governance. Every single one of those things is changing. And then there's new domains that are coming into the sphere, like context engineering and and understanding black boxes that are being called through open AIs that are being used by people in the enterprise that you don't even know are using them.
So there's a whole new set of challenges for the CIO that we don't even have words for yet.
If you go back and look at, for example, like social media, social media was really great because it gave everybody a voice. It lowered the playing field. Everybody had had a megaphone, had a a platform in which they could share their perspective. The downside is it gave everybody a voice, and so we're starting to see the same thing play out with AI. And especially when you look at not just vibe coding, where someone can vibe code an application, that is not a they're not a developer, they're a they're a line of business user.
Um, but they're able to actually create code that gets deployed within the enterprise or used within the enterprise. All the way through to tooling that, uh, when you look at things like open claude and nemo, claude and Claude co-work, and now Amazon quick and even, uh, SAP's jewel. Work. Uh, that they just announced this week. All of those products are starting to enable every user access to AI with varying degrees of guardrails and protections around it.
And so for the CIO, this opens up a vastly wider, uh, th- threat paradigm that you have to get your arms around. Not just in terms of what's appropriate and what's not, but also in terms of the risk profile. And how do you start to? How do you start to ensure that you're still maintaining the right protections for your organization?
One of the tenets of, let's say, ten years ago, uh, the little thing called regression testing. You just did something new, and you've proven that the new thing that you built works, and you've proven that it integrates into the environment. You've done all the things you need to do. One of the things you have to do is also prove that things that used to work still work. That's a really difficult thing to do when all of these foundation models are changing, when the training data is changing, when the way in which the environment that you're relying upon is changing at the same time that you're making changes to how you use it.
There's another dynamic in addition to what Anthony mentioned, which is just thinking about identity and credentials, so authorization as well as identity. When you start thinking about agent identity and who's responsible for that, and which agents have access to which credentials.
We're talking about this shift from essentially automation managing systems to autonomy. Let's talk about risk. How should CIOs reframe AI risk? And also think about the value of AI. How do you balance that? Uh, Tim, you wanna jump in?
We're still figuring this out. Um, you know, this isn't something that we can say, okay, it's just like, um, it's just like we did with cloud, or it's just like we did with the internet, or just like we did with just distributed computing. There's no similar framework in our past that we can use to apply to AI. It's a whole it's a whole new space. And so one of the challenges for CIOs is kind of working through this process, both internally in terms of governance models and how we think about governance, but also with their partners, both their software partners and hardware and services partners, as well as their, uh, consulting partners.
So it's it's a really complicated, uh, question that has to be answered. Now. This is one of the reasons why quite often you see some of these AI projects being much smaller in scope, um, with the exception of some very sizable outliers, but it's smaller in scope. Because that's the way that they protect themselves against exposing the blast radius and and potentially opening themselves up to, um, undue risk.
There are some sort of learned, uh, um, tenets that we have as leaders in this space, and they haven't gone away. So governance is still there, the need to have a consistent experience unless you intentionally are changing that experience. Uh, provenance, knowing where you got your answer from, permissible use, knowing that you're allowed to use it for the purpose that you're using it. The difference between a risk, which is something that might happen, and an issue, which is something that is that is happening.
These are still there. So there might be a tendency to say, oh my gosh, and I'm not accusing Tim of saying this. Oh my gosh, LLMs are in the room now, and everything is blown up, and we need a whole new set of capabilities. I I think it's it's those things and some new things. We don't have to throw away the experience that caused us to all lose hair and get gray hair. We need to build on that a new set of capabilities around being able to understand agency, being able to understand authentication and validation in the context of a non-human.
On the other end of that, uh, the ability to understand the impact of shadow AI and what people might be doing. And we shouldn't necessarily try to stop them because it's like trying to stop oxygen. So there's some new capabilities that we need to name, but those old ones are still there, and they will serve us well if we pay attention to them.
What about this issue of shadow AI and the fact that AI is proliferating? AI, AI, the use of AI, AI coding, uh, every aspect. The proliferation is very rapid and essentially uncontrollable.
You're right, the way that AI is being used has expanded greatly. You know, everything from ChatGPT all the way through to things like Claude, uh, co-work. And so it's not just vibe coding code, which has its own set of challenges, but it's also how users are using this technology and the ease of use to use this technology. Such that now we can give agents, and this kind of gets to Anthony's point about agency, we can give agents our credentials and have these agents do work on our behalf.
The challenge with that is that can happen outside of the enterprise walls, meaning the enterprise may not have visibility to it. And so that's part of the concern is that what do these agents have access to? How are they working? And there's a whole downstream piece of this, which is agents have access to more than what the user thinks they have access to. Because of the way that we structure our back systems and protections within the enterprise,
As I'm listening to you talk, I'm thinking about earlier in my career, when there was a big push to outsource, outsource development, outsource, oh, outsource everything. If it's not nailed down, outsource it. Um, we had some of these same issues of how do you know that they're doing what we asked them to do? How do you know that? Uh, we're not giving them access to more than they need, or or, or, uh, more than we realized?
Um, some of these are not new problems. Obviously, they're happening at a much larger scale now. There are some best practices that are emerging for shadow AI. Um, one of I don't necessarily love them, but I get them and I'll I'll probably come to love them. You know, create a sandbox, put the tools there, put trusted tools there, put trusted data there. Um, teach people how to use the AI that that they're trying to vibe code their way through life with.
So that they get better at at prompt engineering, they get better at testing. Teach them some of the the five biggest things that you can do to make a mistake. So instead of trying to stop this, because I don't think it's stoppable, nor do I think we should, maybe we embrace it and find better ways to control. Is the wrong word, but to walk alongside it and and understand the change that's coming along, rather than constantly reacting to it.
It's a whack-a-mole game right now, and and that is not a sustainable strategy.
Now, let's just think about Joe or Mary in the finance department, or Joe or Mary in marketing, who are coding their own tools. And the lawn is growing and the mushrooms are coming up in the lawn, and they're using corporate data, which is fantastic. We have so as a company, we have so much data we should share it with everybody.
Let me give you an example. Um, a a person whom I greatly respect, um, who has been a guest on your show, Steve Daphron, once said the most fundamental thing in data science is counting things. And I just love the simplicity of that statement, right? So if we start with counting things, if you just let people have at it and give them the customer, the CRM list, for example, and they wanna count, how many customers do we have?
And they just count the rows, they're gonna grossly overcount because there's more than 1 row per customer, right? So you start by teaching them about relationships. Now, at the end of the day, th- there's gonna be a, um, a reality here that they don't wanna be taught. They don't have time. That's why they're vibe coding in the first place. They don't wanna hear you. It's a lot of want want. And so you have to put some controls in place.
And I certainly, one of the biggest controls I would have in place is how much data are they pulling in, how much data is going out, where is that data going? The good news is we can count tokens, we can look at things to see who the sort of mega users are. We can put guide rails in place, like no PII. You know, I'm not just saying take whatever data you want from the enterprise and do whatever you want with it.
Now I understand. So you're going to issue a mandate and everybody's gonna do what you want.
Nope, I'm gonna have controls in that environment, including AI agents, by the way, that watch what's going on and understand, and I'll put understands in quotes cause that's anthropomorphizing, but contextualize the biggest risks that we've identified. So let's let's just take one PII, right? We wanna make sure personally identifying information is not somehow leaking out of the enterprise. I can easily put a control in place to watch for that. I can put a control in place to watch for, uh, data that has value leaving the organization without money that has value coming back into the organization.
There are controls that we can put in place, but we have to give people a a way. We have to have a way to watch what they're doing and not just hope that they're following the guidelines and the edicts, cause we know they won't.
Let's take some questions, cause we have a bunch of questions that have come in. And the first one I'm gonna go over to Twitter X, and this is from Arslan Khan, who says, If we treat AI agents as quote, almost human, why can't we use existing governance models, apply HR models, for example, to AI agents? Tim, thoughts on this one?
they are not like humans. Um, and I think this is one of the one of the challenges. I mean, there's been this conversation for some period of time. Um, you know, and Moderna's the poster child of this, you know, should we be taking HR policies and applying it to agents because they're just digital agents versus human agents? Um, the answer is no, because it's a different, it's a whole different ballgame. So, for example, you have everything from just scale and speed, as well as complexity that come into play when you get into autonomy.
And you start talking about complex agents and complex workflows. Now you have this concept of super agents that can go through an orchestration layer and call sub agents, and you have to have governance layers at each of these, um, steps, as well as understanding. And it's not always for nefarious purposes. That's the other piece that I think we often forget is that it's not just because someone said, you know what? I'm here to screw the corporation and I'm gonna do something, um, nefarious, or or just completely go off the reservation.
I'm I've been wronged, and this is the way I'm gonna right it. But it also could be because agents and AI is imperfect. I mean, let's not forget that we are still learning how AI works. So kind of to one of Anthony's points, I mean, we. We have to be able to have explainability, especially for audit routines, in terms of how AI works, and so we don't necessarily have that in these algorithms. Today, it's more than just putting policies in place and the guardrails, but we also have to think about how we're setting up these workflows, what the data, what data it has access to, but what it could and couldn't do as well.
And that's that's a far broader reach than any one individual.
Let me add to that couple things in my practice that are very interesting in this space. One is to have adversarial agents that are trying to break things, and they can do that at scale, and that can be very interesting. So, the red team concept, another thing is ethical checks. So, uh, we can look at espe- This is very helpful when you're generating text, when you're generating words, you can look at mean shift and sentiment, you can look at multi-modality.
Are the leaders leading the group, or is the group leading the leaders? Those are very valuable things to look at. Uh, provenance, making sure that there's a way to go back to a fact for any kind of new nouns that you start talking about. Um, veracity, looking at misinformation and disinformation. When people repeat things that seem true, they're not necessarily lying. If those things are lies, those lies seem believable, right? So getting at the source of those lies, uh, there are lots of very interesting things that we can do in this space that don't really have names yet, but they have agents, and we can get those agents to do some very clever things if we teach them.
This is from Chris Peterson, totally different topic here now. And he and Chris says, Historically, we could read the release notes and hope to know important changes in software in a new software version. Is there any useful analog in the LLM GPT world? And if not, how do we? And he says, This is to Anthony's point. How do we regression test for safety? It's a really important point.
the large LLM providers, and it's not just the Anthropics and Open AIs. But when you look at what IBM's doing, when you look at what Amazon's doing, when you what Google's doing in terms of building their own, um, their own models. So whether it's granite from IBM, Nova from AWS, Gemini from, uh, Google, for example, they are putting safety measures in the mix. Is that enough? No, absolutely not, because we've already seen examples of where these models have kind of gone off the rails and done some pretty bad things.
And so we have to be real cautious about this. Um, and the same goes with the agents too, because we have to understand how these agents work, and that's something that is a new skill that we are just learning about today.
The old approach to regression testing, you build a bunch of scripts, you say, you know, all of these things should still work. When we're done, here's the data that you use with these scripts. When you're done doing whatever it is you were doing, run these scripts with this data and make sure it produces these answers. You can still do that. Uh, and to Tim's point, the the major models at least do tell you when they're changing.
They don't tell you what they're changing, but they tell you that you're they're changing. So that's a good time to start doing this. Um, there are some other dimensions in an LLM and an agentic world, uh, I'll say, in a gen AI agentic world, there are some other dimensions that you need to think about. Um, one of them is misuse. Are people able to either through prompt injection or stupidity, or, um, you know, kicking the tires, able to get your environment to do something it was never intended to do, and have you just made it easier for them to do that?
Another one is misadventure. You can't say hallucination anymore because everybody just gets triggered. But, um, it is absolutely possible that you may have enabled your system to speak for your organization in ways that you never intended. Um, there are other dimensions like that that we can test for. We have to name them, we have to build modalities for testing. Note that I didn't say scripts, because this will not be linear. This is very much a non-deterministic kind of thing we're trying to do.
But if you say that the first thing we're trying to do is test against misuse, all right, that's a good start. At least now we know what mindset we're in when we're building out regression approaches to misuse. So it's absolutely not a linear process anymore. You can still use that linear process, but you need to build onto it a sort of a non-deterministic approach to some of these other ways of messing up with gen AI.
Tim, this is, uh, a a CIO career question, and this is from Preeti Narayan, who says For aspiring CIOs, what capabilities and mindset shifts will matter most in an AI agentic enterprise world? And how difficult will it be for today's CIOs to transition to transition from traditional IT operating models to AI-native ways of working?
I often talk about this differentiation between the traditional and the transformational CIO, right? A CIO is not a CIO, is not a CIO. If you're an aspiring CIO, you need to focus on the behaviors of the transformational or business-oriented CIO. In an agentic world, you have to understand your business. And I don't mean just the the superficial layperson's explanation of we're a clothing retailer and we sell shirts and pants and socks and that sort of thing.
But you need to understand the underpinnings of how the customer engages with your business. You have to understand the entire workflow of how your business operates end to end. You have to understand some of these dynamics and nuances within how your company operates, not just the cultural and institutional knowledge, but also the industry and where it's going. And so part of this, uh, answer is that you need to be a business leader first that happens to have responsibility for technology.
When we move into this agentic era, it has to understand the business. I mean, if you don't understand your business, it's gonna be really hard to truly lean in on these agentic workflows. More than just the simple, hey, we've taken a process and we've automated that process. So, for the aspiring CIO, first and foremost, understand your business deeply. Second is reimagine your business. Think about how you would do it differently and where technology can play a role.
And I think this is where some of the agentic workflow flows, especially the more complex ones, become really interesting and, surprise, surprise, more valuable to your organization.
Anthony, here's a question from Hui Wang, who says, How important do we see the need f- of approaching a demand to have a very clear and comprehensive blueprint in our solution designs in order to prepare teams to be operational and red team ready?
Talk about hand grenades, hand grenades, that's a that's a fascinating question, and you could probably write a book on the answer. But, um, the the essence of it is, how do you get people who are in this mindset to be ready for this red team concept while everything's changing at the same time? And and that is a very, very big question. So the first thing I would say is going back to the comments that Tim just made about transformational leadership.
One of the essences of of transformational leadership is not to just be changing things for the sake of changing, to be going from some place to some place with overt intentionality and understanding why you're doing it. So if you're bringing red teams into the room, you don't wanna just bring red teams in the room and say, all right, see if you can break it. That's a ridiculous mandate. What you wanna say is we're worried about 5 things.
We're worried about data exfiltration, we're worried about malware injection, we're worried about changes in the, yeah, whatever those 5 things are, right? There should be 5. And then the red team is focused on those 5 things, the non-red team, the good guys, know what they're sort of defending against. And it's a bit of a fair fight because they're not just coming in and trying to break stuff while you're trying to build stuff and fix stuff. There's gotta be some discipline to this.
It can't just be a a food fight. What happens when you let it just be a food fight? It's fun, right? People enjoy it, and especially the red team enjoys it, cause they don't have any rules. Uh, but it's just so disruptive. And I think we've all worked in large corporate environments where you're just trying to get your job done. And emails are coming in that are fake and they're trying to trick you, and there's pen tests going on, and I, eh, and at the you get to a point where you don't trust any email and then you don't respond to a customer because it looked like a pen test.
W- We gotta be a lot more thoughtful about this, and I think it starts with Que's question about how do we get them ready to do this? We've gotta teach them what they're doing and not just charge them with breaking stuff.
I want to build on this a little bit because there's an additive piece, especially when you start talking about AI, that I think, um, some people are kind of missing along this pathway, which is the automation that's coming soon is not gonna be optional. It will be a requirement, whether you want it to be or not. And what I mean by that is that the s the sheer speed in which we need to operate is increasing so quickly that we just can't put a human in the loop on some things.
And so when you think about red teaming, and if you're not familiar with the concepts of blue teaming and green teaming, definitely get your arms around that. But, uh, for those that aren't familiar, you know, a red team is the attackers, right? We're simulating those attacks and trying to understand kind of what's happening. The blue team are the folks that are looking to d put up the defenses around those attacks. And then the green team says, Okay, we found these attacks, these vulnerabilities.
How do we fix them? And and how do we start to, um, ascertain? Well, we have 2 different ways we could go about this. What's the right way to do it? That's no longer a group of humans doing it, nor is it a group of tools doing it. It has to be a group of agents that are automatically working on your behalf within your constructs and are doing this almost continuously. That's the speed in which we have to get to.
And so here's the thing that I would pose to your audience, Michael, is that if you had to automate any said process, whether it's in cybersecurity, whether it's in, um, a business process. If you had to automate that fully, no caveats, no ifs, ands or buts, you had to automate that. How would that change your thinking around all of the other aspects, everything from how you develop the application, how you use the application, how you update the application, like, for example, patching?
We're having this conversation with a group of CIOs just this morning. What happens if you have to take patches automatically? You no longer can do testing before you deploy it, because that's that might be a reality for us in the very near term. This is something that that I think people have to start getting their head around is if these automation routines become the norm, meaning we have no choice but to have to do it. How does that change our thinking around other aspects and other decisions and choices we've made within the enterprise?
And that's what I would challenge the audience to think about.
What could possibly go wrong with automation that you over which you have no choice?
We've all been there, right? We've all run organizations. I get it, but we may not have a choice. We may have to try and figure out how to. So, for example, thought experiment here, just pick one piece off, pick off the patching, for example. If you had to automate all of your patches because everything is a zero day, how does that change your thinking within the organization and how you operate?
I would suggest that there's some of that going on without your knowledge anyway. If you're using anything in the cloud or any API or any external black box, stuff is getting patched, stuff is changing, it's part of your stack, you're relying upon it. You, you may not even know it changed. And there's that other dark side of, um, I don't wanna name organizations, but there's some very big ones in the news recently where they didn't do a patch for whatever reason, and it created some huge vulnerability, and now you're explaining why you didn't do the patch.
So I, you know, the the answer isn't to patch or not to patch. Your your point about the, um, the red team and the and the at least the red team and the blue team being automated clouds of agents, I think it's both, right? So there's also orchestration from, you know, Lex Luthors in those cohorts that are people that that know how to think new thoughts. Otherwise, we only learn how to protect ourselves against pre-existing vulnerabilities that we knew about.
And some of these new things that come up, I I am amazed at the incredible creativity of bored 14-year-olds. Uh, some of the stuff they come up with is just genius in its simplicity. One of the most recent major oopses that hit a big part of the world happened by releasing an empty file and and it caused all kinds of problems, cause nobody ever thought of that. So, um, yeah, we need people and machines doing this.
Let's go to another question, and this is from Swami Vaidyanathan. And he says, earlier, uh, we spoke, one of you mentioned, uh, social media, and he says it was a great analogy with social media, where the floodgates opened, sort of. But if we don't have a close parallel from history that can be used that can be used as a model, do you have a specific path or thought process you can share for CIOs and teams to approach this?
It was my comment that, um, social media gave everyone a voice, and that's the upside, and then the downside is social media gave everyone a voice. Um, we have the we have something similar with AI. I think the construct here is, unlike social media, number 1, there's more at risk, um, and the speed in which it's at risk. Because with social media, it took time to build an audience and and have people follow you. And then there's a human in the loop.
Now we're talking about agents that can run off and and potentially do tasks and do a lot of damage before anyone is even to the wiser. Um, it's actually one of the challenges of agents right now is some of these more sophisticated agents are moving at such a speed. They're breaking APIs, they're breaking websites because they just weren't, they were not never designed for that kind of scale and that kind of speed. Um, there are constructs you can use fundamentally to understand kind of how agents work, how agentic workflows work, and then step into it.
You know, I often say that one of the things you have to think about when it when you're, um, when you're thinking about your organization and how you accelerate your organization in an agentic era, is how do you start with one thing and then move to a second? But with each iteration of this flywheel, you're accelerating the organization. The same thing is true with agentic workflows, is that you need to start small, get your sea legs, but then build upon that and accelerate your knowledge and accelerate your understanding of it.
Um, and there is a train the the reality is there is a training aspect of this. We need to educate people on the importance of both the opportunity and the risk so that they can make good business decisions for their organization as well as their role. And that's something that we haven't necessarily stepped into as much today. It's not as much of putting the policies in place and bringing the hammer down, but rather, you know, we've got some intelligent people within the organization.
Let's lean into that, let's help them make good business decisions on our behalf.
When we think about making mistakes faster with agents, we also have to keep in mind that the reason we call it gen AI is it's generating its own content. And a lot of the content that AI is consuming was produced by other AI. So there's a multiplicative effect of this problem. It's not just making mistakes faster, it's making more chaotic mistakes faster. There's a geometric effect here that we have to keep in mind. We can get out of control at a rate that we cannot recover from at the same rate.
And so it's really important to have these conversations about guardrails and red teams and all of that to to to sort of thwart that kind of multiplicative effect. That those are the things that kill us.
Let's shift gears and talk about one of the great buzzwords of our time, which is human in the loop. So when is human in the loop meaningful and when is it theater?
If you look at the regulatory response to this right now, if you look at the the the application and what the application is doing, the more critical the use, the more mandatory it is that you have a human in the loop. That is the general theme of all of the governance that's out there. There's a little bit of, uh, blinders in this, because I think Tim pointed out earlier, there are things that happen way too fast to have humans in the loop, and that's happening more and more now.
And so requiring a human in the loop might make you feel good, but humans can't make decisions fast enough when the environment is changing at certain rates. And so the response to that is sort of audited humans in the loop, like the humans will look at some percentage of the decisions being made. That's not a sustainable answer, either. Um, the reality is that even humans in the loop, all humans don't have the same reaction when they look at things.
So the gold standard, this is called a heuristic frame. The gold standard is to have similarly instructed, similarly incented people looking at things. Well, you can't get that in the real world because people don't understand things at the same level, they don't have the same background, they don't have the same stake in the game. When they're making a decision, some are trying to get done, some are trying to get done right. So the important thing in this is not to say, well, gee, Michael, it's really complicated, and I wish I could explain it to you, but there is no answer, which is the answer, by the way.
The answer is that wherever you are in this human in the loop, e- be it because of regulatory constraint or because of ethical AI, or because of good design principles. Be where you are on purpose and constantly re-evaluate that in terms of the rate of change in the environment and the elasticity that you have for risk.
Tim, given that, what does AI governance actually look like in practice?
It's complicated, and if you think it's simple, full stop, you probably don't understand all of the aspects that are needed in complex agentic workflows. Because you have you have a level of of governance, for example, and I'll just use a very simple example where Michael has access to certain things, Anthony has access to certain things from an agent, Tim has access to certain things, and that's what that's what it presents. And we t often talk about RBAC as the method to say, based on what Tim has access to, that's what we're gonna present.
Well, then the agent governance layer sits above that in terms of what the agent itself has access to be able to do. Above that is a knowledge governance layer, which is how do we start to amass this knowledge and where does it come in and what agents have access to what from a knowledge perspective? So that's just 3 layers by itself. Oh, and by the way, where does this data that we're gonna tap into reside? Because when we start talking about business processes, we're talking about processes that span multiple departments and therefore multiple systems.
So we might have some data in ServiceNow, we might have some data in Salesforce, we might have some data in SAP, and so each of those systems have their own governance layers. So how do you start to amass and rationalize these different, uh, governance models to be able to present the appropriate data to both the agent, but then also to the end user? And this is where things get really complicated, because it's not just a really complicated math problem, but there's a l- there are a lot more aspects to it, cause it's not just simply bits on the page of data elements.
We have to even look beyond metadata and semantic data and start looking at context around data, which is something of a new concept for us. So how do you start to put all of that together? If that already makes your head explode, you're not alone. And so this is why we're starting to work through how do we think about it? But in the meantime, we have to kind of dumb down the the m dumb down the architecture a bit because we can't solve that problem today.
If you live within a given ecosystem, you have, um, additional opportunities that you don't have when you start working across different systems. But that's also why you have to, you have to kind of wrap your head around, okay? So opportunity risk here. I'd like to be able to do that, but I have to understand how do I start to play these other pieces into the into the equation?
One of the first things that I do in my practice is I I s everybody wants to rush to the tools, you know, w- what, what AI are we gonna use? How are we gonna generate the code? And I usually try to slow it down and start with, what's the problem we're trying to solve? How would we know we solved it? What are the the things that can happen that are sort of the unknown, um, in unintended uses of what we're about to do?
So I have some fundamental questions that I ask, and w- if we can't answer those questions, there's a bit of malpractice in taking that next step. So, first principles really important. There are certain approaches in, I'll just use LLMs as an example. Um, you can use a RAG approach if you if you're trying to do something and you know that these documents represent the truth that you understand right now. Start here instead of starting with the internet.
Um, you can look at veracity, adjudication, you know, how would we know that something is true? How would we know that it doesn't contain, you know, part of the truth to mislead us? So there are definitely mindsets, and there are definitely epistemological models, mod- you know, ways of knowing that you can ground yourself in that help defend against the chaos that Tim is talking about. It's not hopeless, but you do need adults in the room when you're doing this.
You can't just throw tools at it.
So, are there new roles and new organizational structures that CIOs need to be thinking about now or starting to build right now? Okay, so what are
Yeah, I mean, the one we keep talking about is context engineers, right? People that understand the way in which the AI is given context of the problem that it's trying to solve. That's definitely an emerging role. Um, you know, we we're talking a lot about red team. That's a a a role that preexisted, but it's way more important now. Tim, I'm sure you have a few more.
Even beyond red teaming, blue teaming, green teaming, um, from an automation standpoint, and then look at the finance piece of this. Well, you know, we haven't even touched on cost. There's a whole talk track around cost and understanding what the tr- and, more importantly, understanding the true value, right? Let's be f- let's be focused on this. Value is an equation that is value equals opportunity minus cost, right? So it's not just about the cost, it's about what you're able to do for that cost or what you're trying to accomplish.
And so there are new frameworks that are starting to come out. Uh, FinOps is is working on versions of focus and their framework to be able to accommodate understanding the true AI cost. There are some startups that are just about to announce, um, some of the things that they're working on around this. Uh, the TBM Council, from their perspective, kind of that umbrella of all IT, um, understanding value and being able to work down. But this isn't as simple as just tagging resources and saying, okay, well, this resource is this application and this resource is this application.
We're talking about agents that, let's say, the 3 of us all have access to this agent. How do we build those resources to the appropriate person, right? If this is an autonomous agent, and if it's a complex agent, where it's an agent that is learning and it's also discovering other agents, which, oh, by the way, might not be within your organization, might be a third party and it's able to to build ephemeral agents. There's a whole nother layer of complexity that comes into the mix, right?
We've largely been talking about a very simplistic way around agentic AI, but these other pieces are actually in market today. They're just in different forms, and it's only a matter of time that we're gonna see them all weave together.
Anthony? We are just about out of time. Very quickly, you're gonna have the last word. What happens to talent and jobs inside IT as a result of all of this? Very quickly, please.
I don't think AI is gonna take all the jobs away. I think that we're gonna have lots of new, exciting jobs. And I think that it's very important that we don't try to just vibe code our way through, you know, evolving our career, that we learn these first principles, we learn how things are changing, we learn how to take advantage of it. We start working on new problems that we couldn't solve before because we have this abundance of capability.
Don't just look for the easy button, look for the the next thing that you can do to add, you know, much more value to the organization in a responsible way.
Well, that's pretty good advice, and unfortunately, we're out of time. A huge thank you to Tim Crawford and Anthony Scriffignano. Thank you both for being here. I'm very grateful to you
Thank you for having us. Thank
and a huge thank you to everybody who watched, and especially you folks who asked such excellent questions. Before you go, subscribe to the CXO Talk newsletter. Go to cxo talk dot com. Two weeks from today, we're talking with the chief technology officer of Mozilla. So get our newsletter so we can notify you of upcoming shows, and we'll see you again very soon, everybody. Thank you so much. Take care.