# Modernizing Models: Google's Behnaz Kibria on MRM for GenAI

- **Show:** Barefoot Innovation Podcast
- **Published:** 6 Dec 2024
- **Source:** https://barefootinnovation.libsyn.com/modernizing-models-googles-behnaz-kibria-on-mrm-for-genai
- **Transcript:** https://podcasts.opennash.com/shows/barefoot-innovation/modernizing-models-googles-behnaz-kibria-on-mrm-for-genai-80ff52/
- **Provider:** OpenNash Podcasts (machine-generated)

## In brief

Existing financial regulation, built on broad, technology-neutral principles, can still govern generative AI without brand-new laws. Behnaz Kibria of Google argues that the core model risk management (MRM) framework still works. Banks need to adapt how they apply it, because generative AI models are too complex to open up and inspect the way older models were.

The tension is between continuity and change. Kibria says the old framework survives, but proving a model is safe now depends on output testing, grounding and human oversight. Jo Ann Barefoot adds that regulators face a harder problem. They must issue clear guidance on a fast-moving field, and they must also modernize their own technology so they can use AI to see outcomes, not just check inputs and controls.

## Key insights

1. **Existing principles-based rules can govern generative AI** Kibria says the paper concludes, as the first one did, that new laws are not needed. Most financial rules are principles-based and technology neutral, so they can adapt as technology changes.
2. **Generative AI breaks the 'open the box' safety check** Older models could be unpacked step by step to confirm they were sound. Generative AI models are too complex for that, so Kibria says assurance shifts to testing outputs and techniques such as grounding.
3. **Human oversight and data quality become central controls** The paper's four risk areas include governance and data controls, reliable model development, validation with humans in the loop for critical decisions, and clear shared roles with technology providers. Kibria stresses that good data in means good outputs.
4. **More parties now share model risk responsibility** Banks increasingly build models with technology companies, so the supply chain has many actors with different expertise. Kibria says the framework must set out each party's roles and a sense of shared responsibility.
5. **Regulators must modernize their own technology stacks** Barefoot argues agencies need to move to cloud computing and bring all their data into usable form. That would let them judge real consumer and risk outcomes instead of mainly inspecting banks' inputs and controls.

## Questions this episode answers

### Do we need new laws to regulate generative AI in banking?

Kibria says no. Most financial regulations are principles-based and technology neutral, so they can be adapted as technology changes, and the paper's analysis of generative AI supports the same conclusion as the earlier paper.

### What is model risk management, and why does it matter?

It is the set of practices banks and regulators use to make sure the models behind decisions are accurate, well governed and not misused. Kibria says it covers validation, governance and controls, and risk mitigation for issues like bias and bad data.

### Why is generative AI harder to check than older models?

Older models followed traceable if-then logic that experts could inspect. Kibria says generative AI is so complex that this is no longer practical, so firms must rely more on output testing and techniques such as grounding.

### What should a bank do when adopting generative AI?

Kibria describes four areas: strong governance and data controls, reliable model development, stronger validation with humans in the loop for critical decisions, and clear shared roles with technology providers.

### How could AI change the work of regulators?

Barefoot says AI could help regulators spot patterns like money laundering and bias, and measure whether consumer protections actually work. To do that, agencies need modern cloud-based technology and leaders who understand it.

## Facts

- **Behnaz Kibria** [0:08:33]: "we did a study back in July 2023, a benchmarking study, and asked organizations about the impact they felt Gen AI would have."
- **Behnaz Kibria** [0:08:33]: "82 % of the organizations responding that were considering or using Gen AI believed that it would either significantly change or transform their industry."
- **Behnaz Kibria** [0:07:15]: "generative AI is a type of artificial intelligence that can create new content or data."
- **Behnaz Kibria** [0:14:02]: "while that was updated in 2021 to issue more specific guidance on AI"
- **Behnaz Kibria** [0:00:00]: "most are principles based and technology neutral. And what that means is that they are capable of being adapted over time as technology changes."
- **Behnaz Kibria** [0:12:33]: "what we called at that time risk use cases, you know, the use of traditional AI technologies for fraud detection, anti-money laundering, etc."
- **Behnaz Kibria** [0:15:09]: "it's really anything that's got inputs and outputs could potentially be a model."
- **Behnaz Kibria** [0:15:09]: "things like model validation, assessment of a model's accuracy, reliability and limitations."
- **Behnaz Kibria** [0:09:58]: "And we've got a number of customers at Google Cloud who are already using generative AI for those purposes, like Deutsche Bank."
- **Behnaz Kibria** [0:05:16]: "I am the head of a team called Product Initiatives at Google, which is part of the Government Affairs and Public Policy team."
- **Jo Ann Barefoot** [0:32:09]: "Our listeners know I'm a former regulator myself. I was deputy controller at the currency."
- **Jo Ann Barefoot** [0:01:43]: "These models perform analyses that generally can't be done by the people using the outputs."
- **Behnaz Kibria** [0:22:43]: "in the old days when you had traditional models, you could just unpack how the model worked."
- **Behnaz Kibria** [0:23:14]: "one of the core proposals of the paper, for example, is that you really have to look more at things like testing, output testing."
- **Behnaz Kibria** [0:29:36]: "we sort of bucketed into four issues, four areas in which we think there's a variety of best practices that financial institutions should be considering"
- **Behnaz Kibria** [0:30:17]: "the really important aspect of human-in-the-loop systems for critical decision-making to make sure that you do have the necessary extent and level of human oversight"
- **Behnaz Kibria** [0:31:15]: "there's got to be a clear delineation of roles and responsibilities and a sense of sort of shared fate and shared responsibility as well."
- **Behnaz Kibria** [0:39:14]: "we used to be often outside of conversations, you know, in financial services, those conversations often were between the regulators and the regulated entities"
- **Jo Ann Barefoot** [0:35:26]: "but they all have to move to cloud computing in one form or another."
- **Jo Ann Barefoot** [0:37:56]: "a great deal of our regulatory philosophy is based on looking at the quality of the inputs and the quality of the controls"
- **Jo Ann Barefoot** [0:37:56]: "the regulators expect to see a robust compliance management system and the three lines, if it's a bank, you know, the three lines of defense and all of that."
- **Jo Ann Barefoot** [0:03:52]: "our new white paper by my colleague Nick Cook, the Financial Regulators Odyssey, talking about why and importantly how regulators will need to transform the technology that they use themselves."

## Transcript

**[0:00:00] Behnaz Kibria:** And the question that we asked was, you know, with that set of emerging AI use cases as a sort of a basis, did we feel that we needed new laws and regulations? And the answer that we came to in that context was no. And the reason is because one of the great characteristics of the laws and regulations in the financial services industry is that most are principles based and technology neutral. And what that means is that they are capable of being adapted over time as technology changes.

**[0:00:36] Behnaz Kibria:** Every new turn in technology doesn't require that we throw out the tools and start again.

**[0:01:07] Jo Ann Barefoot:** Barefoot Innovation starts right now. The future of finance, and therefore financial regulation, is technology, and especially AI. This means that the regulatory and risk community has a lot of work to do. I'm joined today by Behnaz Kibria, Director of Google Cloud Public Policy, to talk about one area that needs to undergo deep rethinking. Google and AIR have just released a new white paper on model risk management as the second in an MRM series that we've developed together.

**[0:01:43] Jo Ann Barefoot:** The earlier paper looked at MRM generally. This one drills down specifically on the MRM issues surrounding the most mold-breaking tech change ahead, generative AI. I will link to both papers in the show notes. Using data models in financial services has always produced risks, partly because of what they call the black box factor. These models perform analyses that generally can't be done by the people using the outputs. The results the models produce tend to look very definitive, even though we know that their accuracy depends on the quality of the data that was fed in and on the design and upkeep of the model.

**[0:02:25] Jo Ann Barefoot:** This risk of relying on flawed models has generated long-standing best practices for model validation by both industry risk managers and regulators. Today, those risks are compounding as more data is used, more models use AI, the AI becomes more sophisticated, and more people use models, and increasingly with user-friendly interfaces that can sometimes cause the uses to get detached from the model design controls. In our conversation, Behnaz starts by explaining how Gen AI differs from other forms of AI and the resulting challenges it is presenting.

**[0:03:06] Jo Ann Barefoot:** She lays out how financial companies are beginning to use Gen AI. She talks about the fundamental principles that guide MRM, which are largely timeless, and then walks us through the unique issues emerging from Gen AI. One key area is whether current regulatory frameworks are fit for purpose as banks increasingly use AI. We talk about what steps financial institutions need to take as they integrate Gen AI, and we talk about advice for regulators themselves. In our discussion, we try to strike a balance between practical advice that listeners can put to use right away and the efforts to stretch all of our thinking about the possibilities of a full paradigm shift in the years immediately ahead of us.

**[0:03:52] Jo Ann Barefoot:** Be sure to come to the show notes to get both white papers and also AIRS-related work, which includes our white paper on generative AI itself and our new white paper by my colleague Nick Cook, the Financial Regulators Odyssey, talking about why and importantly how regulators will need to transform the technology that they use themselves. I know you're going to enjoy my conversation with Behnaz Kibria. We have a really exciting show today. My guest is Behnaz Kibria and she is the Director of Cloud Public Policy at Google.

**[0:04:31] Jo Ann Barefoot:** Benaz, welcome. I'm so excited to have you on the show. Thank you, Jo Ann. I'm excited to be on. We're going to talk in a few minutes about the fact that we together are putting out a second paper on model risk management with AIR and Google working together on it. And it's been fantastic to be able to work with you and your team on this. these cutting edge topics. And I will link in the show notes to the first paper as well, and the second paper.

**[0:05:04] Jo Ann Barefoot:** But I wanted to start out by asking you to talk about yourself. What's your background? And then tell us about your corner of Google. Sure. Thank you. Thanks so

**[0:05:16] Behnaz Kibria:** much. Thanks for having me. And very, very happy to be working with you again on this paper. So I am the head of a team called Product Initiatives at Google, which is part of the Government Affairs and Public Policy team. And we cover Google Cloud issues in particular. And as the name suggests, Product Initiatives, we work very closely with the Google Cloud product teams to help them navigate policy issues. And as you can imagine, as generative AI has become an increasingly important part of our product suite, the policy issues around generative AI are also becoming increasingly important.

**[0:05:55] Behnaz Kibria:** So we are spending much more time on these issues ourselves. Right. And what's your own background? So I have been at Google now seven and a half, almost eight years. I came out of I used I'm a trade lawyer by training. I used to work in the administration in the Obama administration doing trade work. And then before that, on the hill doing trade work and before that in private practice doing trade work. So tech really was sort of a bit of a pivot for me.

**[0:06:30] Behnaz Kibria:** But it's just been it's a it's been a really interesting eight years or so in tech, as you can imagine, a lot has changed over that period.

**[0:06:38] Jo Ann Barefoot:** That is for sure. So I want to start by talking about everyone's favorite subject, generative AI. And in the first paper, we didn't talk much about AI. It was sort of the basics of model risk management, MRM. but obviously generative AI is changing everything around us. And so I think it'd be great if you would explain what it is, how it's different and how it's being used in financial services so far.

**[0:07:15] Behnaz Kibria:** Well, you're, you're asking the critical foundational questions. And, and, you know, as you know, this is the, the sort of the foundation of the paper too. So the key to understanding what generative AI is, is that term generative, right? So generative AI is a type of artificial intelligence that can create new content or data. So instead of just analyzing or acting on existing information, it can generate things like text, like poems or articles or summaries or even code.

**[0:07:51] Behnaz Kibria:** It can generate images, pictures, artwork, photorealistic scenes. It can generate audio. It can generate video. And what really sets it apart is that generative AI models learn by being trained on massive amounts of data. And these models identify patterns and structures in that data and then use the knowledge that they acquire through training to generate new, sometimes similar content. And you mentioned, you know, the first paper that we did last year and now the paper that we're working on.

**[0:08:33] Behnaz Kibria:** In that period, one of the things that's really changed in the world is really the bursting onto the stage of generative AI sort of evolving from earlier traditional models. And when you look at, we did a study back in July 2023, a benchmarking study, and asked organizations about the impact they felt Gen AI would have. And 82 % of the organizations responding that were considering or using Gen AI believed that it would either significantly change or transform their industry.

**[0:09:12] Behnaz Kibria:** That gives you a sense of how the industry, how industries are looking at the technology. In terms of the benefits of generative AI, it really creates the possibility of enhancing individual productivity, strengthening security operations, and advancing data-driven decision-making. So all of those capabilities really come together in terms of financial services to create a number of compelling use cases. One you can imagine is enhancing customer service. So I'm sure everybody's experienced AI-powered chatbots and other sort of personalized recommendation engines and things.

**[0:09:58] Behnaz Kibria:** There's the possibility of streamlining operations, automating tasks, streamlining workflows. And we've got a number of customers at Google Cloud who are already using generative AI for those purposes, like Deutsche Bank. There's the possibility of improving risk management. So for example, fraud detection and risk assessments. There's the possibility of using AI to boost investment research. So analyzing market data and generating investment ideas. And then there is also the possibility of creating personalized content, such as marketing materials and financial reports.

**[0:10:31] Behnaz Kibria:** Those are just five sort of general categories in which you can see real potential in the financial services industry.

**[0:10:39] Jo Ann Barefoot:** Yeah, absolutely huge. You know, I think our listeners have heard me say this before, but I've never seen a tech trend that hit, that was big and hit everyone at the same time before ever. If we think about the other big ones, you know, they kind of gradually came into our lives over years and this one within a few months you know that little few month period two years ago or so uh everybody was figuring out what this thing was and playing with it and thinking about whether it was going to change what they're doing so absolutely i mean if you compare a lot you know

**[0:11:18] Behnaz Kibria:** january of last year to january of this year it's it's really just a a really significant change

**[0:11:26] Jo Ann Barefoot:** Yeah, absolutely. So you work in public policy, as do we here at AIR, and talk about the regulatory environment around AI, but specifically generative AI. Is it fit for purpose for what we're going to need in terms of the legal frameworks and the regulatory frameworks, and particularly in financial regulation?

**[0:11:55] Behnaz Kibria:** That's a great question, Jo Ann. And we actually started getting that question even before this burst of energy and excitement around generative AI. As you know, because we worked together a lot when this issue first started bubbling up in the context of more traditional AI models, we started getting questions both from the regulators on the one hand and from our customers on the other saying, with all of this new and emerging technology at that point still sort of traditional AI, do we need new laws and regulations?

**[0:12:33] Behnaz Kibria:** And we took that question on board. And the first paper that we wrote, as you know, really was an attempt to answer that question with respect to a subset of traditional AI use cases, you know, what we called at that time risk use cases, you know, the use of traditional AI technologies for fraud detection, anti-money laundering, etc. And the question that we asked was, with that set of emerging AI use cases as a sort of a basis, did we feel that we needed new laws and regulations?

**[0:13:15] Behnaz Kibria:** And the answer that we came to in that context was no. And the reason is because one of the great characteristics of the laws and regulations in the financial services industry is that most are principles-based. and technology neutral. And what that means is that they are capable of being adapted over time as technology changes. Every new turn in technology doesn't require that we throw out the tools and start again. And as we looked at the use cases in that first paper, we studied in particular SR117, which is the model risk management framework in the United States, and, you know, which was issued in the wake of the global financial crisis long before AI ML models were in the mainstream

**[0:14:02] Behnaz Kibria:** use. And, you know, while that was updated in 2021 to issue more specific guidance on AI, the principles themselves in that kind of framework, as we looked at them, really did seem sufficiently high level to provide the kind of meaningful guidance that we needed in the assessment of risk, even as technology advanced. And so the reason for this second paper now is really to look at that same question again and, you know, ask with the continued evolution of technology, is there a need, you know, does the same conclusion hold?

**[0:14:41] Jo Ann Barefoot:** You know, it might be useful to our listeners to talk just a little bit about So even though many of them I know already know a little bit about what we mean by a model in this context and what, you know, what are those basic principles of standards and expectations for a sound model or model risk management process?

**[0:15:09] Behnaz Kibria:** It's a great question. You know, I think that the way that models are defined, especially in the U.S. system, it's actually quite broad. It's, you know, it's really, this is not the legal explanation for it, but I think the way that it's been applied, it's really anything that's got inputs and outputs could potentially be a model. And the way there are certain categories that model risk management frameworks look at, things like model validation, assessment of a model's accuracy, reliability and limitations.

**[0:15:50] Behnaz Kibria:** That's one category of issues that model risk frameworks look at. Governance and control, so roles and responsibilities for model development, implementation and monitoring. That's another category of issues that model risk management frameworks tend to look at. And then risk mitigation, which is identifying and managing potential risks such as bias, data quality issues and misuse. That's another category. And so the general framework of model risk management that we have in the U.S. and replicated elsewhere, as well, is sort of passing new technologies through that frame in order to assess and mitigate risk.

**[0:16:37] Jo Ann Barefoot:** tell me if you agree with this but i think one of the issues that people are kind of waking up to is that not too long ago model risk management and model validation was sort of the province of a expert group of data science and tech people who really understood how a model is built and what it's doing and what it's supposed to do. And as these tools get better and stronger, it's becoming more and more possible for non-expert laymen to use them.

**[0:17:11] Jo Ann Barefoot:** And that causes a new set of challenges in terms of things like governance or things like, you know, who's keeping this model up to date and so on. And more risk probably that the person using the model may not be fully alert to what could go wrong with it. And so I think it's extremely timely to be doing what you're doing.

**[0:17:36] Behnaz Kibria:** Yeah, I agree with that entirely, Jo Ann. And I think you're headed towards one of the issues that we highlight in the paper as being unique or different with respect to the world of models that we're dealing with now, which is that you also increasingly have more collaborations between technology companies like ourselves and financial services firms developing these models. And so you've got, you know, sort of very similar to what you said, you've got many different actors potentially in the supply chain or the use chain.

**[0:18:10] Behnaz Kibria:** And, you know, with differing understandings or differing sets of expertise. And so you've got to have a framework that covers all of that. So, Jo Ann, I wanted to ask you, because we've been collaborating in this space now for a number of years, and AIR has been doing work, you know, even other work in the AI and Gen AI space. What are your takeaways and drivers for further work? How important do you think it is more broadly to share MRM frameworks and principles based on the conversations that you've been having?

**[0:18:47] Jo Ann Barefoot:** Yeah, thanks for asking. I cannot think of anything more important than the whole ecosystem that we're working in, leaning forward and preparing for the changes that are coming with this technology. And it'll be like anything else. It sometimes seems really fast and sometimes it seems slower than people predict and so on. But there's so much work ahead to figure out how best to do this for all the players in the ecosystem. We think that generative AI in particular is really going to bring an entirely new paradigm of risk management and financial regulation because people will have more opportunity to see more information and analyze it easily.

**[0:19:40] Jo Ann Barefoot:** I mean, the biggest, if you think about the core function that risk managers are performing, is they're trying to figure out what is happening that could be causing problems. And it's hard to do that. You know, information is hard to get to, and it's hard to analyze it efficiently. Generative AI changes all of that combined with the, you know, the greater digitization of data, which has been exploding, doubling frequently for a long, long time. And one of the things that we think is going to develop is a lot of use of agentive AI, AI agents that are helping people do what they have to do, including find out what they have to find out.

**[0:20:33] Jo Ann Barefoot:** So the ability to query with using an AI, a complex set of data and get insights and advice out of that without being stuck in sort of static reporting and that type of thing. But the challenges are just massive, and that's including for the regulators. And I'd like to throw it back to you again. When you think about the challenge here for policymakers and regulators, what are the most, and we do have a lot who listen to the show, by the way, what are the most important things for them to be thinking about?

**[0:21:10] Behnaz Kibria:** Oh, that's a great question. And I think, you know, one of the things that I think is really important is to remember that we can have new technology, we can have very, very advanced technologies that really challenge, you know, old assumptions and old ways of doing things. But that doesn't necessarily mean that the tools that we have can't be useful. That's really just a core premise of this paper is that even though the technology is different and really just revolutionary, the ways we've been looking at structuring our processes, our thought processes around risk are still pretty valid.

**[0:22:09] Behnaz Kibria:** And what's really important is to break down the issues into component parts and then to assess, you know, what's truly unique about the technology that requires a different look, a different way of addressing things. And that's, you know, as we think about this new paper, that's really what we've tried to do is to take the frame of model risk management that's been developed, impose it upon the technology, and then identify the very unique aspects that require sort of a special consideration.

**[0:22:43] Behnaz Kibria:** You know, one example is just, you know, in the context of safety and soundness, you know, in the old days when you had traditional models, you could just unpack how the model worked. You know, you could just say if X, then Y, if Z, then Q. And, you know, that's how the model was built. and if the model, you know, the inputs and outputs kind of match what you would expect based on how the model is built, then that's sort of sufficient evidence of safety and soundness.

**[0:23:14] Behnaz Kibria:** With the evolution of technology with generative AI, that kind of unpacking of the model to understand its guts becomes a harder task. These models are very, very complex, and so that kind of assessment of safety and soundness is no longer valid, then you have to move to other, you know, what are other ways of assessing safety and soundness against this new technology? And, you know, one of the core proposals of the paper, for example, is that you really have to look more at things like testing, output testing.

**[0:23:49] Behnaz Kibria:** You have to look at, you know, techniques that are utilized in the process of model development, like grounding. to really assess safety and soundness. So, so, you know, those are new concepts, but the frame against which they're put, the way that we look at evaluating safety and soundness, the structures, the way of thinking about them is still a really useful way. It's just that these sort of, you know, the best practices of course will evolve as technology evolves.

**[0:24:21] Jo Ann Barefoot:** Yeah. When we think about the challenges facing regulators in particular, and our listeners know, I always say, I think that regulators have the hardest job. you know they really don't have leeway to try things out that aren't going to work and that type of thing other than in their sometimes in their labs and sandboxes um i think one of the challenges that's going to really be hard for them is putting out guidance that is both clear but can be made agile too.

**[0:25:04] Jo Ann Barefoot:** They don't like to put something out before it can be clear. I mean, sometimes they'll put out very broad guidance that requires subjective judgment, but they're trying to go for as much clarity as they can, generally speaking. But this area is changing so fast. And I don't know what the solution to this is, but I think they need to be thinking about it. you know, the ability to put out guidance on what they expect in say MRM for generative AI.

**[0:25:37] Jo Ann Barefoot:** And then as they learn more and as the field evolves further, put out something else without creating too much chaos and too much sort of version control, because, you know, what if the regulated entity was following the guidance of the time, but now, you know, the guidance has changed. that's fine if you have pretty stable situations, but when you have a very dynamic situation, it all becomes harder. And I think that's going to tend to make the regulators take longer to reach clear guidance.

**[0:26:14] Jo Ann Barefoot:** And I think that in turn will cause a lot of problems in, you know, in practice. So I think there's a tremendous challenge there, but I'll say at the same time, go ahead, go ahead.

**[0:26:29] Behnaz Kibria:** No, I was just going to agree with you. I think that's right, but I want to hear what you're,

**[0:26:33] Jo Ann Barefoot:** what you were suggesting. I think the point that I made about transforming the risk management paradigm, I think that goes doubly for the regulatory and supervisory processes themselves. and the fact that there will be so much more transparency in these complex financial systems and transactions, we're going to be dealing with the challenges in crime and fraud that are being caused by AI, which is definitely a huge issue, but we're also going to have massively enhanced ability to see them.

**[0:27:14] Jo Ann Barefoot:** It's going to get harder to hide them in the system if the regulators are equipped themselves with AI detection tools that can find patterns that we humans don't have enough, you know, days in the hours in the day to look for. And I'll throw in one more idea that we've been thinking about, which is if consumers become equipped with their own AIs to help manage their financial lives, their own AI agents, that could transform the way we think about consumer protection.

**[0:27:51] Jo Ann Barefoot:** If there were an ability for the consumer to easily see through a predatory practice or a product that was a poorer fit for their situation or a risk to their privacy or something like that, you could imagine that some of the things that agencies do now to protect consumers might be less needed. They maybe should focus on regulating the AIs and making sure that they've got the consumer's best interest at heart. So, you know, I think it's a whole brave new world of possibilities as well as threats for the regulators and their missions.

**[0:28:36] Behnaz Kibria:** That, you know, that's such a good point. And actually, you know, you asked me earlier, how is AI being used in the financial services industry? My favorite use case is actually one that it echoes what you were just saying, which is the risk management uses of AI. You know, we're talking, we spend so much time talking about managing the risk of AI. And yet, AI has such a useful role to play in helping financial institutions manage risks as well.

**[0:29:11] Behnaz Kibria:** So I'm very glad you made that point.

**[0:29:14] Jo Ann Barefoot:** Yeah, good. So let's go back to the financial institutions. You talked about some of the emerging use cases that are exciting. What are the things that you and the folks working on the paper have been coming up with in terms of what financial institutions should be doing?

**[0:29:36] Behnaz Kibria:** Well, you know, as we look at risk management, we sort of bucketed into four issues, four areas in which we think there's a variety of best practices that financial institutions should be considering and that regulators should be maybe acknowledging as evidence of good activity or best practices. Developing robust governance and ongoing controls, things such as integrating socio-technical considerations and stringent data governance. Because again, as we talked about, One of the really unique things about AI technology is just the importance of data.

**[0:30:17] Behnaz Kibria:** Good data in means good outputs as well. Another area is supporting reliable model development. And their techniques such as grounding techniques and outcome-based evaluations, as I said earlier, take on real heightened importance. Strengthening model validation and oversight. And here, in particular, the really important aspect of human-in-the-loop systems for critical decision-making to make sure that you do have the necessary extent and level of human oversight over these processes. And then the third we've also talked about before is, you know, it is very likely with, especially with complex models, that the relationship is, there's going to be some sort of a third party involved in the mix.

**[0:31:15] Behnaz Kibria:** You know, some sort of technology provider is likely going to be providing some support along the way. And so as we look at risk management and, you know, and the relationships between all of the participants in the process, there's got to be a clear delineation of roles and responsibilities and a sense of sort of shared fate and shared responsibility as well.

**[0:31:41] Jo Ann Barefoot:** Great. Is there anything we haven't talked about that you'd like to add?

**[0:31:50] Behnaz Kibria:** Well, just maybe another question for you, if you don't mind. Following your recent work on modernizing regulatory agencies, what steps do you think regulatory agencies can take to keep up with the kinds of advancements we've talked about?

**[0:32:09] Jo Ann Barefoot:** Some of it is common sense. Our listeners know I'm a former regulator myself. I was deputy controller at the currency. and at the financial regulatory agencies, all of them have very strong cultures and deep expertise that they're very proud of and kind of a sort of dominant professional way of looking at types of things and so on. And I think one of the most important things they need to do is very explicitly and proactively bring technology right into the center of those value systems and ways of thinking.

**[0:32:59] Jo Ann Barefoot:** And it's not easy to do, but it's probably getting easier because of demographic change. As more young people are moving up the ranks at these agencies, It just sort of automatically makes it a little bit easier. But the technology, the future of finance is technology, and therefore the future of financial regulation is technology. And this technological tool is the single most important one operating as part of a larger thing in terms of creating this ability to leap forward.

**[0:33:39] Jo Ann Barefoot:** So I think the agencies need to be training on it, learning about it, hiring people who understand it into leadership roles, experimenting with it, putting together experimental lab type situations where they try using some new things and see what happens. learn along with the industry go to all the events listen to the podcast shows you know just do everything possible to bring this into the DNA of these regulatory bodies and get everyone to immediately think of of tech you know working in this field as I have been doing for so long I'm you know I never have more than a few days go by where I don't hear somebody talking about something in the regulatory realm, whether it's a private

**[0:34:39] Jo Ann Barefoot:** sector or public sector person, where to me, the issue obviously has a tech potential solution. And it's not in the toolkit of the people who are working on it. They don't mention it. And usually, again, it's these kinds of things. It's just the ability to learn more faster and understand what it may bring faster. You know, that's what regulators are doing all day long. Everything they're doing is really at that at heart, understanding risks and noncompliance and so on in the system and trying to find it.

**[0:35:26] Jo Ann Barefoot:** So I just think that there just needs to be an all out campaign. And it also needs to include, and this doesn't get discussed enough in my view, it needs to include that the regulators have to update their own tech architecture. They are still trying to operate as many banks have been trying to do, grafting new tools on top of an outdated tech stack that you get to something that's better than nothing. but they all have to move to cloud computing in one form or another.

**[0:36:03] Jo Ann Barefoot:** And they all have to get all their data into and all the data they're using, internal and external data, structured and unstructured, into their ability to work with it and really overhaul their own tech. And that's a project that's going to take some resources to do that. to me that's all the more reason to get started sooner rather than later and they all are i mean we have some u.s agencies that are very sophisticated with ai and we have some that are just starting on the cloud journey and a lot in between but nothing is more important for regulators it seems to me than getting on top of the both the risks that are emerging with technology, but also

**[0:36:59] Jo Ann Barefoot:** the ability of technology to, A, solve for those risks, and B, solve for problems that we've always had in the past and have not been able to solve, timeless problems like how difficult it is for the current system to detect money laundering, for example. The numbers on that are just shocking as to how much money laundering goes undetected in the system. The ability to find patterns of potential bias, the ability to find emerging risks to the safety and soundness of an institution or the systemic risk of a sector or you know, a sub sector in finance, the ability to make sure that the consumer protection rules are functioning the way they're supposed to be.

**[0:37:56] Jo Ann Barefoot:** We have a, you know, we didn't used to have the ability to do any better than to use the tools that we had. And therefore, a great deal of our regulatory philosophy is based on looking at the quality of the inputs and the quality of the controls, as you've been saying. So we, you know, the regulators expect to see a robust compliance management system and the three lines, if it's a bank, you know, the three lines of defense and all of that.

**[0:38:33] Jo Ann Barefoot:** And again, I mean, that was the best we could do before, but now we're getting to the point where we're going to be able to see outcomes. We have consumer protection laws. How are they doing at protecting people? Might we be able to do better? We should be really rethinking the whole thing in light of these powerful new tools that have been put into our hands. So I'm very optimistic and excited about it. Regulators all over the world are thinking about these questions and the momentum is all toward figuring it out, but it's a mountain to climb in terms of exactly what to do.

**[0:39:14] Behnaz Kibria:** Yeah, no, I couldn't agree with you more. And I do think there's a paradigm shift probably that's needed, not in terms of necessarily the frameworks, but the way of thinking, right? I mean, and it's been changing already, but even if you think about the period of time we've been working together or that I've been working on cloud issues, the nature of the conversations really changed. I mean, you know, just even as a cloud provider, we used to be often outside of conversations, you know, in financial services, those conversations often were between the regulators and the regulated entities, the banks, the financial institutions, and we were sort of an extraneous to that conversation, that's really been changing.

**[0:39:59] Behnaz Kibria:** And I think that's positive. I think, you know, there is a three-way conversation that must happen as technology becomes more and more integrated into banks and financial institutions. And then, you know, just the nature of the conversation, too, where it's much more fluid and collaborative rather than, you know, sort of being like a tennis game where one side hits the ball, then the other side hits the ball, then the other side hits the ball. It's a much more fluid conversation, you know, between three people or three entities rather than that.

**[0:40:41] Behnaz Kibria:** So anyway, I'll just say I agree with you entirely. And I think there's a lot of good stuff that

**[0:40:47] Jo Ann Barefoot:** that is on the horizon. Great. So we will put in the show notes at regulationinnovation.org, the new paper, the previous MRM paper. We'll put in a couple of papers that AIR has just put out, including one we did a podcast on recently on what we've entitled the Financial Regulators Odyssey written by our colleague Nick Cook and some of the other tools that we've been working on here at AIR and anything else that you have to share from Google.

**[0:41:27] Jo Ann Barefoot:** And again, that'll be regulationinnovation.org. Behnaz, I can't thank you enough for joining me today. Behnaz Kibria, it's just fantastic to have you. And I hope the paper will have a big impact.

**[0:41:44] Behnaz Kibria:** Oh, thank you. It's just been a real pleasure working with you and being on the podcast.

**[0:41:48] Jo Ann Barefoot:** I hope you enjoyed today's show. Again, come to regulationinnovation.org to see the new paper, the related papers, and get much more information about these challenges. We have wonderful episodes coming up. We're going to talk with Daniel Gorfine of Gattaca Horizons about our upcoming white paper on earned wage access and how it fits into broader financial innovation and consumers' financial lives. We'll have a show that we recorded live at the annual Washington Summit of the National Bankers Association, focused on what else?

**[0:42:25] Jo Ann Barefoot:** AI. We have an episode coming up with Sakat Narayan of Amazon discussing modern technology applications required by regulators. And we'll have a fascinating conversation with Supes Ranjan of Sardine on fraud. AIR is doing a lot of major work on fraud. I will be speaking this one at DC FinTech Week. Looking forward to seeing fans of the show there. Next week, I'm excited to see you all at Money 2020. and then in November at the spectacular Singapore FinTech Festival.

**[0:42:58] Jo Ann Barefoot:** We will also have my air colleague, Mariama Jawa Hayward, speaking on October 30th at the Africa Anti-Fraud Conference hosted by Proveneer and the Africa FinTech Network. Be sure to leave us a five-star rating on your favorite podcast platform so more people can find Barefoot Innovation. And please also find me on social media to continue the conversation.
