Auditing Machine Learning Systems in Financial Services - with Andrew Clark of Monitaur AI in Financial Services ยท 1 Oct 2020 Source: https://aiandbanking.libsyn.com/auditing-machine-learning-systems-in-financial-services-with-andrew-clark-of-monitaur Transcript: https://podcasts.opennash.com/shows/ai-in-financial-services/auditing-machine-learning-systems-in-financial-services-with-andrew-clark-of-mon-8cd9d1/ (via OpenNash Podcasts, machine-generated) This is Dan Faggella and you're listening to the AI and Financial Services podcast. When I'm asked about our AI opportunity landscape research with banks, with big insurance companies, people often presume that when we lay out our opportunity landscapes for the enterprises we work with, we showcase the areas of high return on investment, that the only thing we're really focused on and the only thing our clients are focused on is really taking over the market, winning market share, growing. And certainly that's part of it. But in financial services, risk is a big part of it, reducing risk, particularly in a regulatory sense. And as it turns out, machine learning can be used to help solve that problem. Andrew Clark is our guest this week. He runs a company called Monitaur. He was previously the principal machine learning auditor at Capital One, a company that I happen to respect. I think they're a little farther ahead than most when it comes to AI deployments and the credit cards and financial services space. Andrew speaks with us this week about how artificial intelligence can audit and Monitaur machine learning algorithms so that they don't pose regulatory risks. Why is it necessary and how does it work? Kind of a unique use case, something we haven't covered in the past, but Andrew does a good job of summarizing it. If you're interested in more financial services use cases, be sure to download our AI and financial services cheat sheet PDF. If you don't already have it on hand, be sure to go ahead and download it that you can find it at emerj.com slash FS1. That's FS as in financial services. And then just the number one, emerj.com slash FS1. Go ahead and download that PDF brief and get some of the key terminology and use cases for artificial intelligence in financial services. This week, we focus again on a unique one here with auditing machine learning algorithms. Without further ado, we're going to fly right in. This is Andrew Clark with Monitaur here on the AI and Financial Services Podcast. So Andrew, we'll dive in and talk about this concept of auditing in machine learning. I think for the financial services audience, there's concern about the great black box of AI and all the dangers there. When you define the term machine learning auditing, how do you like to explain it to business people? I like to call machine learning assurance and auditing is essentially providing an ability for you to trust your algorithms and your models that they're doing like they should be doing that you're meeting the business objectives and you understand the whole process and know how everything fits together you can kind of like you would have with a financial audit for instance where we're used to knowing like to trust that a balance sheet is saying what it should be doing you have auditors that are checking this and you have internal control processes with your finance team and things like that that you are confident that if you have on your balance sheet, $200 million in your bank account, that's the amount that's actually there. When you're talking about machine learning and machine learning auditing, you want to make sure that what your models are doing are what you think they're doing, and you're not going to have some sort of bias or things pop up over time. Got it. And what might be an example of that? I imagine some people listening in could imagine, but you've probably got some good examples from your rich experience in this space. What are a couple of things we wouldn't want to happen? Right. So a lot of times what I've seen is you have a model that's been trained for, let's say credit cards. It's going to determine if an individual should get a credit card or not, and then what credit limit they should have. A lot of times these models are trained with a range of features. You know beforehand individuals with an age between X and X, ideally you won't be using age, but I'm just using that for sake of this illustration right now. Yeah. But for sake of this argument, you'll be using age, you'll be using FICO score, you'll be using things of that nature. And you're used to having, if you're a credit card company, is going for maybe slightly lower FICO scores. You're used to having between three and 600, for instance. So you have a model that's built on these ranges of data that you're used to having this sort of outcome for. You've trained a supervised model. You know what you're expecting. What you want to be careful of over time is if you built your model to expect to have about people with an average of 400 FICO score, and over time your FICO score starts increasing or decreasing, your model outputs might be different than what you're initially expecting. You might also have more of an imbalance in your data set of a lot of people that have different classifications of income and things like that. So the drift in what you're expecting into the model is different than what the model was trained for. And the problems can arise, and Apple had a problem with this recently with Goldman Sachs, is you can sometimes get unexpected outcomes and bias starts getting introduced into the model. And just because the model, the environment it was built in, is not the same environment that's operating now. And being able to know when those changes happen and being able to correct those are very key in being in the truster systems. Yeah, yeah. So for, you know, we'll get into some specific use cases by the time the episode's out. I'm going to make some analogies for the audience, Andrew, and clarify this idea even for myself. You know, you mentioned, well, the environment you built it in isn't the environment you live in. I think about a lot of machine learning systems, particularly in big tech, where that's kind of ubiquitously the case. For example, you know, an Amazon is constantly what is selling in one season versus another season. OK, now it's back to school season and all of a sudden backpacks are getting bought way more when we've just updated all of our systems based on this data. But that's more correlated to the holidays. So we always have to sort of Monitaur. Are we calibrating to the now to some degree? So the outside world is always shifting. When you talk about FICO scores, maybe something crazy happens in the real world. You know, you and I are going through that right now with this coronavirus. where maybe they're just more cash-strapped altogether. And so you didn't map that into the algorithm. The algorithm was baked on, let's say, the last five, last 10 years of activity when we didn't have coronavirus with 25 % unemployment. It feels like the world is changing constantly, so we would already need monitoring. Is this a differentiating factor even on top of that? So in many cases, the world is already shifting. What makes this different or more? It's just in these types of regulated industries and these sensitive deployments, it's important to have the types of controls around to know how your algorithms are working and to know when they start decaying a little bit. For instance, using the Amazon example, it's a little less important if they slightly over-order Pop-Tarts one year or not, right? Or one month or whatever, based on forecasting algorithms. The key difference here is where we're starting to use modeling these days with machine learning and things affect end users and consumers directly, not just knowing if you ordered a little too much product or not. And that's where before we can have self-driving cars or really using loan underwriting and radiology scans, machine learning algorithms, those sorts of things that are really affecting the end user for maybe a long period of time, diagnosing and things like that. That's why we really need to have these systems control before they can really get trusted and accepted by the public. So it's different than just making sure your univariate forecasting model is as accurate as it can be, which is always a goal. Yeah, if your Google ads aren't as relevant as you would like, A, is there really a way to measure that? And B, is the consequence all that rough? Same thing with your Amazon recommendations for your next book in Kindle or something like that. But if it's whether or not you get a loan, if it's whether or not a diagnostic is correct, then of course we're impacting lives. And I think the business audience is well aware that many of those barriers are exactly what's a big part of what's helping to hold AI back from actually being deployed in the real world. When you think about what this actually looks like, you know, I'm trying to close my eyes here and for the business audience tuned in, do the same thing unless they're driving, of course. Close my eyes and imagine what it is that this looks like. So, okay, we've got to set in place some auditing. We need X number of new team members doing Y. We need to be testing our outputs by doing X. And we need to maybe even be throwing some test data in the front end to test the back end by doing Z new process. What is that new jumble of processes of checking and of people that is going to make auditing work? Definitely. Well, it's very important to have a strong, in finance, we like to call the three lines of defense. So you have the first line of people actually doing the work and the operational, for instance. Second line develops risk management frameworks of what are the key things we need to control? What are the risks? For instance, you don't want someone that's dealing with cash to be able to kind of embezzle. You want to make sure you have specific access controls around information, things like that. Second line develops these frameworks and risk management controls. Third line is the auditors that are actually verifying that these things exist. So really what we need is expanding upon financial and IT auditing, a degree of model auditing. We do have model risk management, but more so of that holistic look at how a machine learning model interacts with the environment. What are the business objectives? Who are the key stakeholders? How is this going to affect the consumer? Building audit frameworks and risk and control environments around that and then making sure there are no gaps in the process. For instance, when you build a machine learning model, making sure you know where the data is coming from, that it's been prepared appropriately, that it's balanced, it represents the use case you're wanting it to do. And then when you move it into production, it's properly managed. You know when the model is going to start to decay. You know when to raise flags that you need to have a new model. Those sorts of things, the holistic governance and process. And it's really a business level. It's not a really a technical problem as much as understanding how the whole algorithm fits into your business and where you need to be concerned about and where you need to have controls in place. So it's just this whole thinking and new mindset shift to be able to really use these algorithms appropriately, environments that affect consumers. Yeah, well, thinking and mindset shifts has to happen for even the most mundane AI applications. And some of those cultural and talent shifts from the normal way of doing IT are already gargantuan barriers to entry. This additional layer obviously doesn't make things any easier from an adoption perspective, but as you're mentioning here is obviously important. How do you know, you know, you'd mentioned, well, it really matters kind of, you know, you have to have the context and the business case to know when you'd actually need this stuff. If I'm imagining somebody listening in and saying, okay, well, I work in a financial services firm, maybe a bank, insurance, credit cards, wealth management, whatever. What quote unquote you're saying like interacts with consumers or something. So where am I having that interface with consumers that are audit worthy in terms of machine learning, A, and then B, How would I know what part of that process requires what kind of double checking, triple checking, et cetera, in order to kind of feel as though I have an auditing system in place? How would I get the context to get that right in my head? Right. It's very seeing what the business objective is, looking at the existing controls that are in place and the existing audit processes. So most likely anybody in those types of firms is going to have a second and third line function that evaluates the risk of the business. you're most likely talking to external auditors and regulators on a daily basis. They'll know what the risks are. Regulators are very vocal on what they see as concerns. So you might have be getting pushback on, as a business, we're wanting to deploy these new algorithms. You pitch that to the regulators who have to sign off on any of these things, and they're pushing back on, we don't understand this. We don't have good knowledge on why this is happening. Sometimes that's just educating them. Other times it's having to address the problems that they see. And in a lot of companies, when this process happens of introducing the idea to the regulators and to the auditors, there's a lot of pushback on, we don't seem to understand this model. Then a lot of times data scientists will go back and start building more explainability. But the whole process of we don't understand how this works, we don't understand how there's controls around it is this process that I'm speaking about, which really needs to be addressed between your model people, your auditors, and the regulators to find something that can make everybody happy and make them feel like you're not being exposed to risk or harming consumers. Got it. So I think probably what's going to make this concrete is bringing up a couple examples where this might come into play. And so, okay, well, in this particular business case, here's how we might think about it. Here's where auditing might be valuable. And one of those domains that you and I just talked about off mic is in the insurance space. So in particular, underwriting. Who gets access to what insurance policies, yes or no? And then what is the pricing? What are the terms of those policies? These are things that obviously will affect somebody. health insurance, insurance on a building, a car, whatever the case may be. Where might auditing fit in? If we could paint it kind of, you know, if people did it ideally, it would be part of how we bake in AI. It would already be part of the process. What would that look like if we took this into consideration? So leading off of that example, let's say that there's a major insurance company that is wanting to now have an accelerated life program. So traditionally, whenever you want to get a life insurance policy, you'll have to deal with some agents and it's going to be a process of invalidating your health. And it's going to be a lot of different steps to go through to know if you get this life insurance policy or not, and what your premium is going to be. There's a lot of companies in insurance now that are wanting to speed this process up by the great fact that we have a lot of data now. We know what our risks and how much we have to pay out over time. We know that actuarial table information. Why don't we have an algorithm that does this and save us a ton of time, get the consumer a better price, and let us shed some staff. A lot of companies are looking into this. When they're interfacing with the regulators, they're kind of like, well, how do we know this is doing what it should be doing and we're not comfortable? So the main process would be you would have data scientists build models that would do what they're wanting to do. Then you set up the control process into knowing that we know where this data is coming from. We have documentation of how it's working. We know who's signing off on this. We have this whole process. Then we have to show a history of this running in parallel with your existing process, showing it's accurate, showing it's fair, and allowing regulators to come in and audit this and see that it's doing what they should be doing. They can look up, we want to see this policy we gave Sally. Let's see how it would have worked in parallel and go back and validate specific decision, how it was made, why it was made, what are the factors contributing to it? So having all that documentation, that full length understanding and working with the regulators would be what you have to do. And I've been in discussion with a lot of these companies. Those are the processes you have to be in place before the regulators allow you to start selling this product. Got it. Okay. And in your opinion, where are the missteps being made in terms of oversights around this process? I mean, it almost sounds like the regulators are the ones that are prompting this auditing to happen because they're saying, well, I don't understand this. Is that in fact the case? You know, these systems are getting rolled out. It's the regulators that push back and then someone says, okay, we have to audit. Or is there some other place where there's an oversight where we forget about this step in industry today? Really, if you don't have a regulator directly on top of you on you shouldn't be doing this, companies will deploy and the media sometimes will catch them if something goes wrong. But a lot of times the regulators are the one prompting you saying like, this is something we don't feel comfortable with. So most companies would not invest in risk and auditors and things like that if they didn't have to. Some companies are very control minded and want to make sure they're doing the right thing. But a lot of companies like that's an extra expense. So most of the time, who's pushing back is the regulators saying like, you have to have these things in place before we'll prove it. But a lot of times the regulators don't necessarily know what they want. They just know, okay. So this is where the big gap is because data scientists will build an awesome model. It meets all the business criteria. It'll get sent to the regulators. Regulators say, we don't understand it. It goes back to the data scientists and it just kind of sits there because there's this kind of this roadblock on what exactly does that mean? And being able to like go back in time and re-perform exactly how a transaction was done, see the inputs that led to a decision, detailed logging, what does that look like, being able to verify who did what, when was the last time the model was updated, being able to see when drift occurs and having controls for bias. Those are all the components that a regulator needs before they can sign off on something. The problem is a lot of times regulators don't necessarily know that's what they want. They just know they don't see what they need. Yeah, this is a really rough chicken and egg issue altogether. I mean, one of many reasons why this stuff is hard to enable an enterprise. I mean, I'm no pessimist, but obviously this is a real hurdle. It almost feels like the regulators are going to have to up their game in terms of their conceptual understanding of AI and the sort of best practices of AI that they'd like to see so that they actually know what the heck they're looking for. And then companies are going to have to get on the same page. Now you had brought up off microphone that I think it was the FDA is developing some kinds of standards to say, hey, you know, in this context, you need to be rolling out AI with these kinds of best practices. It seems like some regulatory shared understanding absolutely has to develop. Otherwise, this is going to continue to be a nightmare. What are those precedents of that already happening a bit? 100 % accurate. Before the regulators understand what's going on, that's why we're at this stalemate. So FDA has been doing a great job of saying, they still have a long way to go, but here's some controls in place. This is what we want to see before we approve. There's the ICO, Information Commissioner's Office in the UK, that has right now released a very good draft framework for explaining AI. What are the operational changes we need to see? What kind of logging do we need? Really digging in for the UK, what that means for them to feel comfortable with machine learning and AI. EU is starting to do a white paper on that. The White House has done some loose regulations on AI regulation, but US right now is actually lagging behind Europe and the United Kingdom on a regulatory understanding of what this means. But really, before we can get to a place, before we can have self-driving cars, and these underwriting algorithms, the regulators really need to up their game a little bit, as you mentioned. And that's the key problem right now is the companies and regulators are looking at each other and don't really know how to proceed. But there is some change. There are some more movement in that space. There's an Algorithmic Accountability Act in the Senate by Cory Booker a while ago and things like that that are kind of developing that will help regulators get on the same page and allow machine learning and AI to be deployed. Yeah, yeah. So some of this is a little bit up in the air about how it's going to develop. And yeah, us Americans are going to be a little bit more concerned about innovation than regulation than Europe, who I think is often critiqued for inordinately focusing in the opposite direction. Many of the startups we speak to in Europe complain either overtly or subtly about exactly that. But to your point, in this case, regulation and regulatory understanding has to happen for innovation to happen. Because yes, these are real legal concerns, but we actually do have to have something in place so that companies can say, cool, we matched it. Let's innovate. Let's get to that benchmark and move beyond it. Drive some business value. It sounds like that's the gridlock. Completely. Because most of the innovation, if we see that's happening in the US around machine learning, everything is ad tech, social media. Get me to click. Exactly. That's where innovation happens. You look at banking, you look at insurance, you look at healthcare, these areas, it's not that they're not near as innovative as Google. A lot of times these financial companies are poaching Google employees. It's not that at all. It's that they don't have, the regulators are regulating these areas and they don't have the understanding to let the companies innovate versus social media and ad tech and things. They don't have these regulations and you can see how wonderful they're doing. So it actually, it sounds opposite of what should be the truth, but we actually need regulatory certainty before we can start having regulators need to get up to speed before these regulated industries will be able to use AI. And so you're actually ahead of us by regulating ahead of the time. Yeah, yeah. Yeah, in some areas, arguably maybe regulating a bit much, but I'm not going to make a judgment call on either side. I think I will just say, I think there's a golden mean to use Aristotelian terms here that we need to strive for, but clearly the United States needs to buckle down. And I think that frankly, given what your company does, I think that exactly how you just explained it is probably the best way to explain it, right? I don't really want to see you as one of the anti-innovation forces. I want to see you as a pro-innovation force. And I think you just made a reasonable argument for it. Last example, I know we're wrapping up on time, is in the domain of credit cards. So this is another really rife area for leveraging machine learning. I mean, there's fraud. There's do we give somebody a credit card or not? There's all these different applications there. Where might this same kind of idea of auditing apply in the domain of credit cards? Very much so in should we give somebody a credit card or not and what should their credit limit be? The exact same considerations apply. And there's a lot of major financial corporations that have already started deploying machine learning for fraud detection, because if you're protecting consumer, they don't necessarily need to know why or how this decision is being made that, hey, this might be a fraudulent transaction. Those are just correlations, really. It's when we're making an explicit decision that affects the well-being of an individual, if we should give them this credit card or not, is where the regulations of fair lending and other aspects are more important and more of the forefront. So for credit cards, it's those decisions about the end consumer that all of these machine learning auditing and assurance really comes into play. Got it. Okay. So similar analogous use case. I think that hopefully for the folks who are tuned in here, being able to see where this is headed and understand where these friction points are, I think that this is kind of a whole other level. And as we've unearthed here, Andrew, some of this is outside of our control as innovators. Some of this actually has to do with regulatory kind of updating. But I think companies that can meet those regulators when those new standards are up to snuff, companies that understand this conceptually can come in, can match them, can check those boxes, are going to be able to actually push forward and deploy. And to be frank, I know for those of you who are tuned in, that's what you damn well want to do. For those of you that are consultants, that's what you damn well want your clients to be able to do. So I think there's a lot of value in that. Andrew, I know that's all we have for time. Thank you so much for being able to join us on AI and Financial Services. Thank you so much. so that's all for this episode of the ai and financial services podcast if you've enjoyed this episode be sure to check out our other show it's called the ai and business podcast where we cover use cases every single week we also cover measuring ai roi and adopting and deploying artificial intelligence we cover sectors including financial services but also life sciences manufacturing, you name it, we cover the full gamut. So if you want a broader set of ideas, a broader bucket of use cases and trends to be able to draw from, and also a broader list of guests who you can learn from, whether it be the head of AI at Raytheon or head of AI at Comcast, then be sure to listen to the AI in Business podcast. You can find it on iTunes, you can find it on SoundCloud or any of the other platforms, or you can just go to emerj.com and you can see podcasts in our dropdown. You can go ahead and subscribe there as well. So hopefully you've enjoyed this episode. I look forward to catching you on the next one on the AI and Financial Services Podcast.