# Solving Black Box Problems in Financial Services with Machine Learning - with Krishna Gade of Fiddler AI

- **Show:** AI in Financial Services
- **Published:** 3 Apr 2023
- **Source:** https://aiandbanking.libsyn.com/solving-black-box-problems-in-financial-services-with-machine-learning-with-krishna-gade-of-fiddler-ai
- **Transcript:** https://podcasts.opennash.com/shows/ai-in-financial-services/solving-black-box-problems-in-financial-services-with-machine-learning-with-kris-409c5f/
- **Provider:** OpenNash Podcasts (machine-generated)

## In brief

Machine learning can take over much of the slow, manual work banks do to catch fraud, money laundering and bad customers, but only if the models are watched. Krishna Gade argues that AI is a black box that decays over time, so businesses need continuous monitoring and human oversight from the start.

The tension is between efficiency and trust. Automation cuts labor and errors, yet models cannot be read like normal code and can quietly get worse. Gade points to Zillow's losses as the cost of skipping monitoring, and to his Facebook experience as the origin of Fiddler's transparency tools.

## Key insights

1. **Machine learning cuts manual financial-crime review work** Gade says banks employ thousands of reviewers for fraud, anti-money laundering and customer checks. Models can score transactions and raise an alert when a threshold is crossed, so humans only review flagged cases.
2. **Customer verification now starts at the funnel's top** Gade says image processing and entity extraction can read driver licenses and address proofs without a person. Airbnb and Uber already do this, and he sees banks adopting it for onboarding.
3. **AI models are black boxes that deteriorate** Unlike traditional software, models cannot be read line by line, and they drift as data changes. Gade says this is why they must be monitored from day one and retrained at the right time.
4. **Skipping monitoring can sink a business** Gade cites Zillow, whose house-price models overpriced homes without proper monitoring. He says this tanked the business and led to thousands of layoffs.
5. **Facebook's internal transparency tool inspired Fiddler** After the 2016 election, Facebook built 'Why Am I Seeing This?' so staff could see why news feed items were ranked. Gade says this led to better models and showed him other firms would need similar tools.

## Questions this episode answers

### Why is machine learning good at catching fraud and money laundering?

Gade says the volume of transactions can reach millions, which is too many to review by hand. Models can score each user on money laundering risk using data like location and transfer amount, and flag only those over a threshold.

### What does it mean that AI is a 'black box'?

Gade says machine learning models store patterns in structures like neural networks and decision trees that humans cannot read the way they read code. That is why separate tools are needed to explain and validate them.

### Why do AI models need monitoring after launch?

Gade says models are not static and can drift and deteriorate as the data changes. A fraud model might start marking bad transactions as good, hurting customers or letting fraud through.

### Do humans still have a role if banks use AI?

Yes. Gade says he is a believer in 'human in the loop' AI, where people inspect outputs, check why something was flagged, and make sure legitimate customers are not wrongly classed as fraud.

### What happened to Zillow according to this episode?

Gade says Zillow used machine learning to predict house prices without proper monitoring and ended up overpricing homes. He says it caused heavy losses and thousands of layoffs.

## Facts

- **Krishna Gade** [0:01:20]: "banks employ thousands of people today to monitor fraud, But AI and machine learning can really enable them to do that efficiently"
- **Krishna Gade** [0:03:03]: "The models can rank transacting users according to a money laundering score and erase an alert when the score crosses the threshold."
- **Krishna Gade** [0:03:42]: "things like user location, how fast, how much amount that they're trying to transfer. All kinds of these things can go into models"
- **Krishna Gade** [0:06:57]: "If you want to become a host at Airbnb or if you want to be a driver in Uber, you need services, you need to go through a customer due diligence process."
- **Krishna Gade** [0:06:57]: "they try to use AI and machine learning to, you know, identify, basically use image processing techniques to see if you have uploaded a legitimate, you know, driver license"
- **Krishna Gade** [0:11:22]: "So they have this process of managing the risk around models called model risk management, where they validate the models even before they launch to production thoroughly."
- **Krishna Gade** [0:11:57]: "unlike traditional software where you can read the code line by line and understand how they work, machine learning models are these arcane artifacts"
- **Krishna Gade** [0:12:40]: "the model that was trained a year ago or six months ago or a month ago may not be relevant anymore, depending upon the use case."
- **Krishna Gade** [0:13:13]: "And all of a sudden it starts flagging bad transactions as good. It's going to hurt your customers."
- **Krishna Gade** [0:14:49]: "Zillow used machine learning models to predict house prices and they did not have proper monitoring in place."
- **Krishna Gade** [0:14:49]: "they were overpricing the house prices and it just tanked their business. They suffered losses. They had to lay off thousands of people."
- **Krishna Gade** [0:16:27]: "I spent about a dozen years, more than a dozen years, in fact, working on consumer internet technology companies, all the way from Bing, Microsoft, to Twitter, Pinterest, and Facebook."
- **Krishna Gade** [0:16:56]: "by 2016, by the time when I joined the company, we were using very sophisticated, deep neural networks to rank the newsfeed"
- **Krishna Gade** [0:17:27]: "So in the aftermath of 2016 elections, it became quite important for us that, hey, we really need to understand how these algorithms are working ourselves first"
- **Krishna Gade** [0:17:27]: "we built tools internally at Facebook called Why Am I Seeing This? that helped us internally as developers to understand how newsfeed algorithms worked."
- **Krishna Gade** [0:18:04]: "if they didn't find the recommendation relevant, they could file a ticket or complaint to the newsfeed quality team, which would then get triaged."
- **Krishna Gade** [0:18:31]: "I felt like, hey, Facebook could build it, Google could build it, Microsoft could build it, but not every organization could build it or may want to build it."
- **Krishna Gade** [0:10:19]: "it's essentially Fiddler is like a watchdog on your AI and it helps you make sure that your AI is working for you."

## Transcript

**[0:00:04] Matthew DeMello:** Welcome, everyone, to the AI and Financial Services podcast. I'm Matthew DeMello, Senior Editor here at Emerj. Today's guest on the program is Fiddler AI CEO Krishna Gade. Fiddler AI builds custom monitoring platforms specializing in risk and compliance, as well as lending and trading spaces. Krishna joins us on today's program to discuss why machine learning, in particular, is so effective in streamlining surveillance workflows and why financial services compliance is increasingly being geared towards verifying the customer from their introduction into the services funnel.

**[0:00:41] Matthew DeMello:** Krishna also talks about the various black box problems in financial services and how his previous experience working on the Facebook news feed after the 2016 election informs Fiddler's approach to model transparency. Without further ado, here's our conversation. Thank you so much for being with us, Krishna. Thanks, Matt. Glad to be here. So diving into Fiddler AI, where do you see AI impact customer experience and financial services today?

**[0:01:20] Krishna Gade:** Yeah, great question. AI and machine learning are the new technologies of this decade, and they are proliferating across the board. especially in financial services. When you talk to banks or insurance companies, fintechs, they are looking at AI and machine learning to automate a lot of business processes from fraud detection to KYC, anti-money laundering, credit scoring, customer acquisition, lots of use cases. For example, banks employ thousands of people today to monitor fraud, But AI and machine learning can really enable them to do that efficiently, monitor transactions, looking at various different ways that the fraudsters could be trying to game the system.

**[0:02:09] Krishna Gade:** And in AI, machine learning is a very effective way to understand what's going on in fraud and really mitigate it. Same thing applies for anti-money laundering or KYC processes, which are hugely laborious, human-oriented tasks that can be automated with AI and machine learning.

**[0:02:30] Matthew DeMello:** Diving into how machine learning accomplishes this, we see this use case all over financial services. But what makes machine learning particularly well-suited for that monitoring and surveillance task? KYC is a little bit different, I know, but anti-money laundering and fraud in particular are both reliant on detecting deviant behavior in like a receipt or a chain. Just if we can put a finer point on where the rubber hits the road in terms of machine learning and that specific workflow.

**[0:03:03] Krishna Gade:** Yeah, absolutely. You know, so when it comes to, say, anti-money laundering, right, the operational chart, These processes are very time-consuming. They require a lot of manual review. Volumes of transactions can be in millions. It can be an expensive process to hire lots of manual reviewers to see which transaction could be fraudulent. So machine learning models can automate it. So models can be trained to recognize money laundering and alert on them. The models can rank transacting users according to a money laundering score and erase an alert when the score crosses the threshold.

**[0:03:42] Krishna Gade:** The model can be trained with data collected from users, product activity, as long as also transaction data. For example, things like user location, how fast, how much amount that they're trying to transfer. All kinds of these things can go into models to use, to signal whether this is actually a fraudulent transaction or a legitimate transaction. And then you could then basically try to make sure that you are reducing your manual workload and prevent criminal activity around obtaining illegal assets and funds as legitimate income.

**[0:04:26] Krishna Gade:** This is where we see the emergence of deep learning and AI to detect these issues. The problem, though, is these machine learning models are a black box, and therefore, that's where tools like Fiddler can help financial services to operationalize AML with a lot of confidence where we can monitor the performance of the model, make sure that there is ongoing performance monitoring that the model is actually working as expected.

**[0:04:54] Matthew DeMello:** Indeed. I want to come back to that black box problem in a moment. And just focusing on the workflow here, I know for fraud detection in AML, KYC, they're also similar, but in where the pre-AI workflow has problems? Where do you see those pain points?

**[0:05:13] Krishna Gade:** Yeah. So let's take, say, some of the customer due diligence process. Businesses might ask customers to submit ID proof, address proof, and many such documents to make sure that they can check the veracity of customer's details. And this is a pretty large manual process. Same thing, figuring out if the transaction is criminal activity, if the assets are being transferred to evade taxes or it's a public fund corruption activity or trading of illegal goods. All of these things require people to put humans to check these things.

**[0:05:51] Krishna Gade:** This is actually a very time-consuming activity. It requires a lot of manual review. I've heard of banks that employ thousands of people to do this activity today. With AI and machine learning, what you get is you You can actually model what is actually a fraudulent activity, or this is actually a legitimate customer that you can trust and acquire and onboard them. And so that's where you can actually use AI and machine learning to automate it and reduce a lot of human labor.

**[0:06:18] Krishna Gade:** You can still have the humans oversee the machine learning models, but the amount of efficiency that you can drive is very, very large.

**[0:06:28] Matthew DeMello:** Right, right, indeed. And to what extent is verifying the customer, not just the KYC compliance and regulations, but just verifying the customer from the minute they walk in the door of the business at the very, very top of the funnel, to what degree is that becoming more important to the rest of the process in terms of AML and making everything kind of more sanitized from a data standpoint if you can verify that customer off the bat?

**[0:06:55] Matthew DeMello:** Where are we seeing that in financial services?

**[0:06:57] Krishna Gade:** Absolutely. I mean, it's not just in financial services. These days, customer due diligence and KYC is so important across the board. If you want to become a host at Airbnb or if you want to be a driver in Uber, you need services, you need to go through a customer due diligence process. And most of these modern tech companies, what they do is they try to use AI and machine learning to, you know, identify, basically use image processing techniques to see if you have uploaded a legitimate, you know, driver license or if your address proof is correct, right?

**[0:07:31] Krishna Gade:** sort of extract information from these things. So that's entity extraction is huge, right? So where I submit my address-proof documents or a driver license and being able to extract my name, my address, my location, and without having a human do it. So there's a lot of work that is involved today in terms of identifying the customer that can be automated with AI and machine learning. And so that's where I think I see a huge opportunity for financial services as well to make some of these modern tech companies and employ AI-based, KYC, AI-based anti-money laundering at the top of the funnel.

**[0:08:09] Krishna Gade:** And so you can still have the humans in the loop, but they can be inspecting the outputs of these AI and machine learning systems to make sure that there is some human oversight, but you can lend a lot of heavy lifting to the underlying machine learning-based system. The other thing with manual process is like it can lead to errors, right? And it can also hurt genuine customers. So you're not only, you know, there's also a problem where, you know, the manual processes actually have inefficiencies.

**[0:08:39] Krishna Gade:** So what AI and Machine Learning can solve for it is also, you know, instead of, you know, reducing expensive process to hire lots of manual reviewers, they can also reduce the errors that people make and make sure that the customer service is top notch.

**[0:08:53] Matthew DeMello:** In terms of the after impact of AI on this workflow, where does human decision making get allocated to or get delegated to in terms of the decisions that are being taken over by AI automation and what's deemed sufficient enough for a human being to handle where their talent will really come out? Absolutely.

**[0:09:17] Krishna Gade:** So there's always, I think we are big believers in human in the loop AI. And I think even though AI will reduce a lot of human burden and increase efficiencies, you would still need humans to inspect the output of AI and make sure that it is still working. That's where we actually as a tool would come in. So we bridge that gap between human and machine. So for example, let's say you have a fraud monitoring system built on AI, and it's actually monitoring the fraud and alerts whenever there is a transaction that is fraudulent.

**[0:09:48] Krishna Gade:** In some cases, you may want to have a human inspect if the transaction was really fraud or not, really understand why the system flagged it as fraud, and make sure that legitimate customers' transactions are not being classified as fraud. Because AI and machine learning systems, although they are very accurate, they're not perfect. So you still need like humans in the loop to make sure that, you know, you know, that, that there's an oversight on how these systems are working.

**[0:10:19] Krishna Gade:** And so that's where tools like Fiddler can help, you know, they can monitor these AI based models. So it's essentially Fiddler is like a watchdog on your AI and it helps you make sure that your AI is working for you. Right. And so you can, you can say that, Hey, you know, is my AI still accurate? You know, is it still catching the right kind of fraudulent activity in my team? You know, it actually does catch fraudulent activity.

**[0:10:44] Krishna Gade:** Why does it actually classify something as fraud? You know, so a human can go through Fiddler dashboards and understand how their fraud AI is working or anti-money laundering AI is working or KYC AI is working.

**[0:10:56] Matthew DeMello:** Indeed, you touched a little bit right there on the black box problem, as you were even saying before. But I'm wondering if we can put this in a financial services context. What do you see as the common challenges, black box, et cetera, with maintaining AI models in production? What problems are banks running into? I know overcoming that data team business leader gap is a big one.

**[0:11:22] Krishna Gade:** Yep. I mean, the banks operate in a regulatory environment. So any new technology that they embrace needs to have regulatory oversight, especially machine learning and AI technology, whether they're using it for fraud use cases or credit scoring use cases, needs to have a human oversight. So they have this process of managing the risk around models called model risk management, where they validate the models even before they launch to production thoroughly. And then they have to monitor these models from a regulatory perspective, as well as an operational perspective.

**[0:11:57] Krishna Gade:** So it's a requirement for them to do that. And where Fiddler helps us, basically helps them create these insights. So these machine learning models are not like traditional software. So unlike traditional software where you can read the code line by line and understand how they work, machine learning models are these arcane artifacts that encode the patterns that they read from data into structures like neural networks, decision trees that are not interpretable by humans. So this is where you need tools that can explain these models, that can help you validate these models, validate assumptions that they're making, so that you can safely launch them to production use cases.

**[0:12:40] Krishna Gade:** And then once they are running in production, the other problem with machine learning models is they drift and deteriorate over time. Unlike static software, machine learning models are dynamic entities. So the model that was trained a year ago or six months ago or a month ago may not be relevant anymore, depending upon the use case. As the data changes, the models need to be retrained and kept intact. And so knowing when to retrain the models, knowing when the model might be deteriorating, knowing if it's actually deteriorating because of the data that we are sending it is so important.

**[0:13:13] Krishna Gade:** So this is where continuous model monitoring is very, very relevant, not just from a regulatory perspective, from an operational perspective. Say, for example, example, you're not monitoring your fraud model. And all of a sudden it starts flagging bad transactions as good. It's going to hurt your customers. Or if it has a lot more false negatives, then it can actually really leak some fraudulent transactions to happen through the bank, which will be worse for the bottom line.

**[0:13:41] Krishna Gade:** And so this is where it's important to monitor these models to make sure that they're doing the right thing, what they're trying to do and see when you need

**[0:13:49] Matthew DeMello:** to retrain them. Indeed. And I know this has to go into a lot of thought for what Fiddler puts into their products, but I'm wondering if we can center this around what non-technical leaders should know ahead of any AI or customer experience program to be prepared to maintain or monitor performance. What should they know? And how does that go into how you plan out the visualization of your dashboards and your products?

**[0:14:19] Krishna Gade:** Yeah, I mean, simple fact is, as I said, AI and machine learning software is a black box. It's not code that your humans can read. Number two is AI and machine learning models are non-static entities, which means they can deteriorate over time. These are the two things I would make sure that every business leader trying to operationalize AI remembers. These two aspects make AI different from traditional software and they mandate monitoring from the word go because it's so important for you to do that.

**[0:14:49] Krishna Gade:** I mean, we have seen companies who have not done model monitoring and had faced tremendous business losses, right? In a recent example is Zillow. Zillow used machine learning models to predict house prices and they did not have proper monitoring in place. And so what has happened is they were overpricing the house prices and it just tanked their business. They suffered losses. They had to lay off thousands of people. So you don't want to put machine learning models into production without oversight.

**[0:15:19] Krishna Gade:** Otherwise, your business can really suffer. And that's something that every business leader needs to remember.

**[0:15:26] Matthew DeMello:** Basically, finding out what's the appropriate role of humans in oversight in order to provide context. Because I don't think we've reached, especially not in any of these use cases, but I don't think we've reached the point with AI where it can large scale provide its own context. So that responsibility becomes more acutely apparent to the human beings responsible for AI every day. Just in speaking of that responsibility, while we have a little time and I know we got through the body of our questions, I saw that you began at Facebook as an engineer in 2016 when they had major black box problems, to say the least, with their news feed and under a lot of media scrutiny.

**[0:16:06] Matthew DeMello:** What did that experience teach you about Blackbox from both sides of the table in terms of the public and both an enterprise organization that needs to clean house in terms of a platform that is public and everybody is trying to figure out how it works?

**[0:16:27] Krishna Gade:** Yeah, absolutely. So I think I spent about a dozen years, more than a dozen years, in fact, working on consumer internet technology companies, all the way from Bing, Microsoft, to Twitter, Pinterest, and Facebook. A common theme in these companies is to use data, large and large amounts of data, and use sophisticated analytics and machine learning technologies to create insights or create recommendations or search results, create a feed of things that users can interact and engage with.

**[0:16:56] Krishna Gade:** And so as AI and machine learning became more and more popular, these companies started indulging in lots and lots of sophisticated machine learning and AI models. And that's what we had seen in Facebook. And so by 2016, by the time when I joined the company, we were using very sophisticated, deep neural networks to rank the newsfeed, to rank what content is going to newsfeed, which recommendations that you would see, what friend stories you would see, what kind of news that you would consume.

**[0:17:27] Krishna Gade:** So in the aftermath of 2016 elections, it became quite important for us that, hey, we really need to understand how these algorithms are working ourselves first so that we know if there is any discrimination happening implicitly, what kind of data elements that they're using to rank content. And so we built tools internally at Facebook called Why Am I Seeing This? that helped us internally as developers to understand how newsfeed algorithms worked. It created transparency within the organization from top-level leadership to every employee.

**[0:18:04] Krishna Gade:** They could scroll through their newsfeed and understand why they were being suggested a particular news story. And if they didn't find the recommendation relevant, they could file a ticket or complaint to the newsfeed quality team, which would then get triaged. and then if there are any bugs that we find in our software, we will go and fix them. So it created this really nice virtuous loop where we were collecting feedback from employees. We were creating transparency around how newsfeed worked within the organization.

**[0:18:31] Krishna Gade:** And we were able to help build better models because of the tools that we developed. And so that's what spurred me to start Fiddler because I felt like, hey, Facebook could build it, Google could build it, Microsoft could build it, but not every organization could build it or may want to build it. And that's where we found the opportunity to fill that gap.

**[0:18:50] Matthew DeMello:** Yeah, you're seeing a parallel just even in that story you were telling about just knowing who users are, just like knowing the customer, at least in banking and financial services context. Knowing the user is the top of the funnel for that process and everything becomes more transparent when you know the users and the news feeds that in the information that they're pulling from everything else kind of trickles down to a pasteurization, if you will, from the top of that funnel process.

**[0:19:20] Matthew DeMello:** So it's really interesting to see that insight gain there. Krishna, this has been a fantastic conversation. We really thank you for being with us this week on the AI and financial services podcast.

**[0:19:31] Krishna Gade:** Thank you, Matt. Great questions and great to be here.

**[0:20:01] Matthew DeMello:** then you're in for some serious black box problems. On behalf of Daniel and the entire team here at Emerj, thanks so much for joining us today and we'll catch you next time on the AI and Financial Services podcast.
