Overcoming AI Hallucinations: Truist’s Chandra Kapireddy

Despite the tendency to hallucinate, generative AI solutions have promise. Hear from one banking executive about how his company uses both traditional and generative AI to enhance operations and to engage customers and employees.

I'm Chandra Kapireddy from Truist, and you're listening to me, myself and AI.

Welcome to Me, myself and AI, a podcast on artificial intelligence and business. Each episode, we introduce you to someone innovating with AI. I'm Sam Ransbotham, professor of analytics at Boston College. I'm also the AI and business strategy guest editor at MIT Sloan Management Review.

And I'm Shervin Khodabandeh, senior partner with BCG and one of the leaders of our AI business. Together, MITSMR and BCG have been researching and publishing on AI since 2017, interviewing hundreds of practitioners and surveying thousands of companies on what it takes to build and to deploy and scale AI capabilities and really transform the way organizations operate.

Hi everyone, we're back for another episode. Thanks for joining us. Today, Shervin and I are talking with Chandra Kapireddy. He's the head of generative AI, machine learning and analytics at Truist Bank. Chandra, thanks for taking the time to talk with us.

Hey Sam, good to see you here, happy to be part of the podcast.

Let's start with Truist and your role. I'm guessing many people likely know of Truist. It's one of the largest financial services companies in the United States, formed in 2019, I think, by the merger of BB T and SunTrust. Chandra, can you explain your role at Truist?

Sure. For those who may not be familiar with Truist, I think, as we introduced, this is a top ten financial services company. We operate in seventeen states, 15 million customers, five thirty billion dollar assets. We offer products and services in consumer banking, wholesale banking, that includes wealth management, corporate investment banking, commercial banking, so pretty much all of the different products and services you can think of from a top ten financial services. I'm the head of AI, machine learning and analytics, as you can introduce at the beginning.

My group broadly does three things. Number one, we provide the AI strategy and drive AI policy for the company. Let's think of that as a first bucket. The second bucket, we build platform and capabilities for data science community. And both from a training the models as well as a serving the models, also the model risk management teams. The third bucket is really developing advanced machine learning and gen AI capabilities, including the applications themselves. Think of this as more like a AI research and cutting edge and next gen kind of solutions for the company.

Those are three areas that my group is accountable for.

Those three areas are pretty large. You've got strategy policy, you've got platforms, you've got advanced research. Can you give us some examples of some initiatives within the bank? How is the bank using artificial intelligence?

We broadly classify AI into two buckets traditional AI and generative AI. Within traditional AI, we have number of AI models that are already in place from fraud detection models, we have some customer segmentation models and so on and so forth. The second bucket is generative AI. We break that into three sub-buckets. Number one, there are models that we can actually build, and that's what we do. There are fine-tuned models we can actually build based off the foundation models.

And the second is using the services, APIs, Azure Open AI is as a service, for example, the bedrock as a service, uh, with the anthropic behind the scene. And the third one is applications themselves, for example, Microsoft three sixty five copilot or an Amazon Connect transcribe application.

Chandra, one of the things that this distinction between generative AI and what you call traditional AI brought in my mind is, you know, Sam and I have been looking at this space for almost a decade. And the fraction of companies that would say that they have a robust AI strategy has steadily increased over time. Now, one thing we saw was companies with the advent of generative AI took a step back and they said, well, maybe we don't really have our strategy as together as we thought we did.

I'm just trying to get your sense on how real that is. And you don't have to comment specifically on Truist if you don't want to, but just sort of generally want to understand from you, from your peer group, folks you talk to, what do you see?

Look, I think AI strategy is something that most financial services companies have been working on and constantly evolving it. I don't think that evolution has slowed down or has not changed. I think it'll continue to happen. Generative AI definitely brought awareness to a broader community. I think for folks who have been doing natural language processing and NLP, they probably knew this kind of innovation would come at some point in the future, but it just happened quickly.

And as a matter of fact, 2017, I think, is when it all changed with attention is all you need and the transformer architectures. At that time, people had access to GPTs. It was not GPT three, it was GPT one, GPT two, and the responses were really not anywhere close to where humans could rely upon. For financial services, as you know, we always have been in the cutting edge, leveraging cutting edge innovations technology to offer better financial services and products to our customers.

We've been watching this field for some time. And so definitely the shift has occurred with generative AI only from the standpoint of bringing the latest and greatest innovations and see how we could replace some of the existing NLP technologies that was built with the limitations we had. I mean, given the accuracy of the models, where we could use gen AI to surpass some of those and bring efficiencies. And so that's kind of where we are. I don't think the strategies are changing because of that.

That could give a perception that, hey, we probably are moving away from AI strategy. When we make statements like that, it's probably meant that we're moving away from relying just on traditional AI to really adding just generative AI capabilities.

And to build on that point, even some leaders that I speak with now are still thinking of these two as very separate domains of capabilities, traditional AI versus gen AI. And I'm curious on your thoughts about the combination of both, right? Do you have any comments on that?

You're absolutely right. I see them as complementary. I think our audience and everybody in the community who is part of machine learning, we all know that all models are wrong, right? Some are useful, and similarly, every gen AI model hallucinates, right? We know that for sure. And the rationale behind whether we should use or not use, uh, sort of depends on the understanding of the degree of uncertainty that really lies with either of those. Imagine a scenario where a customer applying for a credit card, and behind the scene, there's an underwriting model that really either helps with the decision making of approving or rejecting a credit card.

The level of explainability that has to be built into that model is quite humongous. This is where the regulations come into big play and how we test the models. That model was specifically built to either give a score to aid the decision for credit card acceptance or rejections. And that model was supposed to exactly do that and nothing else. But if you take the generative AI, it's really you could use it for multiple use cases, and that's where the challenges are.

Maybe you want to summarize the text, maybe you wanted to extract entities out of the text, maybe you want to get the sentiment of the text. Previously, we used to have individual specific models for generative AI. We could actually use one single model to do that. So with that, there's positives there. We don't have to worry about training multiple individual models. But the downside of it is really, hey, single model is trying to do multiple things.

Can it actually can we rely on it? And how do we make sure that we validate it and test it to ensure that we stay within the accuracy levels that we would expect it to behave? And also responsibly in a way that we're not really making decisions that otherwise would be unfair and not transparent and so on and so forth. And that's kind of where I see there are use cases for both, and both will stay together.

I don't think we'll ever, ever remove traditional AI. We would actually continue to have that while we continue to build on gen AI, and I think those two will actually converge at some point.

I think that's particularly interesting. I mean, you mentioned SR eleven seven, which is a supervision and regulatory letter. That guidance came out for the banking industry maybe fifteen years ago, and that was really before our new wave of artificial intelligence. How do you reconcile a regulatory framework that was developed and put into practice well before the modern developments in models? It seems like a fundamental issue of timing is difficult there.

You're right, Sam, SR eleven seven, as rightly said, it came in 2011. SR eleven seven was all about effective challenge of any model that gets built. It revolves around check for conceptual fit, which is super important whether you use a gen AI model or a non-gen AI model. If you take a call center summary, if a customer calls a call center and really talks about a specific product, and maybe the customer is called to kind of express their dissatisfaction about a particular service and so on and so forth.

How do you make sure that the summary is transcribed? That's a technology we've been using for some time. We've mastered it. But how do you take that call summary and then summarize it in your way that represents the call itself? Previously, we were actually using NLP technologies. Let us say we replace it with gen AI. And how do you make sure we rely upon that summarization piece of gen AI capability? If that model is in place, you would expect to have the same ongoing monitoring, same effective challenge concepts that SR eleven seven dictated, right?

That's why it is if you talk to regulators today, and some regulators really think that, hey, the technology is changing behind the scene, but what we really want all the banks to do from a due diligence and using these models and providing financial services, that's not changing. And so I totally agree with that thinking, and I'm sure you also have seen the NIST AI risk management framework that came into existence. After that, gen AI had gone live, which had GPT, and then NIST had to release a generative AI profile to aid the NIST AI risk management framework.

But if you take all of the NIST AI risk management framework, one would expect that SR eleven seven is still intact in terms of really how you do the model risk management.

Yeah, the framework is there, the skeleton is there, and the pieces may have changed, but the bones are still the same, perhaps.

I want to talk a little bit about people in this equation. We've been talking about AI and technology, and I would say with predictive AI, there was an impact on people, whether it was in underwriting or in customer servicing or in marketing. And the impact was making complex decisions, if you will, or complex analysis easier by processing information or predicting or optimizing that, which is impossible for human to do with billions of data points and all that.

However, I would say the extent of that impact was more limited than what we see with generative AI, where the models, as you said, are doing tasks that naturally were human tasks. Whether it's in, you know, you talked about summarization or understanding the tone or or even logic or planning and some of the complex even analyses or reasoning that humans do. And so the impact on people should be far greater with generative AI in terms of roles and nature of work and skill sets and that sort of thing.

What are your views on that and what are you seeing happen in the industry?

Right, so Sharwin, I think you hit up on a very important point. At Truist, we know that generative AI, as I said, it hallucinates. And I think we know for sure that is the case. If you truly understand the math behind it. Fortunately, we've been able to implement the seeding mechanism. So it actually at least gives the same output no matter how many times you ask the same question. The same question, same output. Previously, it was actually difficult even to get to that.

You ask the same question twice, you get a different answer. So we've come across many hurdles. I think we've made significant progress. At Truist, what we do is we take responsible AI approach. What does it mean? What does it mean? Take all the dimensions of responsible AI from a privacy perspective, explainability perspective, transparency, accountability, safety, and security. So what we did was we codify, we kind of did all of that into a policy. The AI policy is kind of how we drive the conversations at the company, and we translated the AI policy into set of processes.

And understand if you look at how we implement at Truist is really take the life cycle approach. So every actor in the life cycle is made aware of what stages of life cycle exist, what their roles are in each of the stages of life cycle. It starts with ideation. So if somebody in the company has an AI use case, they start with ideation life cycle stage, and then we actually do the risk assessment, the risk tiering, making sure that we truly understand the risk of the use case.

That's kind of the second stage of life cycle. Then we actually begin to the development, testing, independent validation, and implementation and ongoing monitoring. This is seven stages of life cycle. So we take the policy with all the responsible AI dimensions, and we make sure that every single stage of the life cycle, those dimensions are accounted for. And how do we translate that to the actor that has to adhere to it is through the standards. So if somebody is at the ideation stage, they really have to go through ideation standard, make sure they meet all the requirements, and they go to the next stage.

And that's how we implement it. And so the reason why we have taken such an approach is really making sure that whenever we build a gen AI solution, we have to ensure the reliability of it. We have to ensure there is a human in the loop that is absolutely making sure the outputs, especially when it's actually making decisions, which we are not there yet. If you look at the financial services industry, I don't think there is any use case that is actually a customer facing affecting the decisions that we would make without human in the loop as an industry.

When we present those gen AI built applications or driven applications, we will make it clear that, hey, you're interacting with an application that is AI, and we also warn them that it could hallucinate, and so people can take the appropriate actions. The number one, they have to be trained on how to use it. Number two, they know the consequences of relying on that output without you applying their judgment. That's what we do at True is every time we have a gen AI situation, and those things are mandatory steps that we would take.

Hm, hm. I get it that it's absolutely important not to rely on gen AI to make critical decisions, given the state of the art and hence all of the responsible AI framing that you referenced. But I have to believe that not so much of what everybody in a company does is about making decisions, right? I mean, so much of it is also about gathering information or doing analysis without necessarily making decision. In a year or two, when these models get better, when the frameworks get better, what is your view on what will happen to the nature of work effectively in many of these places where gen AI is taking so much of the work?

Maybe even say none of the decision making, but so much of the work that leads to decision making away.

You're absolutely right. So there are lots of productivity type applications that we can actually build using gen AI, and even that productivity based applications, there has to be training given to the teammates on how to use it. A simple examples like copilot to use copilot. If you do not understand how to write a prompt to get an answer, and if you don't understand on how the system would actually behave, there's a limitations on what it can, what it cannot.

Then obviously, either you give up using that capability, or you either use it in the most inefficient way, even though the product could actually do a lot more. And so that's one area where we would actually take that into account. In terms of how do we train employees to use gen AI based applications, because this is a brand new uh technology, and also most teammates who use those gen AI applications are knowledgeable. They're the subject matter experts, and they can figure out when and when not to, with the understanding of the fact that they're interacting with a large language model that could hallucinate.

So the over reliance on that LLM over a period of time is a challenge, but at this point, I think that's probably the least risk based application. We see those happening at Truist as well as, uh, in the industry. I think the next step that we want to take is really how do we ensure that we do it more economically? How do we do it more efficiently? How do we scale it across enterprise? That's where the challenges are for those applications.

So a lot of what you've been talking about is, I would say, operational focused or internally focused on productivity, and that's important. But at the same time, I wonder about the competitive environment and just the business environment that banks are finding themselves in now that now has artificial intelligence. And so you must be at the forefront of seeing a lot of the difficulties that AI has brought. Can you talk to us a little bit about the kinds of things you're seeing change, and, you know, in general, roughly what you're doing about it and how you're thinking about it?

Yeah, I mean, you're absolutely right. So, cybersecurity, right? Gen AI is definitely most favorable tool for folks who really want to take advantage of it to increase their attacks, right? The fake voices, right? I think, you know, and fake invoices, I mean, not just the voices, but also invoices if you have business. But that has been going on for some time, and the only thing that has changed is the sophistication at which those documents are generated, those voices are generated has gone up, and the technologies are out there to constantly upgrade on how to catch those, and we definitely have to take advantage of those.

I'm sure you're all very familiar with the GAN models, the generative adversarial networks model. As they get sophisticated, it becomes extremely challenging to discriminate what is real versus what is fake. And so that's one thing that's obviously an area of focus for us, just like other banks. So that's number one. But number two is really around innovation in the gen AI space. We definitely wanted to take advantage of generative AI capabilities, whether in terms of fine-tuning the existing models out there or leveraging the services and also applications, as we talked earlier.

But if you look at the evolution now, we are now into agentic frameworks, right? Most people, I think, who are unfamiliar with the terminology here, everything to them is a large language model, and everybody refers to ChatGPT. But if you look at the implementation of gen AI capabilities, it's really an orchestration of number of different components, and it's not really a single model that's going to really help us. It's a combination of different services that really have to work together.

So right from when somebody tries to submit a prompt and understanding who that person is, authentication and understanding what this person can do using the LLM, the authorization piece of it, all the way through parsing the input and guardrails on how this person is really interacting with and whether we can actually catch any insider threat, for example, as a case. So those kinds of parsers have to be built, and then how the agents really take all these inputs, and then how we decide on which technologies that agents could use to interact with.

It's not about LLM. If somebody types in a prompt and say, I want to know my checking balance uh for my account, and if you try to use a generative AI to generate the balance, I'm sure the answer is going to be either approximate or completely wrong. And this is not how we're going to build applications. So we're going to have a deterministic API call that agent has to make to get a specific answer for that.

And how do you orchestrate when to use API, when not to use API? That's where the real challenges are. And if we do that, Sam, I think it's the same exact thing for security as well. When the attacks go up, there's a combination of both the generative AI as well as the traditional AI techniques and models that we use to really counter them.

Yeah, I think you're hitting the nail on the head.

I like the example of the balance. It's you don't want the approximate balance, yeah.

The biggest unlock here is the ability to orchestrate across a variety of AI tools so that the machines are working together as well as they can. And on the other hand, to orchestrate the machine human interaction so that the humans also help versus hinder, and they're also helped versus hindered, right?

Exactly. If you look at any implementation of gen AI that is probably working closer to what one would expect, is that careful orchestration of different components that I just kind of alluded to in the beginning. It's not only that, but also the validation of the outputs, right? I mean, at the end of the day, I think we talk about the various RAG patterns. So when somebody types in an input as a prompt, and you obviously contextualize it by extracting information relevant to the prompt, and then you send it to LLM, including making API calls to get more deterministic answers, but LLM responds back to you with an output.

And how do you make sure this output is valid? There's a whole validation techniques that you had to put in place, and then and this is where the guardrails come into play, and sometimes, you know, guardrails are given by default by service providers, but companies like Truist, we build our own guardrails to make sure that the output is thorough and ensure that this output can be trusted and relied upon by the end user. If we are not confident about the output and our response is going to be sorry, we can't answer this question at this time.

That probably is a much better answer than really giving them some output with some probability of really confidence. I think that's an important step that most of the companies that are implementing gen AI at scale is thinking about and considering about. That's where the real challenges are right now. So hopefully we'll get there soon, but as it stands right now, it's really highly speculative at this time.

Chandra, we have a section now where we talk about a series of questions. These are short questions. Just answer with the first thing that comes to your mind. What's the biggest opportunity for artificial intelligence right now?

Right, I would give a little longer answer. Here. There are revenue making opportunities, loss minimization opportunities, teammate productivity opportunities, and operational efficiency opportunities. The sky is the limit on what we can do with AI, and AI is real. To me, I know some folks think this is hype, but it's actually real. The underlying technology is getting better, the data is getting better, the application of the technology is getting better. So I see a huge opportunity across all those domains.

So what's the biggest misconception about AI? Is it that it's all hype? What's the biggest misconception that people have?

Yeah, I think the misconception is the fact that there's not enough evidence that it could work reliably over a longer period of time. The misconception is it may fade away, just like some of the other technologies that had a lot of hype in the past but had not lived up to the expectations. Generally, the misconception is about the lack of sufficient proof that it would actually work over a long period of time.

Hm, what was the first career you wanted?

I wanted to be a movie director.

I was not expecting that.

I know.

What happened? What happened?

I think when I was in India, I was contemplating to either to come to United States to do my master's or to go and join a film technology institute in India and get a formal training on becoming a director. And I couldn't resist, uh, coming to US because of the opportunities that I had and my passion towards database management systems at that time. My plan was to come to US quickly, you know, learn the latest and greatest, but I never went back.

And, uh, you know how it how it happens. I mean, it's not easy to go and become a director at this point. So I had to drop that idea, at least for this life.

Maybe that's next. When is there too much artificial intelligence?

Too much of AI is when it understands you without you explicitly giving permission to it. That's when I kind of feel like this is where too much of AI is happening without all the stakeholders knowing about it.

When it understands you more than you understand yourself, then there's way too much.

What's one thing you wish AI could do that it can't currently do?

To be honest with you, from a corporate world, look at the number of emails that we get, and there's an attempt that's being made by a number of players in prioritizing the emails, summarizing the emails, even taking the meeting minutes and really figuring out actions. We've made a lot of progress in that. I wish we could do a little bit more. That is one area that I definitely see. Number two is really I want to see a day when humans are not driving, and will that will that day ever come, and what's the consequences of that happening?

We still have a ways to go. I know we talk about AGI and ASI. I don't know we are there yet, given how the models work behind the scene. Once we get to that level of uh sophistication, I'm sure AI could do a lot more, as simple as managing your finances, managing filing taxes, and if AI could do my taxes, I could save tons of money, and there are many examples like that.

That's a good segue, because I think we'll probably publish this episode right around tax day. Thank you for taking the time to talk with us today. I think the ideas of orchestration are particularly resonating with me right now. The idea of the coordination involved, not just humans and machines, which is what we talk about a lot, but machines to machines and different types of machines, deterministic, generative machines, and how these all work together. It's not just one piece, lots of pieces to put together.

Thanks for taking the time to talk with us.

Thank you for this conversation. It's been really, really illuminating.

Thanks, Sam. Thanks, Shervin.

Thanks for listening. Next time, Shervin and I speak with Steve Preston, the CEO of Goodwill Industries. Please join us. Thanks for listening to me, myself and AI. Our show is able to continue in large part due to listener support. Your streams and downloads make a big difference. If you have a moment, please consider leaving us an Apple Podcast review or a rating on Spotify, and share our show with others you think might find it interesting and helpful.