Building Better Answers With AI Agents: SAP’s Walter Sun

How is one enterprise resource planning company thinking about language models, both large and small, to benefit its employees and its customers? Find out on today's episode.

I'm Walter Sun from SAP, and you're listening to me, myself and AI.

Welcome to me, myself and AI, a podcast on artificial intelligence and business. Each episode, we introduce you to someone innovating with AI. I'm Sam Ransbotham, professor of analytics at Boston College. I'm also the AI and business strategy guest editor at MIT Sloan Management Review.

And I'm Shervin Khodabandeh, senior partner with BCG and one of the leaders of our AI business. Together, MITSMR and BCG have been researching and publishing on AI since 2017, interviewing hundreds of practitioners and surveying thousands of companies on what it takes to build and to deploy and scale AI capabilities and really transform the way organizations operate.

Hi everyone, thanks for joining us. Today, Shervin and I are talking with Walter Sun. He's a senior vice president and head of AI at SAP. Walter, great to have you.

Yeah, great to be here. Thank you for having me.

Before we start, I wanted the listeners to know that we had a bit of a small world moment when I met Walter. We'll get a little bit into his background later, but we discovered we're both Georgia Tech grads in the same scholarship program. So I'm wearing a Georgia Tech shirt for our recording session to make Walter feel at home. I hope it helps.

Yeah, excellent. Nice to see you representing the the university.

All right, before we get into your background, let's start now. SAP is a European multinational software company based in Germany, largely known for its enterprise resource planning, or ERP. I'm sure we're gonna use that acronym today. So Walter, first, tell us a bit about your current role at SAP.

Sure, thank you for asking. My role entails working and leading the engineering team to build reusable technologies across the different lines of businesses at SAP. As you mentioned, we have ERP, but we also have many other areas of business applications, including like human capital management with success factors. We have a CRM solution, as well as procurement with Ariba. We also have travel, which a lot of people are familiar with, Concur, our travel applications. My team builds technologies, AI technologies in particular, which are reusable by the lines of businesses.

And we also build them for our developers as well through our business technology platform, where we actually make more than thirty different large language models available. We have this Jewel consulting capability, which we released, and it helps consulting partners work with our joint customers on things like implementations and cloud migrations. And so the idea is we effectively uploaded with retrieval augmented generation our help documents, our consulting docs. And once as a consultant, you can ask in natural language questions and get responses.

And that feature I mentioned, it's timely because we're having our annual conference, Sapphire Conference, May nineteenth to twenty first next week. And we'll talk a little bit more about that, as well as other other capabilities of the agents. But that capability makes it easier for people to onboard to our SAP products.

Walter, it seems like you're focused on two use cases for AI, or sort of two themes. There's the internal use, and then there's also the external use, because you have so much presence within so many different organizations. Let's pick one to talk about first. What are you most excited about?

Thank you for asking. I think maybe we'll start with just the fact that we're building access to large language models, but we're not building our own LLM because that area has been well researched and well developed. And so what we feel like we wanna do at SAP is we wanna make the best of the technology and make our customers as productive as possible. What we have is basically a generative AI hub, which is in a business technology platform, as I mentioned.

The hub has access to more than thirty large language models, and our internal developers can access it, as well as external developers. We provide business benchmarks to different cases and then recommend the best language model for one's use case. Imagine if you're telling me I have a QA scenario, it's maybe not super difficult, but it happens 100,000 times a day. Maybe I say, Hey, we have of the thirty five language models we have, these twenty seven likely will give you the right answer.

So we can recommend one of these twenty seven. Of these twenty seven, we probably would sort them by price to make them as inexpensive for you as possible.

Huh? So internally, how has SAP thought about upskilling its workforce to use these technologies?

We basically wanna make sure our employees are equipped to handle models. So in 2023, we had what we called AI days, which we delivered a bunch of sessions across different global units, so people could have time to attend at their reasonable hours. And we kind of explained what large language models were. We had a sandbox internally where people could access large language models in an enterprise way, which basically means, if you may be aware, if you use a public consumer LLM, it's free.

Because the trade off is you're actually allowing them to improve their models on your questions, which is fair for them, but it's not good for a business because you may have secrets you're asking. So we provided this enterprise grade LLM. Many people try and people tried again and again, but millions of instances of access to LLMs, so people could try it out, understand what it's all about. I think it's easy to read the news and see, oh, people are liking LLMs, it does a lot.

But then I think getting your hands dirty, trying a large language model, changing the prompts, realizing the more detailed your prompt is, the better the response. We did training to help people understand that, and it went a long way. Because, as I mentioned from the beginning, my team's job is to provide reusable AI to different product teams. And the more that they use LLMs, the more they know what LLMs can and can't do, so they're asked to us to become more targeted.

Hey, you know, build me an example is, you know, job description creation with a feature we shipped over a year ago, where you use large language models and you just use natural language. Hire me, I wanna hire a new tenure track professor at Boston College with, let's say, you know, research one university, PhD, hopefully maybe a couple years of work experience. And that description can be enough for it to create, like a three paragraph, four paragraph job description using a large language models.

And so people know that LLMs, people in success factors team, that's the team that we built this with. They know that large language models can be verbose and they can be descriptive, and they just need some prompting. And so basically, we put this feature saying we provide a few prompts, and from the few prompts, you can describe something that's very descriptive. It can also join against company or legal guidelines. A lot of companies have policies, you know, disclaimers and terms that need to be in a job description.

And so using both the short description, which is called the prompt, right, along with the meta prompt of whatever the company's legal policy or compliance policies are, you can create a very nice response using a language model. And so the AI days allowed us to educate the lines of business, which has made our partnership a lot more productive in the time since.

You focused about large language models here. I don't think we've talked a lot about small language models. Maybe for listeners, what's the difference in small than large, so we can get some background, and then how does that change?

it's a great question. It's a good segue to talk about agents. So the idea with large language models is the number of parameters a language model has. I tell people when you're in school, you learn how to do a best fit line, and a best fit line, if you remember, is y is equal to ax plus b. Your two parameters are a and b. So you take ten or fifteen data points and you put a best fit line, you solved a two parameter model.

And so as language models started getting bigger and bigger, we're getting to like like hundred millions and then hundreds of millions and then got into the billions. And so now we're looking at like 100 billion parameters, so that's considered large. And so basically, you have, as you can imagine, what we mentioned, best fit line is two parameters. We're talking about 100 billion parameters. There's a lot of free flexibility, so obviously you need trillions and trillions of data points to fit that to build that model.

It's expensive computation wise. So the idea is we're looking at building smaller models now, what we call small language models, which are still like, could be in the hundreds of millions of parameters or so forth. And so people are saying today is small, but you know, ten years ago, that'd still be considered massively big. But these smaller models are tuned and focused on specific tasks, and the value for this is they get the job done with better context and they're cheaper.

So the whole idea of agents, which is a big topic today, AI agents, is that these models are tuned to do a task very well. Let's say you ask a large language model a question, like book me a flight from, I think San Diego in in the Boston area. Let's just say, let's say, for argument's sake, Lexington to Palo Alto. Let's say you want to do Lexington to Palo Alto. A large language model, two or three years ago, a year ago, would just say, Okay, I'm booking you a flight leaving Lexington at eleven zero am.

Eastern time, arriving in Palo Alto at two fifteen Pacific time. And the language model would have hallucinated that because they basically took the internet and said, Okay, there's an airport in Lexington. It's not a commercial airport, I think, but it's a small airport. And there happens to be a private airport in Palo Alto. And so the travel time between those two airports is maybe six and a half hours. So that would work out. It's theoretically possible, and if we had infinite wealth, we would just book a plane and say, let's do that.

Most people say that's hallucinations, but the language model doesn't know better. The language model says that's possible. There's two airports, there's a plane you can book at a high cost and do it. And so that's not useful for most human beings. Most of us, like you and I, probably take mostly commercial flights. We actually know there's a limited number of commercial airports in the world, something like 300 or so. You can put a fine-tuned model that says, Hey, don't just know everything in the world.

There's actually 300 airports in the world that are commercial, and people typically fly from these. So when you ask for a flight from Lexington, Massachusetts to Palo Alto, California, we're looking at the nearest airports. And so it's say, okay, Boston Logan is the source side. And then the destination side is interesting because there's San Francisco Airport, which is close, but San Jose is equally close to Palo Alto. And there's also another airport, Oakland, which is somewhat close as well.

So in this case, these agents would know, offer flights, but it could become a multi turn or iterative conversation where it says, Hey, I can give you a lot of choices, but I still need your input. Because there are actually three airports that you can actually land at, or two, possibly, maybe a third. And so maybe you can tell me, are you happy with flying into SFO, which is possibly the closest? San Jose is sort of close, and then Oakland is close as well.

So one of the fun things about these models is they're smaller models, they're focused and tuned, and they know a lot, but they can still communicate with the human and ask them for iterations. Just the same way, like if you had a travel assistant, they might come back to you and say, Hey, Sam, I found you many flight options, but there are different airports you can fly to. Which of these would you like? And that's where the iteration happens.

Okay, so I think that's a great example. I have to say I was worried when you started because you started with Lexington and Palo Alto. And I didn't realize how deeply you thought through that example to pull out that detail. And I think, you know, that really brings out this idea of when we move to agentic, we're thinking some of the probabilistic answers are perhaps difficult. For example, I think a classic thing is for a student to come in and say, you know, what's what is the theme of this paper?

Or what's a business plan? Those are very general sorts of ideas, and there's not a good reference point for checking those. But in your example, you can make up a flight. That flight didn't exist, and that's going to be a bit of a problem when we start thinking about agents.

yes, yeah, I think it's a great question. And the reason why I use that example is that people, uh, criticize for good reason, language models for hallucinating a lot. But if you think about a language model as maybe a general receptionist or a reference agent at a library, if you walk up to a library and who happens to know a lot about all the books in the library, that'd be great. And ask them a question, you know, they might know generally, like, Hey, tell me about dinosaurs.

Oh, dinosaurs are in this section. Tell me about Apple. Okay, well, there's Apple the fruit, and there's a company called Apple. You just disambiguate, I can send you to the right part of the library. But then if you ask that person on a flight from Lexington to Palo Alto, and you say on the spot, you need to give me an answer, they might not give you the right answer, right? And that's where hallucinations come in, because there's a lack of context.

The background model knows a lot, believe it or not, but it just doesn't know exactly what you want. And so then it kind of gives you an answer. It doesn't know your financial situation. It might think you can buy your own jet or whatever. So it gives you this answer of you can fly from Lexington to Palo Alto. And hey, believe it or not, a lot of facts are correct, right? The duration, the time zones, the viability, the fact that their airports exist, although many people may not know that.

But the reality is, once you add some context, I'm a business traveler that travels commercial. I have to fly commercial flights. I want to just book a flight. You suddenly say, okay, I can actually take a small language model and focus it on flights and answer that question. So you don't need to have a general language model serve each of these AI agents, rather, you have and tune each of these agents to be specialized and then let them communicate with one another.

And this ends up becoming, I think the analogy is great for humans, because each of these agents could be assisting a human persona. You know, there are travel agents out there, and there are travel agents that might be airline specialists. So if you can imagine, you can ask a librarian a general question about booking a flight, he or she will not know exactly what you're looking for. But if you ask a travel agent that question, they'll know.

They might not know where to find the books about dinosaurs or apples or whatever, but they'll know about flights. And so what you do is, instead, you take these large models, you fine tune them to make them small models and create them for different agents. You can have an expert on history, historian agent, and that agent can answer questions about history. You can have an agent that's an expert on the Apple, the fruit. Maybe they're based in I'm in the Washington State area, so, you know, there are probably some experts here, humans.

You can actually have agents learn off of them and build specialized agents. Talk about Apple the fruit, and then you have travel agents, and you have hotel agents, and then you actually have them communicate with one another. In this case, your travel agent, you know, you book your flight to Palo Alto. And then the next step is to actually find out more about, like, where to stay in a hotel. And so you can have the airline agent then communicate with the hotel agent saying, you know, Sam is flying in on Tuesday, May thirteenth, and then flying back Monday, May nineteenth.

Okay, and then he needs a hotel for six nights. And that's where you say, okay, the hotel agent now is getting information from the airline agent and communicating and making a full itinerary sent back to you before the response comes back to you. And that's kind of what we're trying to do with our system, where Jewel is our natural language co pilot that communicates with customers or users. And underneath the hood, the different AI agents, which are built on small language models, can communicate with one another, coordinate a plan, return a result.

Hopefully with minimum hallucinations, by virtue of them being specialized instead of them being generalists. But you also have the idea, retrieval augmented generation, context settings, saying I can, with your permission, if you upload your profile to the system, we can know where you live. You know, we can know where you're visiting Palo Alto, maybe you're visiting something in North Palo Alto, so SFO is closer, or maybe you're living somewhere closer to Mountain View or SJC, so that San Jose airport is closer.

And you can sort of think about that whole idea of, like, the more context the language model gets, the better the response. The same way that, you know, if you and I we just met today, and you know, the more we communicate, we know we both went to the same college, Georgia Tech, so we can talk about scholarship programs or football team or whatever. We have this context and we can communicate better. Whereas if I meet you on the street somewhere and we started talking and didn't know the context, the conversation wouldn't be as rich.

Your example of the librarian and the individual travel agents makes sense to me because I can imagine how much an individual human would know. When I think about this multi agent scenario, we're not constrained by the limits of one individual's knowledge, and a specialized person can go so much deeper than a general person. But isn't some of the promise of the largest models that they incorporate both both the depth and the general?

I think it's difficult. I mean, there are some mixture of expert models, which will say, you know, we know everything, but when you start asking me a question, I can find the right expert for a task. So there are methodologies. I think one argument for smaller agent, and people can argue the opposite, but the reason why AI agents are nice is that you can build a smaller agent. So it's a smaller computational cost. And then they can still perform, at, let's say, ninety five or close to the full language models in the area of expertise, maybe even better, as my example is indicating.

But I think that you can actually specialize the same way that humans, you know, when we all go through college, we pick a major and we specialize, and then you have a company and all of us play different roles. I think having AI agents play different personas roles allow them to be better. Believe it or not, this whole idea of reasoning, this test time, computes what happens. Basically, that's a fancy technical term of, like, you ask a language model a question, instead of saying, give me the answer right away.

It's like, take a minute to think about it. And by thinking, the models can actually reason and see, do I need to get help from another AI agent? Do I need to actually iterate? Do I need a clarification between different airports? Do I need to contact the hotel agent in that one case, or contact something else to see if there are other constraints? As you mentioned, constraint optimizations, all that helps to provide a better answer. A large language model in itself, a vanilla model, which is built out of the box, it has a lot of information and becomes kind of like needle in haystack problem, where you actually have this.

There's so much information that it can give you information, but it might overwhelm you with the response. Whereas if you simply have a question about hotels or about flights, or about Apple the fruit, you don't need to know everything else, right? The large language model, when you ask it about, you know, when do you harvest apples? It might know that, but it knows a billion other things as well. And it might get confused. Where, like it happens, an Apple, the technology company, and it might start answering questions about when they release their products, right?

Harvesting could be like when they release their products in September or whatever. And so that's the whole idea that maybe knowing too much can be difficult in terms of, like, giving you a clear and straight answer. I guess that's that makes sense.

That's good. The other part I think about that is with small language models and the iteration and retraining time. You know, when we talk about these large language models, we're talking about literally months and tons of power to recompute them. You know, objectively speaking, the small language models, by definition, have to be faster and less performant. So then it seems like you could iterate better.

That's right, absolutely right. You can improve, iterate, and also fine tune. One thing I like about these models is each company has a different policy and preferred partners, and so you can fine tune these models with actual data from a company. Let's say we have this travel agent model, and then each company deploys this model, this agent. And then, you know, it's doing reinforcement learning. It's learning in terms of how to be a better travel agent for this company, maybe even for an individual, longer term, if there's enough data samples.

And so a company maybe has preferred airlines, preferred hotels, policies against flying business, or policies on not flying red eyes or minimizing connections. The model will learn that, say, okay, I have a general model which tells you how to fly from airport A to airport B anytime you want. Over time, I'm learning this company's employees always pick no red eyes, okay? So if that's the case, then I'll lower the ranking of red eye flights and prefer just daytime flights.

And so the type of things, like you said, you can iterate faster with a lower cost of retraining, and you can do it very quickly. Whereas these very large models, you can't really just retrain them for one company or something without it being a very expensive cost.

Yeah, I think it's going to be a a challenge we're all going to face in the future as we use more agents is to figure out what the objective criteria of that agent is.

Yeah, I think it's interesting. I think that the fact remains the same with web pages, right, and websites and domain search engines, everything we do. I think part of the reason that online companies or software providers, even like SAP, we need to develop, give trust to our customers and our users, right? Over time, you build trust and say, hey, we don't have a ulterior motive. Our goal is to do what's best for you. And it takes time to build that trust.

And same with web pages. Sometimes you go to websites and domains, and maybe they show certain sources of information and not others. And you have to learn over time. Okay, if I'm going to this domain, I have to take with a grain of salt. I'm getting data which is leaning a certain way or the other way, right? Or if I'm looking at certain companies which have other partnerships, I go to a travel agency and they're owned partially by another specific airline or something.

Then you have to take with a grain of salt the results.

And it seems like, you know, to tie back to SAP, maybe a little bit stronger, you know, if I think about SAP, and the strength of SAP is, honestly, just a massive amount of tabular data.

Yes, the more we do, the better we can get correct answers. I think there's the other aspect you mentioned predicting, which I like that topic, because language models, in fact, aren't good at math. You know that there's all these stories about the fails they had, and so a vanilla language model. And of course, people argue that these AI agents, which use reasoning now, are able to solve, like Duo and like Olympiad problems now. So that's true, but the language model when they're built, they learn from the internet, and they learn about language, and so they're very good at language.

And they predict the next word in a sentence. So you say, I want to go to the store and buy some blank, Apple and then it can write a story and say, buy milk or apples, or whatever. And it can go through, and it's all predictive. And it says, Okay, what do people on the internet say? What they want when they say that? But people don't have a lot of tabular data for privacy reasons. Here's my bank account, here's my eight transactions, or I've made these fifteen credit card purchases at the grocery store with the sixteenth.

This information isn't generally available, so language models aren't trained to do that. And so what we've done at SAP, given the breadth of applications and access to PII, basically privacy data removed, we actually can train off of a larger model set. I'll give an example, everyone understands the reason, and then we'll talk about how this can work for more subtle cases. Like, if I take a look at all retailers in the North American area and transactions, I can come up with a conclusion that somehow in late November to mid December, or late December, purchases are a lot higher.

There are a lot more purchases that happen every year, somehow around some late November date. It changes a little bit every year, but it's give or take within a seven day window in late November into late December. And so that information doesn't give away secrets from any one company, rather, it's the retail business in North America. And so if every company is opting in and we've anonymized the data, we don't need to have individual companies data because this is a pattern which works across the industry.

And in fact, we don't want individual companies data providing patterns because that's not useful for all of our customers. And so the example I'm giving obviously is Black Friday to Christmas, but you can imagine more subtle things. Like, say, I noticed that on Thursdays in the South China Sea, shipments are always delayed. And that can only be determined if you have a bunch of data from a bunch of different customers, right? If you have one shipper, that shipper might not work on Thursdays, right?

But if you have hundred shippers and all hundred shippers have some reasons of delays in the South China Sea on Thursdays, you know, I'm making this up. But, you know, it might be that there's some reason, maybe the government's approving imports, exports, or there's reasons that there are delays. And so it's good for a company to know the value of this tabular foundation model that we're building is to help provide insights there. And so I mentioned earlier, we're not building a large language model, but we're actually building a tabular foundation model to complement the large language model.

Using the same transformer based technologies that are used by LLMs for this tabular model, and this model is focused on business data and predictions. I'll go back to the last thought about predictions, is that predictions in the future don't have to be perfectly accurate. I understand the point that, you know, facts about the past, we can't make errors. But one thing that we want to do with this model is also help you with forecasting, you know, in this case of Thursdays, predict that in the future Thursdays, there may be some disruption in the South China Sea.

Those things hopefully are useful for businesses as they understand they're still predictions and not facts.

Our show is me, myself and AI. And obviously, we know that you're brilliant from your Georgia Tech background, but tell us a bit more about the rest of your life between Georgia Tech and currently. What other things have you done?

Sure, all right, yeah, my background. I wanted to get some experience before my PhD, so I did my master's degree at MIT. And then I took a leave of absence for a few years to work and get some experience. And I did some work in Wall Street. I worked at BlackRock doing quant work on fixed income. And then I went to the West Coast and worked at Apple for a little bit, working on some of the early versions of QuickTime, which is now integrated into, like, you know, iTunes and other products.

And then I worked at a startup company called Twi, where we built one of the world's first consumer broadband routers at home. And so then I went back to get my PhD. And my PhD, I focused on image processing and the stochastics or the uncertainties behind it. You had talked about probabilities. I really like probabilities because the world's a very uncertain place. So being able to understand and model it as best as possible, maybe not perfectly, because we can't predict the future perfectly, but modeling in a way that can prepare people like supply chain agents or retailers, the right things, that's valuable.

So I spent a few years doing that, and then I joined Microsoft well, twenty years ago now, I guess, and worked on a variety of products from the Windows based products for the codecs, which is basically encoders and decoders for video and audio, to later working in Bing for many years. I actually started a new product called Bing Predicts, which was to use big data to do predictive analytics, and I think that area was very interesting to me.

Much like we're doing now, everyone knows that data can be used as a fuel for AI. We're actually saying, hey, you can take a lot of searches and glean patterns. I mentioned this example about the Thursday delays in South China Sea, those type of things. We're actually predicting political affiliations based off of aggregated and anonymized web search traffic. So much the same way, the analogy of the tabular model today not being. We can do it without looking at individual companies data, we can actually look at people's preferences without looking at individual searches or individual data.

And so that's kind of the thing that I was doing at Microsoft as well, kind of looking at how to do predictive analytics with big data. And then I joined SAP a couple years ago and have been working on building out generative AI for our customers and our products.

Well, it's been a fascinating discussion. I think that really struck me the depth of your knowledge about how all these pieces can work together in an ecosystem of enterprise resource planning. That, on the one hand, has many customers and is balancing the difficulties of vendors and the changing technology. It's a difficult place to be, but I've enjoyed talking to you. Thanks.

Yeah, thank you, and I've really enjoyed it too. I think that part of the reason that we are passionate at SAP about building these technologies is that, like you said, we have a wide base of different technologies that we provide for our customers. And by virtue of having that base, we can actually try to understand across the board. You know, we have a knowledge graph that kind of connects different applications together, and we want to have this dual AI agent connect with the AI agents across each line of business in order to best provide the best answers for our users.

Thanks for listening, everyone. Next time, Sam and I speak with Josh Weinert, senior vice president of consumer engagement and analytics at CVS Health. Please join us. Thanks for listening to me, myself and AI. Our show is able to continue in large part due to listener support. Your streams and downloads make a big difference. If you have a moment, please consider leaving us an Apple Podcast review or a rating on Spotify and share our show with others you think might find it interesting and helpful.