# GenAI risks and global adoption - **Show:** Practical AI - **Published:** 27 Aug 2025 - **Source:** https://share.transistor.fm/s/522b45c8 - **Transcript:** https://podcasts.opennash.com/shows/practical-ai/genai-risks-and-global-adoption-63dfac/ - **Provider:** OpenNash Podcasts (machine-generated) ## In brief Companies adopting generative AI need both testing before launch and monitoring after launch, and the guests argue the two should share the same measurements. Citadel AI's founders say risks like hallucinations, toxicity and prompt injection are best handled by metrics that mimic human judgment, first used in testing and then reused as live guardrails, especially in cautious large enterprises. The central tension is between fast adoption and reputational or regulatory risk. Japan is described as behind the US in building models but eager in using them, with banks and insurers rushing into contact-center chatbots while fearing government punishment and brand damage. The hard technical problem is that generative AI has no ground truth, so another AI model must be calibrated against human reviewers. ## Key insights 1. **Japan leads in using AI, not building it** Rick Kobayashi says Japan trails the US by one to two years in model development but is among the most advanced in deploying generative AI applications. He ties this to cultural familiarity with friendly robots in animation such as Doraemon. 2. **Large enterprises fear reputation damage more than hacking** Japanese firms worry most about reputational risk from chatbots behaving badly, particularly in regulated finance where regulators could scold or punish them. Startups launch quickly because they have no brand to protect. 3. **AI judges must be calibrated against humans** Because generative AI lacks ground truth, customers use an LLM as a judge. They first have people annotate roughly 50 to 100 examples, then tune the automated metric until it matches those human ratings. 4. **Testing alone cannot replace live monitoring** Real users behave unpredictably and some deliberately attack with new prompt injection tricks that testing missed. Monitoring measures real answer quality and guardrails filter attacks as conditions change. 5. **Citadel pivoted from adversarial attacks to monitoring** Founded at the end of 2020, the company first focused on adversarial image attacks but found companies cared little commercially. It shifted to observability and testing of predictive models, then to large language model testing as interest grew around two years ago. ## Questions this episode answers ### Why is Japan considered a fast adopter of generative AI? Rick Kobayashi says Japan is behind the US in developing foundation models but among the most advanced in using them. He suggests that cultural familiarity with friendly robots in animation such as Doraemon lowers hesitation to talk with chatbots. ### What are the main risks companies worry about when launching a chatbot? Kenny Song lists hallucinations (made-up answers), toxicity, and users trying prompt injection, which means tricking the system with crafted inputs. Rick adds that Japanese firms worry most about reputational damage and regulator reaction. ### How do you measure whether a generative AI answer is good? There is no single correct answer to compare against, so teams use another AI model as a judge with evaluation prompts. They check it against a small set of human ratings, around 50 to 100, until it matches human judgment. ### If a chatbot passes testing, why does it still need monitoring? Real customers behave in ways testers did not foresee, including creative attack attempts, and conditions change over time. Monitoring checks actual answer quality and can block bad inputs live. ### What is Citadel AI and what does it sell? Citadel AI is a company co-founded by Rick Kobayashi and Kenny Song that builds software to test, monitor and govern AI systems. Its products include Citadel Lens, which now combines testing and monitoring, and the open source LangCheck library of text-evaluation metrics. ## Facts - **Rick Kobayashi** [0:03:11]: "there are lots of, say, robot, or some of the say animations, which specify some of the AI or robot technologies. And, you know, Doraemon, or there are lots of animation over there." - **Rick Kobayashi** [0:05:13]: "So they say that a soft approach, so there is no strict say regulation in Japan, but there are lots of kind of guidance in Japan." - **Kenny Song** [0:08:36]: "So what we do at Citadel is we build software tools to help organizations test, monitor, and govern their AI systems." - **Rick Kobayashi** [0:13:14]: "Kenny is worked in, uh, uh, Google brains, and he was one of the core member of the TensorFlow team." - **Kenny Song** [0:14:37]: "when we started the company, it was at the end of 2020, um, so it was before the era of large language models." - **Kenny Song** [0:16:15]: "So in 2017 to 2020, I was working at Google Brain as a product manager, and, uh, my team was responsible for building machine learning infrastructure at Google." - **Kenny Song** [0:16:15]: "this included TensorFlow and also other platforms like TFX, TensorFlow Extended, and some of the work on Google Cloud's AI platforms and so on." - **Kenny Song** [0:17:35]: "when we, uh, started the company originally, we focused, um, a lot on adversarial attacks. Actually, back in 2020, since it was a very hot research topic." - **Kenny Song** [0:18:10]: "We sort of found that after a few months, um, this wasn't really a problem that companies were interested in." - **Kenny Song** [0:14:37]: "And about two years ago, we started getting a lot more interest in our from our customers in LLMs." - **Rick Kobayashi** [0:18:57]: "One of the largest customers in in in Japan is the financial industry, surprisingly so, like banks or insurance companies or security companies are very aggressive to introduce LLM applications into their core operations." - **Rick Kobayashi** [0:18:57]: "So because those financial industry is governed by the the government itself, so in that sense, different from regulation related AI, they have already been kind of regulated by traditional, uh, say, financial regulations." - **Kenny Song** [0:26:25]: "So you can't measure accuracy, you can't measure precision or recall." - **Kenny Song** [0:26:57]: "basically, um, using LLM as a judge, um, so you can craft these evaluation prompts that ask an LLM to evaluate some quality of some generated text." - **Kenny Song** [0:30:08]: "So maybe like 50 to 100 of these manual annotations. And from there, you can design LLM automated metrics and measure their correlation and accuracy against a human judgment." - **Rick Kobayashi** [0:31:44]: "now we are say merging radar UH features into lens, so lens can provide both testing function and the monitoring functions together at this stage." - **Kenny Song** [0:37:12]: "LangCheck is a is our open source Python library, um, that contains, um, a a suite of metrics that are built in that you can use for evaluating the quality of text." - **Kenny Song** [0:37:12]: "I think we launched it in, uh, October of of 2023, roughly." - **Kenny Song** [0:37:44]: "we work with a lot of customers that have Japanese text, or, um, Chinese or German, and and these other languages." - **Kenny Song** [0:27:33]: "Um, you could do that with traditional, um, like a sentiment classifier as well, um. But there are more sophisticated metrics, such as, you know, uh, detecting hallucinations against some ground truth document" - **Kenny Song** [0:27:33]: "So if you have, like, a refund chatbot, you can design a metric that, uh, measures if the chatbot adheres to your, your company's refund policy." - **Kenny Song** [0:40:54]: "I really loved O3 as a model in in ChatGpt, and I love using cursor, and I'm excited for these tools to become more and more agentic over time." - **Rick Kobayashi** [0:39:13]: "yeah, now the Chatgpt five is released." - **Kenny Song** [0:41:21]: "Originally it was just like three, five and GPT four, that, uh, just answer a question based on a forward pass of the model." ## Transcript **[0:00:03] Ad read:** Welcome to the Practical AI podcast, where we break down the real world applications of artificial intelligence and how it's shaping the way we live, work and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn X or Bluesky to stay up to date with episode drops, behind the scenes content, and AI insights. **[0:00:33] Ad read:** You can learn more at practicalai. fm. Now on to the show. **[0:00:48] Daniel Whitenack:** Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard, and I am joined as always by my co host, Chris Benson, who is a principal AI research engineer at Lockheed Martin. How you doing, Chris? **[0:01:04] Chris Benson:** Hey, doing great today, Daniel, how's it going with you? **[0:01:07] Daniel Whitenack:** It's going great. I traveled a bit over the weekend to run a a a mini marathon, which was fun, but I am back safe at home. And, uh, of course, safety is is something that hopefully we can talk a little bit about today. With our, with our guests, who are, uh, Rick Kobayashi, who is co founder and CEO at Citadel AI, and Kenny Song, who is co founder and CTO at Citadel AI. Welcome. **[0:01:34] Kenny Song:** Thank you for having us. Yeah, thank you. Well, **[0:01:36] Daniel Whitenack:** Yeah, it's great to have you both here. Um, good to get introduced to you both. I'm I'm really excited about this conversation. Because, of course, security, uh, and safety related to AI is is very close to my own work, and so it's it's always great to connect, uh, connect with people in this space. And yeah, I, I'm also interested to hear a little bit. And I, I even saw some things on your on your website, you know, talking a little bit about some of the AI in Japan, some of the, I guess, regulations or guidelines for for businesses that have come out there. **[0:02:12] Daniel Whitenack:** So maybe before we get into the the nature of of what you're building and what you're doing. Any thoughts on? For those out there that might be in the US. or in Europe? And and be constantly exposed to the things that are going on in AI in those jurisdictions? Any thoughts about you know what is the same, or what stands out about what's happening with AI in Japan? **[0:02:37] Rick Kobayashi:** so, yeah, anyway, thank you very much for joining us. Uh, and uh, uh, my name is Rick. And uh. As for the Japanese conditions, basically, I'm afraid to say it's around one, two years behind the US. situation, basically up to now. But, uh, on the other hand, surprisingly, as for Gen AI, I think that Japan is one of the most advanced countries, not the development of the foundation models, but the utilization of the Gen AI applications. **[0:03:11] Rick Kobayashi:** And probably it might be related to kind of Japanese, kind of animation or cartoon. So, you know, like, there are lots of, say, robot, or some of the say animations, which specify some of the AI or robot technologies. And, you know, Doraemon, or there are lots of animation over there. And people are very, say, getting familiar with such a kind of a say, talking with robot or some of the say advanced computers from their childhood. So in that sense, I think that many people do not have any hesitation to work with or talk with like a chatbot or any Gen AI. **[0:03:52] Rick Kobayashi:** So in that sense, the hurdle to introduce LLM or Gen AI in Japan is very low in that compared with, uh, US. or other other countries. So, in that sense, development side or technology side, frankly speaking, uh, Japan is behind the US. situation. But as for the uh usage of such kind of Gen AI, I think I hope, uh, Japan is one of the most advanced countries. **[0:04:17] Daniel Whitenack:** Yeah, that's, that's really interesting to hear the sort of perception, uh, side of things, um, and of course, with more, uh, maybe even more adoption or adoption. That's ahead of maybe other places in the world. It could be that, you know, users or usage of the technology has hit some, uh, bumps along the road, or or has hit some problems. Um, which I, I, I know is is a lot of what you all are are working on. **[0:04:49] Daniel Whitenack:** What has been the situation in Japan around kind of regulation and business usage of of AI? Is it, is it on the sort of more regulated side or less regulated side in terms of the the government? And and I guess, uh, regul regulation in terms of security or safety, privacy, these sorts of things, **[0:05:13] Rick Kobayashi:** I think Japan is in the middle between the EU and the US. situation. So they say that a soft approach, so there is no strict say regulation in Japan, but there are lots of kind of guidance in Japan. And people are, say, have already noticed some of the issues, I mean, security side or safety side of AI. And on top of that, because they are say trying to introduce LLM applications for like, uh, uh, contact center applications. **[0:05:47] Rick Kobayashi:** So they are concerned about more, say, kind of reputational risks rather than security risks. So in that sense, uh, they understood the importance of the safety of the AI or trustworthy of the AI, but more say, concerned about reputational risk rather than security risks. **[0:06:08] Chris Benson:** Uh, I'm curious, as we're, as we're kind of talking about the adoption and the environment around that is, you kind of mentioned that that in Japan, it's very, you know, people are really getting into, uh, the utilization of LLMs and generative AI. Do you have any any, uh, kind of thoughts around? You know, what is it? a, what is it that's driving that, you know, compared to other countries that you've observed? If you're looking at that kind of adoption in Japan versus the US. **[0:06:36] Chris Benson:** or versus Europe or whatever, um, any thoughts around that? Because I, I had, I had observed that as as, uh, someone in the US. that Japan seemed to be, uh, implementing. And I was kind of curious if there was, uh, you know, what was the what was kind of the driving force there? That that got the utilization up, uh, especially a among just, you know, your typical, average, everyday folks, they are not, you know, outside of the AI industry. **[0:06:59] Chris Benson:** Uh, would love your thoughts about that. **[0:07:01] Rick Kobayashi:** I'm not so sure the the background reasons why, uh, Japan is more aggressive to introduce such kind of LLM applications. Apparently, as I said, people has lower, say, risk of the per perception, I may say. In some cases in Japan, like a chatbot or kind of say, new technology might be a kind of friend in that sense. Uh, in US side, such a kind of a new AI is kind of an enemy of a human. **[0:07:33] Chris Benson:** yeah, so there kind of a a little bit of a cultural difference in terms of how that that openness to adoption and stuff. So, yeah, that would make sense, that would make sense to me. **[0:07:43] Daniel Whitenack:** yeah. And I guess sometimes your your friends can potentially hurt you, even if they, even if they're not not trying, which I know. Um, you, you all are are kind of involved in both, you know, the evaluation of these systems, you know, running metrics against these systems and kind of building trust. Um, in in a very real way, Kenny, I'm wondering if we bring you in here. And and maybe just, you know, safety and security around AI is such a broad topic now, with so many people addressing it from so many different perspectives. **[0:08:20] Daniel Whitenack:** I'm wondering if you could help us zero in on on the may. Maybe the kinds of problems that you all are are are exploring, and how those fit kind of more generally into the landscape of, I guess, security risks or or threats as related to to AI. **[0:08:36] Kenny Song:** Definitely, yeah, so thank you for having us on the podcast. My name is Kenny, I'm the co founder and CTO of Citadel AI. So what we do at Citadel is we build software tools to help organizations test, monitor, and govern their AI systems. And in the world of LLMs, when customers come to us, they usually have some proof of concept that they've been developing internally. So they've been using some foundation model, they're building some chatbot or some agentic workflow. **[0:09:06] Kenny Song:** And they come to us when they want to make that proof of concept production ready. Um, and um, usually their main problem is they want to mitigate some of the risks that they see right. So things like hallucinations, um, toxicity, or users trying to prompt inject the system, um, they're. They're looking for a solution to these types of problems, um, and that's where we come in, um, as a tool provider. **[0:09:30] Daniel Whitenack:** Yeah, and how does that? I I guess? How are you seeing? When you're interacting with your customers or they're coming to you with these problems? What is what is the impact of those of those problems, like hallucination or injection? Is that something that is causing real problems, or something that they've sort of heard about that they're concerned about, but it's maybe not causing a real problem? Or, y, you know, what are you seeing there in terms of, I guess, the the impact? **[0:09:59] Daniel Whitenack:** Let's say I just wanna throw caution to the wind and ignore these things. What's what's the bad side of this that could happen? Um, were I to kind of take that more, uh, more loose approach, any thoughts? **[0:10:13] Kenny Song:** Yeah, I think it's usually a mix of both, um, and it also depends on the the sort of the risk appetite of the company, um, that's developing the system. And typically, our customers are larger enterprise companies, both inside of Japan and outside of Japan, um, and so they have very mature risk management practices. And before they, you know, launch these POCs into a production service, whether that's internal facing or external facing. Um, they want to make sure that they, um, you know, have appropriate controls in place to manage the risk, and they've, um, cr properly identified the potential risks, reputational, um, data security and and so on. **[0:10:54] Kenny Song:** So, yeah, I, I, I think for the customers we talk to, um, it's generally a big concern for them. And they come to us with the UM problem of mitigating some of these risks that they've already identified. **[0:11:08] Chris Benson:** Uh, I'm curious, as as you've kind of identified, you know what, that kind of the strata of customers that you're looking at there, what, what is kind of? Is there? Is there something that definitively kind of separates that kind of mature, larger organization from some of the smaller ones? Do they have a different set of problems that they're coping with? Um, or maybe just haven't gotten far enough along, uh, in terms of maturity and risk management. **[0:11:36] Chris Benson:** Um, what, like, at what point do you see the uptake kind of falling off within maybe smaller organizations? You know, what's what does that look like as you as you move from that upper strata into the mid tier? **[0:11:48] Kenny Song:** I I think my my general perspective is that there there's two ends of the spectrum of like, startups to enterprise company, and for startups. Um, the the risk is fairly low, cause you don't have a brand to protect, you don't have an existing business to think about. Um, and so you can just launch these POCs out to your customers very quickly, and you can iterate quickly. For enterprise customers, uh, they tend to be more guarded, uh, they they have a closer eye on potential risks to the business. **[0:12:18] Kenny Song:** Um, and uh, those are the customers that sort of really want pretty robust testing before deployment, and also monitoring and potentially real time guardrails, sort of filtering, um, after deployment. Yeah, I think it depends a bit on the size of the company. **[0:12:35] Daniel Whitenack:** Yeah, I, I'm wondering with that, like, as you're kind of looking to those types of organizations, I'm I'm curious a little bit of the backstory of of Citadel. Um, m. Maybe Rick, you could tell us a little bit of kind of how you came to the these problems. Cause you seem like, um, you, you have customers now, you've developed some of these things. W When did that happen? How early was that in? Kind of? Was that before gen AI was that, you know, as gen AI was coming up? **[0:13:04] Daniel Whitenack:** And and how did things develop in terms of your your thinking around these problems? And how you would bring something to the to the market in terms of addressing them? **[0:13:14] Rick Kobayashi:** uh. So, in that sense, it was before Gen AI. And actually, the person who came up with such a kind of a idea is not me, but Kenny. So Kenny is worked in, uh, uh, Google brains, and he was one of the core member of the TensorFlow team. So he's taking a leadership to develop, uh, most advanced AI technologies in Google. And found that there should be a lots of risk about the trustworthiness or say safety of the AI. **[0:13:45] Rick Kobayashi:** And, uh, he believes that probably he is the best person to explain by himself. But but anyway, he reached the idea that, uh, uh, uh. Some of the tools which protect such a kind of a safety or security issues would be, say, should be become popular, especially in the enterprise companies where they may not have, say, a bunch of the uh, ai engineers inside. So that is a background. And, uh, our company name Citadel itself is, say, shown the our what we are focusing, so Citadel is, as you know, say, like a vault or castle. **[0:14:22] Rick Kobayashi:** So we are the company to say protect such a kind of risk, I mean human from AI risk, such kind of a say concept is, say, uh, based on our uh with our company name. **[0:14:35] Daniel Whitenack:** Yeah, anything to add there, Kenny? **[0:14:37] Kenny Song:** Yeah, that that's a pretty good overview of the background. Uh, when we started the company, it was at the end of 2020, um, so it was before the era of large language models. And we were initially really focused on helping organizations monitor their traditional predictive AI models, so tabular models, vision models, that kind of thing. And about two years ago, we started getting a lot more interest in our from our customers in LLMs. And how do we, you know, reliably test this new technology? **[0:15:08] Kenny Song:** How do we integrate it into our workflows and our, our business applications? And so these days, I think a lot of our new customers come to us with, um, LLM types of of questions, and then we still have a lot of existing customers that work more on the predictive AI side. **[0:15:25] Chris Benson:** So, Kenny, I'd like to to follow up. As you guys were were kind of developing the idea, going back to that a moment for the company. Um, and Rick mentioned that you had been there at, uh, Google Brain and that you were on the TensorFlow team. I'm, I'm curious, um, uh, ha, as a, you know, for me, as a as a someone who's used Tensorflow a bit. Um, and and so that was like, oh, wow, you know, one of the people who who helped put that together. **[0:15:52] Chris Benson:** I'm curious what parts of the experiences you had there led into the formation of Citadel. In your mind is is, uh, as you had some of the the insights that you might have developed in your previous employment. Did did any of that? W You know, how? How did that lead to what Citadel does? Um, how did that kind of bring forward when you and Rick started the company up? **[0:16:15] Kenny Song:** Sure, so, um, a bit more about my personal background. So in 2017 to 2020, I was working at Google Brain as a product manager, and, uh, my team was responsible for building machine learning infrastructure at Google. Um, so this included TensorFlow and also other platforms like TFX, TensorFlow Extended, and some of the work on Google Cloud's AI platforms and so on. So sort of the software foundations that power a lot of Google's machine learning applications. And I think Google tends to be pretty ahead of the curve, um, in AI adoption, and I think back then it was more called machine learning rather than AI. **[0:16:54] Kenny Song:** But basically, what we worked on was, um, making models at Google more reliable at Google production scale. And that usually meant building, um, these pretty sophisticated pipelines that not only target training the models, but also serving them in production. Um, so we had pretty robust um systems for monitoring data drift and validating data that enters production models, um, monitors the output of these models. And, you know, at Google, you can afford to have hundreds of platform engineers build out this kind of infrastructure to use internally, um, but for most other types of organizations, they can't make that level of investment in internal platforms. **[0:17:35] Kenny Song:** And, uh, we felt that there was an opportunity to, um, sort of build some of these model monitoring and data validation tools, um, for other companies, uh, and so that's where the idea of Citadel AI kind of started. And when we, uh, started the company originally, we focused, um, a lot on adversarial attacks. Actually, back in 2020, since it was a very hot research topic. So basically, you know, designing noise and images and and other types of input data to trick models into making the wrong predictions. **[0:18:10] Kenny Song:** We sort of found that after a few months, um, this wasn't really a problem that companies were interested in. It's like a very interesting research problem, but less interesting commercially. Um, so after that, we pivoted towards more observability and testing of these predictive models, and these days we focus a lot more on LLM testing and monitoring. **[0:18:29] Daniel Whitenack:** And maybe you could just give us, you know, G going going back to you, Rick, maybe you could just give us a few examples. That stand out of kinds of companies and the kinds of models that they're running and the way that or or the some of the things that they would be interested in tracking or detecting or observing. Uh, a few exam concrete examples might help our our audience kind of grasp some of the the cases that you're working with. **[0:18:57] Rick Kobayashi:** One of the largest customers in in in Japan is the financial industry, surprisingly so, like banks or insurance companies or security companies are very aggressive to introduce LLM applications into their core operations. Uh, such as like a contact center or internal applications. And, uh, when they say they can go through POC stage, and when they get into, uh, commercial operations, they start caring about some of the kind of safety issues. So because those financial industry is governed by the the government itself, so in that sense, different from regulation related AI, they have already been kind of regulated by traditional, uh, say, financial regulations. **[0:19:53] Rick Kobayashi:** So if, say, LLM applications, uh, behave badly, that damage their core businesses. So to introduce such a kind of a AI application is to differentiate or or advance their services compared with their competitors. But by introducing new technologies, if their core damage, uh, core businesses damages, that would be very, say, huge negative impact. So, and also, uh, if the AI behave badly, as I said, their say. The Japanese government may say, in some cases, scold or punish them. **[0:20:35] Rick Kobayashi:** So th that is one of the most largest risks for them to introduce, uh, LLM Gen AI applications. So what we are doing is to making sure that such kind of a say bad behavior, uh, will not happen in the application or not, and making sure that they can safely introduce, uh, say, go into commercial operation. So, uh, some of the leading the bank company or insurance company is our say our customers. **[0:21:05] Chris Benson:** Gotcha, I'm curious if I could follow up on that for a 2nd with when you're. We've kinda talked about the the notion of risk management in general, that that industry is is dealing with. How do they, given the risks, uh, for a large, you know, organization to implement LLMs? Do you do you have any insight into how they're making the J the risk management judgment? Um, on the benefits of implementing maybe a new chatbot versus the potential downside, where, you know, as you pointed out, could be damage to brand, damage to core operations, damage from a regulatory standpoint. **[0:21:42] Chris Benson:** That's like, as I'm listening to you, that does sound very risky, you know, from that perspective, how do they make? How do they evaluate that? Like, say that they know that they wanna come to Citadel and get that help. What do you have a sense from your customers? What a typical balance on that is of the benefit of the LLM utilization versus the downside of if things go off. How do they? What are they thinking when they come to you in that way? **[0:22:07] Rick Kobayashi:** So, in that sense, they fully understand that balancing is very important. So to protect the risk, uh, so much may say, uh, delay their say the service advancement. On the other hand, say, uh, without having any security or safety, uh test, uh, they may get into bad situation. So they need to how to balance such a kind of risks and benefit. And, uh, they have or set up several like, uh, internal organization to manage such a kind of a risk inside. **[0:22:44] Rick Kobayashi:** So, yeah, as you can easily imagine, financial industries come, they they are very structured company, so there are some of like a risk management department inside. And, uh, in some cases, uh, there are like a AI governance team inside or something like that. So they jointly work together and try to manage, uh, such a kind of a risk. And at the same time, they try to say, introduce the advanced system so that they can differentiate themselves from others. **[0:23:16] Rick Kobayashi:** And especially as for the contact center, probably the same situation in the US. But, uh, when the the consumer end user try to reach out the banks or insurance company and call it the free, free dial. In many cases, they can't easily to say reach to the contact center person, they have to wait 30 min or something like that. So, uh, because of that situation, they are so so aggressive. Introduce such a kind of a applications into contact center and internal purposes. **[0:23:51] Daniel Whitenack:** I and I. I'm just thinking about this scenario where you have the contact center, you're introducing whatever it is, a chatbot, a voice assistant, that sort of thing. And then I'm thinking back, um, to what Kenny was talking about, the about, kind of how, the how. The company started evaluating models that, m, you know, were not Gen AI models yet. And I'm wondering, um, Kenny, if you could help us think about, you know what, what needs? **[0:24:18] Daniel Whitenack:** Cause, if I'm understanding, you know, part of what you all are doing. Part of it is evaluating the risks of a particular model or system, and part of it is observing those and monitoring those in in real time. And if I think about like a traditional model, let's say a model that detects tumors in medical imagery, right? I can have a very nice sort of ground truth data set. And, you know, uh, maybe it's hard to get because there's some privacy concerns, but I, I can still get it. **[0:24:50] Daniel Whitenack:** I need to get it to train my model. And I, you know, have very specific metrics right around whatever it is, accuracy, f one score, et cetera. I, uh, can sort of grasp what the, you know, performance of that model is, maybe even compare it to human performance with something like a a call center. In some ways, people might struggle to connect that to real metrics that make sense, right? Because it's like, oh, well, people could say anything right to to the to the chatbot. **[0:25:22] Daniel Whitenack:** How do I? How do I know on what's gonna come in, you know, either from a normal usage or malicious usage, or or whatever. And how do I connect that to any sort of metric around a model? So I think sometimes people struggle with this idea of metrics and Gen AI models or Gen AI systems. Could you help, maybe clarify, like, what? What are some of the relevant metrics that that people could think about in terms of these systems? **[0:25:50] Daniel Whitenack:** That might help them understand, you know, how the how the systems are behaving. **[0:25:55] Kenny Song:** Sure, yeah, that's a very spot on question, I guess. Before I talk about specific metrics, I'll just take a step back. 1st. And I, if if we sort of think at a high level, what is the same between predictive AI and generative AI? Um, I think the structure of how you maintain reliability is basically the same, right? You need testing before deployment, and you need monitoring after deployment, and it's also very similar to, like traditional software applications, right, where you have automated tests and automated monitoring. **[0:26:25] Kenny Song:** Um, and so I think that part is the same, but the part that's much trickier for generative AI is that. Usually, as you mentioned, you don't have ground truth in the same way that you do for a classification data set, for example. And so the metrics that you use for evaluation are not as well defined. So you can't measure accuracy, you can't measure precision or recall. And the output of a generative AI model is also much more complex than just like a probability score. **[0:26:57] Kenny Song:** And so in that environment, it's it's very hard to determine. Um, how do we actually evaluate these things in a quantitative and objective way? So the approach that most of our customers take and most of the industry has gone in is, um, basically, um, using LLM as a judge, um, so you can craft these evaluation prompts that ask an LLM to evaluate some quality of some generated text. So it could be, you know, a very simple example is sentiment, so you evaluate the sentiment of of of some text. **[0:27:33] Kenny Song:** Um, you could do that with traditional, um, like a sentiment classifier as well, um. But there are more sophisticated metrics, such as, you know, uh, detecting hallucinations against some ground truth document, or measuring the relevance of the answer relative to the question. Or you might have more, uh, we call them custom metrics that are sort of designed to be domain specific. So if you have, like, a refund chatbot, you can design a metric that, uh, measures if the chatbot adheres to your, your company's refund policy. **[0:28:08] Kenny Song:** And so these metrics, they're very flexible because you can design the evaluation prompt in natural language. Um, and in our tools and our open source libraries, we have a set of built in metrics. You know, it's like a library of metrics you can choose from. But for many of our customers, they also extend those built in metrics to customize them to fit their business applications. **[0:28:30] Daniel Whitenack:** So, uh, Kenny, you, you were mentioning this sort of idea of LLM as a judge, which is, you know, using a a model to evaluate the model. W in in some sort of axis of of performance or some some quality, which definitely seems like a flexible UH option. But also some people, you know, some people might be thrown off by this sort of circular. Using a model to evaluate a model. Also, there's sort of this, you know, you're. **[0:29:03] Daniel Whitenack:** You then have a model that's evaluating the model, so how do you evaluate the model that evaluates the model? And and you kind of kind of get in in in this. How have you all navigated that side of things? Both kind of in terms of using using the larger model, making sure that the evaluations are are sound and also kind of, uh, transferable, you know, one model to the other. And, you know, maybe benchmarking the system over time, cause also the models you might wanna use as evaluators might change over over time. **[0:29:34] Kenny Song:** Yeah, also a very good question, and it's a question that we get from our customers quite a lot as well. And the way that we generally approach, um, the the evaluation workflow, which includes designing these metrics, is not usually human judgment. And taste is treated as the gold standard. But the problem with having humans evaluate every experiment with your LLM system is that it's very expensive and it's very slow. Um, and so, in the ideal world, you would design these LLM as a judge metrics that can mimic human preferences. **[0:30:08] Kenny Song:** Um, and so in our software tooling, this is what we, um, design specific workflows to help users to do. Usually, when they when a customer starts on a evaluation project, they'll, you know, of course, think about the evaluation criteria. That's important, but then they'll also have humans do a small set of that evaluation. So maybe like 50 to 100 of these manual annotations. And from there, you can design LLM automated metrics and measure their correlation and accuracy against a human judgment. **[0:30:42] Kenny Song:** You usually need to iterate a few times to get that, um, custom LLM metric as close as possible to the human judgment. But then once you have that, it's very powerful, right? You have this automated metric that, um, is a very good proxy for human judgment, and it's automated. Which means you can run it at scale, you can deploy it during evaluation, but also, um, in monitoring as well. And you can also potentially use that as a production guardrail. **[0:31:07] Kenny Song:** Um, in our firewall, **[0:31:09] Chris Benson:** Rick, I was wondering, I know, in, you know, we've kind of alluded to it that, you know, the the two sides of the equation in terms of testing the models and then monitoring. Could you talk a little about Citadel Lens and Citadel Radar? Uh, how you bring them to customers and and h, and what the relationship is between those two, uh, products that you're bringing to your customers? And and you know, how do you? how do you go to? **[0:31:35] Chris Benson:** how do you present them? When? When somebody is is interested in in being able to bring this level of security to the models that they're interested in. **[0:31:44] Rick Kobayashi:** Sure, the 1st of all, now we are say merging radar UH features into lens, so lens can provide both testing function and the monitoring functions together at this stage. And, uh, as for the the balancing or difference between testing and monitoring, as Kenny mentioned, especially in the case of LLM, how to customize human, say, uh, say, setting back. So so our system is in the concept that the system should follow the human rather than human has to follow the system. **[0:32:23] Rick Kobayashi:** In that sense, uh, we set the human annotation or human judge as a 1st priority and try to our metrics to be customized to the human judge. So that is, I say, very important point. And to make it happen, uh, we need to, or the customer need to go to go through testing phase 1st, so that all the say metrics should be aligned with the human judgment, so that and we can make use of the same custom metrics during the say. **[0:33:01] Rick Kobayashi:** The monitoring phase or fire firewall stage. So, in that sense, even though final goal might be a monitoring or firewall, but the to. Before going to that stage, how to test and customize the metrics is very important, I mean, very critical, uh, to protect the safety and security and reputational risks. So we recommend, strongly recommend to start from testing phase 1st. So testing phase is not just testing, but customize your metrics into your say professional, the persons, so that is a testing phase, and after that they can go into monitoring phase. **[0:33:43] Rick Kobayashi:** So that is, uh, our approach to the customers. **[0:33:47] Daniel Whitenack:** And I'm wondering, uh, you know, either one of you could answer this. But why is it? And this may be obvious to maybe more of the software engineering type crowd, but maybe less so to some others outside of that crowd. Why is it important to, once you've tested your model, to actually monitor it online? For maybe the things that are are potentially problematic inputs, whether that's a a security thing like a prompt injection, or, um, maybe something outside of the, you know, so some type of input that you wanna filter out, like IP going in or or something that doesn't fit your policy. **[0:34:28] Daniel Whitenack:** Why is? Why? Is it important to have that monitoring piece and not just the testing piece? Cause if I test my model and I convince myself that it can't be prompt injected, which I'm saying that sort of in jest, cause, um, you know, I, as you all know, every model, there's no perfectly aligned model, there's no every model is vulnerable to various things. But let's say that I convince myself of high performance in in one of these areas, why? **[0:34:57] Daniel Whitenack:** Then is it use useful and necessary then to monitor that over over time or in real time? **[0:35:04] Rick Kobayashi:** Technically, probably Kenny is also again the best person. But, uh, even if, uh, the customers, uh, goes, can go through the testing phase. The market condition or, say, kind of, uh, say, the human reaction may change over time. So if we were safe right now, but, uh, if anything, say something new happens today, what we say guaranteed today may not apply tomorrow. So so it's it's a very general things, but, uh, say, in that sense, the keep monitoring is very important to to protect our customers. **[0:35:45] Rick Kobayashi:** Uh, even if the market condition or the world condition change, or economy condition change. **[0:35:51] Kenny Song:** yeah, and just to give a concrete example of why you may want monitoring, so we we really view them as complementary, and you really need both if you want to make a system reliable. Uh, so, for example, if you have, like, an answer quality metric that, uh, measures how how high quality an answer is. Um, you should, of course, use that for testing to make sure that it meets some, like, you know, eighty ninety bar. But then in monitoring, you actually want to measure quality of the real answers that your chatbot is giving to real customers, right? **[0:36:23] Kenny Song:** Um, so, from a quality perspective, it makes a lot of sense from like a safety and risk reduction perspective. Um, another example is that, you know, uh, as you mentioned, in, um, during testing, you might test a a bunch of prompt injections against your system. But then in in deployment, you have real users. Some of them are adversarial, some of them are actually trying to prompt inject. They may do it in creative ways that you haven't tested before, right? **[0:36:47] Kenny Song:** And so you may want some, uh, some kind of guardrail that will automatically detect those attempts and filter them out. Um, even if you're sure that the model, um, is robust to, you know, 90 of these attacks. **[0:37:00] Chris Benson:** Uh, I'm curious, uh, you guys have some open source available, uh, out there, uh, I know one of the tools, uh, is LangCheck. Can you talk a little bit about your approach to open source? **[0:37:12] Kenny Song:** For context, LangCheck is a is our open source Python library, um, that contains, um, a a suite of metrics that are built in that you can use for evaluating the quality of text. One of our motivations for creating this library is that I think we launched it in, uh, October of of 2023, roughly. And around that time, um, there weren't a lot of sort of industry standard metrics and practices for evaluating text, particularly in non English languages as well. **[0:37:44] Kenny Song:** Um, so there was some focus on, you know, these metrics in English, uh. But we work with a lot of customers that have Japanese text, or, um, Chinese or German, and and these other languages. And we wanted to make a library of these metrics that anyone can use. And we view this as a as a pretty good starting point, right? So if you're, uh, if you just need like one or two metrics, you're comfortable writing code. **[0:38:07] Kenny Song:** You can use LangCheck and integrate that into your test pipeline or your monitoring system. But then if you want something production scale and you want an easy workflow to design custom metrics and test them against manual annotations, um, that's where our commercial product comes in. **[0:38:23] Daniel Whitenack:** Makes sense, yeah. And as we get a a little bit closer to the end here, um, there there's so many things and this is such a, uh, in, you know, so many in depth areas to go in. Which is is why it's great that there's, you know, wonderful people like yourselves exploring, uh, the topic. But I'm wondering, uh, if we could maybe talk just a little bit as we close out here about what you're excited about about. **[0:38:48] Daniel Whitenack:** Kind of, yes, uh, citadel, but maybe the kind of general ecosystem that that you're a part of. As as you look to the future, you know what's what's exciting to each of you about how the ecosystem is developing, what's becoming possible with the technology. Yeah, what, what's inspiring to you, or, or what are you thinking about in terms of the the future? Maybe I'll start with you, Rick. **[0:39:13] Rick Kobayashi:** Okay, so, yeah, now the Chatgpt five is released. And but when we say, look back at today, uh, from say, five years later, I believe that, oh, this is a very premature, say model or something like that. So in that sense, the, uh, technology advancement in the AI field is so rapid, and in that sense, yeah, there may be a lots of risk coming in. But on the other hand, there are in mostly infinite opportunities. So you can find a huge, say, variety of possibilities. **[0:39:53] Rick Kobayashi:** Not only just, uh, say, the uh, ai, directly, uh, related technologies. But, uh, I believe some of the material, uh, products or machines or anything will change, uh, maybe within five or ten years. So in that sense, uh, we are in the midst of the say period where anything can change in that sense, especially technology related. So there are lots of, say, risks may coming, and we like to protect such a kind of risk as a company. **[0:40:28] Rick Kobayashi:** But on the other hand, uh, people can find many possibilities, opportunities for you to try. So I strongly believe that, uh, we. So, even though there are lots of issues in the world, but, uh, people can enjoy, or say, say, uh, can, let's say, make best use of this opportunity. Everybody. Yeah, **[0:40:51] Daniel Whitenack:** Yeah, that's great. What, what about yourself, Kenny? **[0:40:54] Kenny Song:** Yeah, I think as a consumer of AI, um, in both my personal life and work, uh, from a consumer perspective, it's. It's really exciting to benefit from all the advancements in in these new AI tools and and models. I really loved O3 as a model in in ChatGpt, and I love using cursor, and I'm excited for these tools to become more and more agentic over time. And I think that's sort of the trend that you see, um, if you just look at ChatGpt. **[0:41:21] Kenny Song:** Um, originally it was just like three, five and GPT four, that, uh, just answer a question based on a forward pass of the model. But now these models will search the internet and it'll sort of reason and think about what to search next. And as a result, the the outputs have become a lot better. Um, so really excited for that to improve even more from a consumer perspective, and then, um, from a business perspective. I'm really excited to help bring these capabilities to our business customers and help them use AI more reliably and more effectively in in their business. **[0:41:57] Daniel Whitenack:** That's great. Yeah, well, thank you both for taking time to to chat with us today. Um, and thank you both for the work and thought that you're putting into the tools that you're building and the open source projects that you're putting out there. It's a great benefit to the to the community and to the business world, of course. Um, so thank you for the work that you're doing. And um, yeah, we'll look forward to, uh, to keeping, keeping, uh, an eye on on what you, uh, what you evaluate and and protect us from next. **[0:42:28] Daniel Whitenack:** So, uh, appreciate you both and hope you have a great evening. Thank you for joining. **[0:42:32] Rick Kobayashi:** thank you very much. **[0:42:34] Kenny Song:** Thank you for the conversation. **[0:42:42] Speaker 6:** All right, that's our show for this week. If you haven't checked out our website, head to practicalai fm and be sure to connect with us on LinkedIn X or Bluesky. You'll see us posting insights related to the latest AI developments, and we would love for you to join the conversation. Thanks to our partner Prediction Guard for providing operational support for the show. Check them out at predictionguard com. Also, thanks to Breakmaster Cylinder for the beats and to you for listening. **[0:43:08] Speaker 6:** That's all for now, but you'll hear from us again next week.