Citadel AI builds software tools to test, monitor and govern AI systems for organizations.
“So what we do at Citadel is we build software tools to help organizations test, monitor, and govern their AI systems.”
Person
Kenny Song
1 episodes · 17 quoted claims
Citadel AI was started at the end of 2020, before the era of large language models.
“when we started the company, it was at the end of 2020, um, so it was before the era of large language models.”
Kenny Song worked at Google Brain as a product manager from 2017 to 2020, on a team building machine learning infrastructure.
“So in 2017 to 2020, I was working at Google Brain as a product manager, and, uh, my team was responsible for building machine learning infrastructure at Google.”
Kenny Song's Google Brain team's infrastructure work included TensorFlow, TensorFlow Extended (TFX) and parts of Google Cloud's AI platforms.
“this included TensorFlow and also other platforms like TFX, TensorFlow Extended, and some of the work on Google Cloud's AI platforms and so on.”
Citadel AI initially focused on adversarial attacks in 2020 because it was a hot research topic.
“when we, uh, started the company originally, we focused, um, a lot on adversarial attacks. Actually, back in 2020, since it was a very hot research topic.”
Citadel AI found after a few months that companies were not commercially interested in adversarial attack defense, and it pivoted to observability and testing.
“We sort of found that after a few months, um, this wasn't really a problem that companies were interested in.”
Around two years before the recording, Citadel AI's customers began showing much greater interest in large language models.
“And about two years ago, we started getting a lot more interest in our from our customers in LLMs.”
Kenny Song says generative AI lacks ground truth, so classic metrics like accuracy, precision and recall cannot be used.
“So you can't measure accuracy, you can't measure precision or recall.”
Kenny Song says the industry approach to evaluating generative AI is using an LLM as a judge with evaluation prompts.
“basically, um, using LLM as a judge, um, so you can craft these evaluation prompts that ask an LLM to evaluate some quality of some generated text.”
Kenny Song says customers typically create 50 to 100 manual annotations and then compare automated LLM metrics against human judgment.
“So maybe like 50 to 100 of these manual annotations. And from there, you can design LLM automated metrics and measure their correlation and accuracy against a human judgment.”
LangCheck is Citadel AI's open source Python library of built-in metrics for evaluating text quality.
“LangCheck is a is our open source Python library, um, that contains, um, a a suite of metrics that are built in that you can use for evaluating the quality of text.”
Kenny Song says LangCheck was launched in roughly October 2023.
“I think we launched it in, uh, October of of 2023, roughly.”
LangCheck was created partly because industry-standard text metrics focused on English, while Citadel AI's customers also use Japanese, Chinese and German.
“we work with a lot of customers that have Japanese text, or, um, Chinese or German, and and these other languages.”
Kenny Song says LLM-judge metrics can include sentiment, hallucination detection against a source document, and answer relevance.
“Um, you could do that with traditional, um, like a sentiment classifier as well, um. But there are more sophisticated metrics, such as, you know, uh, detecting hallucinations against some ground truth document”
Kenny Song gives a custom metric example: checking whether a refund chatbot follows a company's refund policy.
“So if you have, like, a refund chatbot, you can design a metric that, uh, measures if the chatbot adheres to your, your company's refund policy.”
Kenny Song says he enjoys OpenAI's o3 model in ChatGPT and the Cursor tool, and expects tools to become more agentic.
“I really loved O3 as a model in in ChatGpt, and I love using cursor, and I'm excited for these tools to become more and more agentic over time.”
Kenny Song says early ChatGPT models like GPT-3.5 and GPT-4 answered from a single forward pass, whereas newer models search the internet and reason.
“Originally it was just like three, five and GPT four, that, uh, just answer a question based on a forward pass of the model.”