Arvind Jain on Building Glean and the Future of Enterprise AI

You're listening to Gradient Dissent, a show about making machine learning work in the real world, and I'm your host, Lukas Biewald. This is a conversation with Arvind Jain, who I've known for a while. Arvind is the CEO of Glean, which is one of the most successful enterprise companies using LLMs. Maybe right now the most successful, um, enterprise company that foundationally uses LLMs. And I really want you to not be fooled by Arvind's humble style. He's one of the most successful founders in Silicon Valley.

Before Glean, he founded Rubrik, which IPO'd for, um, I believe, a valuation over $20 billion. So he's been phenomenally successful twice, very low key, but clearly very driven and has some real insights. I hope you enjoy this conversation. Arvind, thanks for taking the time to to talk. It's been a while. Um, I think, you know, you have an incredibly successful company. So I feel, you know, maybe embarrassed asking you this, but I think it's not a consumer brand, so people might not have heard of it.

Can you explain what Glean does? Please?

Yeah, absolutely well. 1st, Lucas, thank you so much, uh, for inviting me to this. Uh, Glean, uh, is a it's an enterprise company, enterprise AI company, what we do, I mean, think of us as ChatGPT, but inside your company. Uh, Glean connects with all of your internal systems, your data, your knowledge inside your company. And then gives you a Chatgpt like experience. Where people come, ask questions, and Glean will answer those questions for them. And it's going to use all of the world's knowledge, but also additionally, it's going to use all of your internal company's data and knowledge and context to help answer those questions for you.

So that's like, simply put, like, you know, that's what we do. Uh, of course, like, you know, as as AI, you know, continues to advance, um, the same platform that we build, that we use to build our, uh, Glean assistant. Um, you can also use that as a as an agent building platform to to build all kinds of AI agents on our platform.

You know, what's kind of interesting about your company is it's been, you know, one of the real successes, I think, or at least from my perspective, one of the real successes. In terms of real LM applications for enterprise that that works well. But, you know, if you look at the timing of when you started it, it was before it was obvious that, um, you know, LLMs were were gonna work so well, at least at least to most people.

Um, how did, how did your vision for the company start, and how did it kind of evolve through this period of of LLMs taking off?

yeah. So Glean started in early 2019. You know, in those days, nobody was talking about large language models, and the term generative AI did not exist to my knowledge at the time. Um, and we were not thinking about that either, um, and we had a problem at hand that we all that we were interested in solving, and which was search. Um, we, you know, we felt like, you know, this is my experience in every single job that I had before Glean.

Um, it's so hard to find things inside the company. When you do your work, you need some information, you're looking for a document. Uh, it's ex extremely hard to actually find information, um, and and and it's because, you know, like, you know, as enterprises, you know, we have so much information, so much data. And it's spread across many, many different systems. Like, as you know, in, you know, in enterprises, you know, tend to have, like, you know, hundreds or thousands of different systems.

Um, and stuff is all over the place. And so when when I'm looking for something like, I have a quick question, I need an answer, or I'm looking for a doc that I know exists, but I still cannot get to it. That was sort of like that. That caused a lot of frustration. And and in fact, like, you know, there have been a lot of research studies that have been done. Uh, people, uh, would spend one 3rd of their working time just trying to find things.

And so that was. That was the 1st problem that we wanted to solve. We wanted to build a Google for people in their work lives. One place where you go, ask your questions, and we want to bring the right information back to you. And sometimes, you know, it may not even be the information. It may be you're just looking to connect with the right people who can help you on a given topic. So, like, we become that one place.

We'll come and find data, find documents, find people, all in one place. You don't have to think anymore where to go and look for things. So that was, that was the 1st product we wanted to build. Um, the interesting thing is that in early 2019, transformers were already a thing, like it was not like the rest of the the the world didn't really care as much. But inside search teams at Google, we were seeing, uh, a really good impact of the transformer technology on some of the core components of a of a search engine.

You know, and and we saw this real promise, you know, in that day, you know, you know, in early 2019, you know, where we could conceptually understand information. Like, you know, we didn't have to actually go and build a keyword based search system anymore. Which is sort of like, you know, like, quite dumb. Like, if you think about, like, how, you know, we match information to people's questions, like it's happening at a very surface level with keywords.

Um, but uh, Google had actually put, uh, BERT based models in open domain in late 2018, and they were able to use them. And then actually go and pretrain these, uh, what we call now small language models on your enterprise corpus. And start to sort of deeply and fundamentally understand what your business is about and what you know any given document is about. And and then do this sort of conceptual, semantic matching of information to people's questions.

So so so. Interestingly, transformers actually played a foundational role in Glean's evolution. Like, you know, we were probably the 1st company to use transformers in the enterprise, you know, in in that day, uh, cause it, it was like, really, really applicable to building a good search experience in the. So that's so, that's, that's how we got started. And then, of course, over the as as these models, like in in in 2019, the transformers had no capability to generate information, they had no, like, none of the reasoning capabilities that models have today.

Uh, but they had this, you know, embedding based conceptual sort of matching, you know, capability, you know, uh, which which we were able to use. But as these new capabilities came, that allowed us to evolve our product. And what used to be a Google for you in your work life, uh, became more like ChatGPT for your work life. You know, you can go one step further, and now we now. We don't have to just surface information to you, we could actually read that information using AI and actually give you precise answers to your questions.

So sort of the the evolution has been like that over the la over the last six, six plus years.

So you're, you're politely telling me, I think that you actually did see the power of transformers, at least in in 2019, when you started the company, which is which is super cool.

Well, I mean, I think, yes, yes, I mean, because because we saw it actually like it was not, it was not a vision, it was, you know, we actually saw, we saw. We saw. Like, how, you know, it could really deeply, like, you know, you could take two documents, uh, documents that had no real word overlap. Like, you know, like, if you look at the actual choice of words used in those two documents, very different. But still, you know, a a BERT based model, you know, trained on your enterprise data, could actually tell you that, like, hey, these two, these two documents are very close to each other semantically.

And so that, so we, that was it, that that was, you know, I in. In those, remember, in 2019, when we, you know, today, we expect a lot from AI, but AI was not there in 2019. And that itself was a big, that was still a big step function. In terms of like, our ability to like, you know, really, uh, conceptually match people's questions with with knowledge. We saw that power, um, and and then we decided to sort of bring it to enterprises.

And so, I mean, you're in this world where, like, you know, these labs are investing massive resources, constantly coming out with, um, with new models. Like, do, do you still, you know, fine tune a custom model for every customer, or is that important to you? Or like, what? How do you think about, like, what parts to kind of, you know, buy or use, and what parts to build in house?

Yeah, well, I mean, the company, like our like. The thing that I actually keep telling our technology team is that do not reinvent things that have been invented already. I mean, this is we will not like, that's not, that's not doing service to us or to our customers. Um, and so our approach always is to actually, um, maximally use innovation that's happening outside because it's available to us. I mean, the good thing is that all of the innovation that's happening in in LLM space.

All of these models, uh, companies like Google and OpenAI and Anthropic, they're all making those models available to us to use. So we actually like to use those models, and we do. We use them quite a bit in our stack. So the way, like the way our technology works, is that we we f. We actually pre train, and, you know, we pre train and build some models. Like, you know, uh, for for semantic matching that that we have to do ourselves.

Because these models are all about your enterprise and your data and sort of building an understanding of what your, what your business is about. And so there we actually will go and connect with your enterprise data and data and knowledge. We'll take all the documents, we'll actually build, train the model on it so that it can start to understand how your business talks and speaks. Um, how, how, like, what, you know, what code words, you know, you use, versus your lingo.

So for that, it's actually important to actually go and train a model, uh, on your enterprise, uh, corpus. But these are very small models, like, you know, we don't actually train large models, and and these models allow us to actually build custom embeddings. You know, for your enterprise content. And so that when people come and ask for things, we can match, you know, their requests with your, you know, documents in your corpus in a in a much better way using semantic matching.

Um, so so those models we build ourselves. But then when it comes times to, uh, um, to actually do something, you know, with the information. So imagine the work, the flow of like, when you come and ask a question in Glean. So the 1st step of it is, well, you know, we see your question. We're trying to understand what it is about, and we're gonna use our semantic, uh, embedding models, the one that we train, and, you know, uh, ourselves, uh.

To assemble the right pieces of information to get the best knowledge within your company that we think can answer those questions. And now we're gonna actually take, take we, we're gonna take this, you know, information. And and ask AI to actually synthesize an answer from it. And then for that, for that, we don't actually use our own models, you know, we use out of the box models like, you know, GPT or Um. And similarly for reasoning, like when you actually take a complex question and you're trying to break it down into, um, into like a multi step agentic workflow.

Again, like, you know, we use the reasoning capabilities of the large language models, so that's sort of the architecture. So, yeah, it's, it's, it's more the don't, don't train, don't fine tune when you don't need to. But but there are certain things, you know, in the in the search stack, where it makes sense to do that. Um, maybe I'll give you one one or two more examples. Please? Yeah, Well, well, w what one so in in search, of course, you know, part of it is like, you know, you, you get a, uh, question from a user, and you're trying to semantically match it with, uh, the right information.

So that's what I described right now, uh. But then there are, like, you know, smaller parts of the stack, uh. For example, where to expand, like, you know, where to expand, like, you know, in retrieval systems. Uh, you do spell check users questions, you actually find synonyms, you have to find acronyms. And and and for those models, what we what we realized is that you can take, you know, these really, you know, sort of small open, open domain, uh, language models, and you can fine tune them for those specific tasks.

So, for example, for spell checking, um, we, we have a fine tuned version of a language model that will do it for us. Um, so so for like, you know, for some point tasks, like small, very, very specific um things, you know, the these fine tuned models actually help us, uh, do those things faster and cheaper. Uh, they don't, they don't add more capabilities. Like, could you could actually take GPT, like a large GPT model, to do like to actually do spell check for you?

Like, it'll, it'll do a fantastic job. It's just that it's so overkill to actually use use that model for that kind of a task. So so there's a little bit of that fine tuning that's happening in the search stack for us. But but otherwise, we are for reasoning for generation, we use out of the box models.

I think another one of your kind of advantages or parts of your technology that that you talk about a lot is is like, the um, the security model that you have, right? So it's like, I, I mean, in enterprise search, there's such complicated rules around, like, who can access what data? And as these models get more complicated, like, you know, like, building an embedding model, you could imagine it somehow, like, leaks information, you know, across embeddings.

Like, can you talk a little bit about how, how that works, how you think about it?

yeah, yeah. So the way Glean works, uh, inside your enterprise, is that as we connect with all of these internal systems like Google Drive or Slack or Salesforce. Um, we are reading, we are understanding content that lives in those systems, uh, but we're also understanding the governance of that data. Like, who are the people who can access that information? Uh, for any given document, for example, we'll look at the list of people who are authorized to see it.

Um, and this is. This is fundamental, because if you are going to, you know, deliver AI in the enterprise in any way, you have to do it in a safe and secure way. You can't, you can't have, like, AI based systems start to leak information internally to your employees who don't have rights to see those information. So so, so the way our system works is that we connect with these individual applications, we understand their permission model, and then we actually, you know, uh, bake that into our um, into our, into our core.

Index indexing technology is the permission for baked in right into our, you know, uh, index indexing system. So now when you come and, you know, uh, ask questions in Glean, we will, uh, we will retrieve information for you, uh, which can answer that question, uh. But we'll actually only retrieve information that you have rights to use. Uh, we know who you are, your sign in, and we'll use your enterprise identity to establish that. These are the documents that you are have access to, and these are relevant to this topic, and then we make actually models work on it.

So that's, that's the so, that's the fundamental. Like, you know, in a RAG style system, you can actually d, you know, make AI safely work on data that only you have, you know, that you have permissions for. Uh, but then you also mentioned this interesting question around embedding, like, if you're actually building custom embeddings, what content do you actually train it on? Because if you train it on all of your enterprise corpus, it will indeed like, start to leak information.

And you won't realize, you know, in what form or shape it actually does it. Because ultimately, you can't understand behind the scenes, you know how AI works. So so what we do is, you know, when, when we're actually training models, custom models, we typically would train them on, um, content that we think is safe to train on. So there's a lot of, you know, um, curiosities like going behind the scenes to to understand, like, what, what information is actually safe to train on?

Um, and and so so, yeah, so if, if you, if you sel, if you, if. You use it only on a subset that is safe, then your embedding models are actually not leaking information.

I see, I guess. Um, speaking of SEC security, I think a really interesting fact about you, at least from my perspective, is you're also the founder of of Rubrik, right? Which is like, uh, maybe, maybe not everyone's heard of it, but an incredibly successful company that, you know, recently IPO'd. So I feel like you're like, you know, kind of twice, like unicorn, verging on like decacorn, maybe, um, founder, and and it's like, so amazing, like you're, you're so, like, kind of humble and understated.

Um, I just wonder, like, if you have any insights into, kind of, you know, what you're doing differently than other people, like, what, what do you kind of attribute these? Um, these two successes? Cause I think, like, you know, I would think, like, you know, 10 billion plus exit has gotta be like one in a thousand, one in, you know, 10 zero, depending on where you start. And so the fact that these two are correlated, I think, is a powerful indicator that there's more than um luck involved, at least in in your case.

well, I mean, I think I think it, it is correlation, and and I think no more than that. But but I think what? What I would say, like, you know, from our experience is, um, we started Rubrik, uh, that was, um, early 2014, and we picked a problem which we knew every business faced. Um, it was really hard to protect your data spec, you know, especially like, you know, as you know, the attacks are on the rise, you know, with ransomware and other things.

And and we felt that, you know, in at that time, like this, like 2013, when we were con con planning, you know, building, you know, starting Rubrik. We felt that all the energy and attention, all the best tech people, were all focused on consumer technology, and enterprise was sort of lagging. You know, at the time, um, we didn't have a lot of great companies, you know, like the SaaS revolution was still fairly early at the time.

And and we felt businesses were being ignored. And the, uh, just building, just just helping them with this very fundamental problem. That, like, you know, like, hey, we'll help you help keep your data safe. Um, was something where there were no products that were being built, um, that were less than like, you know, two decades old. So so we, we, we f. We saw an important, large problem, you know, with large market. And and then after that, it's about just go and a good job, you know, building a product, good job, building a business.

And and and I think we, we were fortunate that, like, you know, we were able to sort of go build a good team. You know that then, of course, um, you know, uh, generated all the success. Um, I, I would say that, you know, with Glean also. Um, similarly, uh, we picked a problem which is not, uh, a niche problem. Like you can go and talk to every single person, anybody that you know, like, you know, or that like, you know.

Ask them that. Hey, it's easy for you to find things inside, you know, in your, inside your company. And they'll all say no. And so it was a pretty obvious problem. Again, nobody was working on it at the time, and we felt that we had to actually be the pioneers and and solve this. I got a lot of advice, by the way, at the time, not to actually get into this business. Uh, Well, yeah, cause, but

okay, sorry, I knew you're gonna give me something like, really humble like this. But I think that what these two have in common in my mind, your two companies, is they're kind of like, um, like ideas that, like a lot of people have that, like, don't work. Like, I feel like I've. I've sort of heard those pitches for your two, you know, companies quite a lot in my life. But then you were, like, wildly successful doing both of these, these different things.

Where I think it must be, um, really like an execution oriented thing. Like it's not like you kind of came in, it seems like with some, you know, kind of wildly new product that nobody was expecting. It's like it seems like you did like a phenomenally good job. In a in a kind of market that didn't seem so great, or it sort of seemed like broadly crowded and specifically hard to execute in. I'm not sure, but I I just I wanna draw some insight out of you.

yeah, well, I mean, I think, like, yeah, that's, that's like, maybe I'll add one thing. Look like, think about search. A lot of people actually told me not to work on search and enterprise search because there there were only failures. Like in in the two or three decades, two or three decades of enterprise search history, including like products from Google. Like, you know, every time, you know, Google tried to actually build an enterprise search product, it was always, you know, always failed.

Um, so so, when, when you see like domain, like, it's sort of like, you know, almost. People start to feel like, you know, this. This problem is like, maybe too hard to solve, or maybe it's not even worth solving. Because, you know, if if nobody builds a big business, maybe maybe there's no need to actually build a product like that. So that, so, that was the mindset. And and that's why there was no innovation in search, you know, in, you know, inside the enterprise, for a long time.

There were no startups like, you know, we started in, we started in 2019. Um, there, there was no search company, um, that you would know of that got started. Like, you know, like, you know, ten years, you know, in that ten year period, like this was a dead, dead, dead, you know, probably the like. One of the worst areas, you know, where no investors wanted to invest. Um, but that this is interesting. Like, you know, you, you see a problem that everyone has, like, you know, every, everybody faces this pain point.

Um, yet there are no good products. And then there are also technology trends like we saw some new things that happened, um, at that time. One of them was SaaS, the the whole SaaS transformation, um, which made the problems so much more worse. But also it actually made the problem tractable. In the sense that, you know, SaaS systems actually allow you to actually go and read data inside those systems. Before we couldn't even do that, it was just so hard to get hold of data in the enterprise.

Um, we also saw transformers, like, you know, as a really core new capability that would help you deeply understand content and actually build a better search experience. So so it was like, you know, we did observe those, we observed those trends. Uh, which made us, f, you know, made us actually confident to actually take that contrarian ba uh bet and say that. Like, look, you know, no, we think, you know, you can really now go and build a good product and and build a huge business.

So so, so I I th. I think part of it is that, like, you know, I don't know, like, you know, I'm, I'm, I'm an engineer, by the way, like, you know, by, you know, in my build up and how, you know who I am as a person. And um, doubts are part of my existence. Like, you know, like, it's, that's how I think, like most engineers are, so so I. So I was surprised at myself, like, you know, when everybody else told me not to do this, I don't know what actually, uh, was it in which actually made me have that conviction?

So I was surprised at myself at that. But you know, like, and that, that maybe that, that's, that's, that's one thing to think about. Is like large companies, if you're gonna build them, like, you know, of course, you have to tackle like, you know problems, you know, which are very universal in nature, like, you know, broad, broad, like, you know, which have broad impact.

Do you think that you, like, operate differently? Like, I mean, another thing that's interesting, right? Is your product is to kind of help other companies operate better, you know, more efficiently. And I assume you're, you know, kind of dogfooding your own, your own products. Like, D, do you think that you like? If I, if I was like a new hire and I was asking you like, should I join? or something like that? Would you tell me, Hey, um, you know, get used to, like, you know, kind of the special kind of culture that we have here at Queen?

um, that's a, that's, that's, that's a good question. The um, I think, well, I don't know if we have a particularly unique culture. Like a lot of our learnings, in my learnings, have come from Google, you know, which is where, um, I spent a lot of my um, you know, work life, you know, at. And. And Google was a really special company, like it was very different from. And. And I'd actually worked at three companies before Google.

Um, um, at that time, and it, it was so unique, you know, in the in the sense that it actually put technology and innovation in front of everything else. Uh, as an engineer, uh, at Google, you're the king, nobody gets to tell you anything like you go. And you know, build, build, like, you know, whatever technology you want to build, like, you know, doesn't matter. Like, you know, whether you get to decide whether it's important, whether you want to build that system.

And you had all the resources to actually then go and build those things. So, um, and so so. There was that Google model, which is that you hire really, really smart people, um, you know, really good engineers who are who are very motivated to do something special. And then just let them be. And like, you know, they'll figure out the right things to figure out, they'll figure out how to organize, you know, with other, with other team members, and and do a great job.

And frankly, like, you know, for a man, for a manager, that also is is a pretty, pretty good situation to be in. You know, you don't have to do any work. Uh, and and so, like, you know, I, I sort of like, you know, I love that model, and we've always followed that model. Like, you know, I do. Now at Glean, you know, where we, uh, we have a really s, you know, amazing team. And and like, you know, like, my role is to generally be not in the way.

Like, you know, I can't help myself. I do come in the way sometimes, but but, but often, like, it's mostly like, you know, it's about that, like, you know, build a innovation, 1st culture, um, and uh, and success will come.

Hm, do you think that having like a much better, um, kind of search? And even now you're building these like agent agentic systems? It sounds like that could be even custom to to what a company's trying to do. Do you do you think that is causing or or will cause companies to function dramatically differently in the future?

Well, I mean, uh. The way we work is fundamentally going to change, both both at a individual level and also at a overall business level. Um, in fact, you know the um, the vision for the future, like, I believe this is g, you know, this is how, you know, our work lives are going to l look like in the future. So for any person, um, regardless of like, you know how senior they are. Um, you will have an amazing team of assistants, coworkers and coaches that are go.

We're going to basically help you not on not only do 90 of the work that you need to do, um, but also help you get better at things that you do. Uh, and like, you know, if you think about today, you know, there are certain people in the company that have that luxury. Like, you know, if you're a CEO of a company, you get to have your coach, you got to have your assistants, you have a chief of staff, you have an executive team, and you have all this.

Like, you know, really amazing set of people around you, that that you're surrounded with, which actually helps you become, you know, that true multiplier, you know, for the company. And and and AI is going to change that. Like today, only the CEO or a senior executive can actually have that luxury and that help. Uh, and AI sort of democratize, democratizes that and brings that help to every single person. So everyb everybody's gonna have this dream team around them.

And you get to, in fact, you know, go and build that dream team. Uh, with, you know, ai, you know, powered, like, you know, agents, you can have some, you know, you can have an agent that works like your assistant, another one that acts like your coach. Yet another one that actually acts like your coworker and shares some workload with you. Um, and and that's, that's the world that we're going to be in. Um, so the so, you're no longer alone, you have, you have this team, like, you know, continuously, like around you, um, and s similarly for like, you know, companies.

Um, most of our business processes that we have today, you could envision that parts of those processes, like, you know, uh, that can actually be easily be handled, like, you know, through some kind of an AI agent. Um, so there's a big transformation ahead of us. Um, it's gonna reshape, uh, how individuals work, it's gonna reshape how a company's, you know, how an organization looks. Um, there may be, uh, people's roles are gonna change. Like, think about software engineers, you know, you hear that in the future we're gonna become more more code reviewers as opposed to, like, you know, people who write code.

So so, so there's, you know, that, that kind of like, you know, change in, like, where we spend our time is gonna happen to all of us.

You know, it's funny. I I think about, like, what's in our UM, like weights and biases, like in our, you know, Notion, and you know, Google Drive and and all these things. And you know, we have a lot of um. For example, like, um, uh, OKRs and kinda like our our quarterly plans. Um, I've, I've always it's I. I keep being tempted to to paste them into, like, you know, Gemini, and now that the the context window is like, big enough and put in my like, current, you know, OKRs.

Like, at sort of the quarter and just be like, which ones do you think we're gonna miss? Or, you know what, what do you think about this strategy? Do you think it's a good idea? Like, if you're, you're kind of like, back at at what we've done, do you do you ever do stuff like that? Like, not like, like tactical coding stuff, but sort of like, start to ask about broader, um, strategic questions,

yeah, actually, the um, so so, like, you know, today with Glean, um, these are the kind of things, you know, that it does a really good job at. Cause, you know, Glean is actually connected to all those systems, so you don't actually even have to cut and paste. You know that OKRs from the, you know, Notion or Google Drive doc, cause, you know, it already is in that, you know, that overall enterprise wide search index that Glean has.

So some of the things that, you know, that are my favorite activities, you know, with AI now are, um, number one. The when I want to learn about technology or what, like a given team is working on inside our company, uh, or how do we actually design a particular system? So now, instead of, like, you know, like setting meetings and asking them to give me an update and walk me through, like, some kind of a presentation.

I actually ask Glean to actually, gen, you know, give, generate a latest, you know, up to date tutorial on that topic for me, and that's all I say, like, you know, generate it, like, you know, I. And I have this agent, you know, where I've actually, you know, where? We've told it to actually say that. Like, you know, when I'm looking for a tutorial on a given topic, like, make sure to actually go and look at all the design docs that have been written.

But then, you know, design docs often get obsolete and and so, like, you know, read the design, see when it was last updated. But then after that, actually start to look at more. Like, you know, Jiras have got resolved, you know, code that was committed into that, into that component. Um, and sort of like, you know, get the latest view of like, you know how things work. So like, don't give me that stale doc, you know, give me the latest on this info on this system and how it works.

And this is actually pretty cool, like, you know, cause now, like, there's this concept of evergreen documentation like I, I, I don't have to deal with, you know, stale stuff anymore. Cause, you know, AI is actually doing that human like thing, where it's reading stale information and new information. And it's actually deciding what's the right and up to date information and gives gives it back to me. Um, similarly, um, like an example similar to what what you mentioned.

So we so we're a new business, and um, new businesses, you know, like, you know, if you build a good product, you don't have a churn problem. And so we. So we actually didn't have any dashboards in our, in our data, uh, you know, like, in in our, in our sort of, you know, data dashboards around, um, around churn. It was just simply a thing that was not. Nobody actually built any analysis on it because they're not relevant.

Um, I love how but that,

humble you are, that's, that's, that's amazing. I mean, you're, you're old enough that you must have a very sticky product. If you, if six years in or something, you're not worried about churn.

yeah, the like, at some at some, you know, so, but well, but but. But like, guess what, like, you know, we had some churn that happened last quarter. And then, like, you know, suddenly got worried, like, okay, what's happening? Like, you know, like, what's gonna happen to all the other customers? And there are, there is the customer health. And like, I'm exaggerating a little bit, I mean, like, you know, some teams have it, you know, you know, like, but we didn't have, like, really good versions of it.

So the we wanted to do an analysis. And so, um, one of our team, one of our team members in finance, you know, they're not, you know, she's not an engineer, hasn't built anything like, you know, from a software system perspective. But she went and actually asked this question and and told, like, you know, in Glean and said that like, go and look at every single customer of ours. So you can actually ask Salesforce, like, you know, one, you can look in Salesforce to get out the list of all the customers.

Um, and for each one of them, go and look at, like, you know, our shared Slack channels with them, go look at what the conversations look like, you know what's sentiment there? Go look into our product dashboards and look at the usage. And then, um, and then, like, you know, take, take all of that data. And based on that, like, you know, now give me a risk profile. Like, you know, whether it's, you know, like, a account, that is healthy, like green, green, you know, or red, or or yellow, and and actually, sure enough, like, you know, AI is actually pretty good at this.

And like, you know, you know. It generated a report which basically tells us, like, you know, all the top risk accounts and and the ones that are not. And actually, frankly, I would say like, it was actually better than like any dashboard that we would have built. Because it was able to actually make use of, you know, subjective textual information like, you know, which, you know, obviously dashboards can't like dashboards can only look at, you know, usage, like, you know, usage trends and some, like, you know, structured information.

So so, so these are like, you know, certainly, like, you know, new capabilities, like, you know, where you could tap into this, like, 95 of your enterprise data is unstructured, uh, and this, you know, is full of insights. Like, only if somebody had time to actually, you know, extract them, and like, you know, that's where AI sort of comes in. Super.

Totally wow, that's such a compelling, um, story. I mean, I I want, I wanna do that as soon as we get Glean installed here. Um, wow, uh, how do you? Um, you know? One thing that I think about with you guys is, you know, your your customers probably don't let you, you know, pull metrics on their data, or maybe they do. But how how do you know if you are like, shipping a new model that's that's better, or if your embedding is better?

Like, how do you actually measure your your progress?

yeah, so so 1st, like, you know, we have like customer zero, which is clean, uh, our own like instance, and as we build, um, as we sort of, you know, bring new models in. Like, you know, we have a evaluation framework, um, where, uh, there is like, what we do is um, uh, for for in any, you know, at any customer deployment, actually. Um, there is plenty of information available where there is like, you can have question answer pairs.

So people have asked questions and you know, somebody's answered them. Um, you don't even have to, you don't even have to actually like, you know, create these golden data sets. Uh, sometimes you can just obtain them from actual, like, real conversations that people have. Like, for example, like in Slack, somebody's asking questions or people are responding with answers. And there's a lot of reaction and thumbs up to that. That information, and that that's like a really good example of a a golden sort of set of like, you know what a question is and how it should be answered.

And so so we build these golden sets, you know, like for each one of our customers. And then when we bring a new model in, we will actually, uh, run, run, like, you know, all those questions through the new model, uh, or the new like. It's not, doesn't have to be a new model, like, you know, it's also any change that we make in our own systems, right? Like, you know, it's it's like our system. You know, there's a lot of, you know, things happening there, including changes in the search and retrieval system and then also the MO models that we use.

So so you sort of you measure, um, you know, like, how the answers are now looking, you know, with, um, you know, on the golden set, you know, with this new change that you're rolling out. Um, and and then you also use AI to sort of, uh, cause, you know, like these are subjective answers. Then you have to actually also use LLMs as judge to sort of see, like, you know, how close you are to what the golden response is.

So that's the fundamental, uh, way for us to, uh, measure, like, you know, our our, like how well our end to end system is working.

Hm, uh, do you have um issues with hallucinations? Like, I feel like in almost every domain, that kinda drives us, um, crazy, like, do, do you take like special measures about that? I would think at from an enterprise perspective, they might feel more sensitive to hallucinations than even a consumer.

yeah, I mean, 100, I mean, I think AI has to, I mean it. And and hallucinations are there. Um, the we we have some safeguard against it, um, but but that that's not to say that Glean is going to actually always like, answer questions exactly the right way. Like, you know, I think the it's not, you know, it's not, it's not. There. Because there are so many things that can go wrong. In terms of like, when somebody comes and asks you a question, like, can you answer it properly?

Sometimes information is not even there to answer, sometimes information is stale or out of date, and that you're using to answer, you know, questions for people. So the the failure scenarios are so many. Um, the and one of them is actually the hallucination of the model itself. Um, but for that, for that, for the hallucination part of the model. Like, what we do is remember that in our system. When you come and ask questions, we are the ones who 1st assemble the raw materials.

Like, you know, the knowledge that we're gonna use to answer that question. And we give all of that knowledge at the prompt time to the model, along with the question, and the model comes back with an answer. What we do after that is that line by line, you know, we'll take every, um, you know, the response of the model. And we'll try to see, like, you know, if we can actually find, you know, that same piece of information in, you know, in the input that we gave to the model.

And then we'll sort of like, you know, in fact, you know, we'll use that for our core citations, uh, framework, and we'll cite on a line by line basis. Where is that human generated information that was used to produce, like that AI used to produce this, you know, this particular line? And when? When we don't see it, we either suppress it or we actually would. Don't show, don't show the citations. But so that way, you can actually, like, tame the model a little bit.

Like, you know, if you're not s. If you see, you know, information that's being produced, which has, which is not present in the input that we gave. Um, then we can sort of give up. And, you know, tell, tell the user that, look, you know, like, you know. The question has some relevant piece of information, but we actually were not able to figure out an answer. So so, so that's how we suppress, uh, hallucinations a little bit through this sort of citation and reference checking.

Um, but the. But the bigger problem, I would say, in the enterprise, is not that the bigger problem in the enterprise. When people ask questions is, can you even, um, you know, get that right information from all these different systems that you have? And can you get the information that is actually up to date and actually correct? You know, written by a subject matter expert? Most of the losses actually happen on that side.

I see, do you, do you imagine a world where, um, the these agents start just asking people if they can't figure something out? Like, just like, kind of routing to, I mean, I think I would kind of like that if I could just, you know, deploy a bunch of agents to go collect information from my employees.

Yeah, well, that's an that's an interesting use case. So so, uh, in in Glean, so we so. One thing that we do is we are. We are like, like, we don't think of our product as, hey, take all the information in the company, understand it, and then answer questions for people. It's very much a people oriented product. So, like, when we go and, uh, like, deploy Glean inside a company, we actually build a deep understanding of the enterprise itself.

Like, what the business is about, also, who are the different people, what do they work on? And and in fact, it's a common scenario. Like, you know, when somebody comes and asks questions and we can't answer, sometimes our answer will be that. Like, look, you know, we don't like have the information. But these are the people who are working on this topic and you can go and connect with them. Um, but then, like agents, like, you know, sort of you, you flip the the question, like, you know, where it's like, instead of humans asking agents to do work, like you're saying, agents are actually pinging.

You know, people to retrieve some information on them, from them, like, actually, like. That also seems quite legitimate because the um, that's probably the that's probably the future world, like, you know, where there is. There are these deeply intelligent agents, you know, which are sort of like a they work alongside humans. So, yeah, like, you know, humans can ask agents questions, and agents can also ask humans questions.

Do you have, like, other thoughts on where, um, like, organizations go, as they interact with AI more deeply?

Well, I, I, I think I, I'll just add one thing there. There's a lot of um, this ROI conversation that tends to happen with AI, that the like, you know, I'm gonna have AI come and like, rep, you know, do, like, a lot of the work that we do with humans today. Um, and we'll be able to replace them. Maybe I'll have a small team, maybe, you know, we'll have a single person company, you know, that is $1 billion in business.

And and and there are all these sort of, um, um, sort of viewpoints around, like how AI replaces humans. And and I feel like, you know, that's that's missing the point in a big way. Um, because, like, ultimately, like, a business and its value comes from its people. I mean, that's, you know, that's, that's, that's where the intellect, that's where, like, you know, the capacity to work is. And it's, it's. It's so much better to think of AI as an enabler to all of those people.

So, like, you know, as a business, like, you know, I, I, for example, I don't actually shrink my team size. Um, uh, I sure do grow it, but I want every to be able to do ten times more work so that we can, like, you know, we can succeed in a much bigger way. And so that's, that's, that's so. I think, like, you know, when you think about enterprises and how they change with AI, you know, people are gonna have like, one of the two mindsets, like, either they're sort of they're feeling the pressure, they want to reduce the team size.

Uh, significantly. Or you can actually go and keep investing in people. And just like, you know, in improve their top line significantly.

Do you think that the skills that make people successful will change in this world?

I think the fundamental skills remain the same. Like, I think what makes one successful, in my opinion, is, you know, hard work, dedication and desire to succeed, and I think that doesn't fundamentally change. Um, but of course, like all of us have to adapt, like, you know, one of the things like, you know, for like, think about software engineers, like, you know, the um. Maybe it's no longer necessary for you to sort of remember, like, you know what all the different APIs and frameworks are available.

Because, you know, like, AI can always sort of bring the right information back to you. So you can spend more time thinking, um, more time designing, less time writing code, maybe so, like, you know, there'll be, but I I feel like, you know, those are sort of like incremental, the these changes will happen on an incremental basis. And we won't realize, you know, by the time we have transformed ourselves. Like, it's sort of like, that's my, that's my feeling.

It won't feel like a step function change. Like, you know, like every day there's one new task, you know, that we will start using AI for. And and and before, you know it, like, you know, like, you're fundamentally different from what you were two years back.

All right, well, that seems like a nice place to end. Thanks so much for your time. Arvind.

Yeah, thank you, Lucas. It was really fun to have this conversation. Yeah, appreciate it, thanks, Yeah.

take care. Thanks so much for listening to this episode of Gradient Dissent. Please stay tuned for.