Snowflake’s CEO Sridhar Ramaswamy on 700+ LLM enterprise use cases

You're listening to Gradient Dissent, a show about making machine learning work in the real world, and I'm your host, Lukas Biewald. Today's guest, Sridhar Ramaswamy, is someone I've been looking forward to interviewing for a long time. He's currently the CEO of Snowflake, but before that, I actually met him in his role as founder of Neva, a next gen search company. And before that, he was the leader of AdWords at Google, a division that generated over $100 billion of of revenue yearly.

He started Google as an IC engineer and quickly rose to prominence, and then did the same thing at Snowflake. So he's not only an ML engineer, he's also a founder and executive. And he kind of brings all these perspectives to bear in this conversation, where we talk about everything from the future of foundation models to his leadership style to Snowflake's AI strategy. I hope you enjoyed this conversation as much as I enjoyed it. First of all, thanks again for joining us today.

I I know you're super busy. Um, and y- you know, in in learning about you, I found that you had such a fascinating path from basically an IC engineer, I think at at Google, um, not even that long ago to to running, um, you know, the vast majority of Google's um revenue to starting your own ad free uh, search engine to to getting acquired by Snowflake and then um, running Snowflake. And and I guess, um, one of the questions that came to mind, um, for me is, you know, it seemed like over and over at Google, you were um, you were kind of picked to to kind of move into bigger and bigger roles.

And then again at Snowflake, um, you know, you were, you were picked to to run the company. What do you think you're you're doing in your career that instills so much confidence in you to to take on these bigger leadership roles? Over and over?

Uh, this has happened three times. I was at a company called Epiphany, where I started as an engineer and then led, uh uh, most of the engineering of about hundred, something people there, and I went back to being an IC. Um, at, uh, at Google, uh, grew, grew that team, uh, and then at Snowflake for the first nine months, famously, um, I had literally no one reporting to me. I used to used to joke to people that I was a minister without a portfolio.

I was SVP of AI with zero reports. Um, but I would roughly say that, um, uh, I like, love, uh, being part of groups, um, where, uh, like the company, the group is making a big difference, uh, both to itself and to its, uh, its customers. Uh, I was a reluctant leader, but what I have really gotten to learn, um, and love is being able to put uh, technology, a great team and customer needs together into something that creates magic.

Um, and I didn't come into Snowflake thinking that I would become the CEO. Uh, it was more, uh, the ending point for the Neva journey, but I got to learn more about the customers, more about the company. Uh, the space is one that's deeply familiar, uh, you know, to me, um. And part of the benefit that you get from leading a large team is a part of soft leadership, you know, yeah, if you're SVP of AI with zero reports, that's what it takes to, uh, to move teams.

In that sense, it was, um, um, it was, it was, it was a good test. Um, and so I think it is this affinity for combining technology and people with customer problems. Uh, that energizes me, and, uh, look, all of us, um, progress in our careers when people see more in us than we can, uh. And I am eternally grateful to lots of people, Alan Eustace, Larry, Sergey, um, and here, of course, people like Frank and Benoit, uh, for taking that, uh, taking that chance on me.

So that's a good segue to another question that comes to mind for me, which is that, you know, I've. I've met Frank a few times and and read his book, and I think he comes across as like the most, um, sales oriented CEO you could ever find. I mean, you know, he's, uh, there's so much to learn from him, but he's very loud and, um, aggressive. I mean, the second you meet him, he's kind of coming at you and and and he's got a lot of fire in his belly.

And, um, you seem like one of the most, um, engineering oriented CEOs that I've met. I mean, you're, you're detail oriented, and you're kind of, um, understated. In fact, you know, you're you're a weights and biases user, which is a real honor. I think you might be the the the person running the largest company with an actual weights and biases account with real real runs in it. So, you know, it's like you, you clearly are willing to kind of go back into the details.

And you came up through engineering and I was kind of curious, um, you know, kind of stepping into a role as a totally different kind of leader. What kinds of things did you want to keep and and what kinds of things did you did you want to change?

I mean, yes, I'm detail oriented, but to me, um, the, uh, you know, the people aspect of leadership, uh, is really important. As a leader, um, you have to inspire, um, as a leader, you have to, in my opinion, lead by lead by example. Um, and so, yes, I have an engineering background and I love, love tinkering with products. But I've also done probably 250 customer calls over the past, like nine-ish months. Why?

Um, because, you know, I'm the face of the company. Um, and it's important that all of our in all of our customers feel like there is an actual person that is going to back them up with everything that, uh, you know, everything that we can. And so I would say there are different styles of leadership. Frank was an amazing leader. Um, I am, you know, just as pushy, probably more pushy in, uh, uh, in private. Uh, I'm also analytical, uh, in a way that, um, uh, you know, I'm pretty comfortable, uh, and uh, uh, for example, with, uh, on the topic of sales, um, I have, uh, you know, partly engineering, partly science.

I've had conversations now with Chris Dignan, our head, uh, you know, our CRO, about how really we are trying to measure, you know, processes, um, and uh, entities like accounts. And I introduced the idea of like boolean metrics, like, how do you describe the state of a world so that you can then describe the state of 5,000 entities in your world? It's like it's a way of thinking about numbers. Um, and so I think it is that it is that combination of people skill, um, and yes, a background rooted in analytics and engineering.

That makes me who I am. Honestly, I'm very comfortable with who I am as well.

That's great. Um, are there any parts of the culture that you, uh, you're trying to be intentional about changing right now? Or or when you stepped into the role,

Uh, a few things, um, one, I think it, um, uh, I I'm, I'm very open about it. Um, it is the amount of intensity that we bring to bear. I am, uh, unapologetic about work and the need to work hard to succeed. It's not a dirty word. I also tell people that, uh, it is not necessarily in contradiction with things like needing to look after yourself and, uh, needing to look after your family. And so that intensity of, um, of of work, um, across the board is something that I, um, very much emphasize again, you know, being an engineer, I'll, uh, sort of caution and say, don't infer counterfactuals from what I'm saying.

But um, the the intensity matters, um, excellence in everything that we do. Um, there is a tendency for teams to declare themselves to be engineering teams or sales teams and say other stuff doesn't matter. As a leader, I find that to be like a pretty like bad thing to do because you demotivate large portions of your company. And so I demand just as much from the, uh, HR team as I do from the engineering team. I'm also stressing virtuosity by that, I mean, the ability, um, to make changes, um, to try things out and be deliberate about when something is not working.

So you can actually course correct on the way. Lots of successful companies know only one way forward. It's like straight, they just keep going, they pick a topic and come what may, um, they go at it. Uh, I'm trying to get the team to calibrate more for what is success, what is excellence, and what happens if, for any number of reasons, you're failing. Like, how do you actually deal with it? Do you double down? do you walk away?

I wanna make sure that we have things like that in our, uh, in our vocabulary. You can walk back from everything. And here is where things like, what's, uh, you know, what's a trap door versus what is something that you can walk right back out of? That's the kind of thinking that I'm that I'm trying to instill in the place. If I were to summarize, you know, I would say I'd like to see, uh, uh, greater drive, uh, greater virtuosity, just in getting stuff done, combined with a like, a realistic view of where are we very successful, where are we not?

Where do we need to double down? Where do we need to walk away? Um, you know, it's it's. Hopefully that gives you a little bit of an idea into how I'm thinking about what are the things I wanna see in this team.

where do you think Frank would be different in emphasis? Because that actually sounds a lot like, um, you know, the the book that he wrote?

Uh, Frank, for example, you know, um, by his own admission, uh, uh, like, thought of product strategy as different from, like, you know, sales strategy. It made sense at a time, when when Snowflake was such a world leading product that you didn't need to touch it like it was, it was perfect. But that was six years ago. Um, and, uh, we are doing a lot of new things, you know, we'll talk about it eventually. We're doing a lot of new things in AI.

It's new for Snowflake. How, you know, many of the customers we talk to are genuinely surprised when I talk about what we can do with AI on top of Snowflake. Um, what this means for the company is, and as you know, AI is is a fledgling field. The science is still being born, regardless of what we think in terms of how do you produce, um, a reliable application that uses AI, so everything is new. Um, and so I stress much more of a culture of rapid iteration, yes, from the engineering and product teams, but a much tighter relationship between the engineering product and the go to market teams, the marketing teams, the sales teams.

In fact, one of the pretty definitive changes that I've made is, um, you know, you can call them, uh, whatever cross functional teams we call them war rooms here, because, you know, Frank liked martial analogies, um, and so we have a war room for AI, we have a war room for data engineering. Um, and, uh, you know, I'm stressing much more of a culture of we need to work holistically together. Um, and then finally, as I said, I bring a much more analytical approach to.

How does one measure activity? how does one measure states of accounts? What is healthy, what is not? Um, and again, that is from the lens of how do we make sure that the 3,000 person team that we had in sales is effective? Um, but I would say the big delta change is Snowflake has to move from we have a product that is ten years ahead or five years ahead of everybody else to we are creating, um, entirely new product lines, and rapid iteration is very necessary.

Uh, I'm curious if you've come across that, that Paul Graham essay, uh, founder mode, that everybody is talking about, and you know, you are a founder, edited executive, do do you have an opinion on it?

Uh, no, like I ran 10,000 person team, and anyone that has worked with me in that context, you know, will inevitably tell you, um, you know, this is an intense person. I think it has less to do. I I would think of founder mode more as a personality, uh, trait, rather than, um, you know, like a positional. Where are you? uh, you know, within a company trait. Um, I recognize foundationally, for example, that we are all lucky to be in the world of technology that is so rapidly expanding.

I mean, think about you and I were starting a bank. It's like, you know, they only grow at certain rates. There's not that much you can do about it. You have to hustle pretty hard unless you have a special in. And until about fifteen years ago, you and I would not have dreamt of starting a car company. Um, but, you know, I think like in tech, for example, you need a certain attitude that is all about creating opportunity, seizing opportunity, visualizing the future, then letting reality correct that future.

And being intense and intentional every day, every week, about how you get work done. So absolutely, I saw founder mode. I think it has less to do with founders and more to do with the kinds of people that thrive in changing, rapidly growing environments. I think all of those things could be applied to like, I don't know, successful generals.

But I guess you agreed with the idea that it's more effective to kind of sweat the details and and kind of get involved in all your direct reports activities. Versus giving people an outcome and giving them space to hit that outcome.

Those are all situational. I don't believe in hard and fast rules about it. Um, and I, uh, I'm also very, very cognizant of not mistaking my activity for outcomes. At the end of the day, it's one person. Uh, lots of people, founders included, get into hero mentality of I am the only savior for, you know, the company. Snowflake has 7,500 people. It's absurd if I believe in nonsense like that. Um, and so absolutely, the details matter, but like levels of abstraction also also matter.

Um, we wrote OKRS for the company for the first time. Why? Because I wanted that level of abstraction. I also wanted clear accountability. There are names next to those OKRS. Um, and just like I feel pressure for delivering revenue and growth for Snowflake, I want people at Snowflake to feel pressure about, hey, we got to hit these revenue targets or we got to ship this product. And so I wouldn't get carried away with, like, you know, detail for detail's sake.

Um, you need to use a variety of techniques, especially when you're dealing with like S7500 person companies sitting in, gosh, I think like thirty plus countries, um, and so you really need like a suite of tactics to be successful.

Yeah, so one of the really interesting initiatives from Snowflake that was, you know, very visible to us and our customers was was launching a foundation model Arctic. I. I was a little surprised that, you know, S Snowflake would kind of undertake the, you know, the expense and and the um, the effort required to to launch a cutting edge foundation model. Can you talk about what's the benefit to Snowflake? And and why you decided to to build your own foundation model?

Well, I mean, first of all, um, we, I think by now accept that different kinds of AI models are going to have, um, fairly large influence on tech at multiple levels. Um, at a simple level, with existing technology, not even, you know, GPT five or six about, uh, language models being a bridge between, um, unstructured data, like spoken wa, you know, spoken text, uh, to structured data. Uh, I tell people they should be like, a, uh, basically, a team that would pretty much work on.

When should the weather one box trigger on Google search? Um, like, when should you show show the weather one box? It's more complicated than it sounds, because people are like, weather yesterday, weather tomorrow, weather a week from now. Um, you know, Providence rainy weather, um, it's it is complicated, but all of a sudden, problems like that can be solved with language models very effectively. Um, so I think I think of AI in that sense as foundational technology, um, as this bridge between structured and unstructured data.

And this goes like pretty deep. You can now go from images to characteristics of images very, very quickly. Um, or you can use a company like Landing dot AI and say like, huh, I can train a model to detect manufacturing defects in pictures that are taken pretty easily. So I think that part, uh, people are beginning to get, uh, to me, where things get really, really interesting, um, is planning and tool use. If you can now take that, uh, the foundational capability that you have of going from text to structured data and say, I can now use that to call APIs.

Um, and all of a sudden, other language model can now plan and do additional things. Um, you know, that's early, but you can do a lot with it. So that's the reason why we said we need to be good at this technology. Um, as you know, we are going through a pretty uncertain phase. I would say, like, starting roughly, uh, two something years ago, from June, 2020, uh, twenty two, when GPT four was, uh, announced internally, or was used internally, uh, to now, you know, that's been the state of the art.

Uh, so late last year, it looked very much like many teams could catch up to GPT four. That's when we invested in it, but then we also realized that there was going to be benefit for us. If you're going to do SQL generation, these models could help. If we wanted to ship a model as part of Snowflake's AI layer, these models could help. We also got a bunch of it's not just the training people. We also got a bunch of inference people, um, who are very, very important in how we run these systems, you know, at runtime, as part of, um, as part of Snowflake.

Um, so these were all the reasons where we said we need to have a foundational capability. The thing that I tell people is like, in the 2000s, if you wanted to be, you know, an internet company, you had to know distributed computing. Remember, there was no Hadoop, there was no Spark, but you needed to know distributed computing. In 2023, I think like having a lot of like expertise with how foundation models work, how you can, you know, tinker with them, how you can play with them, how you can make changes to them.

I think of it as a foundational skill for any serious data company. Um, on the other hand, I tell, as I tell my team and all our customers, uh, I. We don't pretend we are open AI, we don't pretend that we can invest $10 billion into training the new model. And so there's very much like a find your place, figure out the maximum points of, uh, leverage. But the simplest answer to your question is, I think of this, um, as a basic technology that any company that is serious about data and computation, like we are as a data platform, need to be good at.

But it's kind of interesting, right? Like a lot of, you know, data companies would probably agree with everything you said, but then they they wouldn't build their own model. They might, you know, use Meta's Llama or Mistral or something like that, figuring that, you know, let someone else bear that big expense of building the model, and we'll become experts in in using the model. So I feel like you made an interesting different choice, and I'm wondering why that is.

Like, look, we did all that invested. Um, our model, it is something that we could absolutely do. Um, and I'll also be honest, there is a little bit of you need to show your curdle in the space, um, without getting carried away about it's the it's the it's the end all, uh, it's the end all beyond. Absolutely. We have partnerships with the folks that you, uh, that you talk about. We have a partnership with Mistral, we have a partnership with, uh, um, with Reka.

You know, all of those things are true. Uh, but having the in-house capability is also very, very helpful. Because, as I said, uh, the kind of people that understand how models are trained often are the ones that understand, you know, how you can do post training on them. Remember, it's a suite of different skills. Maybe one day you're very focused on how do you train a foundation model from scratch? Maybe another day you are much more into, you know, how do I generate some really good post training pipelines that perhaps we can collaborate with Meta on about the problems that matter together, uh, to both of us.

Um, there is an amount of flexibility that, uh, you know, that that that you get, um, from having these skills.

And and is the Arctic model itself? Did you design it for specifically for the text to SQL case that I know is is really important to you, or did you design it as a general purpose system?

Yeah, with the general purpose system, and it's really in things like post training. We said, um, the the two most important applications that we care about a lot, um, which is, uh, which is retrieval and things like, uh, you know, cited answers, uh, and yes, absolutely. SQL generation, including Snowflake specific SQL, those were things that we addressed in post training. We continue to have interest in things like, uh, things like API calling, uh, but the um, you know, the core model itself, um, was trained to be a good general purpose model.

And and you know, because you know this better than others, um, we don't know, um, you know, how to break models apart into their foundation. You and I would love a model that does all of the thinking and the planning of GPT four without the kind of cost that comes, uh, you know, that that comes with it. Um, we just quite don't know how to do that.

Okay, well, I found a another interview, uh, with you, where I think you mentioned, tell me if this is right, that there there's 700 gen AI use cases inside of, uh, or making their way to production inside of Snowflake.

Numbers a lot larger or not? Yes, absolutely.

Can I mean? I guess there's different ways to count that, but that seemed like an astonishingly large number, even to someone like me who thinks there are tons of um, use case cause. Can you? Can you talk about how how that works and and kind of what the the biggest buckets are of of use case for you?

Yeah, yeah, I would put them into a few different buckets. Um,

Well, wait, what's the number now? If it's a lot larger, what would you? What? What? I, I Where would you place it?

it's over, uh, you know, several hundred applications in production, over a thousand applications, um, that are, uh, that are, that are in implementation. The these projects are a little bit faster, um, because some of them are easy, and I'll talk about the easy part in a in. In a second, I would, uh, uh, describe AI at, uh, Snowflake, which is obviously built on top of the data foundation that we have, where our, uh, mission very much is to be, uh, the enterprise lakehouse, where, um, we have, uh, a ton of data.

Either that's been ingested into Snowflake or is sitting on cloud storage and is accessed with things like, uh, uh, things like iceberg format. That's the that's the foundation, there's a ton of data in Snowflake. Um, the first thing that we did was, um, we wanted to make it easy, uh, to use the the the transformative powers of language models, which is the ability to go from unstructured to structured things like that. We said, we wanna make it broadly available, um, that we called Cortex AI.

It's a model garden that goes with every Snowflake deployment, not rocket science. On the other hand, it is deeply integrated into, um, into every aspect of Snowflake. So that anyone that writes SQL can now call these models. Um, we provide a bunch of basically syntactic sugar for certain kinds of, uh, uh, language model uses, whether it is summarization, translation, sentiment detection. You can just think of it as a as a function call. If you can write SQL, you can do these things.

Um, so there are a class of, um, you know, use cases that you can simply think of as data transformation use cases. All of us have inevitably worked on projects that, you know, um, that are software engineering projects that are the equivalent of. I need to ingest these documents, um, or perhaps pictures, and I need to extract this, this kind of structured data. Um, that is as simple as setting up a pipeline in Snowflake, which is pretty much you write a piece of SQL and you say, please run this every six hours on all new things.

Um, so that's one class of applications, which is basically data transformations. It's the bread and butter, um, and that's the benefit of Cortex AI. Uh, all of a sudden, using a language model like you don't need to do anything additional. If you know how to use Snowflake, how to write SQL, you got that done. Um, we have also introduced additional, I call them primitives. Um, one is search, which, as you know, is the basis for RAG applications.

Um, and, uh, you combine that with a language model and a rapid prototyping environment, uh, called Streamlit. Uh, you can write a chat bot in five minutes. Um, and, uh, you know, I downloaded a. I think it was a Kaggle data set, uh, yesterday in the morning, because I want. I keep trying these things every now and then, and I had a chat bot ready in ten minutes. Um, it's like you download the data set, you stick it into a table, um, you, you create the search index, um, fire up a streamlit app, and off, you know, off you go.

did. Did you publish it to the Streamlit public cloud? Could we? Could we post a link to it?

Actually, it was super kick that I actually wrote streamlit apps, um, because I was like, you know, um, it's a big deal. But clearly I should, yeah, right, uh, but to me, that's the power of Snowflake. Our take is that anyone that wants to build a like a corpus backed chat bot, um, you know, is able to do that with a minimum of fuss. You don't need to know search, you don't need to know vector indices, you don't need to know which model you're using.

We wanna make that easy out of the box. And sure enough, there are classes of chat bot sort of applications, um, that people use. They can also turn into like question answering applications. If you now imagine that you are searching over a set of documents that are coming, you have a question you always want it answered, you know, that's fine. You can use the search index to generate answers, um, to generate answers for those, uh, we also have something called Cortex Analyst, um, which is more of a structured data uh product.

Uh, the idea is that, um, uh, like people, you know, for a while now, since the since the advent of BI tools, want to give business users direct access to data, so they don't have to go through the priest, the data analysts are and the tools that are BI. They're pretty hard, cumbersome, and if you want something different, you have to go back to your priest and ask them how you, how you're supposed to get at that.

So companies like Bayer have, uh, are experimenting with, um, with what we call analyst, which is a text to SQL. Um, our secret sauce in all of these is we wanna stress high reliability. We are pretty gung ho about publishing the precision of all of the products we create. Because we think part of the disservice that the AI community is doing to itself is not being clear about when can you create a reliable application and when is something a toy?

I tell people that, uh, we better not, uh, give Scarpelli, our CFO, you know, a talk to your data product that is not reliable. If he not, if he's gonna be answering que, you know, asking questions about how much revenue did Snowflake make? Um, but the fun thing about all of this is, uh, you know, if you ship these primitives, people end up composing applications. They're like, Okay, here's the little data store that's going to have a bunch of questions, and depending on the answer to that question, I'm going to use your structured data to go look up something specific.

So there's a lot of hybrid apps that people are creating, uh, but the broad primitives that are driving these use cases are, um, use ai, um, as an addition to SQL, um, so that it becomes a data transformation tool. Um, make things like cited chat bots really, really easy to create out of the box and give people the primitives for structured data access. Um, and there's a lot of magic when you give people creativity, uh, and even within Snowflake, people are coming up with use cases where I go like, huh, I wouldn't have thought of that.

like, what I mean is when you talk about like 700 to a thousand use cases, are these like internal use cases, or is this

These are, these are, these are things our customers are implementing,

oh, these are customer implemented use cases?

customers are implementing, they're different. Um, these are what we record, you know, sometimes customers, uh, like, we roughly know that they want to get started on a project and we help them get going. They're under no obligation to tell us exactly what they do. It's their data, it's their system, they, they, they do stuff. These are things that, like, we know that we have talked to them on and won POCs and things like that.

I see. So are there any that you can talk about? Because I think people are so hungry for these use cases. Like you talked about a theme of sort of like unstructured to structured data, which absolutely, we see tons of that. You talked about, kind of, um, you know, chat, you know, kind of chat applications and like analyzing data through a conversational interface. And yes, we see that. But I is there any areas where you could get a little more specific about how people are are using these things?

Uh, so, you know, in terms of the concrete examples, the talk to your data application is from Bayer. Um, and, uh, this is it is, uh, um, with analyst, uh. We basically try to simplify the problem of SQL generation, so you can have a highly reliable application. Um, and Bayer is an example of somebody that is giving business users access to data. Another example that I can talk about, uh, is from Elevance, which is a big, you know, uh, like healthcare provider, um, that, uh, is using chat bots, uh, basically to answer questions about plants.

They supply healthcare plans for a ton of companies, um, have over 150,000 admin users, um, for for these plans, um, and as you know, um, just being able to figure out which, like, if a new request comes in, uh, which plan is this about? Is a particular treatment covered or not? Can get pretty pretty, um, detailed, um, and what, uh, Elevance is doing, uh, is basically they're indexing all of the plan information for their, you know, I forget the number, it's thousands of customers, um, and hundreds of thousands of users.

And simply making it easy for their call centers to be able to answer questions quickly and effectively. Siemens has a similar pro project, um, but that one is for, uh, is for manuals. Um, they make hundreds of thousands of devices that they ship everywhere. And if you want to go repair something, you know, as I'm sure you've tried to repair your washing machine or something like that, need to find a super specific manual before you can do anything.

Uh, but these are some of the examples, um, of people that are using our tech building applications on top of it. We've also had hedge funds, I forget the name, um, that are basically using, um, like Cortex AI, the SQL layer, uh, to be able to analyze streams of reports. They basically dump a bunch of PDFs or dump a bunch of text, um, coverage text into into a table that's accessible from Snowflake. And then you can have a series of questions similar to Google Alerts, except that you can get as cited answers to these questions.

Um, and there are people that have those kinds of use cases as well. It's the I will stress the the flexibility, the operator like model and the composability. Um, that's what makes these kinds of applications possible. In fact, our vision is, uh, that if you can think of a declarative language like SQL now, having access to this, um, LLM functionality. But imagine a world in which you can also say, hey, run this piece of SQL, um, over all of the images that are sitting in this particular stage in this particular table, and run this kind of transformation for me.

Um, we have not built that, but to me, that's that's the kind of place that is going to make, um, AI like, very, very broadly usable, because you're not doing fancy applications, you're composing things in a pretty easy way. And as you can imagine, you can also, um, like, wire up a UI that lets you create these pipelines, like many of our partners do.

Yeah, I mean, I, I also love these kind of practical, basic AI applications. They they also get me really excited. And, you know, in in every interview that I found with you, you talk about reliability, and you talked about reliability earlier, and it seems like a really amazing value prop. Because that clearly is like a thing that just holds this technology back from so many companies. But I I'm kind of curious how you approach reliability, because from, you know, from our perspective, um, one difference in gen AI versus ML is customers aren't even typically, you know, measuring, um, you know, performance.

I think MLs been unreliable for so long that everyone working in ML, you kind of know you need to make these benchmarks and measure accuracy and gradually increase that. But that's not natural, you know, for someone who's who's doing these kinds of applications that that you're talking about. So, and and I do think 100% reliability. I don't know, is that is 100% that really

reliability is not a thing, it's so? How that's like,

do you? how do you? Um, how do you approach that? And and how do you make customers feel like your your solutions are reliable without disappointing them?

uh, it's a, it's, it's a good question, and sort of my first level answer to that. At the level of AI, you know, I obviously run into friends like, you know, I'll like doctor friends, for example, they're like, Oh my god, like this is, you know, amazing, or it'll replace my job, or, oh, come on, it can't like, possibly match a human. And what I try to tell them is like, listen, all of us have the tools to be able to evaluate these kinds of things.

You can ask very simple, even medical questions like, you know, um, uh, what's the accuracy of this diagnostic? Like, if you're going to use it in a medical context, they have, they have a vocabulary. It's like, it's not like interpreting radiographs is hundred is 100% science, even when done by humans. They get it wrong all the time, they miss stuff all the time. And so I think, basically, having an engineer's mentality to how you approach software is also true for AI applications.

The first thing we I try to do in all the conversations and our teams try to do, um, is, you know, there are trade offs involved, and you should, you need to. You need to have a basic understanding of, like, what right and wrong is, and stop thinking of models as oracles that are just going to give, um, answers, uh. And then if you talk about chatbots, for example, the thing that I tell people again is like, if you were to create an internal search engine for a particular use case.

You wanted to look through insurance plans, you said, Okay, absolutely, you're going to come up with, okay. Here is a list of 200 queries that I'm going to test this search on. And I'm going to say, um, you know, yes, you can get into more complicated NDCG metrics and stuff like that. But you can answer the basic questions of did you deliver satisfactory answers for a fraction of your questions? Um, and that's the methodology that we have taken.

Um, we, uh, you know, have acquired a company called Truera that we are in the process of integrating with Snowflake. Is how do you make something like that? By the way, this is just like software engineering testing. This is the bread and butter that you live on, um, which is you need to be able to actually re experiment and measure, um, what different systems do. Uh, similarly, when it comes to Cortex Analyst, uh, we tell people there are precision recall tradeoffs that are involved here.

Uh, the simpler you make the schema, the more cl clear the description that you give for the business metrics, the better the system will do. And absolutely, you should write down a list of your top hundred questions that you want to ask this thing and have an analyst vet it and say where we get things right, where we get things wrong. By the way, this process can actually become an aid in making these systems better. So if you have a set of examples that are positive examples, if you have a set of examples that are negative examples, we are then able to take them and say.

If a new question comes and is very much like a positive example that we already saw, you can then begin to get into the real science of, like, okay, should you answer this question or not? Or if this is a dead ringer for a question, you know you should not answer, you can refuse to answer that question. It is it is that, you know, it's that iterative methodology, um, that I think is important, um, and, uh, honestly, I think that is what is going to make people deploy applications that they can believe in.

What else do you think holds back your customers adoption of these LLM applications and and features?

Uh, you know, now comes the hard part. This is all cool stuff, it's all cool tech, uh. But at the end of the day, the thing that I tell all our customers also is like, please make sure you, you are thinking about business value. Please make sure you're thinking about what is making money for you, what is saving money for you. I mean, at the end of the day, like, those are the things that are, you know, that that that sort of matter.

Uh, and uh, uh, and that work is really hard. As I said, the average Snowflake project takes like three to six months to get done from from start to finish. Um, and uh, that's the sweat that we are going through in terms of working with customers, um, in order to identify situations in which AI can make a difference, but deliver a tangible value for, uh, for the business. Um, and this is the good news about a lot of this stuff is none of this requires a GPT five.

Uh, and uh. And to me, that value creation is the is is the slow part, uh, but it's very much there.

How far do you think Snowflake should go in terms of delivering like complete package gen AI solutions to customers? Like, do you think it's your role to give composable pieces like, you know, you mentioned, um, you know, certain types of search? And you know, like, obviously, streamlit's like a wonderful canvas to to build things on top of. How, how, like, far up the stack would you go to to make something more complete that customers could could use?

No, I don't think there is a hard and fast, uh, uh, rule that we have about how we, um, how we think about this. Absolutely, we are a data platform. Our strength is in creating technology that is broadly applicable. Um, we work with a number of partners in different areas. People build, uh, you know, security systems on Snowflake, they build, uh, composable data, uh, customer data platforms, they build observability platforms, um, and obviously many concrete use cases like, uh, you know, AML or fraud detection.

Uh, there's, you know, we are very much a platform, um, and, uh, uh, you know. But when it comes to the primitives that we we provide, um, all of us are on this journey of discovering what these right primitives are. Uh, and for example, like, you know, with search, um, which is, uh, one of the things that, like, we have worked on, it's close to GA. Um, the thing that we are realizing is that having a SharePoint connector that is then hooked up to search so that you can now come up with a rest endpoint for how do you search over all of these documents is actually super valuable.

Um, and the next delta in that is, well, uh, you really need a need to figure out a way to obey the ACLs of these documents, like in drive, for example. Um, people, people will say, yes, I want you to index all of the drive documents and have them, you know, have an interactive access via like a chat interface to it. But you need to make sure that you obey, um, the ACLs. Is that part of the platform?

Is that an application? Um, that's actually a question that we are still debating in terms of where should that, where should that live? Um, but we don't really have a religion about where this boundary, uh, boundary stops. We work with a number of folks, we want to be, you know, good partners to a lot of folks, uh, to all of the folks that we work with, uh, and we think very much in terms of what are horizontals that apply, uh, to many different verticals, to many different countries.

Uh, but outside of that, I don't think anyone should be thinking about drawing lines for what they should do and what they should not do. It's just too early to call those things.

My sense is that maybe this is wrong, but my sense is that other kind of companies in your space, like Databricks or Microsoft, um, is kind of more willing to compete with their partners and more willing to add on things. Where Snowflake is kind of stayed more contained in the data layers. Is that is that accurate?

Um, like with the Microsoft, for example, you know, they have office, um, and if you think about it, is like one of the biggest outlets for AI. You know, I want to click on twenty different, like, uh, you know, word links that, uh, no one, uh, you know, they want that, they want that answer, and so I think the situation is, uh, is is different. Um, and, uh, you know, every, and obviously, Windows, as a platform, has a long and checkered history of what it is and what it is not.

Um, every company has to find its, uh, uh, its way. Um, we've been pretty happy, um, with our approach to how we do platforms, and we have, uh, wonderful partners that are extending, uh, snowflake, like, uh, like Kumo with predictive analytics or relational AI with, like, you know, um, with like graph SQL and, uh, you know, graph databases. Our landing, which I talked about earlier, Andrew Ng's company, um, that's doing vision models. Um, it's, uh, it's, it's, it's, it's a journey.

But when it comes to, you know, things like, uh, AI boundaries, my take is like, it is really, really too early to say, you know, where is the platform and where is the application just yet?

Where do you think we are in the AI hype cycle? Like, I, I feel like inside of Silicon Valley right now, there's a lot of sense that, um, you know, maybe there's some kind of disappointment. Like, you know, we've tried all these things, there's like a culling coming. Sometimes when I go outside I think you of Silicon Valley,

I think your culling has happened.

you think the culling has happened? Okay, interesting. I think that the the volume of investment in AI is still pretty high. And when when you go to enterprises, it does seem like they're still ramping up their, um, their AI efforts right now. That's that's my experience, but you're actually

enterprises are absolutely, they are slowly but surely ramping up, um, their AI investments. As honestly, like, the tech also gets simple. The idea of needing to, like, extract a bunch of documents, um, put them into a vector store, figure out what model you're going to use, stand up a new binary to actually, like, host your chatbot. That's a lot of work. Uh, the technology is getting simpler, and so I think that's the reason you see, uh, that you will see broader, you know, broader adoption.

do you think there will continue to be more and more foundation models? Like, are you one of those people that imagines a world where there's thousands of them for different use cases, or do you think that it will narrow in the next year or two?

Uh, there is the um, I'll, I'll, I'll put it into three categories, um, which, uh, you know, one is, uh, what do I want as a person? um, and that's number one, uh, and number two is like, what is the current conventional wisdom? Uh, and number three, I think, is like, just as possible, is just kind of boring and no one wants to talk about it. Um, okay, first,

Wait, sorry, what's number three?

uh, you know, a an equally likely possibility that which is sort of boring and no one Oh, I see, really wants to talk about.

I see, you know, I I think it's great that you separate those things. Because I always ask people on this podcast and I think people conflate what they want and what they think is gonna happen. Or that always matches in everybody's, in everybody's answer, you know, it's like, you know, surprisingly, everyone believes in the trend that that actually benefits their specific business. So I'm I'm excited to hear your, um, your measured response here.

I have a I have a quote for that, and the quote for that is like, the best religion to believe in is the one that's aligned with your business interest.

You're right, right, exactly.

Uh, what do I want? Uh, you know, I want a world in which there is healthy competition. Call it at least dozens of companies in language models, uh, and, uh, or AI models, and the reason I say that is, um, you know, like, like monopolies monopolize, it's just like how stuff works. Uh, any ecosystem in which there are two or three players, um, reaches a point of stagnation. Okay, this is like hallelujah, toothpaste. It's just like there are lots of people that, you know, that that make it, and so I think

Yeah, it's interesting that Sridhar, it's interesting you say that as the person that ran Google ads for for a long time,

I also left Google Ads and started a company to compete with Google search. Because I said, there are not enough search companies. I can at least argue that I like, you know, I live my religion,

there you go.

uh, so I think like that's a healthy thing. That's the outcome that I would, um, that I really want to happen. Cause I think a world in which there are four companies making founda, or three companies making foundation models, that, let's face it, that's just a tax. It's just like, that's the practical, you know, aspect of how those things end up, end up working. So that's what I want as a person. I think current conventional wisdom is that, oh my god, there's going to be a GPT five, and there are going to be basically three players that can invest that kind of, uh, that kind of money, um, which is open AI and Anthropic, and, uh, and and Google, and

maybe there'll be one or two others that catch up. Um, I think the third sort of voting, uh, outcome that, like, we really don't want to talk about, because, you know, it's kind of boring, is, yes, GPT five comes out. It is marginally better, but not like a thousand times better. Um, and other people, but there are cool new things, even GPT four O did things that GPT four did not, you know, kudos to OpenAI for consistently pushing the boundary of what is, uh, what is possible.

And a bunch of other companies, they sort of like grind and claw their way to parity, and then we take half a step more. And, you know, rinse, lather, repeat. Um, I actually think that the third one is the is is a likely outcome.

Nice, I got a hot take from from Sridhar, I love it. Do you do you have? Okay? What do you think about, um, agents? I mean, it's clearly like a bunch of agent companies with astonishing valuations. Adapt you mentioned would be an early one that maybe wasn't so successful, I'm not sure, but do do you think agents relevant today? Do you think agents will be relevant a year from now?

I think there's value in the concept. If you think of it, um, as a, I don't know, a souped up Google alert, um, that can do interesting things, actually take actions on your behalf. It's like, that's kind of a cool thing. Um, well within the realm of existing technology. This is why I put such an emphasis on what does it take to have reliable primitives? Um, I'll tell you, like an agent that I'd love to see, you know, we've built prototypes for this.

I look at revenue reports every morning, five before five am, which is when I start work. I've told my team is like, you need to tell me, like, I need to get a report that says how much money did we make yesterday? Um, and, you know, look at it every morning. And sometimes something is off with it. Um, usually have to wait or poke around, go look at some dashboards, then figure out where it is going.

Uh, most of the time, it's fairly rote work in terms of, okay, if this happens, then you need to go do these other things. Um, now, imagine if I could just codify a bunch of rules just in text about, okay, here's a report, please look at this. And if these metrics are off in this fashion, these are the drill downs that I want you to do. Um, and depending on the drill downs, you might need to look, look some more.

Um, it's not that hard to describe the process. Um, now, if this could all be turned into, you can, you can call this an agent, it's an analyst agent, it's an autonomous analyst. It's not going to solve every problem for you, but if you could solve 60% of the work, which is unique every time, by the way, so it's not like you can write a trivial program to do this. Um, because it would have all of these different cases, but if you can encapsulate this very nicely and say, okay, instead of the report, you get the report, you get a notebook that has done the initial analysis, so you can actually go look at the data.

And you or somebody in your team can pick up from there. That is actually moving work forward in a pretty meaningful way for how you think about analysis. Um, is that an agent? It is beginning to be an agent. Um, to me, the thing that people have trouble with is if you put highly unreliable systems together, um, you pretty much get into the world of like, stuff is multiplying and is just very unreliable overall. The more reliable you can make the core components, whether it's search or getting structured data right, the more likely it is that people can develop meaningful applications, can take, can develop meaningful, you know, call them agents, meaningful pieces of software that can take a set

of actions that are useful and contribute towards a goal. Like the core concept makes a whole lot of sense. Uh, you know, the devil's in the details, hopefully not like the self driving cars, where, you know, yes, the the the intellectual path is clear, but the details get significantly in the way. Um, we need to develop a vocabulary for how to describe agents, what is a reliable agent, um, and how does, like, one move progress forward again?

Um, but I think the core idea has a lot of merit.

It's really interesting, I would think. I mean, I almost feel like, if if you can write your own streamlet apps, it seems like you should be able to cobble an together an agent that asked questions about your revenue report. Have you? Have you tried that? D does it break down somewhere?

Uh, I have not tried it. Uh, I, we, uh, you know, we are, we are. We are sort of work, you know, obviously, with things like our company revenue metrics, we have to be careful about, uh, like, how broadly we make it available. Uh, we are working on things like having having Cortex analysts work on that on the data, and then the analytical parts absolutely will come, you know, will will come after that. It's not, uh, as I said, it's not that hard to describe a recipe for how do you debug problems in these spaces, but you need the generosity of language models to make these recipes work.

hm, okay, so last question when when you look forward into the next eighteen months, there seems to be so many tailwinds behind gen AI and so many reasons to get excited about it. And and you all are obviously really well positioned. What are the things that that worry you in terms of your your gen AI strategy? Like, I, I, I see, you know, there could be like high profile, you know, kind of enterprise failures. It could be that it's impossible to get the technology reliable enough to get real adoption.

What, what, w what? What keeps you up at night when you think about Snowflake's gen AI approach?

I mean, as I said earlier, we are not the hyperscalers, we are not one of the big three when it comes to when it comes to AI. Uh, a real fear that I have, you know, combined with a little bit of excitement about what is possible, is like a GPT five. That is indeed way better in planning and what it can do than anything else that, uh, you know, that there is. Um, we have, uh, we have good Um, but that is something, like any sort of unipolar world is one that we should be concerned about.

As should other data companies that don't have, um, access that don't have access to this. Um, and I worry less about spectacular failures with AI. That's just not the approach that we recommend for our customers. Um, it is very much a incremental, you know, start with customer value, don't make giant investments before. It's the same as, you know, how I I would I would tell people to approach building a new system, um, which is, you know, it needs to be built iteratively.

Um, I worry a little bit less about, you know, about that part.

I said, thanks so much for taking the time. This is a real a real treat to get to talk to you about all this. Thank you.

Thank you, Lucas. It was great to chat. Loved your Question. Thank you.

Awesome. Thank you. Thanks so much for listening to this episode of Gradient Dissent. Please stay tuned for future episodes.