23 Oct 2023 · 32m · Kawal Gandhi, Sarah Guo, Elad Gil · 22 quoted facts
Kawal Gandhi of Google Cloud argues that enterprise AI succeeds when companies move step by step, not when they chase the newest model. Teams should start with efficiency, then productivity, then creativity, and build trust along the way. Cloud's edge, he says, is secure data handling, model choice and easy operation, not any single model or chip.
The central tension is excitement against caution. Engineers want to prototype quickly, while enterprises must think about guardrails, logging, cost and safety before scaling. Gandhi says costs are falling and the GPU shortage matters less once the conversation turns to deployment. He also admits that multimodal AI and inference at scale are still early and unpredictable.
Gandhi describes a path from efficiency, to productivity, to creativity, with trust growing at each step. Customer support is his example, where a team can watch whether customer satisfaction scores hold before expanding.
Workspace features such as summarizing, drafting email and suggesting slide images were tried inside Google first. They later launched externally as Duet AI, which shows how internal use shaped the products.
Gandhi says customer data, fine-tuned adapter models and weights are kept in the customer's own tenant and treated as their IP. He frames this as a reason to move carefully, since rollbacks after a mistake are expensive.
Google offers its own models, domain models like Med-PaLM and Sec-PaLM, and open models like Llama through its Model Garden. Gandhi estimates the model is 50-60% of the work and the rest is integration, upkeep and operations.
Gandhi was surprised that coding assistants spread so quickly among engineers, who were already productive. Elad Gil says startups show the same pattern, with developers building for themselves first and mid-market companies next.
Duet AI is Google's generative AI layer in Workspace, which covers documents, email, slides and spreadsheets. Gandhi says the features were tested inside Google first and then launched to customers.
Gandhi says it depends on how mature the organization is and whether it has a team that knows how to build and tune models. He also says companies often prototype with an API first before they think about managing and training their own.
Gandhi says it comes up in conversations, but the focus soon shifts to security, access control and deploying AI inside the company. He says the conversation often doesn't return to chip availability.
A TPU is a chip Google designed for AI workloads, while a GPU is the more widely used alternative from NVIDIA. Gandhi calls them reasonably interchangeable, and says Google wants to hide that choice from developers and invest in making inference cheaper.
Gandhi refuses to predict specifics, saying AI often surprises people. He points to Gmail translation, which he uses so his mother can read his emails in Hindi, as an example of progress.
Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.
Google's first internal generative AI uses were in Workspace, covering documents and email, such as summarizing and personalizing text.
“Was was around workspace, just using our documentation, our email. You all, I'm sure, use it. So it was like, can you summarize this better?”
Google's internally tested Workspace generative AI features were launched externally as Duet AI.
“Was gradually launched, and now it's Duet AI, uh, part of workspace.”
Kawal Gandhi uses Gmail's translation feature so his mother, who speaks Hindi, can read his emails in that language.
“I like the translation feature for my mom because, uh, she speaks Hindi, she's really fluent in it.”
Google Cloud's Vertex AI Model Garden includes domain-specific models such as Med-PaLM and Sec-PaLM.
“That's where we have domain specific models, so it's Med Palm, uh, sec palm.”
Kawal Gandhi said open source models such as Llama and Stable Diffusion were coming to Google Cloud's platform.
“I'm really excited to see, like Llama, stable diffusion, uh, other variations coming onto the platform”
Gandhi said Google Cloud has invested in model operations, including handling model drift, tooling and safety.
“Because we've spent a lot of time really specializing around the model drift, operations, tooling, safety around it.”
Elad Gil relayed Ankur Goel's view that large enterprises adopt LLMs in sequence, often jumping to fine-tuning before first proving an idea with an existing API.
“there tends to be a very sequential adoption of LLMs by larger enterprises or people training models for the first time.”
Gandhi described enterprise AI adoption as moving in order from efficiency to productivity gains to creativity.
“it's like efficiency, productivity gains, and also creativity. And I did it in in order because inside the enterprise”
Gandhi gave customer support as an early use case, where the key measure is that customer satisfaction (CSAT) does not drop.
“How can I make my support better? So and not drop my CSAT, that's the KPI I'm going to look at.”
Gandhi said enterprise interest in generative AI started in sales and marketing.
“It started from sales and marketing, like as soon as these came out.”
Gandhi named gaming as an industry already scaling multimodal AI.
“I think gaming is a really good industry which is kind of scaling out multimodality.”
Gandhi said more than 10,000 people had been certified in generative AI on Google Cloud Platform.
“We have more than 10,000 people certified now, just more on GCP, uh, generative AI globally”
Gandhi said fine-tuned adapter models and related artifacts stay within the customer's own tenant on Google Cloud.
“The adapter model, the certification, everything stays in their tenant.”
Gandhi said engineering teams were adopting AI coding models faster than he expected.
“I'm seeing engineering adopt it much faster internally now, uh, with the coding models, with the open source”
Gandhi said GPUs are widely available and many engineers are trained on them, which is a consideration when choosing between TPUs and GPUs.
“Because there is this that was, you know, externally available, and a lot of people are trained up.”
Gandhi said Google was investing heavily in its next TPU generation, which he referred to as V5, with a focus on scaling inference.
“We have V5V, etcetera. We are investing, uh, heavily, and you know, others are doing.”
Gandhi said the NVIDIA GPU shortage comes up in customer conversations with Google Cloud.
“It definitely is mentioned, uh, as we think about it in the conversations, uh, it's always the the discussion that comes up.”
Gandhi said Google Cloud aims to make new models such as Llama available in its Model Garden so customers can use them within existing applications.
“can we make it available like Llama inside our model garden? And then they can use it inside their application”
Gandhi estimated that models make up about 50-60% of the work in an enterprise AI project, with integration and operations making up the rest.
“Um, and I think models, in my mind, are like, fifty, sixty is the is the work around, and how do you leverage your current investment?”
Gandhi said Google Cloud is exploring synthetic data, including for use in regulated industries.
“And we're also looking at synthetic data as well. Can we use in regulated industry some of the synthetic stuff”
Google Cloud offers data labeling and related services through a partner marketplace integrated into its platform.
“So we have, uh, the partner ecosystem through that, we provide a lot of partners, so exactly the marketplace.”
Gandhi said data from partner labeling services is kept in the customer's project and not shared with other customers.
“So we make sure those pipelines and entitlements stay in the project.”
Gandhi explains how advertiser needs for data and compute led him to Cloud, and how Google tested Workspace AI internally before launching it.
“we gradually launched it externally. Because we see a lot of progress that we make internally”
Covers domain-specific and open models, and why prototyping with an API comes before fine-tuning and deployment.
“It's when you really start thinking about deploying it, managing it, monitoring it”
Gandhi lays out the staged path enterprises follow, and which departments adopt AI first.
“it's like efficiency, productivity gains, and also creativity.”
Text leads today, with audio and images next. Success depends more on a company's ambition than on technology.
“It's less about the technology, it's more about like, where they see themselves in the industry.”
Costs are falling and certifications are rising, while data security shapes how carefully Google moves.
“We have more than 10,000 people certified now”
Gandhi frames chips as an infrastructure detail to abstract away, with the next focus on cheaper inference.
“I think inference will be something really needed.”
The shortage fades from customer conversations as deployment takes over. Google is also exploring shared and synthetic data.
“we're also looking at synthetic data as well.”
This week, Elad and I are joined by Kawal Gandhi. He works in the office of the CTO at Google Cloud, where he's the lead for generative AI. Gandhi comes from a long history of working on search and ads at Google before Cloud. Welcome, Gandhi. Thank you, How did you end up working on cloud and then AI in particular from, um, from other projects at Google?
sure. Um, I really worked deeply with a lot of our advertisers around search and ads for shopping and travel, uh, especially commercial based queries. And as working with them, they required a lot of storage, compute infrastructure, constantly around making their ads perform better, which led us to cloud. And then cloud solutions around using some of that to create smart analytics, machine learning pipelines, more around documentation, AI, conversational AI. And here we are, uh, we've been doing it for a while, but now it's generative AI, uh, where?
How can you make that customer experience, uh, much better with the information that they have?
And and just in terms of beginning to broadly at Google, incorporate AI into GCP, like, what was the origin story of that? Is it TPUs, APIs, some customer need that you specifically saw?
As we were, uh, getting into the Google Cloud, this goes back into, uh, how can we provide them with, uh, latency, high response, uh, better experience with data? Is how our customers started kind of leaning towards Google, in my mind. From the beginning, it was more around machine learning and AI as a differentiator to work with Google, and how could they use their data better in on our platform was a constant ask. So as we were kind of on the journey of Google Cloud, it was all about data, AI, storage, privacy, security, and kind of having that same deep technology that we used inside Google.
How could we leverage that and offer that in market? Uh, so lots of learnings, because what we built internally, we had tools, frameworks, etc. that took us time to kind of make our platform rich for our customers. From regulated to non regulated environment and they and how to leverage some of their current investments on our platform.
What are some of the internal use cases that have really driven that behavior in terms of the stuff that you ended up building for your customers? I know that, you know, a lot of what Google does is sort of dog food its own APIs or products, and then it starts launching them externally as sort of a service that other people can use. What? What were some of those first applications of generative AI that occurred internally that then caused you to decide to do these things externally?
Yeah, the only ones I think is it's all public now. Uh, was was around workspace, just using our documentation, our email. You all, I'm sure, use it. So it was like, can you summarize this better? can you personalize this better? can you offer me a suggestion? And all this kind of gradually, as we la thanks, as we, you know, tested it internally in dog food, we gradually launched it externally. Because we see a lot of progress that we make internally in terms of efficiency, productivity gains, folks can use it in their, you know, spreadsheet creation, etc.
Uh, was gradually launched, and now it's Duet AI, uh, part of workspace. So, uh, these are constantly being dog food and tested, we call them experiments. And as the research team leans in and looks at some of these, we add it to the platform and bring it forward in our products.
Is there a single um feature or a product you launched within the internal Google version of Duet, the workspace AI products that have gotten the most uptake or that you're most proud of?
Yeah, I think we're seeing it across the board, uh, aha moments as as we launch, we're seeing it in documents, uh, in terms of generation, uh, summarization, we're seeing it, uh, now in slides with suggestions on images and new image creation. Which used to take time for someone to go ask a studio or an agency. And you have a prompt that you can give and say, here's something I'm thinking about. So, Sarah, we've seen kind of intake on all those features.
Also, email generation has been super helpful from productivity perspective, not only for the consumers, but for enterprises as well. Uh, security, we don't talk about it a lot. I think it's super secure how it's sent, how the links are used. Um, those are something we really take it seriously.
How far, uh, do you think we can take it with email generation? Cause I maybe spend four hours of my workday just trying to keep up with my inbox. So this is of great personal importance to me.
I'm not gonna predict because you predict something and something always comes back and surprises you from a technology perspective, from a model perspective. So I am looking forward, um, to see what surprises you. And I'm sure we'll speak again in six months and you'll be like, Gandhi. I saw this feature inside Gmail, it's really helpful. I like the translation feature for my mom because, uh, she speaks Hindi, she's really fluent in it. And when I write my emails now, I can just say, translate this when she reads it.
Now, it does it automatically because she's using that app, so which is like, fantastic. So I think it just bridges the gaps for a lot of our users internally and externally.
I know that there's a lot of different services that Google Cloud provides, particularly around generative AI. There's a series of models, there's lots of domain specific models that Google's really been forward thinking on, things like Med Palm two or Sec Palm or other things like that. Which of those are currently available, and how do you think about that roadmap of things to expose over time?
Sure, and I think, uh, it's helped. A lot to know that we've invested on the AI infrastructure. So you've seen people training not only using our first party models, but also training their models and bringing it onto our platform. The next level up is the the vertex, what we call is capabilities of models that you can use from our model garden. That's where we have domain specific models, so it's Med Palm, uh, sec palm. These are early days, domain specific.
You can chat with it, you can kind of get validation of things. The notes that are written out from a nurse to a patient, healthcare from a doctor, and it kind of puts it in the right format that can be, you know, saved up. Uh, so it's early days. We're seeing it from just foundational model capabilities to models that are built by our users and deployed on our platform to also open source models. I'm really excited to see, like Llama, stable diffusion, uh, other variations coming onto the platform, um, but what's key, lah, in all of this, is your data is secure.
You wanna be like, you know, fine tuning it on the platform, and then all the model operation, uh, should be easy. Because we've spent a lot of time really specializing around the model drift, operations, tooling, safety around it. Super early, but I think those are the elements that will differentiate on the platform, uh, from the from the next few quarters.
That's that's super interesting. Yeah, there's, um, this guy, Ankur Goel, who runs a company called BrainTrust, that's focused on Eval and a couple other things. And one of the points he makes, which I think is pretty interesting, is that there tends to be a very sequential adoption of LLMs by larger enterprises or people training models for the first time. So often they jump straight into fine tuning, and then they suddenly realize, wait, wait, wait, let me just prove it out in GPT four or, uh, Bard or some other API.
And then let me iterate towards something that actually works, and then maybe I go and train my own model. Do you see something similar in terms of the pattern of behavior that a lot of your partners adopt, or do they just jump to your APIs? Or what's the most common sort of sequencing of people really adopting this technology?
Yeah, I think, uh, it's always the case. If you look in history, it's like, Hey, I got an API, I can build a prototype and it creates a lot of excitement. It's when you really start thinking about deploying it, managing it, monitoring it, using different models to train them, have responses that really answer for capability. You start thinking deeply about it inside an organization. So I see this as early excitement. It's not hype, so I want to separate couple that it's real excitement because engineers we love it, like I've been one.
You want to grab something, you don't want to have restrictions around it, and you want to show the art of the possible. And I think from those experiments, we're going to see in the next year or so capabilities that are, you know, available, uh, to the users and to inside the inside, the enterprise, where they will see differentiated output and capabilities. Uh, that's how I see it, so I'm excited in all those, all those phases, by the way, uh, because it's like the best creative time for engineers to solve problems that they've been thinking about, solving for a while.
Uh, and now they have the tools that they can download, whether they can grab an open source one or a closed one. And we can then kind of see how they're using it and evolving, evolving the platform along with it.
Do you advise, uh, your, your customers, and, you know, major Google Cloud customers, how to think about when they need to train their own models, or when they should be using domain specific models, like, what advice would you have for them?
Yeah, sure, Sarah. So we we deeply think about it because it's it's. You have to be responsible. Once you're building and fine tuning a model, you know what are the guardrails, what are the use cases, what is the cost you want to put behind that. Because it's a continuous, uh, learning process, and it also depends on the maturity of the organization. Do they have a a team that understands how models are built, tuned, and then brought forward?
And this? It doesn't mean that you cannot invest in it, it's just that where you are in the cycle is very important. So we lean in, talk about that from a, you know, board level perspective, from a strategy perspective, and then thinking through cultural transformations as well. So it's not limited to just one dimension in my mind. And it really helps them to think through how they see this transformation also happening inside their organization.
Do you track the common use cases that your customers end up focusing on? Because it seems like there's almost three different things that people tend to do. There's, you know, let's go experiment and just see if this is interesting. There's I'm gonna use it for specific internal tools. I wanna make customer success better, or I wanna make, you know, some ops workflow better. And then there's the people who are actually doing it for external products. You know, I'm actually gonna launch a feature that includes generative AI.
Do you have a sense of sort of how that breaks out across your customer base and the proportion that's doing each at this point in the in this early cycle?
Yeah, it falls into like, for me, it's like efficiency, productivity gains, and also creativity. And I did it in in order because inside the enterprise, folks want to be creative, but every time they think about efficiency gains inside their workflows. So how can I make my workflow better so I can invest back into my group? And then how can I make productivity better? Then I can make them creative. And I think we're seeing the flow inside the organizations.
So what you touched on, in that is right. How can I make my customers successful? Like, let's start that use case efficiency. How can I make my support better? So and not drop my CSAT, that's the KPI I'm going to look at. And if see if that matches, and I go into productivity gains, offering promotions, next recommendations, then the trust increases. So let's put another dimension there of trust. So now you have your efficiency, you're going into productivity, you trust more and more.
And the world we look at is like, how are these things going to work in the system of intelligence or agents in the future? That the trust goes up and then you really free up your human capital to be creative, right? So I think we are on that trust cycle of like, how do we trust these models? They do what we think they don't go off, they don't hallucinate. All those things are important. So that's how we, how we think about it.
And then gradually make progress towards it and not get super excited. And then find that the KPIs are not working, for example.
Are there particular verticals outside of technology you see the strongest adoption in so far, or the strongest interest?
Yeah, I see a lot of interest. It started from sales and marketing, like as soon as these came out. From a horizontal perspective, I think it, that's an intersection between horizontal and vertical. For every department, whether regulated or unregulated, there's like bottlenecks in content creation, distribution, all that became like you could see the efficiency gains in immediately and creative gains, uh, and then making that process and workflow shorter. So that was one customer care, you all touched on.
It is another one that goes vertical across horizontal. Even if you're a B2B, you're touching a supplier and a supplier wants a good experience with their current, you know, provider. So how can that experience get better? And then now we are seeing them verticalize on the internal experiences. So how do I make my employees experience better? So you're kind of seeing that same technology applied internally. It's like, Oh my god, I'm looking for my HR benefits and do this XYZ.
How? Why can't you make that easy, right? So so, Sarah, we're seeing the internal process now, whether it's regulated or unregulated. And I think there are these huge, um, areas that we can, you know, provide opportunity with AI, that's my belief.
I'm assuming that the vast majority of this starts with text, right? text analysis, generation, etc. Where are you seeing, if at all, multimodality, voice input, text to speech, output, any sort of other, even within sales and marketing, perhaps other modalities in people's use.
Yes, uh, so I think the core right now is language and text, and conversation fits really well into it. Like, you know what we're doing. Can you convert it into a script? Then you do translations around it, and you have a distribution mechanism. Sounds easy now, but was really hard three, four years ago. Multimodal, I think we are early stages now, so you're gonna see the early models, which are audio based. So I think about it like text, then images, then images and text with audio, and then you're kind of combining these medias together.
So, as I said before, the trust on these, as it increases, you're gonna see multimodal coming forward. There are things to think about, like identification, creation, deep fakes that get created, voices that are used out of band. How can we provide that layer of safety around that, especially when it's used internally? And the data and the creation becomes, uh, an element of cost. How do we store that? how do we retrieve that? how do we scale that?
So if, if I give you an example, I think gaming is a really good industry which is kind of scaling out multimodality. And we have a lot of customers who are using that now. Think of that when you're going shopping and then the latency and the progress bar, like, why can't I see the video coming, right? So that's why you see it's centralized with some organizations who are experimenting, but we got to bring it at scale out, which will be like, fantastic to see across, not only the web internally as well.
Are there any particular patterns that you see, either organizationally or the types of projects that people start with that make your customers more and less successful with their AI efforts?
Yeah, I think, uh, it goes back to kind of the how much investment, how much belief, how much, you know, they want to be in the vision quadrant. First of a kind versus fast follower, uh, all plays into it. It's less about the technology, it's more about like, where they see themselves in the industry. And if they're seeing themselves as like, hey, we want to really build something and then scale out and then keep investing in it.
I've spoken with a lot of customers, even governments, around the interest in AI. I have not seen this, it's so high, we have just scratched the surface of this right now. So we can see a lot of these gains, which then can get invested back in the in the business. Um, so I I see, I see a lot of positive signs around that.
What's the most expensive part of the investment cycle? You think? I'm, I'm sure, it varies dramatically from customer to customer, case to case. But when you say, you know, it depends on commitment level and investment, like, what, what's expensive?
The expensive part is now becoming cheap, it's the the models, the availability, the usage of the platform. Those were the things that were really expensive. I think you, you. If you do a a cost craft, uh, you know, curve right now and the investments you all are making into the ecosystem. I think we're seeing that just phenomenally coming down, which will allow people to adopt more. So I think we are. We are really entering a phase of Sarah, and in that like a growth phase of just people adopting this more and more.
We are seeing signs of just like, how fast can you do? how fast can we learn? Um, we've had training classes and we've had people certified. I've never seen this for a year. We have more than 10,000 people certified now, just more on GCP, uh, generative AI globally, right? Ready to, you know, write code, take advantage of code capabilities, uh, engineers, productivity going up, which used to take time, migration of systems. Can we make that easier?
There's ton of use cases that are helpful with we're just bringing this forward on the platform.
Are there any anti patterns or big mistakes you guys have made internally, or that you see customers make? When they're trying to get these, um, efforts into production, or even choosing use cases?
Not mistake, but we, uh, think deeply about the data that customer bring to our platform, and they're not mistakes, but, uh, we worry about any of this. Like, you know, not being used in a way or something happens, so we take that very seriously. Uh, so we run a lot of drills, we make sure that data is kept secure, the models that are trained, even if there are early days, no leaks happen. The adapter model, the certification, everything stays in their tenant.
So from the beginning, we just make sure that all of their data, all of their models, all of their weights, that's like their IP, and we want to safeguard that. So none of the mistakes, just deeply have to think about how fast we move and and what are the checkpoints we need to make in between to make sure we don't move too fast that we get into mistakes. Uh, those rollbacks are expensive, by the way, then you have to again educate, like the industry, make sure they understand.
Rather, not commit those is where I'm going.
Yeah, Google's always been very good at, uh, sort of data security and ensuring that, you know, customer data is well secured, I guess. Related to data, there's a number of verticals that are very data intensive, or alternatively, where the data can be quite sensitive. You know, healthcare is almost a canonical example of that, where, you know, bespoke data can really help build a more interesting AI service. Sort of like what you've done with MedPalm, but at the same time, you want to secure it properly.
Are there specific verticals or use cases that you see adopting AI soonest? For example, do you see healthcare moving ahead of education, or do you see fintech or financial services as sort of the early adopter wave?
I think they're all kind of early and adopting in all those verticals that I was mentioning, or horizontal like sales and marketing, creative approaches. Uh, I'm seeing engineering adopt it much faster internally now, uh, with the coding models, with the open source, uh, it. It's a surprise, by the way, if I had to share my personal, I thought engineering was so productive. And they're writing their right, just design elements of UX and the discussion you have around that.
I've been there, takes a long time. But then having, uh, a a bot in the room saying, here's how you can approach it, or here's how code could be written, uh, capabilities, it's fantastic to see that. So I see, uh, a lack of, you know, resources, which was causing engineering projects to be bottlenecked. Now the transformation is gonna be much faster, so bringing that across to vertical is like, fantastic. That's the IP that's getting created now inside those organizations.
Yeah, I'm seeing something very similar in the startup world, where a lot of the early startups are basically technologists building stuff for themselves. Or mid market tech companies, as sort of the earliest adopters outside of a Google or a Microsoft, or sort of the the really cutting edge, large tech companies. And so it seems like, um, you know, at least on the startup scene, there's a very similar pattern being mirrored relative to what's happening on your cloud.
In terms of, you know, developers are building for themselves, first, mid market companies are kind of next, simply because they're often driven by a developer or CEO. And then it kind of is starting to eke into the other parts of the world in terms of adoption patterns. So it's it's interesting to see that parallel.
It's very interesting and think about the the investments you all make. Now. I have a platform in GCP which gives me models, I have a coding model that helps me code, and now I just need capital allocation to go experiment. I think that's great. And then now I need to make it secure and scale it up. That's where we come in with our infra and then make it cost effective. I think we are like early stage of new era of even startups and big companies coming out of this, uh, and different solutions getting built out.
What you said earlier was really interesting about having a bot in the room, so to speak, helping us with UX discussion or specking a product. Where does that live for a Google product team? Does it live in a chat interface? does it live in documents? does it live in, um, the IDE, or, uh, in version control? I think a lot of people are trying to figure out how this should integrate into existing developer workflows.
Yeah, I think today, in my mind, it lives in docs, so you get a lot of, like, you know, benefit in the documentation and that's shared across. I think you've seen it. Like any project, it's like, let's start a doc, uh, any hypothesis or experiment, let's start a doc. I really think, uh, there will be an evolution as the platforms mature, where the docs can make suggestions, and those suggestions could be here. Based on, you know, what you've specked out, here is output.
Now you can choose to ignore it. It could be a notebook as well, you said IDEs and UX, it could be just code written with put the API key here. It's great to kind of do that, but then when you're scaling up, that's where I get scale, like, how does it kind of scale up? You don't want these things to become something that's doesn't have the guardrails around them. Who is entitled inside the organization to do this?
So we have to make sure logging is on. Simple things like logging, we will not think about, but has to be who accessed, when did they access, et cetera. But I think once we get over that, if you're writing a monitoring tool and it knows the schema, you don't need to go back and look at it and have an engineer work on it for like, four weeks, right? Why can't it be generated out of? I just need a monitoring tool for model drift.
So those benefits are going to just accelerate adoption in my mind.
One of the things that Google really pioneered, on top of all the cloud services and APIs and everything else, is actually the silicon layer. I think it was seven, eight years ago, when TPUs were first really rolled forth as something internal, uh, to Google. And they said, let's invent silicon that's really good at dealing with AI and ML systems. And so that was my understanding of the origin of the TPU, and it obviously was, um, dramatically more performant than GPU for a long time.
Could you talk a little bit about the TPU versus GPU trade offs and how Google Cloud approaches that?
Yeah, I think how we think about is like, can the platform be capable enough and have, uh, features, as I call them? That takes away the developer or engineer or a team thinking about a project from, you know, the infra piece. Early days, like, you know, we used to write Windows app, I've written like Java application, and it's always about abstracting it away so you can manage it, scale it, and then roll it out faster. So I think we'll see a world gradually, other than training, and where it's trained and deployed, is it abstracted out from that layer?
In a way that it gives the fungibility and the adoption and the scale out, um, that helps you kind of do that, that'll, that'll speed up, in my mind. And then, uh, to your point, innovation on that layer is going to also continue.
Are there specific trade offs you see in terms of when you should use TPU versus GPU, or do you think they're reasonable, they're reasonably fungible?
They're reasonable. In my mind, it's just, uh, more around how many folks know how to do it directly is something, uh, you know, top of mind for us. Because there is this that was, you know, externally available, and a lot of people are trained up. As more people get trained up, we're seeing cost and benefits across the board, right? So having optionality is what we wanna kind of bring forward constantly.
The people that I know who know how to use TPU very well are very comfortable in that environment. And to your point, that's often people who are at Google. And therefore, they're trained to know how to actually use that underlying silicon versus the GPU. In terms of optimizations, or, you know, little tricks that make things more efficient.
How can we kind of make that more prevalent and available is something we should kind of invest in in the future. And you're going to see that more than that, I'm, or the team as such is thinking about inference. Because it's, you know, you get the train, you get the application ready, and then the scale is where you you don't know. Or character. AI is like, really thinking about scaling it out. We have other examples in Cora and Po.
Adam's thinking about scaling out the application. So now you're going to see the shift on like, we have V5V, etcetera. We are investing, uh, heavily, and you know, others are doing. Is how do you scale that's? And then keep the cost curve kind of going in the other direction. Uh, we're going to kind of invest and kind of put our effort around that.
yeah, and I guess since Google trains its own models, um, and they have an advantage in terms of thinking through how that scalability could work for others.
That's right.
And that may be an interesting differentiator in terms of that, um, that understanding.
Yeah, I think inference will be something really needed. Um, you can see the numbers, right? Like an application launches and it goes from like, five, ten, fifteen, hundred, right? And you're like, how do I scale this further out? And Sarah touched on it. What happens when we get into multimodal world, right? And if I'm a educator thinking about an education site, I'm not having that kind of luxury. So that's like, really something we think about. We didn't talk about the vertical at all, is, you know, education, and people want to be learning, deploying, etc.
How do we help those organizations come on board and take advantage of this as well?
Are you seeing the current NVIDIA GPU shortage change your customers perspectives in training or inference processor choices today, or how does that change your conversations at all?
It definitely is mentioned, uh, as we think about it in the conversations, uh, it's always the the discussion that comes up. But as you look at the applications and the usage and what they're really trying to solve, you then get to the next step. I think it just the conversation shifts around capabilities and platform. To, as we were discussing over here, how do you keep the data secure? how do you make sure the right teams have the access to it?
how do you make sure the data is regionalized, for example? And those answers are not easy to answer for some of the the other providers, for example. And then it comes to like, if models are going to be available, do we really see that shortage playing out, right? And I think it's we are seeing the concentration on training side. But then the world focuses on a different problem, which is like, let's focus on the rail. How do we deploy it inside your organization and make it successful?
That's, I think, where the narration changes, and then it doesn't even come back to talking about the the chip shortage and availability, etc. Uh, but having said that, we are, we are absolutely focused on, like, making sure that the platform is available for customers. And we see the demand, uh, coming as well towards us.
Yeah, I, I mean, uh, there's different ways to look at the demand, uh, for training. So, you know, one point of view in the world is that most of the demand will end up quite concentrated in people who provide model services, right? Versus at least quantity of compute used in fine tuning or training outside of a few large labs. Like, how do you think about this, and how should customers think about Google's model quality versus others?
Yeah, absolutely. So we think about, you know, going back that the customer should have availability and optionality. So if there's a model that's coming out, it's been trained up, can we make it available like Llama inside our model garden? And then they can use it inside their application and not worry about the platform capabilities at all. And it kind of fits into currently and leverages their current investment. We think about that a lot, so just starting from the customer first and then making the technology available.
Rather than thinking about here's another set of models that you need to kind of go take and do the scaffolding and the build out around that. So absolutely, like, as these come out, Sarah, and you know, there will be new ones, uh, you know, launched with some capabilities, different ways of training and approaches, optimization. And if we see our customers in our vertical kind of leaning towards that, we want to make sure it's available on our platform.
Um, and I think models, in my mind, are like, fifty, sixty is the is the work around, and how do you leverage your current investment? Is thirty, forty, uh, work that goes in, uh, from the groups, from our customers as well, and then the upkeep and maintenance and all the operational elements are also important.
One or two last questions for you, Um, what is the the thing you're working on right now within Cloud AI? You're most excited about that we should be looking forward to, or or more customers should know about.
I'm working across the board with multiple customers on just making these user experience the next generation multimodal. And how should we think about that? Um, and what are the real first use cases that we should kind of deeply work and partner together with them? So that keeps me really excited around that. And you know, how can we, uh, bring these to our platform and make it capable? Uh, second is just the amount of shared data some of our customers want for their model training or available on our platform.
That's the next thing we think about, like, are there data sets that we can offer out? And we're also looking at synthetic data as well. Can we use in regulated industry some of the synthetic stuff and like, recreate those, uh, simulation modes that can give them a good insight into some data that's missing?
Does Google Cloud have a data marketplace, a labeling offering, labeling tools, anything like that today?
So we have, uh, the partner ecosystem through that, we provide a lot of partners, so exactly the marketplace. And if they wanna take advantage of, uh, some of those providers, they can absolutely do that. They're integrated into our platform. So you can kind of say, I want this service for our LHF, and you know, reach out to them and then use it for your model training, etc. The the most important thing, Sarah, is to make sure that they're working in congestion and whatever they are, uh, you know, getting from the customer, it's secure, so it's not something used with another customer.
So we make sure those pipelines and entitlements stay in the project.
Gandhi, is there anything that you wanted to cover today that we didn't? It was a great conversation, I feel like we covered a lot of ground.
Yeah, I think it was, thank you. I think we covered all of it. I would love to come back in six months, take look back and see, you know, how we move forward.
I know making predictions in AI is very hard, but I heard from you that my, um, my Gmail suggestions are gonna get a lot better quickly, so I'll, I'll take that.
Yes, absolutely, hold us to that, yeah,
Yeah, great. Thanks so much for the time today. I really appreciate the conversation.
thank you.
Find us on Twitter at No Priors Pod, subscribe to our YouTube channel. If you wanna see our faces, follow the show on Apple Podcasts, Spotify, or wherever you listen. That way, you get a new episode every week, and sign up for emails or find transcripts for every episode at No dash priors dot com.