29 Aug 2024 · 42m · Matt Garman, Sarah Guo, Elad Gil · 28 quoted facts
Matt Garman, CEO of AWS, argues that Amazon's cloud succeeded by offering simple building blocks instead of forcing customers to change how they work. He says the same approach should carry into generative AI. Models become one more component, like storage or compute, delivered securely, with a choice of models and the customer's own data at the centre.
The central tension is between huge demand and limited supply. Garman says most workloads are still not in the cloud and that AI demand seems almost infinite. Chips, power and data centers will stay tight for a while. AWS is spending tens or hundreds of billions of dollars while hedging against forecasts that could be wrong, and it is pushing its own chips and a mix of partner models.
AWS gave developers an ordinary Linux server in thirty seconds, instead of a new programming style. Garman says this let people adopt it without relearning how to build, and newer services like Lambda and Bedrock were added later.
Banks said compliance rules kept them off the cloud, so AWS spent a decade ticking off those requirements. Garman's reasoning is that if JPMorgan and US intelligence agencies can run on AWS, other companies' objections are small.
AWS assumed there would be big and small models for different jobs, and that customers' own data would be their real advantage. So Bedrock offers models from Anthropic, Meta and Amazon itself, and Garman says Amazon's own models are important but not critical.
Chip factories, memory suppliers and data centers all take years to expand. Garman says some models need gigawatts of power and the capital involved is tens to hundreds of billions of dollars. AWS tries to balance that spending against uncertain demand.
Garman expects most firms to buy applications and models rather than build their own, just as few build a CRM from scratch. He also says that managing GPUs is not what enterprises want to be good at.
Garman says Amazon first had to rebuild its own systems as separate services, after a mandate from Jeff Bezos around 2003. It then offered the same building blocks, like compute, storage and databases, to outside developers.
Garman says about 85% of workloads still run on company-owned hardware ('on prem'), by most estimates. He sees at least ten times growth just from moving existing workloads, before counting new AI ones.
Bedrock is AWS's service for building AI applications using many different models. Garman says customers just send text in the form of tokens and get tokens back, without managing the GPU chips underneath.
Garman says no. He expects things to stay tight for a while because chip factories, memory and data centers take years to build, and AI demand looks almost infinite. He notes that AWS also makes its own chips, Trainium and Inferentia, to diversify beyond NVIDIA.
He says to have a real plan to make money and not assume more venture funding will arrive, because startups only fail when they run out of money. He cites his own first startup, which raised $27 million and ran out in about eighteen months.
Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.
Matt Garman did a business school internship at Amazon in 2005 and joined Andy Jassy's new technology business there, which was AWS before launch.
“I did my business school internship at Amazon, and, uh, as we were looking around for projects, uh, I had talked actually at the time to Andy Jassy”
Matt Garman returned to Amazon full time as effectively the first product manager for AWS and says he has been there eighteen years.
“I came back full time, uh, effectively as the first product manager for AWS, and I'm now on year eighteen”
Around 2003 Jeff Bezos mandated that Amazon move from a monolithic software stack to services, which Garman says led to AWS.
“Jeff Bezos basically mandated across the company that in order to move from a big monolithic stack that wasn't gonna scale anymore for Amazon, we had to move everything to services.”
Garman says an early AWS virtual Linux server arrived in thirty seconds, when getting a physical server normally took about six months.
“it auto scaled and you got it in thirty seconds, which was pretty awesome at the time, where normally it was gonna be six months or something for you to go get a server”
A host says AWS revenue grew from about $500 million in 2010 to about $90 billion in the most recent year.
“I think you guys went from something like 500 million or so in 2010 to about 90 billion last year in terms of revenue for AWS.”
Around 2008 or 2009 AWS staff visited financial firms in New York, including Goldman Sachs and JPMorgan, who were fact-finding about cloud computing.
“We went on a trip to New York, and actually, there's a bunch of financial services com companies, the Goldman Sachs' and JPMorgan's”
Garman says AWS deliberately targeted the hardest workloads, such as JPMorgan Chase and US intelligence agencies, so that regular companies' objections would shrink.
“if I can solve the, you know, JPMorgan Chase running in AWS, if I can run solve the US. Intelligence Agency running in AWS, like the the reasons for regular other companies are diminishingly small.”
About a year into working on AWS, Garman told a friend he thought it could become a billion-dollar business.
“Oh, how's that AWS thing going? And I was like, You know what? I think this thing could be a billion dollar business.”
Garman says about 85% of computing workloads still run on premises, by most estimates.
“I think 85% of workloads are still running on prem today by most estimations, somewhere in that range”
AWS won a secret cloud contract with US intelligence agencies, competing against incumbents such as HP, IBM and Oracle.
“we went after the the intelligence agencies for the US. government, and we won that contract and it was secret”
Garman says IBM sued over the AWS intelligence contract, which made the win public, and the agencies then publicly called AWS the most technically sophisticated option.
“so then it became public that we won this deal. And then the intelligence agencies went out to public and said, no, AWS is the most technically sophisticated”
Garman says AWS has invested in AI for about ten years and began building its own processors five or six years ago.
“we've been investing in AI broadly for the last ten years, and that's why we've started five or six years ago investing at the infrastructure layer and building our own processors”
Garman says AWS's AI strategy assumed there would be many models of different sizes and purposes, which customers would combine.
“We thought that there was gonna be a lot of models for a lot of different purposes, and there'd be big models and small models, and people would want to combine them in new and interesting ways.”
Garman says Microsoft has benefited from OpenAI's innovation, and that Microsoft recently listed OpenAI as one of its biggest competitors.
“Uh, Microsoft's done a good job of co-opting, uh, uh, OpenAI's innovation, although in their last, uh, in their last, I saw recently, they listed OpenAI as one of their biggest competitors now.”
Garman says Amazon's Titan embeddings model is the most popular embeddings model in Amazon Bedrock, used for building search indices.
“the Titan embeddings model is by far the most popular embeddings model that we have inside of Bedrock today for people that are building search indices”
AWS made Meta's Llama 3.1 available on Bedrock on the day it launched, and Garman describes a tight partnership with Meta.
“we just launched, uh, Llama 3.1, uh, on the day it launched, and we have a really tight partnership with Meta”
Garman says Adobe built its Firefly models on top of AWS.
“we see folks like Adobe building Firefly as a all built on top of AWS, purpose built for their own thing”
Garman says Amazon Aurora, AWS's managed database, is 100% compatible with Postgres, which he cites as a way to avoid customer lock-in.
“even where we have something like Aurora, which is our kind of managed database, it is 100% Postgres compatible.”
AWS partners with Scale AI to help customers label data for fine tuning or pre-training.
“we have a deep partnership with Scale AI to come in and help you label your data”
Garman says guardrails matter because a financial-services chatbot that gives financial advice could lead to large fines.
“if you're building a chatbot on a financial services website, you can actually get find a lot of money if that thing starts giving out financial advice.”
Garman says AWS has been the largest purchaser of new renewable energy contracts in each of the last four or five years.
“we're the the largest purchaser of of renewable energy, um, over the last, uh, new new contracts, right?”
Garman says NVIDIA runs its own AI training clusters on AWS because AWS has the most stable infrastructure.
“NVIDIA actually runs their AI training clusters in AWS because we actually have the the most stable infrastructure of anyone else.”
Garman says AI workloads are roughly half training and half inference today, but inference must dominate for investment in big models to pay off.
“in order for the math to work out, inference workloads have to dominate. Otherwise, all this investment in in these big models isn't really gonna pay off.”
Garman says some of the largest AI models are reported to need gigawatts of data center capacity.
“they're talking about needing gigawatts of capacity, um, which is, uh, is is kind of a a mind boggling number”
Garman says building capacity for the largest AI models involves tens, if not hundreds, of billions of dollars of capital investment.
“we're talking tens, if not hundreds of billions of dollars of of capital investment”
Garman's first startup raised $27 million, ran out of money in about eighteen months, and shut down when funding dried up in the early 2000s.
“It was 27 million. We ran out of money in, like, eighteen months, and then, you know, the 2000s came around and there wasn't any more funding, and boom, we're out of business.”
Garman says most AI-related value currently goes to NVIDIA, but that this reflects how early the cycle is and there is enough for everybody.
“There's probably enough for everybody. Uh, today, most of it does go to NVIDIA, I think that's that's a lot of it.”
Garman says customers using Amazon Bedrock never interact with a GPU directly; they send tokens in and receive tokens back.
“if you go use Bedrock models today, you never interact with a GPU, right? You just you send it tokens, you get tokens back”
Garman describes joining the pre-launch AWS team in 2005 and why he saw cloud computing as a rare, huge opportunity.
“I did my business school internship at Amazon”
Garman explains how Amazon's internal move to services shaped AWS, and why not forcing new architectures sped up adoption.
“we're gonna build building blocks and let developers and and builders go build interesting things”
AWS targeted the hardest, most regulated workloads first. A secret intelligence contract became public after IBM sued, and that boosted AWS's credibility.
“we spent the next decade just checking those things off the list”
Garman lays out the assumptions behind AWS's AI strategy: security, many models, and customer data as the key advantage.
“there wasn't just gonna be one model”
Garman says Amazon's own models matter but are not critical, and that partner and open-weights models like Llama are a big draw.
“I think it'll be important, but not critical.”
AWS argues that customers should stay because they want to, not because of proprietary licences.
“I want my customers to want to run on us”
Garman says the model will become a smaller part of what customers care about, next to their own data, safety controls and agents.
“how do you have a grounding set of truth that you use?”
Garman explains why AI capacity will stay tight and how AWS balances huge investments against uncertain demand.
“probably gonna be in a constrained world for the next little bit of time”
Garman warns startups not to count on endless funding, and says value will move up the stack to applications.
“any startup goes out of business is cause they run out of money”
Garman predicts inference becomes a normal part of building applications, abstracted from the hardware.
“generative AI honestly becomes one of the compute building blocks that you think about”
I don't know how many times in the first couple of years I had to explain why a bookseller was offering compute services and storage services. There's not a lot of business opportunities that are as big as cloud computing and and as potentially transformational. Original AWS thesis was we take care of the muck so you don't have to. I want my customers to want to run on us.
Hi listeners, and welcome back to No Priors. Today, we're talking to Matt Garman, who took over as CEO of AWS in May. Matt has been with AWS since it was a zero billion dollar business to today's 100 billion dollar run rate business. Welcome, Matt.
Matt, thanks so much for joining us today. It's a real pleasure to have you. One thing that I think is really fascinating is you actually started on AWS in the very early days, it was just getting started, and you did that as an intern while you were getting your MBA. Could you tell us a little bit about the both the origins of AWS as well as your own involvement with it?
Sure, so it was in 2005, uh, I did my business school internship at Amazon, and, uh, as we were looking around for projects, uh, I had talked actually at the time to Andy Jassy, uh, and he was, um, telling me that he was starting a new business inside of Amazon that was technology focused, and he couldn't tell me about it, uh, before I started. But I thought it sounded interesting, so I I joined him and worked on that, and it was, uh, it was it was AWS pre-launch.
Uh, I got to work on that as intern. It was a it was a super cool opportunity. Um, I came back full time, uh, effectively as the first product manager for AWS, and I'm now on year eighteen, uh, or just, you know, so, uh, been working on the the, uh, the business the entire time. And, uh, you know, it's it's a fascinating space. Even back then, as we saw the potential of what what AWS could be, um, obviously, that was, you know, it was a startup, right, at the inside of Amazon, and so it was, you know, we we had visions of what we thought it could be.
But, um, you know, there's a lot of hard work and some good luck and some things that have gone right for us, uh, and a great team over the last, uh, couple of decades to to go build AWS. And, you know, the fascinating thing today is it's still in the very early stages of what the business can be. There's there's not a lot of business opportunities that are as big as cloud computing and and as potentially transformational, um, to to every, uh, industry out there.
And so, um, it's an exciting place to work, uh, just as much as it was, uh, in 2005 when I was an intern, kind of writing the original business plan of of who, uh, when we first launched our services, which companies might possibly be interested in using these things.
What were the, um, other projects that you were offered at the time?
I'd worked at startups before going to business school, and, um, part of what I was looking for is I wanted to see how larger companies did new projects and kind of entrepreneurship, if you will, inside. So there's a company, couple of few technology companies that I looked at and and was excited about Amazon, and there was a couple of other kinda retail businesses that, uh, I mean, there was the internships were mostly in in retail, um, and, uh, you know, there was some new categories that they were starting up and things like that that could have been interesting.
But, um, I always also knew I wanted to go back to technology, um, so this was, uh, far and away. It's what convinced me to come to Amazon, cause it it seemed so exciting.
And how how well fleshed out were their original plans? Because I ended up, um, using AWS for my first startup in 2007 or 2008, and it was pretty new, and there's a small subset of services at the time. And to be honest at the time, uh, Amazon wasn't thought of as a d as deep of a technology company as it both was and is. Uh, and AWS was really kind of like, Wow, Amazon is doing this thing that, you know, at the time you thought maybe it would be natural for Google or somebody else to have started with.
And so I'm a little bit curious about, um, about the that sort of early conception and roadmap and what was in place and what you added over time.
I don't know how many times in the first couple of years I had to explain why a bookseller was, uh, was offering compute services and storage services. I think the approach that we took worked out to be one of the key pieces to our early success. I think if you look at at some of those others like Google and and and later Microsoft, um, they kind of went at this space. Uh, first, first, we were the first ones out there that had anything like this.
But even soon after that, I think they went after this space. Like that was gonna force the developers to change how they built applications and kind of build them in a in a new way. And we went after it as in, we're gonna build building blocks and let developers and and builders go build interesting things. And so our view, and and this this started really from internal Amazon, if you if you scroll back all the way to 2003 or so, Jeff Bezos basically mandated across the company that in order to move from a big monolithic stack that wasn't gonna scale anymore for Amazon, we had to move everything to services.
Uh, and that was really a lot of the impetus for us looking at after that, and we saw the success of that. We said, well, maybe this would work for other people who are likely gonna have the same struggle that we're gonna have, uh, as as Amazon. And so we thought, starting from first principles, what are the things that we would need to go build to help people build a company? And we knew people needed compute, we knew people needed storage, we knew people needed databases.
And, um, so we built those things, right, and we we didn't force people to change how they architected, right, when when we gave you, uh, a virtual Linux server, when you logged in, it was just a Linux server. Like it wasn't magic, you know, it auto scaled and you got it in thirty seconds, which was pretty awesome at the time, where normally it was gonna be six months or something for you to go get a server.
Uh, and this is when everybody was like, when you had a startup, you had to go to Exadata and buy racks of servers in order to get your startup off the ground. Um, and so, uh, so that was obviously transformational. But once you actually got the infrastructure, it kind of operated the same way that you were used to. S three was a little bit different, right? The put get delete was a little bit different, but the the storage concept wasn't too different.
Um, and so I think that was one of the things that really helped us move quickly and helped people not have to have a total paradigm shift of how they develop. It was more how they get the infrastructure to develop on. And then over time, we could add things like Lambda, or we could add things like, you know, Bedrock and AI services and other things like that that are more different that people weren't used to. But but kind of starting out with some of those things people are familiar with, and they could instantly grab on and and build quickly without having to learn a new concept, I think was a a really big, um, accelerator for us at the beginning.
yeah, it was huge. I remember, um, when it when it first came out, I thought there was a few things that were done really well. One, to your point, is just, you know, providing really basic building blocks. Second was just the iterative nature of it, where you launched with a small number of services, and then you kept adding stuff that were kind of the obvious next steps. And, you know, early on, people would wonder, or at least I in the startup community wondered, will they end up with everything I need?
And you very quickly did. But to your point, you also built it in a way where before that, I think most people building today have no idea. You to your point, you literally would have to set up your own server racks or find somebody who would do it for you. It was hugely painful. You had a whole team that was doing that for every company, and in some cases, it caused real problems. My company eventually got bought by Twitter, my first startup, and then at Twitter, we ran into real issues in terms of data center capacity and planning and all these things that you can now just scale on Amazon for.
So it's pretty radical in terms of what's been enabled for startups and the decrease in headcount for a company that's needed to associate with that.
I remember as late as 2010, 2015, maybe you guys are still having this conversation with the largest customers that, um, you know, if you're, let's say, a large financial, you had, like, this whole platform team, several different iterations of it. And the line I would get from people, uh, would be like, we're never gonna do that public cloud thing. From a security perspective, you can't compete with us on cost, our platform is better. Like, you know, it's not reliable.
And and just like there's a very strong orientation toward skepticism of this, you know, even in 2010, like upstart company versus like, oh, we are x large financial customer. I think I think almost everybody has seen the light at this point. But you know, you, you, you described Amazon AWS in particular, still being quite, quite early in that journey. And I think one thing that people, even those in the tech industry who experience more exponential growth than most, like, thinking about how large these markets become is really challenging.
Um, and we might be, you know, I think we're at one of the at the beginning of one of those cycles again with AI. We'll we'll come to that. Um, uh, like, at what point did AWS internally did you guys, like, know that this was gonna be that large? And how did you talk about size and opportunity early on?
And just to give scale real quick on this, I think you guys went from something like 500 million or so in 2010 to about 90 billion last year in terms of revenue for AWS.
that's right.
And so it's just it's this insane ramp of $89.5 billion in incremental revenue over, you know, fourteen years or whatever it is. So it's that's amazing.
yeah, it's it's it's easy to get caught up in those big numbers and that the fact that it's so early. And so so, you know, back to your your question on when we kinda knew we were, um, we were kind of on that track. I I I remember it was probably 2008 or 2009, I I can't remember exactly, um, but I definitely remember the trip. We went on a trip to New York, and actually, there's a bunch of financial services com companies, the Goldman Sachs' and JPMorgan's, and and they they wanted to learn about what is this cloud computing thing.
And and they mostly were just fact finding. They were trying to get information from us on how they could more efficiently run their internal IT systems, I'm pretty sure. Um, but we went in there and we're like, Well, it's worth a shot. And and you know, they were they were like, Okay, look, our our workloads are never gonna run you, like maybe someday our website'll run you or something like that, but but never any of our internal workloads.
And, uh, you know, and and we listened to them, and and we said, why? And and then we we said, awesome, tell us why. And they were like, Okay, we have to have this compliance, you have to meet this rule, you have to meet this thing. We have people that do these audits, blah, blah, blah. And we spent the next decade just checking those things off the list. And and we basically never said, you know, our part of what we did is we said, we wanna know what are the most difficult workloads to run, what are the hardest things to do, and let's go solve those.
Cause if I can solve the, you know, JPMorgan Chase running in AWS, if I can run solve the US. Intelligence Agency running in AWS, like the the reasons for regular other companies are diminishingly small. And, uh, and so that was kind of our mentality as excited and I love getting the startups running up to us, but the big enterprises can kind of easily dismiss. You're like, well, it's a startup, you know, they don't have all these things.
Uh, and so what we did is we just did both. We said, look, we're gonna go in as much bus- business with startups as possible as we can, and we're gonna check off the list all the things that are gonna help JPMorgan Chase or the US. government or, um, Pfizer or whoever it is, run on us in a in a secure, safe way. And, uh, and that's what we did. And and, you know, today, those are all huge customers of ours.
After I was in AWS for about a year, I remember sitting down with a friend of mine, uh, who was a business school, uh, classmate of mine, was also working a different part of Amazon. And he was like, Oh, how's that AWS thing going? And I was like, You know what? I think this thing could be a billion dollar business. And he looked at me and he's like, he's like, Dude, do you know how big $1 billion is?
Like, that seems unlikely. I was like, No, no, seriously, I think we could get to be a billion dollar business. Um, so, uh, you know, I I we, we knew it was gonna be successful, and we didn't know, you know, quite how successful or when, I would say. Um, and so, you know, now that we're at 100 billion run rate, uh, you look at you, you still go out there, and I think 85% of workloads are still running on prem today by most estimations, somewhere in that range, you know, pick your number, whether it's eighty to ninety, whatever it is.
Like, that's enormous. Like, if I have, I still, there's still ten x growth of just existing workloads. Forget all the new gen AI workloads that are being created every day. These are just existing workloads to move. There's a there's a ten x number in there, um, and so that that business is massive. And I think there's a couple of, you know, one of the big inflection points we saw is we went after the the intelligence agencies for the US.
government, and we won that contract and it was secret, um, and it, you know, we we we pushed really hard to go in. That it was against all the incumbents, the HPs and IBMs and on Oracles and whatever. And we won the contract for to do this cloud workload. And, uh, but it was, um, confidential, and we couldn't share with anybody that we were doing it. And, uh, IBM sued
Yeah, I remember this.
to, uh, because they they said it's unfair, it wasn't, and and so then that became so then it became public that we won this deal. And then the intelligence agencies went out to public and said, no, AWS is the most technically sophisticated, they have the most capabilities, they're the mo- operationally strongest, and that's who we're gonna go with. And so we had the government out there now saying that we are the best and most technically capable to run these high, uh, highly important workloads.
And and that was a huge, um, kind of stamp of approval for us, if you will. And and I do think that that's one of those moments that, in some ways, we kinda got lucky that they s sued, otherwise it might still be secret that, uh, that we were doing that. And I do think that that helped gain a lot of credibility in the enterprises.
What do you think is, um, you mentioned that, you know, eighty of workloads still haven't migrated over. Um, what what do you think are the main blockers to that today? Is it just momentum? Are there specific features? Are there big things still to build?
There's some technologies that, you know, I think it look, if I had a easy button and and by the way, we're trying to build an easy button. But, uh, but if I had a easy button that would just migrate mainframes to a modern cloud architecture today, almost everyone will push that button, but it doesn't quite exist today. And it's not as simple as, like, great, I'll go run your mainframe in the cloud. Like, that's not what customers want.
They wanna actually modernize those workloads and have them into, you know, microservices and and containerized workloads and other things like that. So so, yeah, that's one is there's just a bunch of workloads like that that are old and and their customers running a big SAP thing, and they wanna move it to the cloud. But it just takes time cause it's tied into a bunch of other things like that. There's also a bunch of workloads that, as you get out of core IT workloads, that are in line of business, that are the next set of things.
And whether that's, um, you know, say telco workloads, right, that are that are running kind of the the five G infrastructure around the world. Um, we've slowly been moving those to the cloud and helping those customers get that flexibility and, uh, and that agility of of running those in the cloud as well, but they're slower to move. Um, if you think about all the compute that runs, uh, factories out there today on factory floors, most of those have not been modernized, most of those are are thinking.
And and there's a huge opportunity, by the way, for AI to to totally revolutionize how you think about factory workflows and and efficiency there. But a lot of that hasn't moved. Um, and and so some of this is, you know, there's, uh, on prem infrastructure that people are still amortizing. Uh, there's people who's there's still people whose job it is to to run on prem data centers, and so they're kind of resistant to moving things. So, you know, there's there's a bunch of factors in there, and so some of it is just, uh, takes time, some of it is technology pieces, um, some of that is we still have stuff to go build and and innovate and help make it
easier for customers to do that.
I'd love to hear about just the initial, um, investigation of like generative AI as a technology change and like how AWS began to react to it and invest in it. Because to some degree, it puts us all back in the like on prem colo era of the world, where to get one of these, you know, if you're doing any sort of real pre training, uh, to to get your startup off the ground, you're back to. I guess I'll buy a bunch of DGX boxes somewhere and like, I need to think about the cost and management of that.
Well, I and I well, actually, most people are still buying those, but in the cloud. But it is kind of a it's not a serverless type of a thing. It's, you know, mo- most people are still not buying, uh, you know, H one hundreds and hosting them in a colo or anything like that. Um, and increasingly, I think that's gonna get harder and harder as you move to liquid cooling and and and larger clusters. But, um, you know, it is a it's a super interesting space.
I think we we've been working on this space for how many years now? Um, and and look, we we've been investing in AI broadly for the last ten years, and that's why we've started five or six years ago investing at the infrastructure layer and building our own processors. Because we we knew this was coming, we saw this path coming, and we knew that that's also not a short-term investment. So that's one of those things you gotta invest way ahead.
And then we were investing in in building generative AI models, um, and then, you know, OpenAI kind of made a a generational leap forward of what they were able to do and what was possible. And and many people have talked about this, but it it really, in some ways, was a discovery as much as anything about just what was possible and and kind of unleashed a new set of of capabilities. And so we actually, as a business, took a half a step back and said, Okay, these are gonna be transformational abilities.
And and assuming that this technology gets better and better and better over time, how do we make it so that every company out there can go build using those technologies? And so different than how can I go build a consumer application that people are gonna be interested in? We kind of took it from the point of view of of AWS, right? Like just what what are the building blocks that I can help all of our customers, whether they're startups, whether they're enterprises, et cetera, go build interesting generative AI applications.
And so we started from first principles, customers are gonna care a ton about security. They they're that's not gonna change. They're not gonna all of a sudden not care about securing their infrastructure. We also had this hypothesis, two more hypotheses. One that the idea that there wasn't just gonna be one model. We thought that there was gonna be a lot of models for a lot of different purposes, and there'd be big models and small models, and people would want to combine them in new and interesting ways.
Uh, and I think the last two years have probably played played that out, but I think when OpenAI first launched, that wasn't as obvious, but that was kind of one of the bets that we made. Um, and then the third one is that we view that every enterprise that was building on us, the interesting IP that they were gonna bring to the table was mostly gonna be their data. And they were gonna care that their data didn't leak back into a model or or escape from their environment.
And so we built a bunch of what we did, starting from those principles of how do we make sure that these things are secure, that their data is secure, that they can have access to every piece of technology that the customers need to go build interesting applications, and they can do it in a cost effective way. Uh, and so that's how we approach the space, and I think we now have a platform in Bedrock, in Trainium chips and Inferentia chips in, um, and then a bunch of the other capabilities around, as well as the suite of models that we offer, both, um, proprietary as well as open source ones, uh, or open weights ones.
Um, that that I think we're we're starting to see that really, that momentum pick up, and we're seeing more and more customers really like that story. They like that platform to build from, and we're seeing, uh, enterprises really lean in and want to build in that in that space because it it gives them a lot of that control that they want as they go and build applications.
How much do you think it matters that, um, AWS has, let's say, like, first party models it offers its customers? Cause that's clearly a strategy for, um, some of the other hyperscalers.
Google, that's obviously their strategy. Um, uh, they're they're really the only one, uh, that has a fir- a first party model today of the other hyperscalers. Uh, Microsoft's done a good job of co-opting, uh, uh, OpenAI's innovation, although in their last, uh, in their last, I saw recently, they listed OpenAI as one of their biggest competitors now. So it'll be interesting to see how that all plays out. But, uh, you know, for us, I think it's important and, eh, which we do.
So we are building our own first party models, we have our first party models today. In fact, the the Titan embeddings model is by far the most popular embeddings model that we have inside of Bedrock today for people that are building search indices and and thinking about things like that. Um, and we are building larger and larger models as well, first party. I think it'll be important, but not critical. I mean, I think people love using, uh, Anthropic's Claude models, those are fantastic, and right now, those are the best performing models in the world, um, which is fantastic.
Uh, we just launched, uh, Llama 3.1, uh, on the day it launched, and we have a really tight partnership with Meta and their open weights model is fantastic. And, uh, and I and I think increasingly we're seeing customers really love that open weights model because they can go and particularly enterprises, customize it, they can do fine tuning to it, they can add their own data to it and really customize and distill and do some interesting things.
And so I think that is is super critical. You know, we're seeing kind of specialized models, if you will, where we see folks like Adobe building Firefly as a all built on top of AWS, purpose built for their own thing, their own thing that they're building. Um, you know, whether the the Amazon purpose built models are, I I think they're an important part of that. Mostly it's partially it's for us for learning, some of it's for powering our own applications, and some of that may be for end customers.
But, um, it's all kind of that diversity of option, honestly. Like, we we want there to be the best set of options, and we want them all to run in AWS, and so we want those workloads to run there. And so, to the extent that we can do something novel or interesting with our own first party models, we'll do that, and, um, but we're also delighted for our partners to run as well. So I think it's a little bit of both.
Yeah, it's really interesting. Um, uh, Alyssa Henry, you know, longtime, um, AWS leader as well, is a friend. And, uh, over the years, I would like ask her about some interesting new open source project. And she, you know, it was the most terrifying thing, honestly, because she would always be like, AWS loves open source. We make more money on open source than any open source company does. Which is a, you know, if you think about all the advantages that AWS has, even if you're very friendly in the ecosystem, it, you know, can can turn out that way.
And so I think the sea change that's happened in terms of availability of open weights models and multiple players here, Mistral, Llama mix, that are that are very competitive, uh, is like, I think, you know, a a a huge, um, huge boon to AWS's, uh, general open ecosystem model.
We've always leaned into open source, we're huge contributors to many open source projects and and we lead many open source projects. And, uh, and I think we do a good job, um, uh, creating and turning those into businesses for our customers and helping run managed open source projects. And so it's a big area for us. One of the reasons, frankly, is that, you know, we've long said we don't want customers tied to AWS because they're locked into some proprietary licensing, uh, piece.
We want them to be able to we want I want my customers to want to run on us as opposed to kind of locked into a Microsoft license where you're you're held to some different license that you can't get off of easily or or old school kind of like Oracle database that you can't get off of. Um, you know, we want people to be able to run. And so even where we have something like Aurora, which is our kind of managed database, it is 100% Postgres compatible.
And if you take that code and go run it at a Post Postgres database somewhere else, it won't operate as well, you know, cause we do a great job at that, but it runs like in in in theory, if you run it as well, it will, uh, operate as well. Um, and so, you know, that's that's how a lot of our services are built and how we think about things. Um, we'll we'll support proprietary things as well.
And and, you know, at some level, there are some services where you gotta take advantage of the cloud, where, um, you know, there are some proprietary things that customers can use, but, um, uh, things like Dynamo and other other technologies like that. But we really embrace open source and, uh, and I think it's been, uh, it's beneficial to the whole industry, frankly, and it's it's, uh, it's a way to get better security, better visibility, and kind of that license portability, I think is a, um, is a key aspect as well.
You mentioned, um, that, you know, the model side of the AI world, and there's other main AI building blocks. There's RAG and, uh, you know, there's certain aspects of fine-tuning and other things that people are increasingly doing over time. There's other parts of it, like eval suites and what. What are the main building blocks that, uh, you can talk about in terms of things that are either coming to AWS or how you think about that more fragmented world of all these different components and how they fit together relative to AI workloads today?
that is kind of the idea of Bedrock, is that we want to make it easy to do. And I I do think, actually, in many ways, today, the models is the front and center thing that everybody pays attention to. But I think increasingly it'll become a smaller percentage of the thing that people pay attention to, cause people are gonna care about whether it's RAG or some other sort of knowledge base. And we call it knowledge bases, but because it's, you know, the technology may change over time under the covers.
But that, like, how do you have a grounding set of truth that you use? I also think grounding data is an interesting thing for, um, for like, real time information that you want as part of your AI systems. Um, we have things like guardrails, which I is our our customers find incredibly important. Cause, you know, if you're building a chatbot on a financial services website, you can actually get find a lot of money if that thing starts giving out financial advice.
And you d and so you really want to be able to control, let alone, you know, going down and talking about politics or something else that you definitely don't want it to talk about. Um, and so those guardrails are super important as people think about what they want their AI systems to do and interact with and where they want to stay away. This is not controversial. I'm sure you you both hear a lot about this. But again, the next generation of, uh, and the next step forward in what we can get out of AI systems is gonna depend a lot on how well we can integrate, uh, agentic workflows and actually get these AI systems to do things, not just
kind of summarize and tell us information. And so, um, uh, building in the ability to have agents as part of that workflow, uh, is a big area investment for us. And want that to be easy for you to build as part of that Bedrock capability. Um, I do think that pre training and fine tuning, uh, is gonna be something that more and more customers are gonna want to do, uh, as well as distilling over time. Cause I think I was just talking to a couple customers, uh, earlier today that are very focused on how do I get this model down to a much smaller thing?
So I can put it on an industrial edge or somewhere like that. And so how do you think about distilling down? So I get get the value of what I want, I don't need the whole kind of, um, reasoning engine behind that. And I think there's a a long roadmap of, uh, like you said, kind of model evaluation and other things like that. And some of that is us, and also some of that is partners, by the way.
Um, and so we're, you know, AWS has been a place where I think part of the the thing that has made us successful is really embracing the ecosystem to go build around there. And so thinking about labeling data as an example, we have a deep partnership with Scale AI to come in and help you label your data if you're going to be doing any any, uh, uh, fine tuning or pre training or things like that with your data.
Um, we partner with folks like LangChain to help put together some of those agent workflows and other things like that. And not to mention, of course, the model providers, who are are super important partners of ours as well. So I I think it's all of those things. And our job is to how can we make it easier and easier for you to go build those applications in a a a tightly coupled way so that it's easy to go use those different components, easier to innovate rapidly, and, um, and easier to to build the proprietary data that you have as part of your AWS data lake.
So that you can kind of pull that in. Because, frankly, most of these, uh, generative AI systems aren't going to be super useful if you don't have interesting data to go pull from.
The other place that a lot of people are spending time right now, in terms of bottlenecks to utilization or usage or future proofing, is actually more on the chip side, or semiconductor or system side. And then, um, in terms of DC capacity. And obviously, you all have been building trainium chips and other things, which I think is really exciting to see that evolution. How do you think about future, uh, GPU shortages? Does that go away? When I'm sort of curious about how you think about forward looking capacity, and is the industry actually ready in terms of building out data centers, building out semiconductors, all the rest of it, packaging, you know, the whole, uh,
Um, look, I I think we're probably gonna be in a constrained world for the next little bit of time. Just, you know, that some of these things are they take time. Like, like, look how long it takes to build a a semiconductor fab. Like it is, it's not a short lead time, and that's several years. And and TSMC is running fast to try to ramp up capacity, but it's not just them, it's the the memory providers and and the and and, frankly, data centers that we're building, right?
And so as we think about, um, there's a lot of pieces in that in that value chain that I think as you look at the demand for AI, which has been, um, how to exponential might be undershooting it. Some of those components that support that, I think are are catching up. And I think AWS is is well positioned to, uh, to try to do that better than others are. You know, we've we've spent a long time thinking about in the last eighteen years, learning how do we think about smart investing?
How do we think about capital allocation? We've ha- we've spent a bunch of time thinking about how do we acquire our own power? How do we ensure that it's green and carbon neutral power? Um, all super important things, and we're the the largest purchaser of of renewable energy, um, over the last, uh, new new contracts, right? So actually going out and adding and and supporting new renewable energy projects. We're the largest provider, I think, each of the last four or five years.
Um, and so so we've been leaning into that for a while to to ramp up this and and this is just a step up. And so I think we're thinking about, you know, how are we acquiring enough power? Our own chips is a way to support, um, the growth of in NVIDIA chips. And so I think the more diversity there, the the better off we are. We're, um, we're a huge partner of NVIDIA's. We, uh, you know, NVIDIA actually runs their AI training clusters in AWS because we actually have the the most stable infrastructure of anyone else.
And so they they actually get the best performance from us. And, uh, and we love that partnership and and we have a great and growing relationship with them. And, you know, we think things like Trainium are a a good diversification. And I think there will be some workloads that that run better on Trainium and and are cheaper on Trainium over time. And, uh, as well as Inferentia, I think inference is is one of those, uh, workloads that today it's, you know, fifty fifty maybe of training and inference.
But, uh, in order for the math to work out, inference workloads have to dominate. Otherwise, all this investment in in these big models isn't really gonna pay off. So hopefully, uh, for the industry, that that that all happens. Um, but I I think we're probably gonna be tight for the next little bit of time. And so, um, you know, cause the the demand is is almost infinite. I mean, it seems infinite right now.
How does AWS think about making investments in, um, data centers of this scale to train the next set of foundation models, right? And because I think you could take a, you know, AWS is a very educated player. You could take a proactive approach, you could take a customer driven approach. But the idea that there are individual players who want tens of thousands of nodes at a time in interconnected GPUs is like a sort of a new demand, um, factor.
Some of the demand for some of these really large models is, uh, is very large, right? I mean, it's the they're they're talking about needing gigawatts of capacity, um, which is, uh, is is kind of a a mind boggling number that that some of these models need. We're doing both proactive as well as customer driven, right? We try to balance cause, cause there's real capital outlays that are required as part of this, of course. And and we're talking tens, if not hundreds of billions of dollars of of capital investment.
And so, you know, we think about it as how do you how do you make the right investments in things like land and power and other things that are are fungible and and could potentially be used for other things if if eventually demand changes or the the the slope changes. Um, as well as then having, um, visibility into the supply chain for more near term things that you'll need, like servers and chips and and, um, memory and other pieces like that.
And so we balance a bunch of those things, managing the the financial implications of of what we need to go buy, uh, as well as the the long term customer demand. And we try to map out how do we meet some of those match. And, uh, and some of our customers give us long term commitments to help with some of those things, and we give better rates for customers that give very large, long term commitments for for some of that capacity that requires a lot of, um, investment.
But, um, but, you know, it's a there's a lot of air bars on that too. Cause at at anything that's growing at multiple hundreds of percent year over year, um, you're you're not gonna nail that number appropriately. And so we try to have enough buffer in there that we can support upsides when they happen and and manage if it's a little bit less than we thought.
As someone who's seen, um, just many generations of startups decide, like, what investment they want to make in infrastructure, that is suddenly a much more important question to a generation of AI companies, and, um, than it has been in recent history. What, like, advice would you have for them as the the man holding the data center, I suppose?
We've had to go on this before, right? We started from $100 million of revenue to $100 billion or well, actually, we started. I started when we were at $0 of revenue to $100 billion of revenue. And so, you know, we've we've had this kind of rapid growth before. We think about how do you balance some of those pieces? And and I think part of that is is how do you make sure that?
For me, I think as you're as a startup, thinking about this is how are you thinking about investments with a real plan of how do you have monetization? And not, um, assuming that there's always more VC funding to come and and bail you out. And and so kind of having a plan there to flexibility of, you know, what, how how can I start monetizing sooner if I need to? And and where can I keep investing if that's the part that I'm in?
That that it, that that traction makes sense. Um, cause I think, look, the only the only reason that any startup goes out of business is cause they run out of money. That's the only that it's as simple as that. As long as you don't run out of money, you're not gonna go out of business. Obviously, easier said than done, but, uh, but I honestly think some startups kind of forget that. They're always like, that's no problem, I'll just go raise more.
And, uh, and kind of remembering that just because there's like a hype cycle, doesn't mean that someone's gonna give you a bill another $1 1,000,000,006 months later. And I actually earned learned that early in my, uh, career, my my very first startup, uh, we raised, I think at the time, was a lot of money. It was 27 million. We ran out of money in, like, eighteen months, and then, you know, the 2000s came around and there wasn't any more funding, and boom, we're out of business.
We we assumed that we could just go raise more money. And, uh, and so I think that was a good early lesson that you can't always do that.
If I look just at, um, you know, my own portfolio or friends companies, what what is interesting is if, I like, of course, you have a few examples that everybody is looking at, like OpenAI, um, where, um, value creation and dominance, or, at least, like, lead in a market, is highly correlated with the amount of money they're spending. It's not true across the portfolio, and I'm like, only investing in really AI companies, uh, in that, uh, yes, e- everybody absolutely needs computers.
And I'm talking about companies that are doing their own, um, training or fine tuning and and. But some of our companies that are making the most progress and, like, you know, we're still talking very early days, you know, zero to tens of millions in their first year or two. Um, eh, I I think one of the other open questions that people have wondered about is, um, does all of the value creation in the ecosystem go to, um, your compute vendor and eventually a big piece of it over to Jensen and NVIDIA or to the model vendor?
And I I think the the answer, at least to like right now, is clearly not right. I think there's different there's cap capture different levels.
There's probably enough for everybody. Uh, today, most of it does go to NVIDIA, I think that's that's a lot of it. But I just think that's cause is where it is early in the cycle. Uh, you know, I think, um, uh, and they've built some incredible technology that's enabling some really cool stuff. So I think that that's, it's, it's, um, it's fine. And at at some point, it's gonna be the the companies that find out how you actually go solve real problems and deliver real value to enterprises and to customers and other things like that.
And that's gonna be that, you know, I I see a lot of if if I if I take a step back and see who's implementing AI out there, it's a lot of enterprises that are doing proof of concepts and and a lot. And sometimes they'll find one that really works well and it'll go to production. And I think if you can have a startup that can make that part easier, that says, like, this is a real value, right?
It's not a chatbot on your website, but it's something that helps you go faster, make sales, better, innovate more rapidly. Um, you know, do something you were never able to do before, uh, improve manufacturing efficiency, whatever that is the startup is focused on or the company is focused on, for that matter. It's it's that it's gonna be an application level, right? It's most most people don't, um, build a CRM from scratch, they go use a Salesforce or something like that.
And and most companies don't build software from scratch. And so I think most companies are not gonna build their own models over time. They might tweak them a little bit, but I think a lot of companies are gonna build are gonna use the applications that use the software and the models underneath. And, um, so it's not surprising to me that you don't necessarily have to spend the billions of dollars to go build your own model, you can build a small model.
seems like there's a lot of precedent for this just in terms of prior waves of both software and internet. Where I think Kotter or somebody had some very good slides where they broke down the relative value accrued by each layer. And to your point, each layer sort of ends up benefiting over time. So it it seems like that question is almost overstated in terms of the importance of it, you know?
Do you have a prediction if you just look at what happened with public cloud, if, um, what happens with AI platforms, uh, evolves differently? And the the reason being, you know, I just spent a little bit of time talking to large enterprise customers again. And exactly as you said, there's a lot of activity coming from, um, different POCs. There's investment in the area, there's top down interest, and I I do think there's a real conviction that the value will be real.
There's also, like, I feel a little bit of deja vu and bunch of large organizations saying like, nobody meets our needs and we're gonna have to build our own platform. And it's gonna be everything from like, data management to, you know, um, GPU in management for training and inference to like, eval suite to, you know, compliance and audit. And so I kind of wanna say I've seen this story again, uh, before. But you know, how would you predict it plays out?
it's exactly that. I think every time there's a new space and some of those things don't exist, people are like, well, I've got to go build my own. Well, it's like, or, you know, likely there's a bunch of other people that are actually building that too, like us. That once it exists, and I don't know how many times customers have told me, like, oh, if I, you know, if you had already had this, I wouldn't have gone and built my own.
And now I can stop investing in it, cause it's actually not the thing that gives me value. And so managing GPUs, I bet very few of those enterprises, that's the actual thing that their stockholders are like, yeah, that's what this enterprise is super important and good at, is managing GPUs. Like, outside of, like, you know, hyperscalers or somebody like that, they don't really want to, right? If they could use something like SageMaker and it has all the capabilities that they can go and build the the services that they want.
And it it doesn't today, we know that, but we're iterating incredibly rapidly and launching new stuff every month, uh, every week, really. And so, um, you know, I feel pretty convicted that that, for particularly for that level of the stack, like, it just doesn't make sense for people to do that part. And it's not surprising that they do it today because some of those things don't exist. And they wanna go deliver on something, and, you know, their priority was slightly different than when we go deliver it or something like that.
But having it integrated as part of your platform, having it integrated as part of where you have your data lake, having it, all of those things tied together, kind of as a as a infrastructure piece, likely makes sense. And and s, and doesn't mean that we'll do all of those things, by the way. Some of them will be partners that are built on top of us, um, which is great too. But, um, but I do think that it's the the people that are specializing in those places are probably the ones that are gonna eventually kind of support that space, not individual enterprises.
If we were to abstract out a a level and, uh, you know, ask what you, what your vision is, or how are you thinking about the next three to five years of AWS more generally as a business? What are the key things or, um, areas of focus for you?
Well, this is one. I think, I mean, I'm I'm just as excited about generative AI and and and AI broadly as as you all are. Um, I do think that it's an enormous opportunity for us and for our customers to. And and I think it actually, in many ways, it has a positive flywheel effect and is can be a tailwind to some of that first stuff that we were talking about a little while ago about helping customers move to the cloud.
You know, I think if we think about where can generative AI help, some of that can be like, how do you make that go faster, right? How do you take some of that more, you know, our our our original AWS thesis was we take care of the muck so you don't have to. There's still a lot of that that customers have to do today that I think generative AI can help with. And so over the next three to five years, there is a big investment for us in both building that tool set, building that whole platform that we're talking about.
So that customers don't feel like they have to go manage a bunch of these pieces, they don't. And and they don't have to think about, you know, GPUs, or they don't have to think about how do they think about kind of tying these clusters together or whatever. All of that can be abstracted away. If you think about the start of what Bedrock is, if you go use Bedrock models today, you never interact with a GPU, right?
You just you send it tokens, you get tokens back, and you can eventually you're gonna be doing things like fine tuning and and pre training, where you're sending it information and training the models under the covers. But you're still just then sending it tokens and getting tokens back. And so the more we can abstract that to, whether it's serverless or or an application platform that people can go build, and that's just, you know, again, we're at the early stages of what that is.
And so I think that as you move forward, generative AI honestly becomes one of the compute building blocks that you think about. You're gonna need storage, you need compute, you need databases, you need inference, if you will, for your application, largely. And I think that's just gonna be and and networking and a bunch of the other things. And inference is just gonna be one of those building blocks that people come to expect. And just like with compute, where you may want a Intel processor or a Graviton processor, or you may want block storage, or you may want object storage.
You you may want different models behind the inference, and you may think of that, and it'll have slightly, you know, different databases. Um, inference is gonna have different flavors to it, and it'll be big models and small models, and you'll trade off cost and latency and capabilities and things like that. But I do think it's part of the applications, and so we're trying to build it as part of that platform. When you're just building your application, it it comes with it.
And then there's a lot to get there between now and then, but I I imagine that's how most applications are going to be built going forward.
So we started this discussion talking about, like, the early days of AWS and sort of how you were discovering the the sort of, uh, you know, unthinkably large requirements from, um, large financials. And yet, you know, $100 billion of revenue later, you are still, uh, a huge partner to startups. Like, Mm, that goes against some of the conventional wisdom of, like, choose one audience and just, you know, slowly move up market as something that, you know, many startups themselves choose to do.
Like, why continue to work so much with startups? And I know you personally still think about this a lot.
It's super, in- uh, important for us, and and startups are the lifeblood of what what helps us grow. And we get so much benefit from the learning from startups, and and so they will continue to be incredibly important for us. And and we're gonna, if anything, lean more into startups and supporting startups, um, as part of what we do.
Thanks so much for, um, providing your perspective over the last, you know, twenty years or so of AWS. It's been really fascinating talking to you, so thanks so much for the time today.
Thanks, well, to you for having me. Appreciate it. It's been fun.
Find us on Twitter at No Priors Pod, subscribe to our YouTube channel if you wanna see our faces, follow the show on Apple Podcasts, Spotify, or wherever you listen. That way, you get a new episode every week, and sign up for emails or find transcripts for every episode at no dash priors dot com.