Why AI Compute Comes in Rectangles — Erik Bernhardsson, Modal CEO

Thanks for coming on, Erik Bernhardsson from Modal. Could you explain, uh, the business a little bit, and maybe how you got here?

Yeah, so we, uh, we build, uh the easiest way to put it is we build AI infrastructure. So we we offer, basically, compute, uh, a compute platform and a storage platform that enables a lot of customers, like Decagon and Suno and Runway and Cogni- Cognition and many others to to essentially build AI applications. So what that means in practice is a lot of inference, a lot of sandboxes, sometimes training and fine-tuning, uh, sometimes batch jobs and and other things.

So we have basically built a um, a software SDK that enables customers to move really fast and and build this. Uh, we manage all the capacity underneath, make it easier to deploy and and scale and and run these things in production.

So you were on Latent Space, like, 2 years ago. A lot has happened since then. Like, how big were you back then? Like, whatever numbers you can share, and comparatively, yeah, how big are you? W- now?

I actually don't remember where we were, but it's, like, it's probably, like, at least, like, 50x in terms of, like, revenue and users and usage stuff like that, yeah.

I use it actually for, like, a side project that I have, where, uh, yeah, yeah, it's like, uh, yeah, turn, like, 2D images into 3D models. Uh, so, yeah, we're just hosting, like, a

That's awesome.

couple models there.

What do you think? Any feedback?

I mean, I love it.

Nice. Good.

Um, I mean, yeah, that's why I use it over, like, any competitors or anything. Good. So before you started Modal, you were at Spotify and also, like, Better.com, I think.

That's right, yeah.

Um Yeah, what were you doing at Spotify?

I actually did a lot of different things at Spotify, but my main contribution was I built a music recommendation system, the 1st version. I I joined after school in 2008, 2009, maybe, uh, and spent 7 years at Spotify, and and most of that time was building, basically from scratch, a music recommendation system and built up a small team, and we launched a bunch of features, like Related Artists and Discover Weekly and and a bunch of other stuff.

So a lot of lot of matrix factorization, a lot of compute intensive stuff, and and and arguably, like, a lot of what sort of later became the genesis of Modal was, um, you know, just just building AI applications and realizing, like, the infrastructure fucking sucks, like, there's gotta be a better way to do that. So a a lot of sort of, you know, the rationale for for Modal was, like, kind of building a tool I always wanted to have throughout my years at Spotify.

Yeah, so Modal was more of an offshoot of the stuff you learn at Spotify, not much from Better.com.

Better Better.com was a little bit of a side tour. It was kind of a rollercoaster, uh, long story about the company. Talk about it separately. But I was the CTO for for 5 years of Better and grew the tech team from, I think, you know, 2 to 3, 400. So, uh, learned a lot about management and building teams and hiring, and also learned a lot about developer productivity, thought a lot about tools and things like that.

Did a lot more, like, full stack stuff, but, yeah, I kind of came back to my what's close to my heart, which is infrastructure and AI and machine learning after that.

What was Better.com doing again? Mortgages. Oh, okay. How did you get into that? I mean, we don't have to spend too much time on that, just

Yeah, no, it's fine. I like, this guy, Zach Frankel, uh, who I, uh, didn't know at that time, ended up recruiting me to Better as the CTO, and and the company was 10 people, and, you know, for whatever reason, you know, the the founder, Vishal very interesting guy, very, very smart, little crazy, in in mostly positive ways. Um, we got very well along. So we, uh, yeah, we we started building it in in 2015, and I spent 5 years, uh, growing that company.

And then, you know, eventually turned out that doing mortgages is pretty hard, and, you know, interest rates went up very quickly after the pandemic, and and that that caused a lot of issues at the company. But by then, I had already left.

that was like 2020, 2021 when you left.

Yeah, something like that, yeah.

So, okay, you left before you had the idea for Modal, or you left and then you started I ha- it's working

hard to say. I I had a lot of random ideas when I left Better, and and and I I, I actually wasn't sure if I wanted to start a company. I was kind of just, you know, during the pandemic, I I sort of almost took, like, a sabbatical. I had 2 kids. I I mean, I still have 2 kids, but, uh, I mean, at that time, there were there were, I think, 2 and 4. And so I decided to hang out and took some time off and was kind of working, like, part time, like, hacking on certain things and wasn't sure.

Maybe I should do some consulting, and I started building stuff, and I just got, like, super pumped about this idea that that eventually became Modal. But actually, I built, you know, Modal was, like, this I think the 3rd thing I built that I got, you know, that so so, s there were, like, a few attempts before then that that went nowhere, but but Modal was, like, the 1st idea. I was like, This is kinda cool, like, I should do this.

This is, like, this could be a really cool platform.

were the other ideas, like, also in a similar vein, like, developer tools?

The 1st thing was, like, some sort of web scraping thing, which went nowhere, but actually, you know, in hindsight, maybe could have gotten somewhere.

And was this before the ChatGPT launch?

Yeah, yep, 2021. So ChatGPT was late 2022, right? Then I I started building s so, maybe, actually, to rewind a little bit context. Like, when I worked at Spotify, I released an open source framework called Luigi, which is like workflow orchestration. So it's sort of the same category as Airflow and and Prefect and Daxter, sort of dated category to some extent, like, people don't talk about as much, but, uh, for a while, it was, like, you know, got a lot of attention.

Uh, anyway, I I never turned it into a business, uh, but a couple years later, when in 2021, when I was thinking about business ideas or or things to work on, uh, I, I kinda came back to workflow orchestration, and I started hacking on a bunch of ideas of, you know, running code in the, you know, like, basically orchestrating code running in the cloud. And what I realized was, like, you know, at the end of the day, like, a core puzzle piece is not just, like, orchestrating the compute, it's, like, executing the compute.

And I realized, like, that's kind of a bigger problem actually to solve. So in a way, I actually, like, you know, left it on the shelf and then said, like, Actually, I'm gonna, like, go to this other thing over here, and I'll go back I'll get back to the workflow scheduling a little bit. But but, like, you know, I never got back to it. Long story short, and and and ended up s building Modal, which is, like, the the focus is more, like, let's just figure out a way to make it easy to run code in the compute i in the cloud.

And and the focus was initially, like, fairly general purpose, and and it actually, in a way, like, it's always we've kinda gone full circle more recently, but but we we focused on so many different applications. It was kind of the idea was, like, let's just make it super easy to run code in the c in in the cloud. Could be GPU code, could be AI, machine learning, could also be, like, data stuff. Um, and, uh, focused a lot on, you know, how do we scale it up?

How do we make the developer experience awesome? Realized super early that, like, in order to do that, we'd have to go very deep in the stack and build our own file system, build our own container runtime, uh, and and did a lot of, you know, solved a lot of hard engineering challenges. And then you mentioned ChatGPT. That was actually not quite the killer app, but, uh, like, if you remember, like, that fall, another thing came out around the same time, which was Stable Diffusion.

And Stable Diffusion turned out to be a killer app for our for what we were building at that time. We were only 4 or 5 people at that time. So, uh, we started seeing a lot of people coming to us and wanting to run, uh, Stable Diffusion on Modal, and then we real that that that made us realize, like, maybe gen AI inference is actually a really good use case for us. And How were we're sort of

they finding it?

I'm this, like, compulsive tweeter. I I was even more, like, now, like, I'm too distracted by building a company. But, uh, I always, like, tweeted a lot. I always blogged a lot. So I I think I had a little bit of a following. And so, you know, in the early days, I like, a lot of what drove the growth is just, like, me, just compulsively, just, like, tweeting about Modal and, like, what I was building and getting people excited and writing long blog posts about where the cloud is going and all these things.

And I call it a bit, like, founder-led marketing. I always, like, feel like you kinda have to lean into, like, what you're good at, and, like, what I'm good at is, like, I guess, like, just shitposting and being terminally online. And and so, you know, like, work it, it's it's still, like, you know, I think a lot of our growth is we're just kinda good at social media and just, like, being, you know, online a lot.

Nice. How long until you had your 1st customer?

It took a while, cause we started

Also, also, did you raise, like, early on?

We didn't raise for a while. I actually self-funded it for a while, uh, even with up until the first few employees, uh, which, in a way, was, like, kinda nice cause it made me really, like, frugal about money and, like, wanting to see, like, ROI. I wasn't taking out a salary for a while for a long time, actually. At some point, it it became a problem because I needed the money. But we didn't raise for a while, and, um, uh, until early 2022, we raised a seed round.

So, so, at that point, I'd already been working on it for about a year. Uh, Akshat joins as the CTO and co-founder in in about about 5 years ago, about now, August, maybe 2021. But then it took until, like, basically, like, late 2022 for us to see any meaningful, uh, revenue. And I guess that's and

when ChatGPT came out.

Yeah, and and Stable Diffusion, then, in in our case, uh, what was the main thing that's sort of so so we went from, like, you know, $100 a month in, like, I think, like, November 2022, and the next month, we did, like, $2,000 a month. And then, like, January 2023, we did, like, 30,000 all of a sudden. And, you know, so so finally, I felt this, like, massive, just, like, anxiety thing off my chest.

I'm like, Maybe this is actually, like we're onto something, it feels like we're onto something. And then 2023, we grew very fast, and I guess we've been growing very fast since then, basically.

That 30K a month moment, was that just, like, a couple big customers, or was it, like, very, like, diverse?

It's just 1, and I don't I don't know if I ever talked about, like, who that was and if they even know it, but I think they suspected it was a company in in New York called Mage. They're doing, like, a Stable Diffusion company. So at some point, you know, in in in the first, it I think, first quarter of of 2020, uh, 3, I think they were, like, eighty, 90% of our revenue. But but then after that, you know, it started diversifying a little bit, and late 2023, we we got Suno as a big customer.

They were a big chunk of our revenue for a while, too, but then, you know, it's over time, like, we actually have really good, uh, revenue diversification now.

Yeah. For the Stable Diffusion stuff, I remember using it for, like, Comfy UI. I'm assuming a lot of people u- were using Comfy UI, like, running Comfy UI workflows on it.

Some people were, but but I I think the people that were using Modal, they tended to be more, like, low-level stuff, like, they did, like, complicated stuff. What they like with Modal is, like, it's always been pretty hackable. So they they deploy their their own custom workflows with adapters and custom inference and and a lot of optimizations, and and all the all the stuff we had done in order to get the developer experience to be really good turned out to be really good for also running production workloads.

Uh, c- cause auto scaling turns out to be actually a very similar problem, like, you wanna scale up containers up and down very fast. Uh, so we we just happen to be, like, very well positioned to to capture a lot of those workloads in the early days.

Did tools for auto scaling, like, not exist? Like, why did you have to build those yourself?

In general, like, my thesis is, like, you know, every new sort of era of, like, you know, technology, kind of runs up against limitations of of tools, uh, you know, whether that's you look back, you know, like the web or the cloud or, you know, mobile or PCs or or, you know, AI or whatever. So so I I I think, you know, i it's of often the case that tools exist, but but they're often not, like, particularly good.

And and and often you you kinda need to rethink a lot of those tools in order to reap the benefits of, you know, and and really s you know, turbocharge and start building for this new world. And and you saw this with cloud, you know, we had to invent things like Kubernetes and Docker and things like that, and and it in a in a similar way, you know, like, with with gen AI, you could build these things on the sort of prior generation of tools.

Like, you you can you can build these things on top of Kubernetes and Docker and and these things, but but but those technologies are built for a very different type of workload. They're built for a very sort of steady work steady backend workloads. They have an even, you know, steady number of requests, like, running at pretty low utilization, cause CPUs are normally pretty s cheap, and so people don't really ner- worry too much about cost. And but now you're entering this new world, and you have, you know, very expensive GPUs, you wanna manage capacity and and utilization.

You have very large models that you need to load over network into CPU, into GPU, and and it and it turns out, like, a lot of these tools, like, you know, like, they're they're just not built for that. So so we felt that, you know, we we had to rebuild a lot of that, you know, building our own. I I mentioned the file system, but we also spent a lot of time over the last few years, like, building, you know, some really cool stuff around, like, CPU snapshotting, GPU snapshotting, and aut- we built our own auto scaler and scheduler, and and all kinds of capacity management tools.

And we kind of feel like you have to do that, because, like, you can't build, you know, for this new generation of of software on top of you can't build, like, a wrapper on top of the old stuff in order to to do the new stuff. You kinda need to start over a little bit from scratch.

So I guess, like, the auto scaling stuff, like, did you did you see any, like, um, weird behaviors on the platform? Like, was there anything that, like, broke while you guys were scaling?

I mean, capacity has always, from time to time, been been a challenge, like, how do you get, like, you know, hundreds and thousands and tens of thousands of GPUs? So that's that's always challenging. We had to go multi-region very early. So I think that was, like, kind of a fateful decision, in hindsight, like, doing that and kind of spiting the bullet and building it. Like, we always thought of, like, our our capacity management as, like, 1 big pool, like, it's, like, 1 big pool of it over the the entire world, like, we don't think of it as, like, separate clusters running in each region, and so that that was, like, 1 of the, like, very consequential decisions we made early

to kind of think about it that way. So I don't know if there's, like, been, like, you know, specific issues. I mean, we've obviously had bugs and, you know, outages, and I think, you know, as much as I, you know, would love for Modal to be 100% uptime, you know, and and I think recently we've basically hit 4 or 5 nines, but throughout the history of Modal, of course, we've had, you know, run up against scaling issues and things like that.

yeah. The GPU market and well, actually, before that, so you guys use AWS, like, GCP, like, all these cloud providers underneath, like, do you also buy any infrastructure or no?

Let's talk a bit about, like, cloud capacity, cause I I think it's actually changed a lot. Like, in in the first few years, we did we got almost all the capacity pretty much from from the hyperscalers, spot and on-demand capacity, mostly spot capacity, running in all the different regions, you know, using, uh, initially AWS, but also GCP and and over time Azure. Uh, we we started renting some capacity from Oracle pretty early, cause we wanted, um, a 80s and H100s, and we got really good rates on those.

Uh, and and over time, we added a couple more clouds, like, doing reservations, but but, like, still up until, like, actually end of last year, like, most of our capacity was either spot or or 1 year reservations.

Spot means you're just, like, renting it right there, and then someone else has it yeah, and it's now.

also preemptable, which we had to deal with, um, but but it sort of works in aggregate, cause you can sort of just, you know, aggregate capacity from many, you know, hundreds of regions, basically. So that that that model worked really well until we started running up against I mean, first of all, like, Modal started growing, like, very fast, and so we needed more capacity. But fundamental, also, like, the cloud market started changing a lot, um, so in the last year, basically, now, in order to get the capacity we need, we have to work with a much larger set of clouds.

We actually work with almost 25 clouds at this point, and in many cases, the in order to get the capacity we want, like, we we have to go and do, you know, three plus year deals, and you have to pay a meaningful amount of money up front, uh, and and you're locking in pricing on a on a three year term, right? So, you know, like and and and so we have to spend the entire, you know, last 6 to 9 months, like, you know, kinda rethinking how we get capacity.

We have to build a whole procurement, like, desk. Like, we have, like, 4 guys working full time, basically. All they're doing is, like, procuring capacity, calling Neo clouds, you know, doing deals, vetting the capacity. We've had to build a whole engineering team, basically 10 people, just, like, managing, like, bare metal capacity from all over the world and dealing with network issues and bad drives and GPUs falling off the bus, and and all these things. We don't own capacity.

We rent all the capacity so far, um, you know, on on on typically these 3 year deals. And that's, like, a big decision, like, should we buy, should should we, you know, you know, or should we keep renting? I I'm actually, like, not, like, mega dogmatic about it. I I think renting is nice because there's a certain amount of capital efficiency to it. If you're buying, you know, like, uh, I mean, first of all, there's an obvious, like, operational complexity of it, like, you have to, you know, deal with vendors and suppliers and and, you know, power and and and cables and and all kinds of stuff.

But there's also, you know, financial complexity to it, like, how do you fund it? Like, should you raise equity, you know, and and you can raise $1 billion and and go buy hardware for it. That's a very diluted way to do it. Um, you can raise debt, but, you know, if you raise debt, you know, you you typically wanna have, like, locked in, long-term revenue to to to pay the interest rates, so that's also not trivial.

But, you know, like, we look at the economics, we look at the the, um, the operational complexity, we look at the risks, uh, it's a complex trade-off, and and we're open to to buying. We're we're sort of there's other constructs too, like, we're, it's to get the capacity we need for for 2027, it's gonna take a lot of work on on, you know, going a lotta different avenues to to to lock it up.

So, do you have to hire a lotta, like, finance people to do all this stuff or not really?

You know, I've had to spend a lotta time on this, and we we've had to to to spend a lotta time on this, and I I do think over time, I mean, there there is a lot of, you know, these are very expensive assets, uh, the hardware, and, you know, I think, I mean, it's kinda staggering. You're starting to see, like, AI investment starting to become, like, several percentage points of of of GDP of the US, and, you know, starting to see, like, interest rates in in America shooting up in the last few weeks, and I I mean, at like, at this stage, like, you know, and and you go and look at the neoclouds.

Like, these are very, very sophisticated financial engineering companies, and so, yeah, we we look at that, and, you know, we look at all the options, and and it's possible we might have to do similar things ourselves. I wouldn't want to, like, I would rather focus on being, you know, like, a very lean software company, like, doing the things we're good at. I don't think it's good, in general, for software companies to own hard assets, you know, in in the same way as, like, should you own your office?

Like, I I don't think that's necessarily the right, uh, solution for for most companies. But, of course, like, in order to get the capacity you want, like, you know, everything's on the table.

Yeah, and I guess the supply is going to be constrained for a while, so you kinda have to do that. Like, do do you agree with that, or do you think this is just, like, a moment in time?

I don't necessarily agree with that. I mean, like, NVIDIA is delivering, you know, several 100,000 blackwells, you know, a month, right? And and I I I don't wanna say, like, the market is, like, 100% efficient, but, like, you know, in general, like, if you're willing to pay a high price, uh, you can get GPU capacity, right? Like, if you're willing to pay more than everyone else, and and, of course, Anthropic and OpenAI is buying up as much as they can, and but fundamentally, like, there's constantly supply coming on the market, and and so I don't I don't necessarily think it's, like, a fully cornered resource.

Prices are going up, and and I do think it's worth kinda reflecting on that is, like, the normal state of hardware is, like, it's, like, kinda like gravity, like, eventually, hardware prices always go down. So we're a little bit in, like, a weird sort of period of time where, like, hardware prices are going up, and, you know, it it's hard to say, you know, h- how long this is gonna last, you know, could be another 6 months, could be another 36 months, like, it's really hard to to to know how how how long and but but I I I fundamentally don't know if, like, it's it's true to say, like, you know, you need to buy capacity in order to

to to me, that's a little bit just sort of almost like the, uh, great toilet paper shortage of, you know, the pandemic or whatever, right? Like, you know, at least this, like, you know, run on the bank type situations, and but that being said, like, you know, capacity is gonna be tight for a while, and and, you know, prices are going up, and, you know, luckily, I think you know, it's not even necessarily bad for us if prices go up, cause cause, you know, scarcity also means, like, we can charge more from our customers, and and, you know, and the fact that we have a lot of reservations locked in at a rate means, like, to some extent, we're effectively,

like, long GPU prices already. Um, so, um, I I don't necessarily worry too much about prices going up. I do worry more about actually prices collapsing. I don't think that's gonna happen anytime soon, uh, but I think that would be a quite bad thing for a lot of the players in the market.

Why would that be bad?

I think it would be bad for if for anyone who's locked in a lot of supply.

Yeah, cause they paid way too much, and They paid they're yeah.

too much, you know? Th- so, there's a risk of buying at the top, right? And and again, like, I I don't worry about that that much, like, I don't I don't think it's gonna be, like, you know, it's gonna happen anytime soon. But, like, you know, like I said, like, hardware prices in the very long run, they tend to go down, and so it's it's worth, you know, reflecting on that, right, where, right, like, right now, I think I think your your capacity is constrained, and and that leads to sort of, you know, in economic terms, there's a scarcity rent, right?

Like, who capture the scarcity rent? I I think it primarily goes to 2 places, mostly NVIDIA, but of of course, but also to some extent Neoclouds. Like, Neoclouds, and in general, like, you know, anyone owning capacity has better economics, like, that's undeniably true today, with capacity being constrained. Long term, you know, in the very long run, I don't think that's necessarily entirely true, right? Like, there's always gonna be a little bit of a premium you can get from owning GPUs, but is it worth the risk?

You know, is it worth the risk, like, holding this, like, very expensive assets on your balance sheet? I'm not so sure, but, you know, we're gonna see. Like, it's gonna take a few years to get through this, and and and in the long run, you know, we might own some, we might rent some, maybe a combination, maybe do some other crazy stuff, joint ventures, who knows? Maybe some complex SPVs, whatever. Um, everything's on the table.

How do you compete with OpenAI and Anthropic in buying GPUs capacity? I'm assuming they'd wanna, like, buy up literally everything that's available.

They are, and and and I think what you're seeing is, um, you know, a lot of that the the price is going up is a reflection of that. That being said, you know, NVIDIA's shipping a lot of stuff. I think also, like, to some extent, like, NVIDIA's, like, kind of prefer a little bit of a diversification, like, they don't want, like, you know, small amount of large buyers, in much in the same way as you see, you know, hyperscalers getting a little bit of a constrained allocation.

Uh, I I think, you know, also, NVIDIA prefers to see, you know, a lot of neoclouds and a lot of alternative clouds, like, getting more allocation, and I think a little bit what we see in the market is also this is more speculation, but but I think we see, you know, smaller clouds that are a little bit more sort of risky to work with, being more of an opportunity for us, like, where, you know, maybe NVIDIA sorry, Anthropic and OpenAI my feeling is, like, they prefer to buy from, like, the the the tried and true.

Uh, part of it is also because they they do a lot of training runs, so they need to have, like, very large blocks, but there's plenty of capacity out there, you know? You can get 32 nodes in Armenia, you can get 128 nodes in Indonesia, like, you know, and, like, we love that kind of shit, like, you know, so so we can go out there and, like, get a lot of, you know, smaller capacity all over the world, pretty good prices.

Is it pretty much just, like, if someone offers you, like, a certain number of GPUs, like, that's all you care about? Is there nothing, like, Oh, you care about, like, how reliable it is, you know? Uh, is it just some, like, janky warehouse that someone has these in? I mean, sometimes You guys looking

it's kind of a janky

at that?

warehouse, it feels like, uh, you know. But but, like, we do have, like, a spec sheet, basically. Like, when we talk to vendors, the first thing we do is we send them a spec sheet. We say, We need this number of IOPs, we need, you know, like, the CPU needs to be at least, uh, at least, you know, have have this many, you know, whatever the PassMark scores, or whatever it is. We need to see, like, you know, minimum, you know, north north-south, uh, bandwidth, uh, egress, and and ingress.

Uh, so so we do have a list of requirements. Often, we try to do, like, an acceptance testing period where we get, like, 1 node and and run some tests. Over time, you know, like, to some extent, it also involves a little bit of a leap of faith. You know, sometimes we sign very big commitments, you know, tens or hundreds of millions of dollars, and we start working with the vendor that we still don't fully know.

And, you know, over time, like, you know, we'll see, you know, and and and I think over, like, eh, in a little bit of, like, it's there's a little bit of sort of just, you know, our own sort of, like, you know how, like, uh, semi-analysis has, like, a Cluster Max, I kind of feel like we almost, like, have our own, like, Modal Max score internally. Like, I know which vendors I like more than others, you know, like, and and and we sort of prefer to do additional deals with the vendors that we've seen, you know, very good reliability.

Not just that also, like, it it's not just reliability, it's also, are they very responsive when when we have s you know, support issues? Do they fix the issues we ask for to turnaround time? If things go down, you know, how quickly do they fix it? Like, things like that.

Do you agree that the demand for intelligence is basically infinite, and if so, like, why wouldn't you just buy up as much as you am as much capacity as possible, and then, like, if you overbuy, you can just, like, sell it to somebody else?

I think it's very easy to say when there's, like, a scarce resource, like, let's just buy as much as you can. But I think, you know, first of all, like, if we bought it today, we couldn't utilize all of it. Like, we could use utilize a lot of it, right? But, like

But you can sell it off to somebody else who wants to use it, right?

I think the challenge is, like, if there was, like, something in the market that happens, there are a lot of correlations that happen at the same time. If, you know, price is going down, and utilization going down, and the market for reselling capacity goes vanishes at the kind of the same time. I'm sitting here and, like, I'm I'm extremely aggressive, like, buying capacity every day and, like, raising money to buy, you know? So overall, I'm I'm very bullish, but, like, I do wanna point out and, like, you know, like, th- there there are tail risks here.

And and when those tail risks happen, like, it could be get very bad. Part of it is, like, maybe I'm, you know, I'm just old, and I've seen this in the past. Like, I actually had my first job in 1999, so I saw the dot-com crisis, and I've seen the great financial crisis, and things like that. And, you know, like, I wanna build a company that that is around for, like, 20 to 30, 50 more years, right?

Like, and and so, you know, I I think we can certainly go out and, like, buy an insane amount of capacity right now and try to just corner the market. But, you know, do you know about the Hunt brothers? It's, like, a legendary story in trading. They tried to corner the silver market. It failed spectacularly. Um, you know, so it's cornering the market, I I think in general, like, it can be, um, not not a good idea.

Um, but but we are in general, very, very aggressive. But but we we're ramping up, you know, like, at a at an aggressive plan that basically matches our demand growth and matches where we wanna try to get to at the end of next year in terms of run runway or run rate, I mean.

Are you only buying up NVIDIA stuff, or also, like, you know, maybe a capacity from other types of companies? Like, I think I mean, it was, like, we're Trainium, and

we're we're looking at it. I I th- I think the alternative accelerators are starting to get very interesting. It's, you know, ironically, like, every every year in the last 5, maybe even 10 years, I've been saying, you know, like, next year, it's gonna be the year of, like, non-NVIDIA accelerators, and and then I'm always wrong. But but I do think 2027 might be the year of non-NVIDIA accelerators finally. So we'll we'll see if I'm wrong this year too.

But so I I you know, AMD's, they're starting to get, like, really good scores on certain things. I I think, you know, TPUs are are obviously used very, you know, widely internally at Google. Uh, seem to work really well. I I don't know much about Trainium. I I don't feel like I see quite as but, you know, the the team is very good. I know they're working on a lot of the software stuff. You know, hardware is hard in a in a certain way, like, software is, like, easier, and that makes me feel like, you know, if we just fix the sort of software layer for these accelerators, like, I think there's gonna be a lot of demand for them.

And and I think fundamentally, you know, with NVIDIA being such a bottleneck, like, we'll probably see, you know, more alternative accelerators. That being said, you know, there's bottlenecks further down the pipe. I mean, you know, there's, like, TSMC and ASML, maybe, like, a deeper bottleneck to all of this anyway. Uh, but, you know, long term, I think monopolies don't always last, and so I I'm I think it'd be healthy for the market to have more alternative as much as I love NVIDIA, like, to make that clear, like, I mean, they're obviously an amazing company and with an amazing product.

Um, I I do expect to see a little bit more divers- diversity in in the market. Well, as well al- also things, you know, like Cerebras and Grok and these things.

Yeah, what are your thoughts on, like, some of those, uh, providers? I think Inference they're pretty exciting, Wrappers? Yeah.

and, you know and and you see them doing really well on on certain workloads, and, you know, there there's some interesting, you know, people doing these, like, hybrid, you know, disaggregated inference, where they run, you know, NVIDIA for the prefill and and Cerebras for the decode and and things Oh, I haven't like that.

heard of that, actually.

Yeah, and, you know, Grok too, like, cause a lot of these things, you know, certain things are compute bound, certain things are memory bound, so you can sort of mix and match. Uh, and and I think that's quite exciting. It it's it starts to get very complex when you're deploying these things in practice. Uh, but but I I I do think there's a lot of there's a lot of, you know, opportunity and and and in these sort of, you know, more complex hardware setups.

Are there any learnings that you get from, uh, some of these neoclouds, like, I don't know, like, Nebius or whatever, and, like, the way that they're doing things versus, like, what you're doing?

I think one of the things that, you know, is very interesting with with neoclouds is, like, how can they expand so quickly? I I think actually the most crazy example is, like, Nscale, which went from, like, nowhere to, like, you know, having, like, you know, whatever, hundreds of billions of dollars in in their pipeline, despite barely having any capacity. How can neoclouds grow so fast? I mean, there's, like, an in, you know and and and basically the answer is, like, you know, and and this might be obvious, but, like, what they do is, like, they go out to, like, you know, Anthropic or Coor- or or or OpenAI, some some big buyer or whatever, and s basically say, like, Hey,

do you wanna buy, like, 5 years of chips? You know, sign this contract, we give you a good rate. And then they take that contract, and then they go to, you know, Wall Street, and say, Look, we got this, like, you know, big buyer, you know? They they committed to $10 billion worth of chips, uh, we just need the money right now. And then they go and raise the money, and then they, like, you know, go to, like, Supermicro and Dell or whatever, and NVIDIA, and, like, get the hardware, and then they install it, and then, you know, now they have, like, you know, delivered to the customer, right?

So you so you have this, like, very nice sort of financials construct where you can basically sort of pre-sell things that you haven't even bought. And and that means, you know, you can scale up extremely fast, and so that's what we've seen with these neoclouds is is is they can scale very, very well. I think Modal, because we've always had a little bit more of a PLG focus and and and not the sort of, you know, ability to lock in long-term contracts in the same way, it it's a little bit more challenging from a financial perspective to scale that fast, uh, even though I think, you know, in the long run, like, we'll get to the same point.

But I think that's been kind of a fascinating thing to see with these neoclouds and especially, like, CoreWeave and and those big ones, um, been very, very savvy with, you know, how they raise money and and how to, you know, think about GPU as collateral for loans and all these things.

What's the relevance of, like, I I forget who it was, but, like, some, like, financial company was like, Oh, we're gonna make Oh, I think it was NVIDIA, actually. They were they were like, We're going to turn GPUs into an asset class. Is that just cause, like, they need to raise money and they're, like, trying to figure out where to get the money from? Like, what's the relevance of that?

I mean, there's a bunch of startups, I don't know if you're thinking about that. There's, like, Silicon Data and Compute Exchange and, like, Orin and and and a few other ones that are, like, basically, like, trying to get to, like, you know, creating price indices and indexes and and and trading derivatives and things like that. I think it's interesting. Like, I I think with compute, like, it does there are a number of challenges with compute as an asset class, like, it's it's definitely not as homogenous as other asset classes.

Uh, you have different network and different locations and and and different disks and and CPUs and all kinds of stuff and reliability. I mean, that being said, like, you can say the same thing about oil, and, like, we kind of created, like, you know, relatively commoditized, you know, tradable assets out of oil. It remains to be seen. Like, I do think, you know, at the scale where we are with the sort of AI, like, data center GPU deployments, like, we do need to be a little bit smarter with, like, risk hedging and and how we share risks and and, you know, having more, like, sort of insurance and and things like that.

So, I I'm I'm very happy that people are thinking a lot about this. That being said, like, I I'm I'm not entirely sure if we're gonna see, you know, GPUs becoming sort of, you know, whatever the sort of Henry Hub, like, you know, Brent Crude type, like, you know, super commoditized, like, tradable asset.

Yeah, I wanna shift the conversation to Modal sandboxes. I guess, like, before we do that, what are the different products that Modal has? It's I guess, like, the main product obviously is, like, the the GPUs. Then you have Modal sandboxes. Is there anything else?

I think of the like, the 2 big products that we have is basically number 1 is Modal Functions, and number 2 is Modal Sandboxes. So so, with Modal Functions, we we launched that, you know, back in 2021. It was the idea of, like, serverless functions that are very easy to deploy and very easy to scale, and we manage all the capacity. That turned out to be really, really good for inference, and it turned out to be really good for managing GPUs and and running on things like that.

But we actually never had, like, you know, dedicated, like, inference product or, like, a GPU product per se, like, we we we never thought of Modal as, like, a neocloud, and and I think there's, like, a risk of, like, getting sucked in that direction and becoming a neocloud, like, just selling, you know, sort of what I think was, like, Rectangular, like, Here's, like, a 3-year, you get this many GPUs. Modal's always been very good at, like, you deploy something, you can kinda scale it up, you have this, like, very cyclical or, like, bursty stuff, like, we manage all the capacity, and the beautiful thing is, like cause we have many, many thousands of customers, we can sort of aggregate that

capacity and get very smooth workloads, you know, uh, very smooth aggregate demand, which we can manage very, very efficiently. So, Modal Functions was the the the core. We launched that in 2021. Like I mentioned, we started seeing product market fit in 2023, and now, you know, it's about 2/3 of our revenue, and and and powering a lot of, you know, pro- in practice, it's mostly inference, but but you can use it for all kinds of other stuff.

Modal sandboxes, we launched that in 2023, and it's actually kind of, you know, took a while to see product market fit, similar to Functions. Why did we launch it? We realized a lot of people we talked to were using LLMs to output code. This is, like, me and Akshat talking over lunch one day. We're like, Maybe they need a place to run that code. And and and we realized, like, actually, we kinda have, like, a lot of the infrastructure already to just take code and run it, cause we already built the primitives for, like, managing containers and isolation.

We already had, you know, gVisor as an isolation primitive, and and and things like that. So we we get kind of excited about that idea, and and and Akshat wrote a a a prototype and launched it in basically 1 day or 2 days, or something like that. 0 Like, we we saw, like, very little, like, growth. I I don't wanna, like, trivialize it cause we did see some growth. Like, there were, like, some sort of, you know, I mean, it was growth.

2023, so Yeah, it was it was 2023, and, like, you know, like, d some people picked it up. Like, we had some early users who kind of enjoyed it for for certain things, but but, like, overall, like, it was, like, a pretty insignificant source of revenue into going into the summer of 2023. So 2 years later, we even discussed, like, do do we even need this product? Like, should we shut it down? Then it just started exploding.

So 2023, and and I think what what really happened in 2023 was, like, LLMs got really good at generating code. in 2023? I guess, 2023, I mean, it was, like, around that time. Like, I I feel like that was, like, when they went from, like, being able to, like, refactor functions and, like, write snippets to, like, generating, like, full apps, right? And and and so we started seeing a lot of growth coming from, like, vibe coding, you know, companies like Lovable and and other ones, uh, and and then, you know, companies like Ramp started using Modal for background agents.

And then we see started seeing tremendous growth coming from a number of neolabs doing things like reinforcement learning on Modal sandboxes. So, sandboxes is now about 1/3 of our revenue, you know, we we consider it, like, basically, like, our our 2nd product, and and to me, it's, like, really exciting cause, like, even back in the early days of Modal, I always knew that Modal, you know, is about, like, creating a whole toolkit of products. Like, we always wanted to be almost like a cloud, you know, you have the storage and the compute and a bunch of different ways to work with it.

We always wanted to be a multi-product vendor, but that wasn't necessarily true in practice until, like, now, where, like, I think now it's, like, kinda clear. We have functions, we have sandboxes. We're working on a bunch of other stuff too. We're working on training, we're working on batch, we're working on a bunch of other stuff. So so, to me, that sort of evolution into becoming, like, a multi-product company, I think has been amazing cause that was that was always so core to the vision, uh, even in the very early decks, uh, I I had a bunch of slides on this, uh, but but it wasn't necessarily true to the external world, uh, until recently.

Do you have anything in the works that, you know, you think is gonna be, like, that next product? Any predictions, maybe about the market, anything like that?

I mean, I'm I'm very bullish on training. I think training is, like, kind of an underserved market. Uh, obviously, there's, like, a lot of focus on, like, very large, you know, training runs, uh, but we actually see a lot of demand in the sort of smaller and and mid-sized training segment. So I'll I'll be talking a little bit about that later this week. We're organizing a conference. Batch is, I think, like, a really interesting market. I I think we see a lot of demand for, sort of, you know, customers that need, uh, you know, we had a customer the other day that needed to transcribe, you know, 5,000 years of audio.

They had, like, you know, millions and millions of audio files, and and we we had a lot of capacity, so we we let them do that, you know, it took a few days. I I think there's some really interesting use cases there. Very bullish on, like, low latency stuff. I I think there's gonna be a lot of, you know, once we sort of get the audio, like, the speech-to-speech models working, like, the voice models, I I sort of expect that to be very interesting market.

Then, of course, like, all this, like, agentic stuff, right? And in a way, like, I I have, like, mixed feelings about, like, the word agentic cause I feel like in in certain ways, like, I think all software will become agentic. So it's almost like, to me, like, what does it mean to be agentic? Like, I think I think all software, in a certain way, like, will become, like, AI powered over over time. But but, like, clearly, there's, like, a set of, like, building blocks of, like, you know, doing more sort of LLM-powered software that seems to be sort of, you know, common across all the different use cases.

And and we're we're like, you know, the way I think about models, like, we're in the business of building building blocks, you know? We we wanna give you, like, amazing, generalizable, battle-tested, you know, general-purpose, uh, composable building blocks. And and I think there's some really interesting stuff also, like, you know, with the with the agent use cases.

That's interesting. You bring up training cause I think in another podcast, maybe, like, a year ago, maybe, like, 2 years ago, you said that, like, training you were not focused on, like, specifically, like, what changed in the market?

I mean, we we've been building a training product for multiple years, so, you know, on and off, so so we we I I wouldn't say we're, like we've never been focused on it, but it's, like it's, like, one of those things where, like, I mean, now we're, like, 150 engineers, but, you know, a year or 2 ago, we're we were a much smaller number of engineers, and so but just virtue of, you know, being smaller, we had to focus a little bit more, and so inference became, like, our sole focus and and was our sole focus for many years, but we're bigger now.

Like, we we we do have opportunities to also place bets in in other markets. I'm also really interested in, you know, kind of continuing the the list of things post-training, reinforcement learning, I think there's some really exciting stuff around that. Um, I I think there's also many other, like, stuff around, like, storage and building data warehousing and feature stores and real-time streaming pipelines and and and all kinds of, like, I I don't know. I I think the long-term view of of Modal is, like, basically, like, you wanna build some cool AI shit, like, you know, we got you covered.

Like, you know, we got all the building blocks, like, come to us and, like, we'll help you, you know, take all these building blocks and compose them, and that's the beauty of software. Is, like, you you can, you know, build almost anything if you have the right building blocks. So that that's very much the sort of company and the product I want to build.

When you say training, are you focused on more of, like, the bigger enterprises doing huge training runs? Or are you okay?

No, they're gonna go and, like, you know, do these, like, you know, big hyperscaler or or rather, neo neocloud mega deals and get, you know, 100,000 GPUs or whatever. But but but there's, like, a little bit of an underserved market of, like, in my opinion, you know, like, researchers iterating or, like, people training smaller models, there's, you know, text-to-speech and speech-to-text models. Computational biotech is actually a market that we see a lot of demand from.

We have a lot of customers in that space, training relatively s you know, modest models. You can train on 64 GPUs or whatever recommendation system. A lot of sort of traditional machine learning. There's there's a lot of things where, like, you don't need, uh, you know, 100,000 GPUs. That market, I think, is, like, we're we're not gonna, like, focus on that for a while, but but I think there's a lot of stuff in the, you know, slightly smaller markets that that that are relatively underserved.

It's very hard to set up training clusters.

Yeah. Have you noticed, like, since, I don't know, like, GLM 5.2 or, like, Kimi 3 or maybe, like, Kimi 2.5, uh, that, you know, a lot of people are using those and, like, fine-tuning them?

If anything, I've actually seen the other direction. I feel like fine-tuning the value prop a couple years ago was, like, Yeah, you know, open source models, they're not quite as good, but, you know, if you take, you know, a small model and you fine-tune it on your specific domain, you can actually reach the frontier in terms of, you know, quality and cost. Uh, today, I I would say, like, the open source models are so good that, like, you can basically just, like, swap them out and just run, you know, instead of, you know, the the frontier ones, you can run, you know, Kimi 3 or or or whatever, DeepSeek 4.1 or GLM 5.3, and they're, like, basically almost as good

as, like, the frontier model at a much lower cost. So we still see a lot of demand of, uh, for for post-training, and I do think it's a really interesting space, and we're working on some some products in that space, you know, both sort of SFT and and but also reinforcement learning and things like that. I would also say, you know, fine-tuning is no longer the thing that you kinda have to do, like, it it's it's very optional.

Like, you you can open source models, like, work really well just off the shelf, too.

Back to sandboxes, actually, I forgot to ask you, so, like, what did you have to update about the sandboxes that you guys, um, were offering, uh, that, like, existing sandboxes did not, uh, did not allow for?

Not a whole lot. I I think the thing we've always been very, very good at, I mean, we're infra geeks, right? Like, we're very good at, like, you know, scaling and reliability and and doing a lot of stuff. And and I think that's why that explains to a large extent, like, why we have, you know, in in my opinion, I don't have numbers, but, like, I think we have by far the largest market share in this space.

Like, we're just very good at running, you know, hundreds of thousands or even millions of sandboxes. I will say we'll have to we've had to rewrite our systems, you know, a bunch of times in order to handle that scale. So so but we were always, like, very good at it. The there are other things, like, over time. I mean, the sandbox space in in general, like, you know, like, it's, like, not the most, like, differentiated space, but, you know, we're working on some really cool stuff, like, over time, like, we've had to add support for VMs, like, virtual machines.

So we we we previously built our our sandbox product on on GVisor, but now we offer full virtualized machines as well. Turns out a lot of for for certain use cases, you kinda need full Linux compatibility. There are other things we've had to build, you know, better primitives for, like, more interactive, you know, piping data back and forth between sandboxes, uh, working with file system. We've had to spend a lot of time on snapshotting. I think a lot of the things that over in the long run will differentiate will be, like, you know, really cool things, like, you know, forking sandboxes, snapshotting, resuming snap like live migrations.

I in general, like, I I think sandboxes the novelty of sandboxes was always, you know, this sort of ephemerality, like, you can spin up sandboxes for a few seconds, and and and that's something I I like, that's a string I I wanna pull more, and I I think there's a lot of really cool stuff you can do, you know, kind of decoupling the agent and and the harness, decoupling it from sandboxes, having agents control many sandboxes, forking sandboxes, you know, snapshotting things like, you know, like, running basically, like, very short-lived, may maybe 1 per command, like, things like that.

So I I think there's a lot of cool stuff there. We've already spent a lot of time doing that, you know, around snapshots, uh, but but I think there's a lot of cool stuff, uh, to be to be built there as well.

You brought up snapshotting a couple times, like, w- what do you see as, like, the use case for snapshotting? Like, like, what would that give us that we don't get right now?

I think the the big thing for me is, like, today, a lot of sandbox workloads are kinda like lift and shift. Like, people think a lot about sandboxes almost like, you know, I have my dev box, now I'm gonna shove it in the cloud. I'm gonna have an agent control it. Maybe I'll run the harness inside the sandbox. I don't know if, like, long term, that's, like, the right way to think about, you know, on, like, compute for agents.

I I think a much better model would be you're gonna have these, like, very long-running agents, which almost, like, to me, are, like, kind of durable execution workflows. Like, they're gonna run, you know, 24/7, you know, every once in a while, they're gonna run commands. And and, you know, having the ability to, like, spin up a sandbox in, you know, 100 millisecond, run a few commands, and then, like, shut it down, and, like, snapshot it. I think the faster you can get that, like, you know, wake up, you know, go to sleep cycle, the more you can sort of create sandboxes that only run for, like, a very brief period, and and that will make compute much more efficient, and

it will also enable you to to do some really cool things, you know, like, forking and and moving things around and and and, you know, having like, imagine, you know, like, Claude Code, instead of, you know, having sort of a very linear flow, like, sequential flow, like, kind of branching and, you know, running hundreds of agents at the same time. I I think a lot about, like, CI is, like, another, like, illustrative example. Like, for most of the the history of CI, uh, you know, talk about, like, GitHub Actions and things like that.

Like, you had a very sequential flow. Like, you you you know, you slowly install a bunch of stuff, then you run the test suite, and then, you know, generate some reports or whatever. And it's like, you can break it up into shards, but it's kind of annoying. Like, what I would love to do is, like, take your entire test suite and just run every single test in its own sandbox, right? Like, and just parallelize the shit out of that.

Run, like, 1,000 sandboxes. Why can't that be done? I mean, the problem is, like, there's too much there's too much overhead around, like, you know, starting and stopping sandboxes. So I think the more, like, you you create, like, you know, frictionless, like, starting and stopping, the more you can enter a world where, like, you can do this, like, extremely bursty, like, large scale, like, run as much as you think want in parallel. So so, uh, you know, I'm an infra geek, like I said, like, I I get very excited about these things.

You know, once we figure out all these primitives and and the right building blocks, I think it's gonna unlock a lot of really cool stuff that, you know, we we previously could not do with with software.

Why hasn't that already been built? I would assume even previously, people would would want to parallelize as much as, uh, of their CI as possible.

You can, but the overhead, it's like Amdahl's law or whatever, right? Like, the overhead of parallelization right now is very high, cause, like, right now, like, even GitHub Actions, for instance, like, we we parallelize in in the modal test suite, right? But, like, what that basically means is, like, now if you take your test suite and you split it up in 2, like, now each of those shards both need to do all the installs and and the test setup, and that takes a long time.

Like, it it can take minutes. But let's say you can do the test setup once, and then you snapshot the container, and now you clone it, and now you can run the test suites independently. I think a lot of a lot of the the benefits cannot be, uh, y- y- you can't materialize those benefits on on until you get the the friction and and the overhead down, the transaction cost.

I think you guys have a partnership with Anthropic, and they're using sandboxes for s something like Modal sandboxes.

Yeah, I mean, the the managed agents with with Anthropic, uh, there's a sort of, uh, bring your own, um, sandbox, uh, integration. So you you can use Modal for those. Yeah.

Oh, cool. How did that come out, like, when? Maybe

I think Anthropic is like trying to create, you know, like, a thriving ecosystem and community, and and so I think, you know, they make money from inference, right? Like, so I I I think it's very much in their interest to promote, you know, like, uh, for everything else, you know, as much of, you know, of an ecosystem as possible, and they wanna have the largest possible surface area as long as you use Anthropic for inference. So I I think it makes, like, very rational sense for them to to to partner with with us as a sandbox provider.

Also, I feel like we have to bring up the Hugging Face incident. Are you okay talking about that? Okay, I mean, it's all cool. there. Yeah, yeah. So yeah, could you just briefly describe, like, what happened? And I think they were using the Modal sandboxes, but it wasn't your you know, it wasn't, like, a fault of the sandbox.

Yeah, exactly. I I I don't I forget the all the details, uh, of of of the exploit, but, um, I I mean, you know, I think what you're referring to is the fact that, like, Modal was involved, and so, you know, just to comment on that, like, basically, the the the agents running inside, you know, that that that exploited Hugging Face, they discovered a a vulnerability in an app that was deployed on Modal, and they used that to sort of as almost, like, a springboard for for getting access, getting full internet access, is my understanding.

I don't remember the full details. To be clear, this is an app running on Modal that was hacked. Like, it was not Modal that was hacked. So it's, like, AWS. A lot of people run stuff on AWS. Occasionally, those app get hacked. Does that mean AWS got hacked? Like, no, of course not, right? And and so there's a lot of people running apps on Modal, and, you know, one of those apps got hacked. And I I think, but but, like, there are some lessons we can learn, and and and I think one of the things we've been reflecting on is, like, could we have more sensible, you know, r- security restrictions by default?

I I think to use the AWS analogy in the pa- eh, as just just as another analogy, you know, S3 used to be this, like, you know, it used to be very easy to m to accidentally make an S3 bucket public, and AWS realized, like, you know, it doesn't necessarily cause a, you know, pose a security problem for AWS per se, but a lot of their customers accidentally would, you know, like, leak data, and so S3, it's much harder now to make S3 buckets public.

And so, in similar ways, like, we're exploring, like, you know, can we make security a little bit more restricted by default without necessarily, you know, destroying the magic that make mo- that makes Modal good? And I think there's also, you know, it's kind of a separate sidetrack is, like, agents are getting really good at cybersecurity. So we've actually also, as a sort of, you know, separate sort of learning from this incident, like, we started using agents internally quite a lot to to to try to find vulnerabilities.

So far, hasn't found anything, but, like, they're very good at, like, probing into, like, very deep parts of the system and exploiting s trying to exploit things and trying to e- escalate and get out of sandboxes. Um, so, you know, i it's it's it's something we're exploring very actively right now.

How exactly do you get the agents to, like, run through the codebase? You're just like, Hey, like, find security vulner- vulnerabilities?

Well, no, you just, like, you just put them in a sandbox and and tell them, like, Can you try to Just run exploit? Yeah, through the entire can you just try to escape

codebase?

from the sandbox? It's like, escape the room, but for agents.

Cool. When did you start doing that?

We started doing it a a few months ago, and it it's pretty exciting.

Like, when Table, I guess, came out?

Yeah, roughly around the same time. I mean, my my view on, like, cybersecurity is that, you know, I actually think they're very good at finding vulnerabilities, but I think you can also use them defensively very well. So I I I think, you know, companies like Modal and many other companies right now, like, I think, you know, i if you go to talk to, like, pretty much every CISO, I'm sure they're all thinking about the same thing, like, How do we use agents defensively to to to, you know, fix all our vulnerabilities?

So over time, actually, in a way, like, I'm not that nervous. I think if if anything, like, agents and and cybersecurity, all the breakthroughs in cybersecurity will will make the world safer, if anything.

Do you think that, like, all of the cybersecurity, the big cybersecurity companies are just cooked?

I don't know if that's true because, I mean, you're you're asking something that's, like, very far outside of my, like, core expertise. But when I think about, like, cybersecurity, a lot of what you're selling is effectively fear, and that's almost like a branding and, like, marketing strategy as much as, you know, obviously, the core products. But I think, you know, brand and and things like that are actually, like, arguably, like, more valuable in the era of AI.

So I I I wouldn't necessarily jump to the conclusion that they're cooked. I think they have, you know, deep relationships with the enterprise customers, and they have, you know, big sales teams who are very good at telling, you know, spooky stories and and things like that. And and I think they're gonna continue to be very good at that.

I just ask cause, like, yeah, you're doing it yourself. You're not going to one of, like, the, uh, Palo Alto Networks or whatever. But I guess, like, it's only, like, giant enterprises that care about that. Also, they can blame somebody if something goes wrong.

That is true. Yeah, yeah, you kind of want to hire, in many cases, just like a established vendor, uh, in order to blame them. I I I mean, to some extent, it it does reduce a little bit the need for us to do to bring in pen testers, like, human pen testers, which most startups, like, do on a regular cycle. So maybe previously, you know, we would do it. I actually don't know what the cycle was, but but I think we used to do it, like, once every 6 months, roughly.

Maybe now we only need to do it once every year. I I don't know. But but, like, I think with all of these things, you know, humans can also use them to get more efficiently. So, like, you know, like, I I I don't know. I'm I'm kind of a believer also that, like, humans with AI is even more powerful now. So, like, maybe there's an argument that, like, we can bring in pen testers who are really good at using agents and, you know, find vulnerabilities even faster.

I I don't know. I I'm not that, you know, I'm an optimist. I started a company, so, like, you know, I'm I'm generally, like, optimistic about everything. But, like, I I certainly don't think humans are cooked or companies are cooked. Like, I I think in the era of AI, a lot of companies are gonna figure out, you know, how to use AI to be even more efficient.

Hm. I wanna transition a little bit to, like, developer experience, cause, like, early on, I mean, even now, like, you guys focus a lot on that. I mean, you've talked a lot about that. Does developer experience even matter anymore?

I think it still matters. Yeah, and and I think you're you're sort of, you know, like, the context, yeah, I think you're fish like, the thing you're talking about is, like, now with agents, right? Like, d does the developer

Like, the docs are already there. The agents are gonna look at the docs, like, all that stuff.

to that point, though, like, almost all the things that make developer experience good also makes the agent experience good, is kind of what we've seen in practice, which is, like, things like having good docs, having descriptive error messages, having fast feedback loops, right? Like, I do think there's certain things have changed. The other day, I was, like, setting something up with Cloudflare, and it's, like, I find Cloudflare, like, notoriously annoying to work with sometimes, and and I threw an agent at it, and it was able to, like, just configure a bunch of, like, ask, you know, things in in Ploomi for me, and, you know, and and so I will say, like, agents does they do make it a little

bit easier to work with things that are, like, harder to work with in the past, you know, like, the sort of human toll of dealing with YAML has arguably gone down a little bit, cause you can just, like, you know, tell your agent to deal with it. But but for the mo- most part, I I think, you know, the things that make developer experience good also make agent experience good. I think to some extent also, like, infrastructure over time, anyway, becomes and needs to be more programmatic.

I think that's always been the thesis about sandboxes is that, you know, like, things that you previously had to configure using YAML, that should just be an SDK. That should be more, like, imperative, like, launch a sandbox, like, do stuff with it, tear the sandbox down, right? Like, rather than sort of declarative, using using, you know, some sort of DSL. It also pushes, like, a little bit of a different interface that maybe previously didn't exist. I think in either way, like, SDKs are as relevant as they will a as they always were, if not more.

Is there anything you changed in your product just because, like, you know, most people are just running agents on on it?

We do, uh, I mean, we we also we actually do run a lot of internal tests where we have agents we tell agents to, like, build a modal app and run it. And, I mean, a a very concrete thing that is kind of funny is that we've actually seen that, like, because of agent training data, it's very hard to deprecate things now. Uh, we see that, like, agents often, like, love using, you know, old versions of modal syntax and and it's kind of hard to nudge them away from that.

So so we've actually had to be a little bit more conservative going forward, like, you know, eh, over the last year or so with with how we deprecate and remove features. It takes we have to be very careful to update every single example that's out there online. And and and, you know, You have And and

Skills, right?

Yeah, there's, like, stuff like that. But but, like, there's also an insane amount of, like, modal code that's now, like, online too that goes into the training data. I've seen, like, that to be, like, kind of an, you know, little bit of, like, an interesting example where, like and another thing, by the way, too, is, like, there's obviously, like, the sort of e- similar to, like, search engine optimization, there's, like, agent engine, you know, whatever people call it, agent optimization.

We often, like, run internal benchmarks where we're like, ask a an agent, like, Hey, like, can you deploy a stable diffusion for us? Use whatever tool you want. And And we see if it recommends modal or not, and those things are very useful for us to sort of track over time. Or, like, Hey, spin up some sandboxes, or, like, Hey, like, rank all the sandbox vendors. So so we we spend a lot of time, like, doing doing those things to to to make I mean, what Sure.

Do you do about that? Yeah, you say, Hey, like, you know, build this stable diffusion thing. Does it recommend modal or not? Like, what yeah, what do you do about that?

Yeah, I I think that's harder. I I think, you know, it's like search engine optimization. I think it's primarily through content. It's like you just have to put out a lot of content. You can also, to some extent, like, sort of see the perception, I think, a little bit more. Like, you can ask an agent, like, you know, how does, you know, rank all the sandbox vendors by reliability or rank all the sandbox vendors by price and and sort of see.

And it does feel like agents are a little bit more prone to, like, mis- understanding pricing and and, you know, things like that. So we also spend a lot of time, like, just making content more clear and more, like, you know, understandable for agents, 'cause we know that all the indexing, like, everything that's out there and And that's talking about modal.

Have you thought about, just, like, making a bunch of GitHub repos and having the agents, like, write a bunch of modal code in there? Have you tried Slop slop That?

Cannon. Uh, I think the truth is, like, that is probably something you can do. My my guess is, like, LLM training data today is probably quite insensitive to, like, what's, like, AI output in the first place, and you could probably, like, Create a lot of slop. Like, we've thought about it, like, I I don't love that idea, but, like, in theory, like, much like, you know, SEO in the early days, and still to some extent, like, auto-generated content is, like, a very effective strategy.

I think it's possible, like, you know, you could you can do those things Too.

I wanna switch to FDEs. Apparently, you guys hire, like, former defense engineers. Like, A lot What do they

Of them.

Do?

We're in a new era of, like, building applications and and and, you know, things that we couldn't build, you know, just a few years ago. And we see a lot of companies that, you know, they need a little bit of help, right? Like, you know, and and so we started investing in FDEs a couple years ago, but it but it's really been this year we just kind of scaled it up a lot. I think we started this year having, like, 4 FDEs now.

I think we have, like, 24 or something like that. So we've grown a lot. And they do it, you know, it's more so exactly the Palantir model, but they do a lot of, like, heavy, you know, sort of customer engagement. Like, they work with a lot of customers on performance tuning, especially with LLMs. We we see that there's so much benefit from, like, you know, sitting down with customers and training draft models and tweaking, uh, you know, the parameters and DLM and SG Lang.

Um, they they often, you know, fix bugs or or not fix bugs, but, like, you know, when when customers run into issues, like, they they often work with customers to try to figure out, like, how do we work around that? Or, uh, another thing we we've used FDEs a lot for is, like, just kind of proactively, you know, if if we see certain customers, you know, over-provisioning with modal, like, customers requesting too much CPU or too much memory, for instance, like, that's kind of expensive, and, like, we we try to catch those things, and we try to sit down with those customers, like, Yo, like, you're spending a lot of money on modal right now.

We can actually help you save money. And then, of course, like, with with sort of active deals, you know, we we help them migrate code from other platforms, uh, things like that. Um, I think, you know, we we we never have the intention of becoming, like, a service-heavy business. I I think the the goal for FDEs is, like, you know, A is to be part of, you know, the the sort of sales process and, like, closing deals.

B is to serve customers and and do some, like, you know, customer support. But, like, I actually think, like, the most important thing is, like, C is, like, bridging gaps in the product that we later want to turn into, like, real features. So, I I think to a large extent, you know, anything an FDE does, you know, my question is always, like, long term, how do we turn this into product so that we don't need an FDE for this?

Yeah. Do you guys hire PMS?

Product managers? No.

But would you say these FDEs are kinda like pseudo-ish PMS?

It's a little bit of a different function. Like, I I think certainly on the more, like, you know, customer-facing side they are, but there's, like, you know, on the more, like, inward, you know, internal side, like, there's a sort of separate part of, like, product management, which is more like, what are the next building blocks modal should invest in? And, like, you know, and then there's, like, something in in in between, like, you know, looking at, like, maybe 10, 20 different customers, they're all asking for, like, s slightly different things, but maybe what are the patterns, like, how do we turn this into product?

We do have people doing product with us, and and it's sort of evolved, you know. We've had people basically, like, become product managers after previously being either devrel or or engineers and just showing strong aptitude on on doing thinking very strategically about what we have to build. So, I'm I'm actually not, like, anti-product management. I do think that a lot of companies in the past hired product management product managers too eagerly, and and I think there's a risk that you you kind of I can sort of see it in a lot of products.

Like, certain products, like, you can tell, like, there's just, like, eager product managers that just thought of, like, kind of, like, widgety things and just shipped them. I think doing product management at an infra company really well is very, very hard, 'cause it take it takes, like, a very deep, you know, level of strategic thinking, thinking about, like, what are the foundational building blocks? Where do we wanna be in 1 to 2 years? 'Cause the truth is building infrastructure is very hard.

And how do we make sure we don't end up with, like, 35 features that are, like, kind of similar looking, but all do, like, a little bit slightly different things? That sort of level of thinking is something that, you know, is very rare to find, and and for that reason, we haven't aggressively tried to build up a product management team yet.

Yeah, that makes sense. How would you say your teams have been evolving, maybe, like, since January? 'Cause I would assume yeah, I mean, you obviously worked at, like, Spotify, you had, like, a certain team structure. Um, have your team structures, like, evolved a lot now that, you know, we have coding agents?

In a funny way, like, yes, I I do think team structures have changed a lot, but I'm actually not convinced it was, like, gen AI driving that. I I think, in particular, like, one thing I've seen a lot is that there's been sort of a focus, like, a swing back towards being an IC. Th- that I actually think is very, very positive for the space as a whole. I think, you know, if you look back at the space, like, 10 years ago, 20 years ago, everyone wanted to be in management, 'cause that was the way you you would get more money and more, you know, more recognition and more prestige.

I think that shift actually started even before gen AI, but I do think it's, like, heavily accelerated with AI. Like, you know, we we we make it very clear to everyone at Modal that, like, manager or IC, it's like there's no difference in prestige. Like, everyone is just a member of technical staff. You know, it's about ultimately your contributions, and we see a lot of ex-managers, you know, becoming ICs and and 'cause they wanna go back and build code.

And so, and, you know, and there's been there's been sort of shift away from, like, you know, non-technical managers. Everyone pretty including me. I I still write a lot of code at Modal and and Oxshott too, and and pretty much everyone, uh, in in the tech team, and a lot of people who are not in the tech team still write code. So that, to me, is, like, you know, a a recent phenomenon, but but I actually don't know if it's, like, necessarily caused by gen AI, but I think gen AI has, like, greatly accelerated it, 'cause it it made it even lower the cost of, like, writing code is even easier.

So if you have, like, non-technical people, like, committing code, like, how do you how do you r- review that? Like, how do you guys do code reviews?

We do code reviews, but it's we we think of it as optional. I mean, for most, uh uh, my my view is, like, code reviews in in large part is, like, people in many cases trying to just escape the the accountability of, like, you know, merging things into production. And and if you have that, like, you know, if you have that culture, like, then to me, that's a sign that, like, you need to invest way more in, you know, CD and and CI and and and things like that.

There's CI in particular, but also CD, uh, continuous integration, continuous deployment. We do code reviews. Like, I mean, like, when I write code, like, occasionally, like, I'm I'm asking, you know, authors, like, Is this in line with how you think about things? I hope I'm not changing the architecture fundamentally. But we also make it clear internally that, like, code reviews are are quite optional. And, you know, if you're just changing docs, if you're just doing, you know, some basic stuff, like, don't wait for a human to look at it, especially in the era of AI.

Why do you need a human to look at every piece of code? You don't have to. But that does require you to have, you know, very good controls, you Know, internally around, you know, Especially Testing things.

At an infra company.

Yeah, and and exactly, like, you know, infra companies, like, we we need 1 more 9 than the average startup, right? In terms of uptime. But, like, you know, these are investments you you just have to make. Like, you have to build systems that are very resilient to, like, you know, you have to have rate limits everywhere, you have to decouple systems, you have to make sure that, like, nothing is a single point of failure, you have to have extremely good test coverage.

Agents are actually really good at writing tests these days, which is, like, you know, a few years ago, I I thought, like, one of the major things missing in agents. Now they're actually quite good at, like, because I think it takes a little bit of, like, sort of kind of, like, need to step out of what you're trying to do for a second and be almost a little bit, like, devil's advocate to write good tests. But now they're actually quite good at writing tests too.

So I I think I think all these things help.

Do you guys use, like, frontier models for coding, or do you guys have you guys experimented with, uh, like, the open source ones?

We experiment with everything. But, yeah, I mean, you know, initially, it was a lot of cursor, then, you know, of course, Claude and Codex, and and now a lot of people are using, you know, Nightshift and and all these open source stuff or or other stacks. So I don't necessarily have, like, a strong point of view. Like, I I want people to experiment and try different things. Like, we, you know, use frontier models, use open source models.

We're trying to do a little bit more dogfooding internally now that we're actually running inference for a lot of the open source models, like, at a very large scale. I think it's also helpful to, like, use certain internal endpoints for a lot of agentic development. Uh, so we try to promote that a little bit.

Hm. Something I read maybe, like, I don't know, a year ago, maybe, like, 2 years ago, you were saying that you didn't think that there was gonna be, like, a big on-prem business for you guys. Um, have you maybe, like, changed your opinion on that?

No, I have not changed my opinion. Um, I think modal to me, I I'm a cloud maximalist, and maybe to take a step back, like, when when the cloud came in 2008, 2009 or whatever, my first reaction was like, That's crazy, like, why would I wanna take my code and run it on someone else's computer? That's, like, insane. And then, like, 6 months later, I was doing it. And I think, you know, like, took me 6 months to change my mind.

It took probably enterprises, like, another 15 to 20 years. But, like, you look at, you know, vendors like Snowflake or, I mean, AWS, and, like, you know, I I think there's many examples of Datadog, et cetera, like, the cloud is clearly winning, right? Like, and and the multi-tenant systems are clearly winning. And, you know, to me, on-prem looks increasingly, like, sort of a legacy business model. And and I actually think, you know, BYOC, you know, sort of VPC peering, like, all of these things, you know, like, to me, they they reflect sort of an outdated security model of, like, we need to protect everything inside our own network, right?

Like, using the network as a moat. And that, I don't think is true anything an anymore, right? Like, we we've sort of moved on to more, like, actually robust security paradigms, like, you know, uh, zero trust and things like that. And so, to me, BYOC and those things, like, they they look increasingly, as, like, outdated, at least from a security perspective. From our perspective also, like, the fact that we have full control over our entire infrastructure stack, we rent all the hardware, we deploy, we manage all the infrastructure, we deploy and, like, manage health and observe, you know, all the metrics on all the hosts.

It means we can iterate very fast and build a greater product, and and I think that's very, very valuable versus, you know, customers in the past who had enterprise editions that they had to deploy on customers' environment. They had to be very, you know, they have to think very carefully about release management and and having depend- like like, we don't care like, if we wanna introduce a new database, like, we can introduce a new database 'cause we have control over it.

Like, we don't need to think about what customers have access to. So, to me, multi-tenancy is is there has so many different benefits. I mean, that being said, like, you know, if we were to work with, like, you know, some very large Fortune 500 customer in the future, and, you know, they ask for BYOC, you know, we've done this in the past. We'll give them a number. Like, you know, that number is very high, and no one said yes to it so far, but, like, who knows?

Like, You know, at at at the right price, I I I would still build it. And and I think there's, like, maybe separately, like, a a another thing which I think is interesting, which is, like, in theory, BYOC might also be a way to work with and and get capacity in certain ways. Like, we we sometimes work with Neolab, and those Neolabs have already have very large capacity reservations. And, you know, the question is, like, you know, how how do we how do we get them to run modal when they already have capacity?

And BYOC might be a solution. We can still keep the control plane in in our environment. And the other flip side of that is, like, we could maybe even run, like, other people's stuff on that capacity. So so there's some, like, kind of fun stuff you could do in theory where, like, you can sort of, you know, like, kind of aggregate all that capacity into 1 big pool. So so we're we're excited about some of those ideas, but but in general, like, I feel like on-prem, I don't know if we're ever gonna do.

But BYOC, like, maybe bring So Your own cloud. I keep saying BYOC, I should Yeah, it's okay, Like, like,

It's okay.

I should explain the acronyms.

Um, let's say, hypothetically, you are going to raise, uh, uh, you should attend more money, like, soon. What would you use that money for?

I mean, like I mentioned, like, in in this capacity situation, the a lot of the money a lot of the cash flow, I should say, that's required is is capacity prepayments, right? And and so that, to me, represents the best use of of of cash today. Uh, we're ramping up capacity capacity very aggressively. Maybe we move into, you know, other types of deployment models in the future- ownership or whatever, but I think there's always gonna be a cash component of that, like some sort of down payment or prepayment of some sort of that.

So that, to me, is is the the greatest use of cash right now. There's, of course, other things like, you know, salaries and office spaces and stuff like that. But that actually, in the grand scheme of, like, our our, uh, financials, like, it's it's basically a rounding error, right?

Yeah, interesting. Do you think that's gonna change?

No. I I think, you know, the the sort of operating leverage that we have in this business is kind of insane. Like, you know, we have 200 employees, you know, they cost a lot of money, but, you know, the the the GPUs cost a lot more money.

Yeah. If you had to say, like, what are the top, like, I don't know, like, 3 things that are your biggest focus or the business's biggest focus right now, like, what would you say those are?

Capacity and the sort of financial complexity and, you know, in general, like, how do we how do we think about that space? How do we think about pricing and packaging? How do we think about the sort of tension between ownership and renting? Should we be going in a more neoclass direction? Should we go in the opposite direction? The sort of counterposition, like, that's probably the biggest thing that I think a lot about. Uh, the second biggest thing I think a lot about is, um, probably around, like, more, like, product strategy.

Like, I think a lot about, like, what is the next building blocks? Like, if if we, mm you know, do more, like, agentic stuff, like, what are the things we need? And, you know, uh, are there certain storage primitives or compute primitives? Or, in general, like, what is the thing that the world is gonna wake up and realize they need over the next few years? And and ideally, we have that at that point. We've been pretty good in general at, like, you know, anticipating demand and building products in advance of it with the serverless functions and the sandboxes.

So what's the next thing? Then I would say, like, actually, the third thing I think a lot about is, like, more around, like, culture and company building and stuff like that. I mean, we recently crossed 200 people. We're we're growing extremely fast. We'll probably hit 250 by the end of the year. Like, when you're growing very fast, you have 250 people that are all trying to figure out how they can add the most value to Modal, and, you know, making sure that they have the right context, that they have the right motivation, that they're excited to come into work every day 'cause they get to build something incredibly cool, and they know, you know, what makes Modal successful, what they

can do, aligning the incentives and and making sure that, you know, we we we people have, like, a very positive culture where they have the sort of implicit trust to do a lot of stuff, but also the accountability to, like, you know, deliver the best things and, you know, and and be held accountable to, like, if they're not doing it. To me, those are, like, you know, super critical things around this scale. Like, when when you're, like, 10, 20, 30 people, you can all just, like, kinda sit in a room and, like, it's all kind of implicit and you kinda just, like, feel it.

People can sort of read read me, like, they can tell me when I get a little bit annoyed. They can tell when I'm happy. But, like, now we're 250 people, and, like, people don't necessarily see it. And, you know, of course, there's, like, a lot of new people. They're, like, maybe a little bit nervous to, like, talk to the CEO and, like, all this stuff. So, like, you know, how do I remain approachable? And how do I, eh, like, it's it's there's, like, the Dunbar number, which I think is, like, more sort of, you know, fake than than anything, like, at least, like, the story behind it.

But, like, it is actually kind of real still. Like, there's something around, like, 150, 200 where, like, laying the foundation of the culture is very, very important.

How many people are you guys trying to hire over the next year?

We've grown about 4x on an annualized basis in the last 12 months. I think that's, like, an unsustainable growth rate on a relative basis. I've seen that, you know, you can double the team every year, and that's not super hard to do, but 4x, what you start to see is, you know, cultural drift. You start to see islands of different working styles. You start to see people doing, like, very speculative things. Like, that that's something I always watch out for in the engineering team.

You go and talk to the eng- an engineer, and I'm like, What are you working on? And and it turns out they're, like, building something, like, kind of, like, fringe, and, like, is this really what you should spend time on? So growing very fast has that cost, and, you know, it has the cost of ramping people up. So I I don't know. I I think by the end of the year, maybe 250, by the end of next year, you know, but but, like, you know, we're gonna keep growing, like, maybe 2.5, 3x, so so it's very likely we'll end up, like, next year, like, 700 people or something like that.

Yeah, I'm gonna assume most of those are engineers. Most mostly engineers. We're Like, what Ramping up sales Kind

A lot, too.

Okay. What kinds of engineers?

It's primarily systems engineers. I mean, we've always been, like, very deep in the sort of, you know, Linux and file systems and scalable systems and networking and and cloud and and stuff like that. So so that that's that's the core of what we're looking for. Like, people are really good at those things. We're also looking a lot at, like, you know, more, like cause we moved up in the stack a little bit. Like, we're doing a lot of work in the LLM inference space.

So now we're dealing with a separate set of concerns around, you know, CUDA and, like, GPUs and and MFU, you know, like, maxing out the tokens that you can get per GPU and and and a lot of that things. And, you know, e- even though I actually, like, will say a lot of the work in that space, in my opinion, turns out is, like, more systems-driven still, cause it turns out, like, KV caching and routing is actually, like, is still, like, very, like, systems-like, like, like, these are very, like, systems-heavy, p- uh, problems.

But, yeah, a little bit more, like, sort of GPU performance, that kind of stuff. We're starting to build up a research team. We're very excited about, you know, g- moving into stuff around, like, you know, post-training and and distillation and and better methods for for quantization and For your Things like that.

Customers or for internally?

Both. I I think it's, like, a little bit like with the FDEs, like, I I I think we initially will do a lot of stuff with our customers to optimize their models, but over time, figure out how do we turn those into products. I think we'll sort of, arguably, fail in a certain way if every time a customer needs to optimize a model, we have to go and work with them, and, you know, long term, it has to be turned into products.

And and so, with the FDEs and with the research team, initially, you're gonna have to do a lot of work with the customers. And we are that. But over time, we're also building a lot of internal tooling and a lot of over time, internal products that are fully self-serve. Uh, so we're working on a lot of stuff right now in in the post-training space, but also in in the GPU performance space. So a lot of cool stuff.

A lot of cool stuff.

Back to the product strategy stuff that you were talking about, Um, yeah, like, what are your thoughts? Like, what do we need over the next year, 2 years?

I think it's incredibly hard to say, and and i if I said, like, I know exactly what we need to build, like, I I think I would be, like, delusional. I think what what I've what we've seen so far, you know, and and what I kind of believe in on a more general level is, like, we need better building blocks. Like, this is a new era of of software, and we're unlocking a whole new type of applications that could not be built a few years ago.

You know, doing, you know, all the gen AI stuff, code that, you know, LLMs that output code that, you know, needs to run, like, in an isolated way. Like, there's so many, like, new primitives that are needed to to work with GPUs and work with LLMs. And, you know, we built some building blocks so far, and and, you know, that's unlocked, you know, a bunch of applications that that the world, you know, has seen over the last few years.

Not just us, of course, but I think over the next few years, like, there's gonna be so many more cool things. And so I I I mentioned some stuff, like, I'm very bullish on training, I'm very bullish on batch jobs, I'm very bullish on, you know, certain things around, like, agents and and low latency and and things like that. But but, like, in the end, we consider ourselves like an infrastructure shop, right? Like, we we're not trying to build, you know, very narrow things.

Like, we try to build, like, very general-purpose building blocks and put them in front of customers is, and then those customers will combine those building blocks and build some incredible applications that we could have never anticipated.

I wanna go through some of your tweets. This will be Like the last part of it. Um, And we can just, like, maybe you wanna double-click on anything. A lot of long-running persistent sandbox stuff right now, uh, today feels lift and shift. I suspect sandboxes will evolve from dev boxes in the cloud to something more ephemeral, uh, 1 environment for every command and snapshotting in between. So you can pause, resume, clone. Yeah, I mean, we talked about this a lot.

Yeah, we talked Yeah.

A bit about it, right? Like, I I I'm fully com- I I think s I think the median sandbox in modal today is, like, 8 seconds long or something like that, like, which is kind of incredible when you think about it. And and over time, I think it might get even shorter. We wanna support long-running sandboxes, too. Like, I think there are some, like, very valid s use cases around long-running sandboxes. But, like, my vision is, like, you know, the world needs also we need to figure out this, like, how how can we just massively parallelize things and reducing the overhead of starting and stopping sandboxes?

I don't know if I ran an IP already startup engaged in catfex arms race with a competitor, it would be nice to coordinate and pace the arms race a bit.

No comment.

The biggest downside of GPU scarcity is not that it's hard to get GPUs, but that almost all capacity is sold as 3 years as a 3-year rectangle starting 3 to 6 months in the future. That's not what demand looks like for most users. I think we talked about that a little bit as well.

Yeah, and, you know, maybe this will go a little bit deeper now. Like, one thing I think Modal has always been very, very good at is fully on-demand capacity, right? Like, if a customer comes to us and need 100 GPUs, like, we can typically give them that, like, immediately. You don't even have to talk to salesperson. And a lot of workloads in in in real life, like, they don't look like these, like, flat, you know, rectangular workloads.

Like, they scale up and down throughout the 24 hour cycle, they maybe grow over time, they maybe have certain burst periods. And to me, that is a challenge with a lot of customers today. Like, the and and I do think it's a legit problem for for the space. It's, like, the only the prime way to get capacity is to to buy these rectangular blocks. It's not great. I think the world needs to evol- and and that's fundamentally, I mean, I'm I'm very biased, but, like, that's why I think Modal can play a very large role in in that sort of, you know, whatever you wanna call it, sort of duration transformation of offering more flexible capacity.

And we can do that, be we because we can aggregate many, many thousands of customers, and the demand gets very smooth in aggregate. So that's a lot of the value prop of Modal is exactly that.

Would you say that's a differentiator? Like,

I think so.

The Oh, really, okay.

Yeah, massive differentiator. Actually, in fact, like, I think models, like, if you wanna be, like, very sort of reductive, like, you can break down models value prop into 2 things. One is the developer experience, the ability to, like, iterate and, like, build things and not have to think about infrastructure. The other thing is the capacity stuff. The more and more I think about Modal and, like, where we're going in in the long run is, like, we're building, like, we're aggregating the world's compute capacity, and we're very good at slicing it up, like, very granularly, 'cause we've built all this work around, you know, scaling up and down extremely fast, you know, all the stuff around GPU snapshot and working some,

uh, some ways to make it much, much faster. And that model works at very large scale, like, really, really well, 'cause you can aggregate lots of customers.

Every month, I keep saying our sandbox revenue might apply to it, but it keeps going up 30% every month.

It's kinda nuts. Like, I don't unders- you know, like, every month, I'm like, Well, you know, Sandbox has grew a lot, but, like, clearly, they kind of plateau, and then they keep growing even more. So so I I don't know. And a lot of VCs, like, they're like, How did you know Sandbox was just gonna be big? Like, and and the truth is, like, I didn't. Like, like, I I thought it was gonna be bigger than it was in 2023 for sure, but, like, the fact that Sandboxes is 1/3 of our revenue today, it's it's insane, and, you know, we're obviously very happy about it.

We we have, like, a very large team investing in Sandboxes. You know, I don't wanna, like, trivialize it, like, we obviously I do wanna s give ourselves, like, a lot of credit for for seeing Sandboxes for what it is today a little bit back in even 2023. But I I I think, you know, if, like, I probably would have never, you know, taken a bet that revenue would be this large from Sandboxes and the market demand would be so much.

There's an interesting structural reason why inf- inference providers should exist separately from model builders. The best model keep changing a lot. Inference providers keep can easily reallocate resources between models since overall demand is quite smooth.

So, I mean, if you think about all the Chinese labs and, you know, in the future, hopefully more US labs, too. Every time a new lab releases, you know, the latest and greatest open source models, like, you see big shifts in demand, you know, for for these models. And and so, I think a challenge with, you know, I if all these labs independently, like, held capacity, they would see when they launch a new model, demand would go up a lot, and then another lab launches a better model, and then the demand goes down.

You know, but but, of course, like, in the background, they're working on the next version. So, like, I think, you know, there's, like, a structural reason, like I said, like, why you're gonna start y- you're seeing demand kinda shifting to, like, the latest and greatest Chinese lab. Mo- I mean, not like, it's not like, you know, this is, like, everything just moves overnight, 'cause 'cause a lot of models and prompts and stuff like like, a lot of things like, get sort of there's, like, a certain stickiness to it.

But I I do think, you know, there's an argument that it's very easy for modal to, like, you know, right, today we're running, you know, x number of thousand GPUs for one model. Tomorrow, we need to shift, like, you know, 50% of that to another model. Sure, that's just, like, that's what the system was built to handle. No one even needs to tell us to do that, it just magically does that.

So you agree that basically the LLMs are commodities? And if you agree, then where do you think the business for the big labs is going to be?

I'm not sure if that's necessarily true, right? Like, is Taylor Swift a commodity? Like, she releases a big album and goes on a tour and makes a lot of money, and then, you know, then there's, like, a down year when she goes and, like, records, I don't know if this is the best analogy. And then she releases a new album, you know? So I I I don't know, like, she still has, like, a lot of brand, and there's a lot of, you know, power and, like, makes a lot of margin.

So, like, as long as the the labs, like, repeatedly can launch new models, and as long as the margin that they have on those models are such that they they can sort of pay off, amortize, like, the the cost of training those models, it could still be, like, a decent business, I think.

So you don't think that the intelligence is, like, plateauing?

No, I don't I don't think so. I mean, I I think it remains to be seen, right? Like, models are today are, like, very good at, like, doing things that can be trained very efficiently, you know, using reinforcement learning. So so we've seen that initially with, you know, coding and writing simple snippets. Over time, it's gotten really good at also writing production-level code. It's gotten really good at math. It's gotten really good at cybersecurity. Like, these are all, like, very quite verifiable things.

I I still think it remains to be seen, you know, are models gonna be really good at whatever cooking food or I I I don't know, like, things that have, like, more, like, diffuse targets, that it maybe longer feedback loops or or whatever. I think humans probably, like my feeling is, like, we'll be quite good and quite competitive with with models for a long time in those domains. But, you know, certainly, you know, AI models, they're they're getting very, very good, As as I'm sure everyone can agree with.

Yeah, so obviously, like, the models are getting really good at coding, they're already really good at coding. What other applications do you think maybe that aren't as big right now are going to become huge over the next few years for I think The LLMs?

I I I don't know, but I think anything that has, like, kind of a verifiable

The computational bio, I think you mentioned Yeah, yeah, computational Before.

Bio, I th- I think those are, like, not exactly verifiable, but there's, like, certain ways you can sort of run simu- like, you can use an LLM to output predictions, then you can do, like, computational, like like methods to, like, simulate the molecular dynamics and get, like, kind of feedback loops. I mean, like, I think, like, material science is, like, maybe another area where, like, you have a little bit of that loop where you can sort of, you know, synthetically manufacture certain materials and measure the properties, and you just kind of put an agent in the loop and, like, you know, do things.

So, you know, I have a physics background, I'm pretty excited about that. So drugs, materials, what else? I'm drawing a blank, but I'm sure there's many other things.

Cool. I mean, those are all the questions that I had. Is there anything you wanted to talk about that I haven't?

I'm good. This is great.

Cool. All right.

This is a lot of fun.

Awesome, thank you.