What Is an AI Agent?

Today, we're discussing one of the busiest and most confusing terms in AI right now, agents. Are they just fancy wrappers around LLMs, full-blown autonomous workers, or something in between? a16z Info Partners, Guido Appenzeller, Matt Bornstein, and Yoko Li, break down the technical definitions, pricing models, use cases, and why the term agent means so many different things to different people. If you're building, buying, or just curious about what agents are and aren't, this episode is for you.

Let's get into it. As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a16z fund. Please note that a16z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z dot com forward slash disclosures.

So I think there's some things which are probably kind of easy to say, which is like, a, there's a good amount of disagreement, what is an agent? We've heard a lot of different definitions of it, on the both on the technical side, as well, I'd say on the marketing and sales side in in some cases, because there's some sales models associated with it. So so let's start with with the technical side. I think there's sort of continuum here, you know, the simplest thing that I've heard being called an agent is basically just a clever prompt on top of some kind of knowledge base or some kind of context.

That has this sort of a chat type interface. So from a user's perspective, this looks like an human agent would look like, right? So, for example, I ask it, Hey, I have a technical problem with my product XYZ. It looks at the knowledge base and comes back with a canned response, right?

But there it doesn't have to be a knowledge base, right?

It doesn't even have to be a knowledge base. I see, got it okay. So maybe it's just a trained model, it's on the model weights, the knowledge, so it's even simpler. So an agent could just be an LLM, but a chat interface or something like that, by some definition, right? I think on the other end of the spectrum, there are some people who basically say, for something to be a real agent, it has to be something fairly close to AGI, right?

Needs to persist over long periods of time, it needs to be able to learn, it needs to have a knowledge base, it needs to work independently on problems. If you take then the most extensive definition, is it fair to say that doesn't work yet?

I think so, it doesn't work yet, although

Will it ever work? We that's a philosophical don't know. All question. right, fair, very fair, very fair. So if we take that continuum in between, is there at least a way to to chop that into a couple of categories of sort of maybe degrees of agentic behavior?

And different types of agent. There's some artsy agent that help artists to come up with new, Bezier curves. There's coding agent, which we like to talk about as the agent of which we use, the day, yeah, I think, yeah, which we use. There's agent that's just a wrapper around of LLMs. that's right, yeah,

I may be the contrarian in this group. all right, Look, I kind of think agent is just a word for AI applications, right? Anything that uses AI kind of can be an agent. Now, before before we started this talk, I actually went online just to refresh myself about some of the more interesting AI agent perspectives out there. I found a really cool talk from Karpathy that he gave a couple of years ago about agents, which I can describe a little bit.

But the really funny part was on the YouTube recommended videos to watch next. It's like, AI agents are going to revolutionize your lifestyle and the rise of super intelligent AI, you know, it's just kind of like marketing. And so I actually do think that's what's going on in a lot of ways. The cleanest definition I've seen of an agent is just something that does complex planning and something that interacts with outside systems. The problem with that definition is all LLMs now do both of those things, right?

They have built-in planning in many cases, and they at least consume information, you know, at least from the internet, maybe from some servers that expose information through MCP or some other protocol. So the line really is very blurry. And you know, what was so interesting about the Karpathy talk is he basically he related to autonomous vehicles and said, AI agents are a real problem, but it's like a ten-year problem. It's like a decade problem that we need to work on.

And I think most of what we're seeing in the market now is like, is not the decade version of this problem. It's like the weekend demo version of this problem. And this is why we sort of generate so much confusion. You have this kind of poorly defined, nebulous thing that LLMs are kind of consuming themselves over time. And so I I don't think anything we have are actually agents is kind of and and agent itself may be a poorly defined and kind of overloaded term.

But if someone's willing to do the hard work and define exactly what it's like to kind of be a human, but in digital form and spend ten years to make it actually work, like, you know, that that's sort of what I'm excited to see.

Okay, so defining agent is a difficult job. Maybe it's easier to talk about how people use the tools they call agents, and what are the different degrees of agentic behavior?

I wonder if part of the conversation is redefining agent, because we all know that agent as a term is just not a great term. It means so many things to so many people. If it's interesting to dissect, like, what do we mean? What do different people mean when they say agents? What are different ways we could utilize this process? We call agents.

so it seems to me there's if we're trying to define agents, or maybe even degrees of agentic behavior, which might be a little easier. There's something like a user interface aspect to it, right, where something, it's a pure co-pilot, but basically user goes back and forth with a with an LLM to work on a particular task that's often not called an agent. Is that fair? There's a little bit the co-pilots versus agents UI models.

Yeah, I guess, like, what are the elements we would think that go goes into agentic behavior? Like Matt mentioned, planning could be one. There could be decisions made by the agent. There has to be LLM somewhere. But curious about your takes.

So I think, um, a another definition we heard from Anthropic recent recently was this idea that an agent is an LLM running in a loop with tool use, right? Which there's sort of two, two important parts of that. One is this notion that it's not just a single prompt, and not even just a single static sequence of prompts, right? But something where the LLM takes the output of a prompt, feeds it back into itself, and based on that, makes decisions on what the next prompt.

And likely also went to a board, so like you went to and went to complete a task. I think that for the re real agents or the more agentic behaviors, I think that's a reasonably good definition. And I think the other thing Well, just that you, oh by that definition, yeah,

isn't every chatbot effectively an agent then in this world, right? Like, if I go just to chatgpt. com and use their latest reasoning model with web search, right? Isn't it using tools? And like feeding, you know, feeding its outputs into a new prompt in order to do kind of chain of thought?

chain of thought is a little bit in between. If it's just a single prompt that comes back with a result, then it wouldn't have this notion of planning and doing a more long-term concept and deciding itself when it is complete, right? If you have a chain of thought reasoning where I'm giving it a more complex task, that's starting to look agentic. I agree, yeah,

I just think it's really tough to define a system based on what someone says to it, right? Because the these are by design unstructured inputs. These systems will accept literally anything. And so sure, if you tell it, you know, what's today's weather? I would agree that's not agentic, right? That's just fetching, you know, from an API. If you ask it, define a new philosophy of weather, right? It'll happily go do it, right. So so it's like an agent if you ask it one thing, but not an agent if you ask it another thing.

I think that's kind of the a lot of the confusion in the market around this. And and you know, if we spoke in the terms that you're talking about, Guido, of like, hey, this is an LLM in a loop with a tool like, that's actually a much more productive way to talk about it. I think

yeah, yeah, I mean, that that said it, it seems like we're seeing, to some degree, a specialization of user interfaces in sort of two directions, right? There's, let's say, the a a cursor or something like that, which really emphasizes the tight loop between the user, the tight feedback loop between the user and the LLM and the thing I'm working on, right? So I'm, I want immediate gratification when I do something, you know, and and sort of response time matters.

Then there's sort of more the backend, as you know, source code management system type plugins, where it's more about throwing something over the wall, right? Maybe answering a couple of questions. And then you try to maximize the amount of time the agent can work independently. So it seems like so, I think you're right, that there's no clean system, def like definition, split between the two, but there seems to be a little bit of an user interface specialization.

Is that a fair statement?

It almost feel like for all the use cases we've described, there's one element that all agents have, which is reasoning and decision. Would you call just a call to LLM to say translate this text to JSON? That's probably not the agent. But then if you ask LLM to say, Hey, decide where, you know, this response goes and route it for me, it feels more like an agent than before. So it almost feel like planning. I'm actually not sure does the agent need to plan or does it need to decide?

Maybe both. I actually feel like it's like multi-step LLM chain with a decision tree. A dynamic decision A dynamic tree? Yeah, yeah, decision tree. Yeah,

I think that's fair.

I think we've all just been nerd sniped. I just think, you know, it's like humanities people love classifying, and you know, they they draw kind of like fine distinctions between different types of things, entities, whatever. We're computer scientists, like, we're, you know, you know, not that there's anything wrong with humanities, but we're just not that. So I think we're not well equipped when it's a bit isn't just zero or one, it's maybe something in between, and we just talk about it a lot.

Until we like, try to, like, coerce it to one value or the other.

Of course, agents are more than pure technology, they're also becoming products, which means they need to be marketed. And how someone positions their product has a major effect on how they price it. What's more, the ultimate value of any given agent, which is still to be determined for the vast majority of them, is to what degree they can actually replace or simply augment human workers.

I think it's an interesting point, which is, I think there is a marketing angle to agents, right? I've heard this narrative from a couple of startups that they're basically saying, like, hey, you know, we can price the software that we're building much, much higher because this is an agent. So we can go to a company and say, you're replacing a human worker with this agent. The human worker makes, I don't know, 50,000 a year, and therefore this agent, you can get for only 30,000 a year.

This sounds really compelling from a first glance. And actually, I mean, there there's some value to it in the very early days, because it essentially it's very easy to understand comparative pricing for somebody who's to make a buying decision, right? You know, on the flip side, we all know that the cost of a product over time converges towards the marginal cost of production, right? And so today, if I used to use a translator, maybe to translate a page of text, today you use ChatGPT.

I do not pay pay ChatGPT like I paid my translator, I paid a tiny fraction of a cent, right, which is the via the API, which is the actual cost. So I sort of wonder how much of the agent debate is driven by marketing and pricing.

I just actually think this is a really interesting topic. What fields can you think of that are actually suffering complete replacement from AI or AI agent? And this is a setup, I'll warn you, I have another extreme point of view that I'll s say afterward. But can you think of fields where this is actually happening?

not completely, but definitely partially, cause there's a lot of, for example, voice agents that replace receptionists. I don't know if we shouldn't. They replace people who would, you know, get back to customers. So there's definitely a lot of workloads that have been offloaded from the folks who traditionally did the job. But I don't think they're, you know, 100 replaced. They can, you know, they can do something else. But we are seeing headcount growths in some areas are slowing, so it's not that existing jobs are being replaced, it's more like they're hiring net new humans slower.

I think it's exactly right. I mean, I think in few cases, humans will get replaced by AI. In most cases, you know, two humans will get replaced, one human, that is more by one human that's more productive with AI, should Or take over then.

yeah, or maybe they keep the two employees, and Or maybe they go and you know,

to three employees because now they're more productive, right? And so

right, it's just a really interesting question. And the reason I think it's really relevant to agents is, I think part of the ethos and part of the confusion around agents is this idea that we actually will develop human replacements, right? And that this thing we called an agent, which, by the way, is a name for a person, right? Before we had AI, we had people called agents, and we still have all kinds of people called agents.

And it just doesn't seem like that's happening, right? Not in the replacement sense, right? You you mentioned Yoko with agents, we've already always had, you know, customer support automation, you know, we've had 1-800 numbers where you like, press one for sales. Plus, you know, that's existed for a long time. This is a much better form of that, obviously. Translation is a great example, too, Guido. These systems can perform translation extremely well, but you're probably not gonna just stick something in a chat GPT and then publish it on your website, right?

There is actually work that needs to take place. And I think the reason for this is there's just fundamental creative work in most things that humans do, right. I I think from our kind of perch in Silicon Valley, we can forget that sometimes that people all over the country and doing all sorts of jobs actually have hard jobs. And not just hard in in the sense of someone's gotta do it jobs, but hard in the sense of it does take thinking and human decision making, which I I just don't know that AI kind of has what we would think of as decision making or intent, right?

It's a system that still somebody has to push the button, right? It may be running somewhere, it may do a great job of whatever, but someone still has to give it a prompt and hit go. And to me, that's a lot of the that's a lot of the confusion around agents. We're all thinking at some point, a human person with intent and creativity and thinking is gonna be replaced. I'm just not sure that even is theoretically possible, right?

It's almost just like a catch-22 to say an AI system is thinking for itself, right? Cause somebody has to have sort of created, you know, this is old sci-fi philosophy I'm getting into now. But like, I actually do think it's part a big reason for the confusion that, you know, we sort of experience now.

It's interesting cause there's two types of agent we're already talking about. There's one type where the agent is replacing humans, work with humans, do things humans can do. There's the other type of agent as more low-level system processes. They work with each other, they hand off tasks to each other. To some extent, agents are like technical details in the system in that way, but we mean both when we talk about agents.

in that case, is actually a difference between an agent and a function.

I think so, I think agent will be multiple functions with LLMs in the middle.

If I have a low-level agent and I'm giving this low-level agent a task and I get back a task result, it looks a little bit like a classic API call.

But was the LLM in the middle to make decisions on what to do for that API call?

So, but uh, I understood, but that's sort of how this function works internally, to some degree, right? So from the outside, would I care?

you wouldn't care. It's like most of the time when we see AISDRs, when we talk about AISDR agents, what we mean by that is when the agent can go to the CRM, pull something out, and then filter the list, draft an email and send the email. So that feels very process level instead of human level. Yeah, so that's what I meant.

If you don't know how this thing works internally, a classic function and agent become indistinguishable.

Totally, I absolutely agree. But when you as a programmer, when you find write a function, you will define agent that does the that's this certain I see, things, right? Implementation.

We'll get back to pricing shortly, but first, let's dive a little deeper into this discussion of how interacting with an agent is different than or similar to traditional software-based functions.

So so uh, so here's one interesting thing to think about on that topic. I totally agree with you, Guido, that and I think you sort of agreed, too, that there's the it's really a function if you kind of just look at it that way. Shareable, reproducible functions have never really been a thing. This has been one of these long time goals that people in the market have tried to say, Oh, I can just write a function and then anybody on earth can use it, right?

Like, you know, we have packages, right, that you can download a whole package with various functionality, but literally just one function that you can share. If you kind of squint a little bit, that kind of exists now with AI, right? Because you have these models that's trained by somebody, somebody else may download it, fine-tune it, train a LoRA, package it up into some new and interesting way. And then it's actually immediately available for someone else to use on hosting services or Hugging Face or something like that.

So while it does seem to be just an implementation detail, whether you're using an LLM or not, it there is this interesting thing where the model itself takes up so much of that functionality in the in in the function. And it's just a different kind of animal compared to normal code. It's actually more it's kind of shared by default in a way, cause nobody's going in and training their own model every time they're writing code. You know, it's obviously heavy, right?

It's harder to move around. There are all these different characteristics from normal functions that some of which are actually very desirable. Some are kind of, you know, bad, right? Characteristics you don't want, but many of them are kind of interesting. And I think we'll actually see new infrastructure, new dev tools kind of built around this in the long run.

I think it would make sense. I mean, when if we go back in time, the last time we sort of invented a major new component for building systems, which was probably networking, right? How we thought about calling a function before networking afterwards changed a lot, right? I mean, the essence like the the complexities of APIs and the infrastructure around it is completely different today.

This is such a good point, cause now I think about, I feel like humans are just functions too. Like, if you have a thought experiment and then replace LLMs in the program to a human, like how the kind of answers we'll give to the program is not that different from what a LLM will give to the program,

So so if we actually all get hooked up to servers one day and can be called as a function from Lambda, then I will agree that agents have been created. That's when an agent

Is isn't mechanical Turk exactly that, or Wait, but or maybe even your email inbox, there's you know? a Amazon, I mean, I would, there's a Amazon

sounds like an agent to me.

Go supermarket a while back in SoMa. I think they were advertising that it's computer vision models behind the scenes identifying what you took from the supermarket. But then people found that they hire a lot of people behind the scenes to actually label the data in real time. So the humans in that case are the functions that today, maybe

secret agents.

right, replaced by LLMs, was

Well, but this was exactly my point, though, right? There actually is important creative work, e even in a a grocery store checkout clerk, right? You could naively think, Oh, this is an easy job, actually, it's not an easy job at all, right? And so you can take this work and kind of shift it right, and you can squeeze it down with automation and stuff, but it never really goes away. Yeah,

All right, so, given all of this, how should companies think about pricing their agents per seat, per token, per task? Hint It might be too early to truly tell.

Usually, if you introduce a brand new product category, right, you often initially put a pricing the prices against the status quo, right? Whatever you replace, or augment in some cases. But let's assume we have a direct replacement, right? So that's, I think, where this idea from, Oh, this replaces a human, which it doesn't, and but if it would, right, then you could charge, you know, x amount for it, right? Usually, over time, competition kicks in, right, and you effectively price by how much your competitors are charging, right?

And you start sort of an an erosion. Then it depends on many things, like how much of moat do you have, do you have customer lock-in, right, and so on. Long term, converge against the marginalized marginal cost of production, right? Which, I mean, look, if I look at most agents today, it's probably very low, right? Any agent you can purely model in software with a couple of LLMs calls, you can run at a very low cost, and the cost is decreasing over time.

and I would sort of argue that's kind of already what's happening, that in practice, most AI applications, and if we, in particular, if we want to call them AI agent applications, you know, they have their sales pitch around. You should pay us X because we're saving you, you know, it's like a classic ROI calculation. Establish value. Yeah, Yeah, exact, yeah, exactly, value-based pricing, you know? But in practice, I think most buyers are actually pretty sophisticated about what's going on under the hood.

And to your point, they know it's like pretty simple stuff happening. And so it's like, Hey, what does it cost you to run all these GPUs? And we'll pay you some premium over that. And I think that's how a lot of vendors are pricing in practice these days.

I mean, y- long term, you'd expect pretty healthy margins, just like in SaaS, right, which software traditionally has very good margins.

It's so funny, cause we always advise companies to not price based on the margin, but price based on the value you add. Whatever that could be, it could be compared to other vendors in the market, it could be compared to just, you know, what it is building in-house. And traditionally, for infra, a rule of thumb, not always the case, is that if the service is used by a human, it's a per seat pricing, and if it's a service is used by other machines, it's a usage-based pricing.

And I actually don't know where to put agent here.

Well, it could be used by either, right? It could be used The agent could by either, use an agent, or a human could use an agent.

Look, I think your analysis exa is exactly right, and the reality is most AI companies don't know what value they're generating yet. This is so new and so nascent that it's like, Hey, we're just gonna charge something that we're not gonna lose money on. And, you know, in the case of OpenAI, they have how many millions of users? They probably don't have a very strong sense of what they're all using it for. And once they do, right, and you see this more, they're trying to verticalize a bit more and have kind of specific products for specific use cases, code obviously being the big one.

You know, then you'll be able to see the pricing kind of catch up is kind of my hypothesis.

This reminds me the OpenAI point you brought up. I was thinking about AI companions, cause that's the closest to per seat, per seat, human pricing. Like, you can't charge someone every sentence they talk to their companion, although some of the foundational models Ooh, they that there do

are there are services that will charge you per response. I haven't used them, but they do exist.

I see. Wow, okay. So usually it's kind of weird to charge someone, like by tokens of how much they talk to the companion, rather than like a flat monthly fee,

it doesn't feel like a true friend that you can charge for that Exactly, it's very right. transactional.

Th- this is look, this is all theory, right? People love sitting around and talking, Oh, we're gonna charge per person, per task, per, you know, world economy that we rescue, you know, it's like, it's all made up, right? Like, I think Guido's thing was exactly right. Let's look at the actual technology underlying what we're calling agents right now, where they're being deployed and why. And honestly, the pricing, the marketing, the sales tactic, all of this kind of follows from what they're actually selling.

If I'm selling something that looks like an agent, but I haven't truly figured out the value I'm providing to my users, how do I justify the jump to a higher price point when I do figure out that value?

You just need to be selling a solution rather than a product, right? This is really well-worn expertise in enterprise go-to-market code. You can somewhat see the decoupling of price from the underlying technology now, because it really works. There's very clear ROI to people who use it. And so as a VP of engineering or a CTO, you can look at this and say, Okay, I'm actually saving a lot of money and my guys are getting a lot more productive.

I can value, I can do a normal

And they're happier.

Yeah, so you're kind of buying a solution, right? You're buying from a vendor something that solves a problem for you, which, again, Microsoft, Oracle, Salesforce people have been doing forever. Once we start to see more of that, it's gonna be these things that become real products and kind of decouple pricing and look kind of like real businesses. I think,

I think it's dictated by the high level application. So I'll give you an example. So I'm a Pokémon Go player, so for those who have played Pokémon Go, once you collect enough Pokémons, you are out of storage in your pocket. So you need to pay extra to buy a new bag, virtual bag, that you can put more Pokémon in. And as the infrastructure investor, I invest in storage businesses. And then when I look at how much I need to pay for, like, 30 extra Pokémon, it was thousands of times more expensive than what storage is.

So it actually reminded

I'm surprised it's only thousands. you It's been getting, know, like I would've, price hike, I guess, ten to the fifteen, there's a or so, right?

whole price curve on Pokémon storage, it turns out. But

cause this is like one JSON blob, right, it's basically, that you're talking

one JSON blob. about. Yeah, I know, yeah,

and they charge you like five yeah. And dollars.

then the Pokémon normal Pokémon players, they wouldn't think about this, like, how much do storage costs, right? Like a normal Pokémon player would be like, Oh, this capability, I would be happily paying thousands of more than I were to have a S3 bucket somewhere. So one of it is monopoly, so it's application layer monopoly that you wouldn't have been able to store the Pokémon anywhere else. And two, it's the use cases. It's for a different audience.

They wouldn't be asking these questions. They would be thinking about, what is the net new value, what's the net new cost? I will be willing to, you know, foot the bill for if I were to get this value. Is it a fun game? It's a fun game. Take hundred more dollars.

Yeah, I th- I think that's exactly right. And and implicit is what you're saying is this idea that the product or the solution has to actually work for them, right? For a less technical person who's, you know, the person who's not gonna try to provision their own storage bucket to self-host You can have their Pokemon.

a brain on a string for Pokémon, yeah,

And it's quite defensible and differentiated, too, because, you know, you know, Pokémon Go is not open source. There's no other replacement of Pokémon Go, there's only one Pokémon Go. So there's only one place where you would be willing to pay so much money for Pokémon storage.

plus very strong brand, plus you have a little bit of network effect because you can play together, yeah, and then you know,

we'll see the agent AI agent version of this. I can't wait to see the AI companion version of this. Paying storage for AI companions wardrobe.

As the AI market continues to shake out and evolve, where will agent capabilities ultimately live? For example, can they live inside LLMs, or must they call external tools? And who's ultimately in the best position to influence this?

super interesting question, right? What's the system's perspective of how an agent is built? And I I personally think that architecturally, there really is no difference between your typical SaaS software today and agent in terms of how you build it, right? And let me explain why, right? So an agent, we said, you have sort of an overall loop with an LLM and prompts that feeds into itself, plus external tool use. The LLM itself, you probably want to run a separate infrastructure just because it's highly specialized.

You need these vast GPU farms, uh, can't easily run today's LLMs on a single GPU, so that's that's a very specialized infrastructure that's externally. So the LLM call is external, the state management well, today in SaaS applications, we do all the state management externally in databases or something like that. So you probably also want to externalize that, right? And then what remains is fairly lightweight logic, right, where I'm basically I'm taking context that I retrieve somehow from databases.

I assemble that into a prompt, I run the prompt, and then I occasionally invoke tools. Maybe I do that with MCP or something like that on an external server, but the core loop is actually pretty lightweight, right? And I can run a gazillion agents on a single server, not a gazillion, but many agents on a single server. Uh, I don't need a lot of compute performance for that. Does that sound about right?

Yeah, yeah, I totally agree. The interesting architectural question for me, uh, has always been, how do you handle the kind of nondeterminism that may come? Many of these successful AI applications that we all use and love really just spit model outputs back out to the user, right? Like a chatbot or image generator. It's like, Hey, I called the LLM, here's what I got, you know, good luck. When you try to actually incorporate the output from an LLM into the control flow of your program, that is actually a very hard, very unsolved problem.

That, you know, to your point, they're relatively minor architectural differences today, but this may actually drive more significant changes in the future.

I actually think the winners will be the specialists, not the foundational models. It's the people who will build on top of the foundational models or fine-tune the foundational models. So, like, a very artistic example of this is that I've been spending the last two weeks just prompting GPT for other image model. It's very good at cartooning, so it's very good at manga, it can spell, so it has the storyline. But then I realized that there's only top two or three style it's good at, so it's good at Ghibli, it's good at manga, and then there's variations of a style in that realm.

So now where art comes in is that the market likes out-of-distribution art. Everyone doesn't want to see the same things over and over again, cause that's how they value art, something that's different. Yeah, I I Ideally, maybe, Ideally, maybe,

so, d. Did somebody recently define art as out of distribution samples? yeah, I That's true, too. Yeah, agree.

yeah. Art can be in distribution, that's pop art, right? It could also be out of distribution. That's like when impressionism came up, many years ago, everyone was drawing impressionism, and at the time, the painters before, they were like, what's wrong with your eyes? Why are you drawing blurry images? So styles come and go, but because of that, I think it's a pushing distribution question. How the foundational model will never cover 100 of everything. So it's really up to the humans and specialists of the next wave to come up with the new data, new workflows, new aesthetics to push that distribution.

Of course, at the end of the day, agents are only as useful as the tools and data to which they have access. So what happens if major web platforms decide they want to keep agents from accessing their data?

it seems like one of the hardest things about agents today are data modes, right? In some cases, just because they're technically difficult, I'm trying to access data, and agents trying to access data, and it's just very hard to integrate with that system. In some cases, it's very deliberate, right? My iPhone, the photos are not accessible via any API because it's a walled garden,

So sort of data silos you're data silos, talking about. Yeah, yeah,

right? The so, is that something that's holding back agents, or is making them more difficult, or m to make it even stronger? Consumer companies traditionally often were opposed to offering automated access to their services because they want the user engagement, they want the time to advertise to the user. Will that limit how much we can deploy agents?

And would that be changed once we have the browser-native agents that can browse the web and browse other Great question. Yes, forms?

I think that I think Yoko is totally right, you know? It's like there's strong incentives for people who own data about, you know, physical entities, you know, people, businesses, et cetera, to keep to themselves, right? It's especially cause they may be scared what AI is gonna do to them, by the way, so they're kinda clinging tight to what they have. And these problems are rarely solved by defining a new protocol and just saying, Hey, if we make it easy for people to give away their core assets, they'll just do it.

You know, obviously, you know, that's very unlikely to work, but someone eventually will solve this by saying, Hey, if your data is publicly visible, we're gonna get it, you know? It's like, by the way, it's not actually your data, it's about me, so why should you be holding on to it?

Yeah, actually, I feel like the new advancement in models may just change the data mode, kinda to the point of today. Web browsing using an agent doesn't work super well. It's very slow, it's very clunky, you have to try multiple times for it to do any task. But imagine if we have foundational model capability of giving an agent ability to go to any website, logging as a human. We'll table that one. I don't know how agent identity works yet.

Or go, you know, go SSH into a server, like execute certain commands, or like spin up a virtual machine for a mobile, or access a device far devising of device farm to play Pokémon Go. Like, maybe those are the the data traditionally only available to humans under that account now may be available to agents.

There's also the opposite that could happen, right? That basically all the consumer sites are starting with more and more complex anti-agent captures, trying to keep out the agents because they only want the humans that have attention to come to those sites. I mean, I recently did use one of these deep research tools, one of the major LLMs, and one of the steps, if you look through it, all the steps it went through was like, you know, trying to see how I can get get around a capture mechanism for a site.

There was an actual reasoning step, right, where basically it found it, know what information it wanted, and it was blocked from accessing it. So is that that, you know, how dystopian

Did it? is the future Did it solve it?

gonna be here? It solved it, actually. Yes,

I mean, it's so interesting. So here's a really early machine learning example of this. I don't know if you guys remember when Gmail first implemented ads. It was a big controversy because they basically said, Okay, we're not going to read your emails, but our algorithms are going to read your emails, and we're gonna suggest ads that you should watch. Ba- you know, or click on based on that. We all sort of, I think, just forgot and got used to it.

I still think we don't love the idea, but we kind of have lived with it. But some of the data providers reacted by removing data from email, right? So Amazon, famously, now, when you order something, they send you a confirmation email that says, Hey, you just ordered something. Click here to find out what you ordered, when it's going to arrive, or any information you might want to know. And so that actually did happen in practice, in that example, that the major data holders kind of found ways to withhold it.

It'll be interesting to see whether that's possible now or not.

but that same data is scraped on the client side from the ad networks I install by the time you ask.

that, yeah, there's always some other Yeah, some, way. Yeah, it's not maybe exactly the same, but pretty good proxy. Yeah, yeah,

yeah, that it may be that it it's much harder to tell the difference between an LLM and a human than a classic, you know, so the API call maybe. mechanism and a human, that may be, that may change the dynamics.

Finally, Guido, Matt, and Yoko answer an obvious question on the longest timeline into which we might have clear visibility. What needs to happen to make agents a truly game-changing innovation within the next, say, two years?

I think the positive vision is that in two years, we figured out how an agent working on my behalf can use most of the tools that I have access to. I think it's also clear what are all the pieces that are missing for that, right? We have not figured out security, authentication, access control for agents working on my behalf yet, right? We have not figured out how data retention works, you know, we have not figured out the relationship with consumer websites that potentially want to block that agent.

But if you had that, it could make many tasks much, much easier, right? Today, if I have data sitting, say, my Google Drive or so, right? How easy I can reason about that data versus other data that's in more fragmented sources, it makes an incredible difference, right? So the I think that that's the bull case, right? Where you have agents that can take all the data that you can access, they can access it on your behalf and perform tasks on your behalf, right, and save you a ton of time, right?

It could make you, depending what you do, like, you know, uh, multiple times as productive as you are today.

My answer to that is actually different modalities on the foundational model. Today is still very much text-based, and that worked really well for coding and text-based tasks. But then for more visual-first tasks, there's just no one-to-one mapping. Even for web browsing, it's like a very clunky experience of take screenshot every couple seconds and send it back to a foundational model. So I will actually bet on multimodality when it comes to if we train the model with different traces of clicking on buttons on the website, navigating the web, using different devices, drawing, producing vector art.

I think there will be net new things that the model could unlock on the agent level.

You can probably guess my answer. If we don't use the word agent two years from now or five years from now, I think that that's a huge win.

I think.

There there's actually a fun paper put out by some folks at Columbia, I think, called AI is normal technology. And they sort of make the argument that there's a false dichotomy out there. It's like, AI is either gonna bring about Utopia or dystopia, meaning everything's gonna be amazing because we have AI, or everything's gonna be terrible. This is kind of the national discourse. But if you just think of it as normal, right? Like water or electricity or the internet or things like that, I think that's the world we're kind of headed towards.

An agent is this kind of way to help us get there. And so that that's my goal. I mean, this stuff is just incredibly powerful, and we understand how to use it, we understand the use cases, and we're kind of, you know, we're kind of putting it to use for us.

Thanks for listening to the a16z podcast. If you enjoyed the episode, let us know by leaving a review at rate this podcast dot com slash a16z. We've got more great conversations coming your way. See you next time.