The good, the bad, and the future of AI agents

As soon as autumn rolls around and you swap your iced lattes for pumpkin spice, that's your signal to start thinking about the holidays, the decor, the gifts, the shopping lists, but more importantly, the holiday treats. Cheesecakes, cookies. Is there anything better? And when the recipe calls for cream cheese, there's only one option Philadelphia, the original cream cheese, made with fresh milk and real cream. This holiday season, whether you're going all out or keeping it simple, make it really Philly good.

Find your favorite holiday recipe at creamcheese dot com.

Running a business shouldn't feel like surviving a software group project. One app for accounting, another for inventory, another for sales, and somehow none of them talk to each other. That's where Odoo comes in an all in one business management software that brings every part of your business together. From sales and accounting to inventory and marketing, all in one powerful platform. No messy integrations, no bouncing between tabs, and best of all, no spreadsheets. Stop managing software and start managing your business with one unified system.

Try for free today at odoo dot com slash vox. That's o d o o dot com slash vox.

Support for the show comes from CrowdStrike. It's not a huge stretch to say that AI is the next major computing platform. Every platform shift changes the way we build software, and it changes security, too. CrowdStrike is defining cybersecurity in the AI era with AI detection and response, or A I D R, a solution for seeing, monitoring and securing AI across the enterprise. Learn more about how leading companies are turning to CrowdStrike to secure AI and secure their business at CrowdStrike dot com.

Decoder

Hey there, and welcome to Decoder. I'm Hayden Field, senior AI reporter at The Verge and your Thursday episode guest host. I'll be subbing in for Nilay for a couple more episodes, and I'm excited to keep diving into the good, the bad and the questionable in the AI industry. Today, I'm talking with David Hershey, who leads the Applied AI team at Anthropic. David works with startups to help them figure out how to best apply Anthropic's tech, plus he tests new AI models to understand their limits.

I wanted to have David on because earlier this week, Anthropic released a brand new AI model called Claude Sonnet 4.5 that's been making waves. Her reference Claude is to Anthropic what ChatGPT is to OpenAI. This new model, Sonnet 4.5, is being billed as a big breakthrough in autonomous agentic AI, especially for coding purposes, which is a big battleground in the AI market right now. All these companies want to get a slice. These types of AI products can, in theory, be given complex tasks and then go off and complete them over the course of many hours, or in some cases, even multiple days.

And Anthropic says this particular model, Sonnet 4.5, can run for up to thirty hours straight without any human intervention, all while working on a singular task, like building a software application from scratch. For the last year or so, companies like Anthropic, Microsoft, OpenAI and others have been promising that this agentic technology would be the next phase of AI, the next big hype filled thing that comes after general purpose chatbots. They say it could really unlock generative AI's potential, and it's true they've made some strides.

But as we've seen so far, agents aren't quite there yet, and they have a ways to go. Most of us are not, in fact, sending agents off on the internet to do our bidding. And we're certainly not giving them tasks that might take twelve or twenty, four, or even thirty plus hours of autonomous work without human handholding, at least not yet. At the same time, many companies are looking at agents as the breakthrough that's supposed to unlock huge productivity gains from AI models, including the opportunity to use them to replace or augment human labor.

So I wanted to sit down with David, who spends a lot of time testing out what models like Claude Sonnet 4.5 can and can't do, to ask him where we are on this promise of AI agents. I wanted to talk about what these types of products are good at from a consumer standpoint, beyond just programming purposes, and also what the path forward looks like as AI agents progress. Okay, here's Anthropic's David Hershey on the state of AI agents.

Here we go. I wanted to ask you about your view of the current state of play for AI agents. Like we hear all the time that agents are the next big thing for generative AI, but are we still in the prototype stage, the testing phase, or what? How would you characterize what AI agents do today, right now, relative to what AI companies actually want to offer in the end? Which I hear from a lot of execs is basically Jarvis from the Marvel movies.

I I am less confident in our ability to predict the end, but I'm happy to talk about it now. I I've seen agents come a long ways in the last year working with customers, and my view is there are places where we're starting to see what it looks like when agents work really well. And there are still a lot of places where they don't work really well. And I think that kinda makes it confusing. I think for some people, code is a great example.

When you're writing code, and especially with Sonnet 4.5, when you watch the model spend a lot of time as an agent developing software itself, it's incredible. It can do a ton. It's like gotten much better literally this week as we release models, it's really visible and obvious how much better they're getting. Sort of if you're plugged into that at being an agent doing really long running, complex tasks. When you look at other sections of the economy or other like jobs that people want agents to do large pieces of.

Sometimes there's stuff they're still not good at. In some cases, they're not good enough at deciphering what's on a computer screen just yet to be able to navigate complex UIs or whatever it is. And so they fall over and stumble over themselves on on something sort of silly, and it's easy to point at that and say, what are agents all about? Like, is this kind of fluff or hype or whatever it is? And I I think the way that I see this sort of generally happening is we've, like, slowly been ironing out the kinks of the stuff that models fall over themselves on.

And we've done that the best so far in coding, I think the industry has done that the best so far in coding. Where we're making really fast progress on how much an agent can accomplish when writing code, and we are, I think, starting to make that progress in a lot of other domains. For example, like I talked about clicking on UIs in a computer, this model, Sonnet 4.5, is way better at that. I don't know if it's like exactly the point where we're gonna tip into people like trusting it to automate a whole bunch of stuff they do when they're clicking around a browser yet, but we're getting there.

And so I think my general view is we're going, we're making really fast progress. It's just not necessarily visible in every part of the economy yet, and in every job and every person and every individual. I have a feeling sort of each model that comes out will get one bit closer to being something that everybody can sort of interact with and see. So it's great at developing software, it's great at coding. It kind of reminds me of, uh, the robot hand moment, you know how robots can do things that are really, really complex and hard for humans, but actually grasping something has always been a real headache.

What about consumer facing stuff? You talked a little bit about UIs, but you know, what do you think agents right now are the absolute worst at? Like, what's the simplest thing that they truly just cannot do?

I honestly sometimes struggle to put my finger on this because I think it's it's surprising little, funky things in a lot of different agents. So, for example, if you're trying to do something related to finance, maybe there's like a bit of manipulating a spreadsheet that is really hard and it falls over. And so it's like ninety nine of the stuff it can do. It can do the math and it kind of understands how a finance model works, but it will stumble over a spreadsheet.

This is the thing you talked about. I don't know if there's like, I I think it's hard to boil down all of the jobs that we want to help people do into one tiny little thing that the models aren't there. If it was that easy, I guess we would probably be on top of it already. And I think like, maybe a better model is for each of these different things that we want models to help us with.

There's like a million little components it breaks up into. You need to be able to, like, see the right cell on a spreadsheet and know how a formula works and know the macroeconomic model. And thing by thing, you can, like, find any different task with a little like thing that's not quite there yet. And so as we think about, like coming from Anthropic, when we think about it, it's like we just have to sort of be able to iron out, like, fix each one of the little gaps in each of these to to help everybody use it.

But I honestly, like, this is probably an unsatisfying answer, but I can't quite put my finger on, like, there's just like this one thing. I think this is why this field is hard, where we're just sort of like, constantly working on the whole universe of the stuff people do when they're on their computer and trying to help them out. And that just means it's a really wide scope, and there's just a lot. It's like, from my side, it's a lot, it's fun.

There's a lot of hills to climb to try to make the models better at all of the things that we wish they were good at. So we're working on all of them at once.

Being someone that works with a lot of different startups in a lot of different industries on how they're actually applying AI. What are the industries that have surprised you the most? Like, what are the you know, what sectors are clients in that you just wouldn't really expect? Or maybe the ones that you've seen the most growth in in the past six months, the past year? You know, what are the trends you're seeing in terms of which sectors are really adopting agents at scale?

I have two versions of this, but I'll I'll give you the fun one. First, I think like one of the domains that surprised me the most in my personal customer work is the legal domain, which is a at some value, at face value, it it's apparent why that can be really useful. There's this all this information you need to know about case law and studies, and and it's just fusing information is something that's like, pretty obvious models are good at.

But actually, there's so much depth and complexity to the legal field, which I didn't appreciate. I'm not a lawyer. I didn't appreciate it until I started working with people in the legal domain.

I used to write a lot about legal AI and how the law sector was kind of the last to adopt a lot of AI tools because they were super old fashioned. I did a couple trend pieces on that, and then when they started to adopt it, I was really surprised at how quickly it came.

Yeah, yeah, that's exactly sort of like what caught me off guard was you would think like there. And and honestly, there's so many things that make it hard writing a really good legal system. You typically need a lawyer to tell you if it's good or not. And and like having to have lawyers in the feedback loop of how to build is challenging. And so I've been really impressed with a lot of the companies I've worked with and their ability to sort of, like, work around that.

Where they like, have this interesting build of companies that have like lawyers on staff to help with product building, like, like, like big amounts of lawyers on staff to help them build products. They build agents, a cool things to, like, be able to, like, comb over case law and look for the right details and and pieces. And then obviously use the stuff that AI is obviously good at of synthesizing answers at the end. But how quickly that field I I think it's just like a it's probably like the the scale of the upside there of how much just the pure volume of work that needs to be done and how much that it can help.

That has driven the speed that people go, but that domain has has surprised me. I promise you a two part answer. My my second part of this answer is, I think the funny thing about this space is it's really hard to guess where the next agent is going to take off. Because of that thing we talked about before, or I talked about before, where it's like, sometimes there's just this like one little tiny thing that's not super obvious to to any of us that's blocking an agent from working.

So, like, if you wanted a model to do your taxes for you, I think that'd be very nice. I don't look forward to doing my taxes every year. And you just find out that, like, actually, what it's really bad at is you upload your W four and it can't quite see the difference between the two boxes on your W four. And that's why the whole thing doesn't work. You know, this is a toy example. I don't think that's actually where models are, but

Yeah, that would be a a pretty complex. I remember when I worked in two states, or I had a job in one state and I lived in another. It was also really hard for me to figure that out, so not surprising that AI can't either.

yeah, it is complicated, but like, I I think we can get there, like, that's clearly it feels in the domain of what the models can do. I've seen them do much more complicat- I don't know. They're better than me at math by a decent amount, so I would s expect them to be able to do this. But sometimes it's just like this little one little thing. And so I'm kind of, like, constantly surprised. And and mostly it's around when we release new models, like it turns out there's some set of things that the model can do now.

And then you see new agents crop up that we're doing things we couldn't do before. And so there's like, like, nice little micro surprises that come out. I don't spend a lot of time thinking about accounting normally in my day job, but then you run into an accounting startup that can suddenly do something new and interesting, and that's pretty cool.

Do you think that data annotation is gonna be a big part of that? Do these models get tripped up because there's some data that they don't have enough of, especially with a niche industry? You know, is that kind of where you see some of the obstacles coming into play?

I think we need to learn from specialists, and we need, like, great ways to learn from specialists. And I don't know if it's necessarily data in the classical sense, that there's some like pile of data that we're gonna learn from. That makes it better, or there are just better ways. We need to incorporate the intelligence of accountants and lawyers and other people into our models. And yeah, I think I think that could come from data, some of it certainly will.

I think that can come from, like, talking to and interfacing and working with people in those domains. I think a lot about learning directly from our customers. I I think there's a future where more companies can contribute more directly to making the models do the stuff that they care about. It would be really nice if someone saw this big important thing that they'd like to achieve. Instead of having to sit around and wait for a lab to hopefully build a model that helps them do it, they can sort of, like, directly work with a lab.

And I think that's probably somewhere that, well, I I know, that's probably somewhere that we'll, we'll head, where we can have more direct mechanisms of of working with sort of experts. And and yeah, it's partially a data thing, but I also think it's just like. I don't think it's surprising that one of the things that models are great at today is software engineering. When the building that I'm in in Anthropic headquarters is filled with software engineers who know how to make models great at software engineering, cause they write software all day, you know?

Yep, that tracks.

And yeah, exactly like it's. It's deeply unsurprising that that's true. And I think as we grow up as a company that's only a few years old and has most spent most of our time hiring software engineers. And figure out how to consult and work with more people in more diverse places and doing more diverse jobs, we'll get better at building models that are great at all of the other things that people wish models were great at.

We need to take a quick break. We'll be right back.

Support for the show comes from Vanta. As AI adoption grows, so do your company's security risks and requirements. Most compliance tools promise automation, but you're really stuck doing all the work by hand, chasing screenshots, validating controls, and playing catch up when audit season comes around again. Vanta works differently. Their agentic trust platform is built to scale with you, not slow you down. With over 1,400 automated tests across more than 400 integrations, Vanta collects evidence and monitors your controls year round.

Plus, you can access the Vanta agent everywhere you work, whether your team is in Vanta every day or living in tools like Claude or Cursor. They work around the clock from full program context, so nothing slips through the cracks. The numbers back it up. Vanta delivers 526% ROI over three years and pays for itself in three months. Over 16,000 fast-growing companies like Ramp, Writer and Harvey save time and resources while scaling fast with Vanta.

Learn more at Vanta dot com slash decoder, spelled V A N T A dot com slash decoder. Support for the show comes from Rippling. You just found out your sales team is at risk of missing quota. Don't panic, just ask Rippling AI. Since it's built on your real-time people and business data, Rippling AI can pull metrics from Rippling and Salesforce into a meeting-ready dashboard, which shows quota attainment, headcount plan, and monthly revenue to quota by region.

In seconds, you'll see exactly what's behind your quota risk and fix it before it's missed. Question answered, action taken, crisis averted. When you have critical business questions that need answers, don't just file a ticket and wait weeks for an outdated report. Describe what you need and have Rippling AI build it instantly from your live people and business data. Whether it's a dashboard with detailed charts or automated workflows with the right triggers, conditions and approvals, ready to rule your business, head to rippling dot ai slash decoder.

Get the only AI built to give you full visibility and take complex actions across your entire organization. That's R I P P L I N G dot AI slash decoder. Sign up for exclusive access today. Support for the show comes from granola. Have you ever left a meeting thinking that was productive? And then 10 min later, when it's time to take next steps, you're at a loss. Or worse, you've been heads down the entire meeting, furiously typing notes instead of actually listening, and now your brain is fried.

Meetings are a mess. Granola fixes that. Granola is an AI powered notepad that makes your meeting notes useful. It capture what happens in your meetings and turns it into clean, structured notes with decisions and action items pulled out and made easy to find. And the best part, granola works through your device's audio, which means it integrates seamlessly into the video conferencing tools you already use. No setup, no awkward bots, it's just your normal meeting with superpowers.

You get to actually listen instead of frantically typing every word, and still walk away knowing exactly what was decided, who's doing what, and what comes next. Once you try it on your first meeting, it's hard to go without. You could try it totally free. Just head to granola dot ai slash decoder. That's granola dot ai slash decoder to get your time back. Go to granola dot ai slash decoder and try it on your next call.

We're back with Anthropic's David Hershey discussing the landscape for AI agents. Before the break, you heard David explain some of the trends he's seeing, both from his work internally at Anthropic, testing new models, and from clients who are now using this tech in their industries. But now I want to ask David about Anthropic's big announcement this week, Claude Sonnet 4.5, and why it's being billed as such a step forward for AI agents. So it's great at developing software, it's great at coding.

It kind of reminds me of, uh, the robot hand moment, you know how robots can do things that are really, really complex and hard for humans, but actually grasping something has always been a real headache. What about consumer facing stuff? You talked a little bit about UIs, but you know, what do you think agents right now are the absolute worst at? Like, what's the simplest thing that they truly just cannot do?

Yeah, I'm very excited about Sonnet 4.5 too, and I'm also very cognizant that when my mom texts me asking about it, it just is a a model with a decimal and a number after it. Until I talk to her too, so there are a lot of things that I think are exciting. One one thing that I wanna say up front before I get into a lot of details that, like, have jumped off the page to me about the model is.

It's sometimes hard to predict, like, what the big impact is gonna be on everyone. These models get generally smarter, they get generally more capable. And not to keep harping on the same point, but sometimes we just like, don't know the blind spots that they had until we get over them. And so when we released a model last year, Sonnet 3.5, and suddenly all of these vibe coding startups happened. I don't think we actually, like, knew that that was true, we didn't know that there was going to be, like, the moment we released that model.

I don't think we could have predicted that so many companies would crop up and be able to help people start writing code with agents in this amazing way that we've seen happen. So part of what is exciting about Sonnet 4.5 is the unknown to me. I'm really confident this model is the smartest model we've ever created. I'm really, I've seen, and I'll get into some of the things I've seen it do that I've never seen another model do before, and that's really cool.

And part of it's concrete stuff we'll talk about. I I'll tell you I'm excited to talk about some of the software engineering stuff I've seen it do. But some of it's stuff that we really don't know until our customers and the people we work with try to build cool new things that they couldn't make work before, and then they suddenly make it work before. And so when I, if I had to guess how my mom might see Sonnet 4.5, and it might make a difference to her, is that there's some product that she couldn't use before or didn't exist before.

In the way that perplexities happened in the past for consumers or or cursor exists now for developers, that it's going to happen, that's gonna impact your life. Because this model is capable of a new thing that we didn't know before. And I have a hard time predicting it. It's part of the the fun part and the hard part about being someone who works with customers now is it's really hard to predict this stuff. The day we launch a model, but it tends to happen and that's cool.

Have you seen Yeah,

trends start? Even though it's been like one day, anyone that's starting to use it, you know, in a new way, one of your clients, or even in beta testing.

I too early for customers, I think, to to see anything brand new. It typically is like, uh, it's a very fast field, but I I have, like a give it a month rule to find out what the new companies are gonna be. My testing, I have certainly seen some new stuff a and with the testing some of the team has done that is really exciting. One of my favorites to talk about is my team has been working on, seeing how much of a software engineering task a model can take on at a time.

And so I think we're all aware that models can be used to write code like that. That's sort of obvious to a lot of people, at least in this industry, and and probably people listening to this podcast at this point. But it's often in pair with a human really directly, like going back and forth using Claude Code or an IDE, like write one thing at a time, check and review it. And the longer you let a model go trying to implement something big and complex, the more likely it is to, like, do a whole bunch of stuff you didn't want to happen.

And so we were, like, really curious, like, what is the most we could stretch that? Like, if I give Claude a really huge task, like, what can I accomplish? And one thing we've seen with this model is in a way that is sort of, like, really not true of models I've tested and seen in the past. There doesn't seem to I if you give like a sufficiently good overview of you what you want a model to accomplish.

I haven't really seen a ceiling on how much it can keep, like, coast to consistently working on making something better. And so my favorite actual example of this. We released a video of this yesterday, but I I like, I'm going to talk about this until people get bored with me. Is we asked Claude to recreate Claude.ai, like our our consumer chat application from scratch. And Sonnet 4.5 just like worked overnight, it we woke up and it just did it like this beautiful clone of of Claude.ai that works incredibly well.

And and my favorite moment from it, there's a feature that people like of ours called artifacts, where when you ask Claude to make a document or make a web page, it will make it and render it next to the chat. So you can, like, play with an app that you built live or whatever it is. And I was saying to the person on my team who was working on this demo, like, work on this thing. It'd be really like cool if Claude could, like, build artifacts itself, that would be amazing.

It's like a complex feature, it, it's pretty hard to figure out. It's like, we should, we should try and see if that happens. And then he messaged me, like, two hours later, and he didn't do anything, he didn't intervene. And he's just like, Hey, Claude just built artifacts on its own, and it's like, currently testing it live, like, it's just like trying it out itself. And I really, it just like we released artifacts a year ago.

It was like artisanally created by a whole bunch of like wonderful engineers around me at Anthropic who were doing all of the software engineering to build this really complicated thing themselves. And then, like, we, a year later, we released a model and we asked it to build claude.ai and like, let it go for twelve hours and just did it itself. You know, it's like, instead of this giant team of people at Anthropic that worked so hard on this thing, the model is just like, really capable of of biting off really meaty, complicated tasks.

Like this is something that would take me months to do if I did not have Claude. And just overnight, we we sort of looked at it and watched it happen. And that progress of just like, like, going from a point where it was neat that a model could write a snippet of code for me a year ago to like, oh, it, it can do like a big chunk of the stuff of the complex developer work that I need to do.

It's just like, so, like, I don't know, it blows my mind. It was like a pretty big wow.

So that took about twelve hours, you said,

That specific one was like a twelve hour thing. We've seen like up to thirty hours of continuous dev work until the model would start to start to struggle.

This is what I was gonna ask you about, because something that unexpectedly went viral from my own article about Sonnet 4.5 was the thirty hour bit. The fact yeah, that Sonnet 4.5 could code autonomously for up to thirty hours with no interruption. So I heard one engineer at Anthropic used it to code a chat app that the company compared when they talked to me to Slack or Teams. Obviously, it was only 11,000 lines of code, so much smaller than Slack or Teams.

But and it seemed like just an example project. But people online are really excited about that detail and calling for Anthropic to release it and they wanna know more about it. So can you give me any details? Was that someone on your team that was testing it, and how m how impressive was it really, or was it just kind of rudimentary? Give us the deets.

Yeah, yeah, yeah, this was the thing that also, I don't know, like all of my excitement. And the reason I'm here is cause I've been working on this exact thing. So it w it was someone on my team. His name's Justin. Shout out Justin, you should give praise to him, he's amazing. Recently joined the team on the side note, he's pretty cool. This was born out of this thing of a lot of people have built demos or proofs of concept.

And there's this sort of vibe coding trope that you can, yeah, you can build it to use it to quickly mock something up, but can it really, like, build a real application? Justin really wanted to, like, test that, and so he was experimenting using our Claude Agents SDK, which is just sort of like, uh, more programmatic version of Claude Code. To some extent, he was testing, like, can I give a full spec of like a complex thing, like an app similar to Slack to the model and and just like, watch it build?

And we did some tinkering and experimentation around it to get it right. But yeah, you asked, like, was it impressive? It's impressive. It has like DMs and threads and channels and a slick search functionality, and you can upload images and gifs and render them. And like multi-user authentication and Claude. We didn't ask it to, but implemented a whole bunch of like AI users for testing. So if you like, log in, you can send a message and there's like, Alice, the PM is in there that you can send a message to, who will respond you back stuff about PM work.

It is it is remarkable. It it is like it is not by any means slack. But like, if you didn't spend a lot of time thinking about it, you'd like look at it and think that was a a pretty reasonable productivity app that you would use to chat with your coworkers.

wow, what, what Wow, did you guys name it?

I don't think we have a name. I need to ask Justin to give it a name. We need to. I, I actually know I lied. He has a work chat, very boring. Yeah, we're engineers, we're clearly not the product folks in the org. So,

since, though, I love I love that. that. Yeah, someone who worked Yeah, at Slack, I think, messaged me and said, You know, our code base is a thousand times that. And I was like, Yeah, it's just an example. But I mean, it happened we need in thirty to take a quick hours, so it's a big deal. break. We'll be right I was also back. gonna ask you what surprised you the most during your own testing of it of Sonnet 4.5?

I, I'm just going to continue double clicking on this thing. The way that it built the app was surprising and really interesting. I'm not surprised that the model's getting better at building complex apps. The thing that was really interesting for the way it does this is it just has a tendency to, like, bite off little pieces that it can handle really well and do that continuously. So a lot of times, like models in the past would get really eager and ambitious.

They would say, like, I'm gonna build this whole thing and I have, like, these grand ambitions. And it would kinda just like meander everywhere, trying to do this miraculous piece of work. And the thing that the the was really cool about Sonnet 4.5 is it's just like pragmatic, kind of like it, uh, it's like, okay, right now I'm going to test like, does image upload work? And then it's gonna do that, it's gonna spend a little while doing that.

But it just like bites off one little chunk at a time, and that feels a lot more like what I want a coworker to work like, or a collaborator to work like. It's like, if I ask you, like, hey, I need you to go build work chat. I don't want you to go off on this like crazy escapade, trying to make everything magical. I want you to just like, bite off a piece at a time, show it to me, like, commit it to git, like that kinda thing.

And it's just a little bit more natural. A and collaborating with feels with it feels more natural. And and funny enough, like, I I think this is unrelated. But we've been chatting in our internal company Slack with Claude a lot lately a and just like chatting with it, and it's been really natural. Like it. It feels just a little bit more like working with a coworker does,

In terms of the tone,

yeah, like the the tone, how it, how it respon- like how it acts in Slack, how it, how it participates in a conversation, what it tries to do. It's a little less like over the top and eager. That has jumped out a little bit, which was surprising. Like, that's not something that I normally expect, but yeah, it's also been funny, I guess, which is a funny thing. Like it, it like cracks better jokes, like it's a little bit more witty, that kinda thing.

Do you think we it also dialed back the sycophancy a bit? You know, I know that's been been a big word in the space lately, in terms of, you know, being over eager, over validating. Did you feel like it was more real and that it was less like that? A little bit

I have felt that, and I think it's certainly something we're pressing on. I I. It's like one of my least favorite things about every model is when they're sycophantic like that. I think it just gets in the way of doing good work. And it's been a big focus for us. Like, I think nobody at Anthropic likes that, and and I also just like, I think it's it's bad for the, it's just bad. So I do think we have made meaningful progress and this model seems to be a little bit more willing to, uh, push back a.

And that's part of that thing. Like being a natural co-worker is is someone who actually can tell you when you're wrong.

when it comes to rebuilding Claude.ai, that's pretty big. Did that worry you at all? Because, you know, it's kinda doing, like you said, you know, organic work that you worked on for months. Does that worry you about job replacement for engineers, anything like that? What were your thoughts?

We have a great team and I'm really not worried about put it. I'm gonna frame this two ways. I'm not currently worried it cl right now. Claude is a collaborator like it, it works really well with me, it it accelerates me, like, I think it makes our whole team better and faster at writing software. I am general, like this is not a now thing, to be really honest. Like watching Claude go for thirty hours, like it, it does trigger a little bit of like, oh my god, like that's like a pretty different thing.

It. It is a meaningful step change, and I think it does. Like this technology, Anthropic is founded on the principle that this technology is going to be hugely impactful on the world. And part of that is it's going to change how we do jobs and something like doing a whole week of work for me. Like, that's just like meaningfully gonna change the industry of software engineering. And so, yeah, there's like a little bit of and and I.

I would be like, it'd be goofy for me to say there's not any amount of like. This is like, how are we gonna work next? Like, how do I incorporate this? What is it? My net net, though, is like, Uh, I just think there's a ton of room to make us better, to make better software for users, to make the world a better place with this technology. And I, I'm really confident in that still. But yeah, there's like a smidgen of like, well, we're gonna have to figure out some new ways that we build with Claude and that we like operate as software engineers to work with the thing.

If it's really gonna like build the whole app itself, like, we have probably a different role that we need to play here.

we need to take another quick break, we'll be right back.

Running a business shouldn't feel like surviving a software group project. One app for accounting, another for inventory, another for sales, and somehow none of them talk to each other. That's where Odoo comes in an all in one business management software that brings every part of your business together. From sales and accounting to inventory and marketing, all in one powerful platform. No messy integrations, no bouncing between tabs, and best of all, no spreadsheets. Stop managing software and start managing your business with one unified system.

Try for free today at odoo dot com slash vox. That's o d o o dot com slash vox.

Support for the show comes from Framer. The future of website building is seamlessly blending all the tools we have available, and you could use a platform that has it all and stays useful even after you launch. That's what Framer is. Framer is the AI website builder that helps creators, teams and businesses ship production-ready sites faster than ever, while getting every detail right. You're able to prompt, inspect, edit, and publish in one place at a whole new pace.

On Framer, agents bring speed and scale, while people bring taste, judgment and control. And that all takes place in the same workspace where the real site is designed, managed, reviewed and published. Everyone's ideas land on the canvas, stay editable and can be published when the team is ready. Thousands of businesses, from early-stage startups to Fortune five hundreds, are choosing to build their websites in Framer, where changes take minutes instead of days. Learn how you can get more out of your site from a Framer specialist, or get started building for free today at framer dot com slash decoder for thirty off our Framer Pro annual plan.

That's framer dot com slash decoder. Rules and restrictions may apply. Support for the show comes from Outshift, Cisco's incubation engine for frontier technology. You can find multi-agent AI in every enterprise roadmap right now, but here's the problem everyone's facing agents can pass messages but can't think together. These systems silently underperform through cognitive failures, misreadings, unverified claims, false consensus, and conventional monitoring can't detect them. Outshift by Cisco is building the fix with their new comprehensive resource, the Internet of Cognition.

It's an open source guide that's a foundation for building multi-agent systems from design to production. The goal is to give agents shared context, shared memory and guardrails, all to drive the results we actually expect from AI. You can lean on this resource to understand agentic AI's evolution and how it shifted from isolated agents to orchestrated superintelligence. You can gain insights for how to future-proof your business by tackling complex challenges with multi-agent systems. You can read the paper, experience the demo, grab the code from Outshift dot com.

That's O-U-T-S-H-I-F-T dot com.

We're back with Anthropic's applied AI lead, David Hershey. Before the break, we were talking about the capabilities of Claude Sonnet 4.5 and whether he sees a future where this technology might even automate parts of his job. But now I wanna ask David where he sees the model falling short and what might be next for AI agents. Well, what are the primary limitations to Sonnet 4.5 that you wish you guys could have offered with this release that you couldn't?

And what was something that you tried to make it do during testing that it couldn't do? Basically, yeah, the things you, the features, or, you know, the context window or anything else that you wish you could have offered that you couldn't. And also what was dumb about it in testing?

I have a pet thing that I always test on, which is I really like to make Claude play games. I'm accidentally famous for creating Claude plays Pokemon in the past. Uh, Oh, that is the project

was you. that that I've heard a lot was the about that.

yes, that is, that is my project, and I have also tried a lot of other games. Like I had Claude playing chess this time, and and one of my favorite comms people really wants me to make C make Claude play Catan effectively. And it's really bad at spatial awareness, still, and this it annoys me to no end. Like it, it just like, basically doesn't know the difference between left and right and up and down. And this is just like, it's just one of those examples that exists in the world of it can do PhD level math and I can't.

And and just for it to not really understand that it can't walk straight through a building hurts my brain. And models are kind of weird and funny that way. So that one, that one was the one that, like, probably jumped out the most. Is I really wanted Claude to be this like great chess player? It's so logical and interesting, and then it, like, doesn't know where its pieces are on the board and where other pieces are.

And it's like, Ah, you're almost there, Claude, one day. And features that I think are still out there. I have been so in on this model that I have to think about the the next level. And honestly, like, the real answer, I don't know, I like, I can't think of some like feature that I wish was there. There's like a lot that I want Claude to get better at. There's like so much that I want Claude to get better at.

I want it to be able to beat Pokemon one day on its own, and I want to see it like how it does it. I want to, like again, there's all of these fields where I'm like, know that Claude still needs to get better. Like I, I. We talked about legal, like, I think Claude's becoming a better lawyer, but I don't think it's like as good as a lawyer as it is a software engineer yet. And like, I still wish we worked on that and and we're making progress.

Like, I know the teams that are working on all of these little things and focusing and thinking about all these little things. And there's always this like, so many hills in my mind that we have to climb still. So I could probably like, there's probably this intersection of there's a million things I wish Claude was better at. Not one that I am personally banking on, except for Pokemon, which is what? One day the research team will listen to me that it matters.

They haven't quite listened to me yet.

Well, when I when I went on a reporting trip to London and visited the office, it was all I heard about. So it's making waves somewhere.

That's good, well, it, it, it is, it makes people happy here. It's a fun experiment. I don't think it has crested the peak of of what we need to train Claude to be good at yet, unfortunately, one day.

Let's talk a little bit about what's next. So talk to me about why you think Anthropic is pursuing such gains on the AI coding front and how it stacks up to competition. Obviously, in testing, you had to, you know, take into account what the market looks like right now and what other models can do. You know, how did you think Sonnet 4.5 stacked up during testing? And, um, you know, what stuck out to you there?

Coding market is really important for a lot of reasons. It it's really well positioned for people to build with our models, it's it's a place where you can have a huge impact. People have figured out great ways to integrate models into how they do a job, writing code, and I think more so than essentially any other industry. And so if there's like a right now where you can make a huge difference by continuing to make models better.

I think coding is probably like the best use case, and so it's a huge focus for us. We wanna keep helping people who are relying on our models to write code be able to get more out of them. And our belief we've talked to a lot of customers, done a lot of testing. Like, I, I'm pretty confident that this is the best coding model in the world. We have a lot of benchmarks and other things to prove that out.

And it certainly has, like, for all of the people here in our early testing, just been a huge step function improvement in what it feels like within Claude Code or other surface areas to to develop and and write code. And and from my perspective, this is like a really big step function. And and personally, like, this is just talking for myself. This is the most noticeable change and and improvement I've seen since we released Sonnet 3.5 last year, which I think is a a funny coincidence.

I don't think we necessarily knew that these 3.5 sonnets were going to be such special models for us, but just mechanically like it. This is some of the kind of our interface of of vibes and numbers and things I've seen. This feels like a really, really meaningful step change improvement. One of my favorites, just to maybe call it out, I love the team at Cognition. I spent some time working with them. They put out a post yesterday about how much their their product devin improved with the model and some of the work they did to to make that product better.

That I loved, it's a cool blog. And that that like, really, I don't know, I I think that just that validation from the customer like it. It's a huge jump in a benchmark that I really haven't seen have a benchmark like a jump for them in a while. I just think that's really cool. Like for the right product built the right way around this model. I have a feeling this is a huge step function change in ability to write coding.

And I'm, and I'm quite confident it's going to be the best model in the world for for people like that.

Yeah, that's really interesting because I usually don't mention benchmarks much in my own reporting. Because sometimes they can be subjective, and sometimes they're, you know, created by the companies that are testing their own models in very specific areas with very specific sets of questions. But I think usually what I hear from engineers is that they go We based on the feeling and based on what we things it can do that we it couldn't do we before in their own anecdotal testing.

we we So that's why it's interesting to see. You know, you have your own pet projects that you like to test on, and you've seen changes in that regard instead of just in the benchmarks specifically. I also wanted to ask you about Anthropic's chasing consumers versus enterprise versus governments. So, you know, all AI companies right now, it seems like, are kind of like working with those three tiers. And I think it's probably because, you know, those are two of them, maybe are more concrete areas for potential profit.

So do you think with Sonnet 4.5 and just all the stuff you're working on in general, I know you work mostly with the enterprise side of things. But is there a specific slice that Anthropic is chasing more right now, and why?

I I think they're basically all really important. And I think one of the things that we have luckily seen, and I think is true for all of the labs, is that when we make our models generally smarter. They service all of those segments, they service enterprises who are building with us, they service consumers who want to use our chat app or write code or get into that, and they're useful for the public sector too. Just to tie a little bit of a link here, I I work with, I've worked with customers and startups that have become big consumer hits.

Like Lovable is a great example, where, you know, it's like, that's an enterprise customer to from my perspective, that I spend a bunch of time working with and helping them be successful. And our team does. But then they go to turn on this thing that everybody can use to build just in time apps to service every part of your life. And so I I think this kind of just depends back on itself, where in reality our focus is building really great models that are safe.

And we have seen time and again that that results in sort of like success in all of these places. And there's just like a lot of different ways from our perspective. I don't know, this is talking a little bit from my personal perspective instead of just Anthropic's. But I think there's just a lot of ways that you can make progress building great models. And whether it's helping enterprises or helping consumers, or helping the government, like these all have a way to warping back in on themselves.

Where the thing that we really do that makes a difference is make great models. And there are a lot of different ways you can impact different segments with that.

And when it comes to consumer use cases, OpenAI seems to be pushing pretty hard into that right now. At least this week, they launched Pulse recently last week, and then yesterday they debuted their instant buy button. You know, I wanted to ask where you think Anthropic is planning to meet consumers. Do you think it's more likely you'll be reaching customers with Claude directly in the future, or is it more likely customers will use Claude through something like Cursor?

You know, and that can also apply to your startup clients too, right? Where are you seeing people really find Claude?

I definitely would think we are growing. It's just like the presence of the applications we build to interface with consumers. And Claude Code is a great example of this. I know it's not like the traditional consumer, but there's like a very prosumer-y thing that we've captured with Claude Code. A And I think we've demonstrated that we do have some of the muscle to capture, like for some sets of clients, a thing that they love that that's real, like sparks, the excitement of a consumer product that people love using.

And I, I guess, like, obviously, we would love to do that fifty times over. Uh, we would love to be able to invent a bajillion beautiful uses of Claude that interact w or that that consumers love and and love to build. And and we're investing and trying to find and build great experiences that help people do important things with Claude. We have a sort of constant tinkering we we we have this imagine demo that came out.

It's just a limited time thing. I don't think that's the product we're gonna launch that is consumer facing, but like, we just like it. It's part of a portfolio, I guess, of like, how we think about. We need to keep building and trying and innovating and inventing products, seeing if there's something there. And we have a good opinion of what models are capable of. That helps us have an interesting lens on what we build. So it's a big focus like we, we need to get there, like it's important for us to build direct relationships and help people have awesome ways to interface with Claude.

That said, like, I don't know, I'm, I'm biased as a customer guy a little bit, but I I also just think like I wouldn't ever. And I don't think, and I don't think we should. And I don't think. Anthropic is betting against the ecosystem of people who are trying to build amazing products. And I think it'd be really silly to to claim that that's all our market to grab. I just don't think it is like, if we build great models, then there's an incredible ecosystem here in Silicon Valley and abroad, building amazing products.

That I think is like, probably bigger upside than our our first party applications will ever be. So I'm like, I think that it will always be a huge focus for us.

Awesome, and then last question for you, back to the AI coding market and how important it is, and how much most AI labs are chasing that right now. We talked a little bit about vibe coding earlier, and here at the verge, a lot of us have tried it and to no avail. We had some success with really simple stuff, but not a lot of success with building large scale applications. A couple of us did, but not as much as we actually expected.

You know, did you see a big change in terms of testing out, like vibe coding with Sonnet 4.5, or was it the same as before?

I have noticed, like my own personal coding, like every model we release. I am trained as a software engineer, so it's just like cheating. I'm, I'm like, not the best guinea pig for vibe coding, but I I still do it on the side sometimes. And you notice meaningful improvements on, like, what, how much it can churn out before it goes off the rails and how much you can trust it. I actually, I I think this is actually an interface problem, and and I think there's something funny about AI, which is that it has this tendency to outgrow interfaces really fast.

So if you look at the history of coding with AI, there was probably like, copilot with GhostText, so copilot would automatically complete your code. For a while, there was you would go to us or ChatGPT or wherever it was. And you would ask it to write some code for you in a browser window if you were a developer. And then you'd copy and paste that into your editor. Cursor figured out how to sort of, like, bridge those two things where it could be side by side.

A lot of people have started building this sort of agent that lives aside alongside your ID that lets you build things. I don't think any of that is quite the thing that we need for everybody to build production applications, though. Like, there's some interface where I actually do think Sonnet 4.5 is the first model that could be that thing where anybody could build a sort of like production ready application. I've seen enough evidence of when left to its own devices, how it can build complex applications.

I've seen it be able to deploy a complex application to AWS, like fully autonomously, and do like a security audit on it. Those are like both really incredible things that like, make me think this is the model that could sort of, like, cross that chasm and get to the point where, like, anybody can make something production ready. I have a feeling, though, we need like one more interface that isn't Claude code and isn't cursor. But like, the next step past that, I think needs to happen to get to, like, it's more obvious to everybody.

Instead of having to, like, try to figure out if you're on the right path, vibe coding,

Awesome, well, thanks so much. I'm so glad we got to talk. And, um, you know, I appreciate you, uh, coming on with so last minute. I really appreciate it.

yeah, it was fun, I appreciate it. Of course, it was really nice to, uh, nice to be on.

I'd like to thank David for taking the time to speak with me and thank you for tuning in. I hope you enjoyed this episode. If you'd like to let us know what you thought about this show or what else you'd like us to cover, drop us a line. You can email us at decoder at the verge. We really do read every email, or hit me up directly on X, BlueSky or Threads. I'm at Hayden Field on all platforms.

Decoder also has a TikTok and an Instagram, and now also a YouTube channel. Check those out at Decoder Pod. They're a blast. If you like Decoder, please share it with your friends and subscribe wherever you get your podcasts. Decoder is a production of The Verge and is part of the Vox Media Podcast Network. Our producers are Kate Cox and Nick Statt. Our editor is Ursa Wright. The Decoder Music is by Breakmaster Cylinder. See you next time.