OpenNashPodcasts

Latent Space: The AI Engineer Podcast

⚡️ The 2026 Software Factory Standard — Zach Lloyd, Warp.dev

3 Nov 2026 · 34m · Zach Lloyd, swyx · 22 quoted facts

▶ Play episodeOriginal source ↗

In brief

Zach Lloyd of Warp argues that a "software factory" should be a concrete, measurable system, not a loose pile of AI automations. The factory is defined in code, runs in the cloud and is fed work through tools like Slack and Linear. Because every agent conversation is stored, a team can measure cost and quality and tune the system the way it would any other engineering project.

The main tension is between rigor and overhead. Lloyd says teams should stop judging coding agents by Twitter hype and measure them with scorers, benchmarks and self-improvement loops. He admits this is overkill for very small teams, which are better off on cheap subscriptions. It is also a poor fit for giants with bespoke infrastructure, and he describes the middle band of roughly 10 to 100 engineers as the sweet spot.

Key insights

  1. A factory must be defined in code

    Lloyd says Warp uses a YAML file listing repos, skills, MCP integrations and workflow steps. Like Terraform, this gives versioning, branching and rollback, and it lets coding agents propose changes to the factory itself.

  2. Centralized agent traces enable measurement

    When all agent work flows through one system, a team can track DORA-style shipping metrics, human interventions per merged PR, and cost per PR. Lloyd says most companies lack this because engineers run Claude Code or Codex locally.

  3. LLM judges grade past agent runs

    Scorers are judge models that retroactively grade stored runs, using a prompt and a sampling rate to control cost. One example flags redundant tests, a common agent failure that bloats the code base and wastes tokens.

  4. Warp cut cost per PR roughly fourfold

    Lloyd says cost started near $80 per PR because expensive models like Opus were used on steps such as triage, and because too many sub-agents ran. Tuning brought it to about $15-20 while throughput rose about 40% and the revert rate stayed flat.

  5. Factories suit mid-sized teams, not tiny ones

    Lloyd puts the floor at about five to ten engineers and calls the product self-serve for ten to a hundred engineers on common tools. Smaller teams should use flat-rate subscriptions, and very large firms like Uber or Ramp build their own.

Questions

What is a software factory in AI coding?

Zach Lloyd defines it as a cloud-based system of AI agents, each with a defined role, that moves software work through a set workflow with some autonomy. It should be written down in code and be measurable, so a team can tune it over time.

How is a software factory different from just using coding agents a lot?

Lloyd says many companies just string together automations and call it a factory. He argues a real one is a specific, defined-in-code system with set repos, skills, integrations and workflow that you can run and measure.

How do you know whether your AI coding agents are actually any good?

Lloyd says you store every agent run and then have judge models grade them on things like code quality or redundant tests. You can also replay the same tasks across different models and tools to compare cost and correctness on your own work.

Is a software factory worth it for a small company?

Lloyd says no for very small teams, calling it overkill. He suggests it starts to make sense at about five to ten engineers, and that tiny teams should just use Claude Code or Codex subscriptions.

Can a factory really lower the cost of AI-written code?

Lloyd says Warp reduced cost per pull request (a proposed code change) from about $80 to roughly $15-20 by using cheaper models for easy steps and fewer sub-agents. He says output rose about 40% and the revert rate did not increase.

Facts

Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.

Zach Lloyd says Warp's cost per pull request was about $80 when it first adopted the factory approach, partly because it used Opus where it was not needed.

“when we first started using the factory approach, it was costing us about eighty per PR because we were using, like, Opus, where we shouldn't have.”

0:23:39 · Zach Lloyd · Cost per PR

Zach Lloyd says Warp tuned its factory until cost per pull request fell to roughly $15 to $20.

“we were able to get the cost per PR down to, like, twenty bucks or fifteen bucks, something like that.”

0:24:14 · Zach Lloyd · Cost reduction

Zach Lloyd says Warp's pull request throughput rose by about 40 after tuning its factory over the previous month.

“we can measurably improve the throughput of our PRs over the last month. It's gone up by forty.”

0:24:14 · Zach Lloyd · Throughput

Warp defines each software factory in a YAML file that lists the repositories the agents can use, along with integrations and workflow.

“So for us, we basically have a YAML file which says, okay, these are all the repos that you can be using.”

0:05:11 · Zach Lloyd · Factory as code

In Warp's model, a company has one software factory for each product surface, such as a mobile app or server app.

“You can orchestrate it, you can measure it. You have one factory per product surface,”

0:04:15 · Zach Lloyd · Factory design

Zach Lloyd says about 60% of Warp's internal factory work starts in Slack.

“the most obvious one is Slack, and, like, we're doing, like, 60% of our work is starting in Slack.”

0:25:15 · Zach Lloyd · Slack usage

Zach Lloyd says Warp uses Namespace as its primary agent sandbox provider for compute.

“We use namespace as our primary one.”

0:22:26 · Zach Lloyd · Sandbox provider

Zach Lloyd says Warp tests iOS apps by running a Mac virtual machine where an agent operates the simulator.

“the way that we test, like iOS apps, is you're gonna wanna have the ability to run a MacVM and actually, like, have it run the simulator and click around.”

0:22:40 · Zach Lloyd · Computer use

Zach Lloyd says Warp factories let customers plug in their own agent harness, including Pi or any ACP-compatible harness.

“You can even plug in your own harness, which I think is a pretty big deal. So, you know, you you can orchestrate things like Pi, like I mentioned earlier, and basically any ACP compatible harness.”

0:23:39 · Zach Lloyd · Harness support

In an example benchmark shown by Zach Lloyd, GPT 5-6 Sol was 95% correct at about $6 per task while GLM 5-3 flash was 93% correct at about $2.

“GPT 5-6 Sol here was, like, 95% correct on this task set, but it costs six bucks on average, and GLM 5-3 flash was 93% and cost two bucks.”

0:17:31 · Zach Lloyd · Model benchmark

Zach Lloyd says factory data can track traditional DORA metrics, including how long it takes to ship something.

“things like, you know, traditional DORA metrics, how long does it take to ship something?”

0:08:24 · Zach Lloyd · DORA metrics

Zach Lloyd says Warp's evals work as an LLM-as-judge setup in which a chosen judge model grades stored agent runs.

“the eval is, again, think of it as LLM as a judge. So you're picking a a judge model.”

0:11:14 · Zach Lloyd · Evals

Zach Lloyd says self-improvement agents can open pull requests against Warp's factory definition to improve the factory.

“because you have your whole factory defined as code, and these are coding agents, they can essentially make a pull request against the factory definition to try to improve the factory itself.”

0:15:21 · Zach Lloyd · Self-improvement

Zach Lloyd gives an example in which Warp's self-improvement agents found 54 triage failures that a sub-agent could have avoided.

“we have fifty four failure cases of this particular form, where it would've been better if our triager had been able to spawn a sub-agent to do its investigation.”

0:14:51 · Zach Lloyd · Triage failures

Zach Lloyd compares defining a software factory as code to using Terraform for infrastructure.

“So, again, the analogy is, like, Terraform here, you get the same benefits, but actually more.”

0:19:51 · Zach Lloyd · Infrastructure as code

Zach Lloyd says larger companies want different MCP access for different departments such as marketing, sales and engineering.

“So they want things like, what MCPs can the marketing team access versus the sales team versus the engineering team?”

0:22:00 · Zach Lloyd · Access control

Zach Lloyd says a software factory starts to make sense at roughly five to ten engineers.

“We start seeing that maybe around, like, f- five to ten engineers would maybe be the floor where this makes sense.”

0:27:41 · Zach Lloyd · Team size

Zach Lloyd names Uber and Ramp as examples of companies that are a poor fit for Warp factories because they build their own systems.

“our our kind of not a good fit customer for this, for instance, would be like Uber or Ramp.”

0:30:34 · Zach Lloyd · Customer fit

The host says a $200 coding subscription plan provides roughly $8,000 worth of API usage.

“you get, uh, for, like, 200 plan, you get 8,000 of API, right?”

0:32:10 · swyx · Subscription subsidy

Zach Lloyd says Cursor made a good exit by becoming a model lab after recognizing the economics of building on third-party models.

“I mean, Cursor was brilliant, they did it, and then I think that they had a perfect exit and became a model lab because I think they fully realized the economics of it.”

0:32:35 · Zach Lloyd · Cursor strategy

Zach Lloyd says factory tooling is used mainly by engineering managers, CTOs and developer-experience or platform teams, not day-to-day engineers.

“they'll often have an internal team that's been trying to, like, it's like the DevEx team or the platform team”

0:12:29 · Zach Lloyd · Target users

The host says AIE Code is the last AIE flagship event of the year and will be held in San Francisco.

“AIE Code is the last, uh, AIE flagship of the year. It's gonna be in San Francisco, where we're gathering all the top agentic engineers”

0:00:54 · swyx · AIE Code

Chapters

  1. 0:02:23What a software factory means

    Lloyd defines a factory as an autonomous, cloud-based agent system that should be measurable and closed loop, rather than a vague collection of automations.

    “agentic system in the cloud where you're defining what are the roles of various agents”
  2. 0:05:11Factories defined as code

    The factory is a YAML-defined unit per product surface, with work entering through Slack, Linear, Jira, GitHub, API or MCP.

    “we basically have a YAML file which says, okay,”
  3. 0:07:11Traces and metrics across the team

    Centralizing agent conversations gives teams delivery, automation and cost data that individual local use of agents cannot provide.

    “these agent traces are like the key aspect of how this system improves.”
  4. 0:11:14Scorers: LLM judges on past runs

    Custom and built-in evals retroactively grade stored runs for failure modes such as redundant tests, giving teams something better than vibes.

    “think of it as LLM as a judge.”
  5. 0:15:21Self-improvement and benchmarking

    Agents can propose pull requests against the factory definition, and teams can replay runs across models and harnesses to compare cost and correctness.

    “make a pull request against the factory definition”
  6. 0:19:23The factory stack and Warp's results

    Lloyd lays out layers from declarative definition to orchestration and access points, and shares Warp's cost-per-PR savings.

    “you define your factory declaratively as code.”
  7. 0:27:20Who needs a factory, and who doesn't

    The approach fits roughly 5-10 engineers and up, but not tiny teams or firms with bespoke infrastructure.

    “Use the Claude Code sub or the Codex sub or whatever”
  8. 0:32:21Subscriptions, economics and what's next

    The hosts discuss subsidized subscriptions versus API cost, why Warp avoids the pure coding-agent business, and Lloyd's upcoming talk on factory engineering.

    “don't wanna be in, like, the pure coding agent business”

Transcript

0:00:00
swyx

Hey, it's me again. I have heard a lot of positive feedback about our episode intros, so I'm gonna keep trying to do them as far as time allows. Today we're gonna dive into Warp with Zach Lloyd. Um, Zach has been working on Warp for a while, and, uh, it's pivoted, uh, into what it is today, which is basically the definitive software factory, uh, that is also open source. Um, and he has this whole definition, I think, which, uh, has done really well.

0:00:24

He's he's basically his talk at, uh, AIE has been going viral as one of the top talks of all time. And I think effectively everyone is trying to set this up for themselves. They have a solution for you, um, but it's also, you know, kind of an open spec that is becoming table stakes for what you're gonna want. Uh, and I think you are gonna want cloud, you're gonna want a control plane, uh, you're gonna want, uh, so all these sort of basic security features and team features that, uh, are you take for granted.

0:00:54

So we're releasing this episode as part of a preview for AIE Code. AIE Code is the last, uh, AIE flagship of the year. It's gonna be in San Francisco, where we're gathering all the top agentic engineers and talking about software factories, among other things. System one models, Diogo is coming back and, uh, raising your ambitions. So t- Tariq and Jason and Brandon from Artcraft, uh, is doing his first conference talk ever. So if you're interested, um, Zach, you will also be giving, uh, a great keynote on software factories and sort of recapping the year and previewing what's next.

0:01:29

Join us and it'll be lots of fun. See ya. We're in the remote studio catching up with Zach Lloyd from Warp. Welcome.

0:01:38
Zach Lloyd

Good to be here. Thanks for having me.

0:01:40
swyx

Zach, you've been going viral on AIE. What's up? What's the It's what's the what? all You've hit a nerve. You've hit a nerve.

0:01:45
Zach Lloyd

people are really interested in software factories, and that is a area that I've been focusing on, that Warp has been focusing on. I think we have some, you know, strong, kinda interesting takes on what a software factory actually is and what it means for being a software developer. So, yeah, that's and that's people have been into it.

0:02:06
swyx

Your H1 on the landing page says open infrastructure for cloud software factories. I think maybe if since you spend more time than I do talking about telling people about factories, what do people think is a software factory, and what do they actually need?

0:02:23
Zach Lloyd

When I talk to companies or developers, most of the world is trying to build some sort of internal software factory or set of software factories. What what I think it means, or maybe I'll just define what it means to me, is like a agentic system in the cloud where you're defining what are the roles of various agents across building software for your team or for your product. And there's a certain amount of autonomy in that system, so it's like the agents can do things on their own.

0:02:49

And that system, to me, should be something that's very measurable, very closed loop, which over time, you as a team can, like, tune, measure, benchmark, and just ultimately get confidence that as people switch to this way of doing agentic development, that they're doing it in the most scalable and efficient way.

0:03:08
swyx

And wha- what do people come to you with that perhaps is a common misconception, right? Like, they're like, Okay, if I if I do whatever, if I buy whatever, I'm gonna come out the other end with, like, this

0:03:21
Zach Lloyd

Yeah, so so when people come to us and they're trying to build their own software factory, like often they'll be like stringing together different automations. Sometimes people are conceiving of software factories as just like just like not a not a specific thing, but it's just like we're using agents a lot to ship software, and we're calling that our internal software factory. And what we try to to do to make that more concrete is be like, no, no, no, you should actually think of a software factory as a very specific thing.

0:03:49

Like, you should define your factory in code, you should make sure that the factory has all the workflows, skills, MCPs, everything is like, it's like a concrete thing that you should be able to run at some point in time and measure how it's doing. Cause people have all sorts of kinda random ideas around what these factories are, and so for for Warp factories, which is our platform, it's like, well, we say no, no, it's actually it's a real thing.

0:04:15

You can orchestrate it, you can measure it. You have one factory per product surface, and we try to, like, take it out of the realm of just being like this nebulous, uh, it's a collection of, like, arbitrary automations and interactive agents into a a thing that's an actual well, the way to think about a software factory, or the way that we think about it, what you're looking at here is is Warp software factory platform is a factory operates on some product surface.

0:04:42

So imagine you're building like your mobile app, or you're building, you know, your server app or whatever, and the factory itself is operating on a set of repos. And it has a set of skills it needs, it has a set of MCPs, and then it has, like, a a workflow associated with it. And this workflow, you know, roughly matches what, you know, your team is probably already doing to make changes to that code base. So there could be a step where you're doing triage, it could be where you're doing specking, implementation.

0:05:11

The important thing is, like, there's some kinda defined workflow, and that that workflow itself is defined in code. So for us, we basically have a YAML file which says, okay, these are all the repos that you can be using. These are all the MCPs, all of the integrations that are hooked up. So you basically have this whole thing defined someplace. The way to think about using the factory is that there's actually sort of two personas. So there's someone who is building the product, they're doing product engineering, they're shipping features through a factory.

0:05:49

And for those people, the way they use the factory or the way that is through their existing tools. So it could be slacking the factory to put work into it. They could be using a task tracker like Linear or Jira. They could be interacting with the factory through GitHub. They could be calling it via API or MCP. And the metaphor is like you're putting a work item into the factory. And when it goes into the factory, that starts like a predefined workflow, and that workflow is defined in code.

0:06:18

Does that make sense so far? I love defining things in code. Yes, so okay, so you have this whole thing set up in code, and then once you have that in place, and so the again metaphor is like someone starts a task in Slack, or in Linear, or through an automation, like, you know, there's a server error. It all goes through one system is the kind of key thing. And once you have that working, you basically get a whole bunch of benefits that flow out of it.

0:06:45

And so one of the big benefits is that wherever these agents are running, and you're sort of seeing a real-time view across Warp's internal team, how are we using our own factories? You get a full trace of what everyone is doing with coding agents across any product that has a factory for it. And this is cool, not just for like, oh, I wanna see what people are doing, like, I wanna see what's off. Ick is doing, which is cool.

0:07:11

But it also starts centralizing all of the agent conversations and agent artifacts in all of the work that are is being done by coding agents across your team. And so when I talk about like having a closed loop system, these agent conversations, these agent traces are like the key aspect of how this system improves. And so, for instance, just to look at one of these agent traces, Sophic is working on our mobile app here. Sophic was doing this in Slack, so I could pull this up in Slack if I wanted to, but, like, the thing is actually generating it has a full record of the conversation, has a full record of all of the these are computer use screenshots that

0:07:53

are coming out of building this thing. And you can see a full record of exactly what the different pieces of the orchestration agent were doing. And so you have this, but you have it not just for one thing, you have it for every single thing. So on your team today, what we when I talk to companies, for the most part, they don't have something like this because most of their engineers are like individually using cloud code in their terminal or Codex, the Codex app or something like that in their own local setups.

0:08:24

But they want to get to a world where they have this centralization. When you have this centralization, you can basically get all this interesting data on how your agentic coding is working across your company. And you could think of this data along different dimensions. So you could think of this data as, like, okay, what's our team's output in the world of using coding agents? And so that's things like, you know, traditional DORA metrics, how long does it take to ship something?

0:08:52

It's things like how automated are you? And so we have a sort of metric around, like, for every PR that gets merged, how many times did a human have to do something with it? What's your throughput? It's metrics around cost, and so we're using this, like, constantly. And when we talk to companies, this cost thing is very, very important, where you want to be able to tune your factory with the right models, the right skills, to basically bring this cost per PR down by time.

0:09:18

You want to be able to slice the cost by person, you want to be able to slice it by automation, by s by where in the SDLC are you spending money? The final thing that's that's really cool once you have this sort of closed loop system going, is that you can actually look underneath the hood at how well your coding agents are doing along different dimensions. And so you can think of this as like online evals, or we call it scorers, and these can go from very general things.

0:09:48

So like, are your agents being too verbose? are they being efficient? Is the code quality good? We got work to do on this. Do I as a user define this, or do you provide this as a platform? Yeah, so both. So we provide out of the box a set of ones that we think are, like, generically useful. But then, you know, as we work with companies, you know, they may have specific failure modes for their agents that they're interested in tracking.

0:10:15

They may be like, Hey, is the agent making a specific type of thing? That is like change that is like breaking our build. They may want something, you know, very specific to their workflow. So, again, this is all defined in code and all customizable. And what you do is, like, when you set up one of these, what's actually happening behind the scenes is that there's some set of runs that you you know. When I say runs, I mean, like, historical agent conversations that have been stored, that an agent then goes and, like, retroactively grades.

0:10:49

And so this is a good example of one where it's grading for redundant tests, which is like a this is a common failure mode for using coding agents. Like, sometimes they write a bunch of tests that you don't need. They write it in the course of, like, figuring out how to, like, implement something, and then they leave it in the code. And so this, like, bloats your code base, it wastes tokens, and so, you know, say, you wanted to check for that, you, you know, you basically create one of these evals.

0:11:14

And just to show you, like, a little bit more about what the eval looks like, the eval is, again, think of it as LLM as a judge. So you're picking a a judge model. You're you're then seeing, okay, like, where did these run? What's the prompt for judging it? And then a sampling rate, because you might not like, these are agents that are running. You might you gotta be wary of the cost of them. But when you have this set up, think of it then as, like, you have all these smart agents that are looking across your team as ha- at how all your your coding sessions are going.

0:11:43

And they're saying, okay, sometimes there are agents that are writing too many tests, right? And so you get this record and this understanding of where you should look to sort of improve the way that your team is using coding agents. So far, so good, cause I wanna show a couple other things just to, like, make it more specific what you can do with this. This is like the the people who use this, it's partly the engineering manager or the CTO, but it's also, like, very commonly, like, the platform team.

0:12:29

But it's not your, like, day-to-day engineer typically. But, like, when we work with companies, you know, they'll often have an internal team that's been trying to, like, it's like the DevEx team or the platform team, and they're trying to figure out how to optimize coding agents. So what I'm showing you here, yeah, is, like, the tool set that they are using to be like, how do we get the most out of Claude Code at our company or Codex or something like that?

0:12:52
swyx

That part is clear.

0:12:53
Zach Lloyd

Yeah, cool, yeah. And so, like, a a couple other things that I would just want to emphasize, not to make this just, like, completely about Warp factories, but, like, the general principle here for anyone who's trying to build their internal factory, whether it's on Warp factories or not, is like, you want to get to a spot where everything is sort of stored, everything is observable, everything is measurable. Because that's when you can actually start to, like, apply engineering rigor to how you make coding agents better.

0:13:26

You know, you probably you see this on Twitter all the time. Everyone is always tweeting about, like, you should check out this new model, you should use this new harness. What's better? Is it Py or Claude Code or, you know, Cursor or whatever? And so, what our opinion on this is and what the the world sort of needs is, like, out of the the realm of vibes for understanding these coding agents and actually into the world of measurement.

0:13:51

And so we think that this is the direction that companies are gonna wanna go. They're they're gonna wanna, like, put in place a system that's like Warp factories, whether they actually use Warp factories or they build their own, which collects all this information and provides all this way to see how your coding agents are doing. I guess, yeah, one other thing I like to kinda take for granted that's part of this system is, like, you can create all these different automations around it.

0:14:13

So, like, I I mentioned people work in Slack, but people are also setting up, uh, different trigger-based ways of putting work into the factory. Once you have this view that I showed earlier of, uh, like, scoring how your factory is doing, you can also set up another kinda loop of agents, and everything is kinda loops these days, where those agents are looking over. They're looking for failures in your scores, like they're looking for things that aren't going well and trying to suggest changes to your system to make it go better.

0:14:51

And so, like, a really simple example here, we call this self-improvement, is like, okay, our self-improvement agents, we're looking at how well our triage step was doing. So we have a triage step, and they're saying, okay, we have fifty four failure cases of this particular form, where it would've been better if our triager had been able to spawn a sub-agent to do its investigation. And so what they're able to do, and, like, you could go look at, like, why it thinks this if you want.

0:15:21

But the cool thing is, like, because you have your whole factory defined as code, and these are coding agents, they can essentially make a pull request against the factory definition to try to improve the factory itself. And so this is like a, like, a full learning loop that's built into the system to try to actually self-optimize. And, like, we don't have this yet, but the next thing that we want to do with this is, like, actually, for any suggested change like this, you could go back, replay the runs and make sure that this actually would have improved them.

0:15:56

So the possibilities of, like, how you can engineer this stuff get pretty cool once you have the right primitives in place. And I'll show one last thing, and then we could sort of maybe talk more generally about the whole approach here. But the other thing that is, um, you know, I think companies should strive for that we find very useful internally is, like, the ability to actually do what we call benchmarking. And so benchmarking, in this context, in the factory context, means essentially the ability to, like, take any coding agent that's run in your system, you can sort of add it to a benchmark suite, which means, like, you know, you might curate a set of coding agent runs that

0:16:33

were related to, like, doing front-end coding or something like that. And then once you have that, you have the ability to sort of replay those same tasks against a matrix of different configurations. And so, you know, I'm a big believer in not just allowing you to vary the model, but also to vary the harness. And so, you know, in Warp, we have our own harness, but we actually will let you benchmark against Claude Code or Codex or Py.

0:17:01

You can pick the model, and when you do something like this, what you end up getting is a result that looks very similar to, like, the published results for Sweet Bench or Terminal Bench, but on your company's own real runs. For instance, we're looking at, like, a sort of cost versus, uh, correctness trade-off for a certain set of things here. And, like, you wanna be in the upper left-hand corner, so you want, like, lowest cost, highest correctness.

0:17:31

These are these models are now slightly older, cause the world's changing so fast. But the idea is basically, you know, we could see, okay, GPT 5-6 Sol here was, like, 95% correct on this task set, but it costs six bucks on average, and GLM 5-3 flash was 93% and cost two bucks. And then the cool thing about this is you can take this information and feed it into a model routing strategy or harness routing strategy, honestly, that's defined specifically for your own workflows and code base.

0:18:02

And so the you know, if I had again to zoom out, the really big picture thing that I think is so cool and powerful here is, like, you're getting out of the realm of guessing and you're getting into what we call factory engineering, where it's like you are taking a real engineering mindset. You're like, how do I, with this amazing new primitive of of coding agents, how do I know as a team, as a company, that we're actually deploying them in the right way?

0:18:27

And so, you know, we're trying to build this infrastructure and tools for companies, uh, to do this. And, like, this approach is really, really resonating with a lot of companies that have been trying to build, you know, their own their own versions of this. So that's, like, a quick tour of, like, what do I mean? When when I talk about factory, I can, I can sort of stop talking for a sec.

0:18:49
swyx

I do think that, you know, the the the power power move, I guess, is to sort of throttle lead so hard that everyone ends up trying to build warp e- even if they don't use warp. Yeah. S so, you know, you have on your landing page something I I like in in your talks, you've d done a few of these as well. Uh, something I really like also is any surface, right? It really has to be any surface.

0:19:11

It has to be factories as code. Um, you have to have the eval loop. Like, what is the whole like? People calling this the ADLC? I don't know if you use that term. It looks like you use SDLc still, you know, all all that stuff.

0:19:23
Zach Lloyd

This is how I would think about it again, whether it's Warp or not, if it's I would, obviously, you know, Warp Warp is this is what we're fully focused on. We have an awesome platform for it. But just in general, what I would think about if you are trying to build something in this space is, A, I would view it as, like, a stack, so as infrastructure. At the bottom of the stack, I think it's really foundational and important that you define your factory declaratively as code.

0:19:51

So, again, the analogy is, like, Terraform here, you get the same benefits, but actually more. So the benefits you get of from defining it as code is, like, you get a versioned view of, like, what was your we SDLC, AI, DLC, P, PDLC, whatever DLC you want. You get a versioned view of that, and then you get the ability to create branches off it, which you can test. You get the ability, if you change it, to revert it.

0:20:19

So all the obvious things about source control that are great. And then you get the added benefit that's extremely cool, where because it's in code, coding agents can suggest improvements to it. So to me, this is, like, the foundation of a self-improving software factory setup is define your factory declaratively. The next step up in this stack, as you think about architecting it, is some place to gather all of the information that agents are producing. Cause what what you wanna be able to do is examine that information, look for failure modes, or replay those same past tasks with different variables to sort of, I don't know, scientifically is the right word, but from a measurement standpoint, really optimize your system.

0:21:10

So that's the mindset. So you need the data to do that, you need the agent conversations, and you need the agent artifacts. You need a record of all of the interactions from all of the tools to get to a result. And so what I mean by that is, like, you might have a a factory conversation that starts in Slack and then continues into Linear and then continues into GitHub. And there's a code review comment. And that whole interaction is gonna result in some piece of M merged code.

0:21:37

In fact, it might even result in something that gets shipped and then has a server crash. And you want the ability to, like, retroactively go back with an agent, most likely, but you could do it as a person and and diagnose how could that have been better? What was the failure in it? You also need this part will seem fairly obvious, but you need a way of getting data into this system. So you need some way of plugging in MCPs.

0:22:00

And then we find that as companies get bigger, they want different access for different parts of their organization. So they want things like, what MCPs can the marketing team access versus the sales team versus the engineering team? And so, like, again, you'll you'll find there's a bunch of complexity if you're depending on how big your organization is, if you're trying to roll something like this out. You really want a compute layer. You should think of that as a separate thing.

0:22:26

And so there's there's obviously a lot of agent sandbox providers at this point. So if you're gonna use a third-party one, I would make just make that pluggable. We use namespace as our primary one. Um,

0:22:37
swyx

Ooh, they launched at, uh, AI Europe.

0:22:40
Zach Lloyd

namespace is awesome, but this should be pluggable. I think that's really important. So if you wanted to use Daytona or Modal or E2B or whatever, like, I would architect it. So this is a pluggable component of it. I would make sure, depending on your requirements, that you can do different platforms. So, again, by default, you know, I think you'll probably wanna use, like, a Linux type sandbox. But for computer use, like the way that we test, like iOS apps, is you're gonna wanna have the ability to run a MacVM and actually, like, have it run the simulator and click around.

0:23:11

We have a very open approach to inference, and this is, like, this is very much with our ethos of trying to be super duper flexible. So we find that a lot of companies right now will have pre existing commits with Anthropic or OpenAI or Google that they wanna burn down. And so we're we provide like a a a very big variety of, like, you can plug in your own inference provider, or you can buy inference through us.

0:23:39

You can even plug in your own harness, which I think is a pretty big deal. So, you know, you you can orchestrate things like Pi, like I mentioned earlier, and basically any ACP compatible harness. And then you want this improvement layer. And so this is some of the stuff that I showed in the demo where it's like, uh, the real power of this whole system is is the ability to improve it. And so, just to give some stats at Warp, when we first started using the factory approach, it was costing us about eighty per PR because we were using, like, Opus, where we shouldn't have.

0:24:14

Our triage agent was using, like, a really high-powered model. M you know, we were doing a bunch of redundant work with our the shape of agents. Like, we were doing too many sub-agents relative to just having fewer agents. And so we basically did a whole bunch of internal tuning and testing using these built-in features, and we were able to get the cost per PR down to, like, twenty bucks or fifteen bucks, something like that. You know, we can measurably improve the throughput of our PRs over the last month.

0:24:43

It's gone up by forty. We measure, like, the revert rate, and it hasn't gone up. And so I really feel like a lot of confidence in this general approach of, like, observe, measure, tune. Finally, you you need some sort of, like, orchestrator. Sometimes people talk about the orchestrator like it's the whole platform. I think it's a piece of it. What I mean by orchestrator is, like, you need some system that will take the factory definition code and then actually move work along the different stages, calling out to the right models, loading the right skills, et cetera.

0:25:15

You need something that will fire the right automations, and then finally, you need a bunch of access points. And again, our philosophy with this, which you can, I could see people have different philosophies. Our philosophy is to sort of bring the factory into a company's existing knowledge work tools. So think of, like, you know, uh, the most obvious one is Slack, and, like, we're doing, like, 60% of our work is starting in Slack. But sometimes it makes sense to start in the task tracker.

0:25:42

Sometimes it makes sense for for a developer who is more comfortable using, like, Claude Code in their terminal to do a bunch of work there and then submit a spec into the factory. And so, you know, our philosophy is to try to s to make this whole thing API-driven and MCP-driven, so that the factory itself is like a piece of kinda headless infrastructure where it has all these different entry points. So I think this this stack-based view of it, when I've shown it, uh, when we talk to companies, seems to land very well.

0:26:14

It's like, okay, yes, this this makes sense. This is a new thing in the, like, developer infrastructure stack. It kind of sits one level back from, like, where CI CD is, but I think most companies will have something like this for software development.

0:26:29
swyx

Yeah, that's a lot of layers, actually. I've never actually seen it spelled out like like you have. I wonder, cause I think most, you know, advise cognition, and I've s I, you know, cursor, what all these, they all try to do some bundle of, like, the ver- like, from bottom to top. And there's a little bit of bundling and unbundling with all software. And so there's the critical question of, like, look, I'm trying to do my job here as CTO of a, you know, a B2B SaaS, like, I do I need, like, all these, like, six layers?

0:27:02
Zach Lloyd

I think it's like, who's this for? Maybe is is the kind of the question. So when we talk to companies, that where this resonates the most is, it has to be a company that has some some amount of scale. Like this is, if you're like a few people, honestly, this is this is

0:27:19
swyx

Whatever, yeah.

0:27:20
Zach Lloyd

overkill. Don't do this. Use the Claude Code sub or the Codex sub or whatever, and just you'll go faster. You probably don't need you And, uh, could use yeah, same this. Yeah, yeah,

0:27:30
swyx

same for, like, any, like, larger team stack, right? Like, you at some point, you switch from, like, fire, YOLO, whatever, to we actually need a platforming system here to accelerate

0:27:41
Zach Lloyd

exactly, yeah. So you have to be a certain scale. We start seeing that maybe around, like, f- five to ten engineers would maybe be the floor where this makes sense. And then all the way up to to very large companies, they want something like this. The bigger the company, the more that this requires, you know, implementation work. Like, I'm sure if you're, you know, you're working with cognition or or I don't know how much Cursor does, but, you know, to get something stood up like this within a company's own infrastructure, it's not gonna be purely plug and play.

0:28:15

There's, like, you know, the sort of complexity of a team's internal developer setup just grows with size within, like. But there's a bunch of companies where, you know, maybe, like, it's like a ten to a hundred engineers, fast-growing, like growth-stage startup, where it is pretty self-serve, actually, where it's, like, you just if they're using, like, the happy stack of, like, Slack, you know, Linear, Jira, hang on, did you call it the happy stack? I call it, like, Yeah, I, that's, like, our I don't know.

0:28:44

No, uh, no, no shade to to teams or whatever, like, sure, but, like, there's, like, a, you know, it's like a set of s integrations that I find very common at startups, where you can plug something like this in and get, like, immediate immediate value without too much setup. I don't know if you how much you're working like this, but if you're working with, like, Devon or something, you can kinda get some of the same experience.

0:29:06

Where when you're working in a some an environment that's natively multiplayer, so, like, Slack, for instance, or Linear, there's all these benefits that you might not realize just by having, like, this central system of record, where you can have, you know, for, like, the Slack integration, for instance, it's like you can get some flow of information coming in through, like, a server crash channel and some f- through a user feedback channel. And you can just feed that directly into the factory to have it do the work.

0:29:35

The factory can loop in multiple people on your team, depending on who knows the right part of the code base. And so, you know, I don't think you have to be that big for this for this to make sense, but it definitely doesn't make sense for a s for a small team. And it really makes the most sense for a company that's like, I wanna I wanna put in place the right infrastructure so that as models and agents continue to change, we have a system that we're building upon.

0:30:00
swyx

there's a lot t to go on here. The the one follow one follow-up question, and then I'm gonna have you plug your AIE talks in just generally where to where to follow up with you. If you, uh, what is the most steel man criticism of setting up all this? Like, when you encounter criticism or pushback, what should people know as, like, here here is, like, what people are running into.

0:30:24
Zach Lloyd

I'll answer this in, like, there's a certain class of customer where I don't think it makes sense to, like, go on to this particular thing. And that's like

0:30:32
swyx

Yeah, and beyond the size thing, yeah,

0:30:34
Zach Lloyd

where we have challenges right now are companies with very, very bespoke infrastructure, or companies with, uh, like, our our kind of not a good fit customer for this, for instance, would be like Uber or Ramp. Like, they're building their own thing, they have very large internal developer teams. But apart from the size thing, I'm thinking, like, I mean, there's definitely, like, the for some people, they should, it's just gonna be more economical to just use the subs, like, for small, very small teams.

0:31:05

But that comes back to the size okay, actually, thing.

0:31:06
swyx

yeah, uh, I don't wanna actually omit the subs stuff because I think a lot of a lot of people go straight to API of pay by API. And that is between five to fifty times more expensive than just getting a whole bunch of subs. Do people do? S that is is that, like, so against terms of service that, like, just don't bother, or, like,

0:31:29
Zach Lloyd

talk to, like, a lot of small companies we talk to where they're like, Oh, this system seems amazing, we want this, but we're on these, like, collection of $200 subs that we're rotating through. I'm like, people are it's probably not

0:31:43
swyx

rotating, like, seventeen different subs, right? I. I And then you can't

0:31:45
Zach Lloyd

can't imagine that's allowed, but I I don't. The truth is, I just don't really know, but, like, they they are, and it's it's like, and it makes, like, honestly, it's like, if you can do that, that's gonna be the cheapest way. But you you, you do lose a whole bunch of other benefits if you do that, uh, and it's not a scalable thing. And if your company scales to a certain point, they're, you know, you're definitely not gonna be allowed to do that.

0:32:08

But it's like, it, it makes sense.

0:32:10
swyx

uh, yeah, I'm just, you know, I'm just saying, like, it, it th- that's the extent of subsidy. Like, you get, uh, for, like, 200 plan, you get 8,000 of API, right?

0:32:18
Zach Lloyd

Yeah, it's crazy.

0:32:19
swyx

That's not that's not money to sneeze at, right? Yeah,

0:32:21
Zach Lloyd

And it's like, it's one of the I I mean, just from, like, a business perspective, it's one of the reasons that, you know, I I don't wanna be in, like, the pure coding agent business, because you you just can't because good luck, build a com-

0:32:34
swyx

uh, good luck fighting.

0:32:35
Zach Lloyd

you, you can't p- you can't build a competitive coding agent on top of a thing that costs so much more money. And, like, that's just the reality of it. I mean, Cursor was brilliant, they did it, and then I think that they had a perfect exit and became a model lab because I think they fully realized the economics of it. And so, yeah, the the cool thing about Warp factories is, like, we're actually not, we're not trying to pitch, like, a token reselling product, we're trying to pitch basically an, uh, you know, a coding agent optimization platform and a coding agent, like, productivity platform.

0:33:08

So we we're we're not in that game with this product, which is a strength, and, like, that game is I. I know there's still a few independent coding agents out there. I don't know how they're gonna how they're gonna do it.

0:33:20
swyx

It's basically cognition and open code, that's it. Uh oh, and and factory, of course. Um, let's wrap up with, like, a little plug for your talk. If people c- will come see you and we'll, we'll livestream it and all all that. But, uh, do give us a general arc of the the sort of series that you've been developing and where, where you where you're going to talk about next.

0:33:37
Zach Lloyd

What the talk's gonna focus on is what it looks like to be an engineer in this world of, uh, increasing factory engineering. And, like, what it what are what are your roles? What's your job gonna be? What are the two types of engineering, product engineering, factory engineering, and then what is this new discipline, if you wanna use the word of factory engineering, actually look like? What are the tools? How do you, as an engineer, build the right skills so that you're engineering the system that builds the software and not just building the software directly?

0:34:06
swyx

That is a perfect recap and overview. If if people wanna talk to you, uh, where should they find you? Uh, if you're hiring, what are you hiring for? Anything that's like a call to action.

0:34:17
Zach Lloyd

Yeah, so come check out warp dot dev. So most stuff is there. Uh, we're hiring engineers, we're hiring on our go-to-market team, we're hiring on the data team, we're hiring a designer, so hiring up for a whole bunch of roles. If you're interested in, like, my my musings on factories, I I put out a lot of content on my Twitter now, so Zach Lloyd tweets, uh, so come come follow that, and I'll I'll keep sharing how I'm thinking about it.

0:34:39
swyx

Will do, will do all right. Thanks so much for the talk. Cool,

0:34:42
Zach Lloyd

Cool, thanks for having me.