Person
1 episodes · 20 quoted claims
Zach Lloyd says Warp's cost per pull request was about $80 when it first adopted the factory approach, partly because it used Opus where it was not needed.
“when we first started using the factory approach, it was costing us about eighty per PR because we were using, like, Opus, where we shouldn't have.”
Zach Lloyd says Warp tuned its factory until cost per pull request fell to roughly $15 to $20.
“we were able to get the cost per PR down to, like, twenty bucks or fifteen bucks, something like that.”
Zach Lloyd says Warp's pull request throughput rose by about 40 after tuning its factory over the previous month.
“we can measurably improve the throughput of our PRs over the last month. It's gone up by forty.”
Warp defines each software factory in a YAML file that lists the repositories the agents can use, along with integrations and workflow.
“So for us, we basically have a YAML file which says, okay, these are all the repos that you can be using.”
In Warp's model, a company has one software factory for each product surface, such as a mobile app or server app.
“You can orchestrate it, you can measure it. You have one factory per product surface,”
Zach Lloyd says about 60% of Warp's internal factory work starts in Slack.
“the most obvious one is Slack, and, like, we're doing, like, 60% of our work is starting in Slack.”
Zach Lloyd says Warp uses Namespace as its primary agent sandbox provider for compute.
“We use namespace as our primary one.”
Zach Lloyd says Warp tests iOS apps by running a Mac virtual machine where an agent operates the simulator.
“the way that we test, like iOS apps, is you're gonna wanna have the ability to run a MacVM and actually, like, have it run the simulator and click around.”
Zach Lloyd says Warp factories let customers plug in their own agent harness, including Pi or any ACP-compatible harness.
“You can even plug in your own harness, which I think is a pretty big deal. So, you know, you you can orchestrate things like Pi, like I mentioned earlier, and basically any ACP compatible harness.”
In an example benchmark shown by Zach Lloyd, GPT 5-6 Sol was 95% correct at about $6 per task while GLM 5-3 flash was 93% correct at about $2.
“GPT 5-6 Sol here was, like, 95% correct on this task set, but it costs six bucks on average, and GLM 5-3 flash was 93% and cost two bucks.”
Zach Lloyd says factory data can track traditional DORA metrics, including how long it takes to ship something.
“things like, you know, traditional DORA metrics, how long does it take to ship something?”
Zach Lloyd says Warp's evals work as an LLM-as-judge setup in which a chosen judge model grades stored agent runs.
“the eval is, again, think of it as LLM as a judge. So you're picking a a judge model.”
Zach Lloyd says self-improvement agents can open pull requests against Warp's factory definition to improve the factory.
“because you have your whole factory defined as code, and these are coding agents, they can essentially make a pull request against the factory definition to try to improve the factory itself.”
Zach Lloyd gives an example in which Warp's self-improvement agents found 54 triage failures that a sub-agent could have avoided.
“we have fifty four failure cases of this particular form, where it would've been better if our triager had been able to spawn a sub-agent to do its investigation.”
Zach Lloyd compares defining a software factory as code to using Terraform for infrastructure.
“So, again, the analogy is, like, Terraform here, you get the same benefits, but actually more.”
Zach Lloyd says larger companies want different MCP access for different departments such as marketing, sales and engineering.
“So they want things like, what MCPs can the marketing team access versus the sales team versus the engineering team?”
Zach Lloyd says a software factory starts to make sense at roughly five to ten engineers.
“We start seeing that maybe around, like, f- five to ten engineers would maybe be the floor where this makes sense.”
Zach Lloyd names Uber and Ramp as examples of companies that are a poor fit for Warp factories because they build their own systems.
“our our kind of not a good fit customer for this, for instance, would be like Uber or Ramp.”
Zach Lloyd says Cursor made a good exit by becoming a model lab after recognizing the economics of building on third-party models.
“I mean, Cursor was brilliant, they did it, and then I think that they had a perfect exit and became a model lab because I think they fully realized the economics of it.”
Zach Lloyd says factory tooling is used mainly by engineering managers, CTOs and developer-experience or platform teams, not day-to-day engineers.
“they'll often have an internal team that's been trying to, like, it's like the DevEx team or the platform team”