Intelligent Orchestration: Better AI Coding and Software Delivery | #CXOTalk #907

AI tools accelerate coding, but software delivery isn't getting faster. Bill Staples, CEO of GitLab, what's the bottleneck?

AI and coding agents like cloud code and codex are accelerating the coding aspect of a developer's job, but it's leading to downstream bottlenecks to overall software delivery. You know, things like code reviews and bug fixing and security checks are all becoming, uh, overwhelmed with code. Teams and, uh, leaders are not seeing an overall increase in innovation velocity. We call that the AI paradox. Fragmentation of in tooling, if you will, causes a developer productivity impact because those tools were not built, you know, by the same company in a in the same platform sharing the same data.

They're obviously disconnected and assembled kind of by hand by individual developers or by the developer experience teams. And that fragmentation causes them to have to switch contexts all day long as they move the software through the lifecycle. It causes them to have to keep the context about that software i in their head.

Dive into how typical developer tools create these obstacles to the software development lifecycle that interfere with developer productivity.

They store their own context within their own data systems and that context you can open that up with mcp. But then you are limited in terms of identity and the corresponding security, uh, potential security issues there. And then you're also limited based on cloud connectivity, you know, bandwidth or network throttling may apply, walled gardens in terms of data access. And what you actually are able to, you know, extract from that vendor's cloud may apply. And so, um, that fragmented context leads to issues.

We also have a trust issue in that, even with great AI tooling, some individuals and teams are reticent to cede control to AI. In some cases, some organizations aren't ready to let certain, you know, aspects of their data, uh, into another cloud or into an AI vendor's, um, control. And without that consistent verification and validation of processes, it's not clear which tasks are well suited for AI, um, and which are gonna continue to require human approval.

There's also regulatory aspects, uh, with AI, you know, uh, there's a growing need, especially in countries outside the US, where there's, uh, heavy data residency requirements and all kinds of new laws are, you know, coming up that are gonna drive governance requirements and standards on, especially financially, you know, regulated, uh, companies to be very careful in how they use AI. And, um, then there's the budget questions, you know, where does the budget for AI come from?

Uh, is obviously often needing to be offset other costs, and, uh, the cost of AI can be very high. And it can be unpredictable because often AI vendors are charging a usage-based model given the underlying cost of service, which then, uh, results in more fragmentation and expansion of budgets. So a all of those kinds of tool, uh uh, challenges come in the way of, um, you know, customers adopting AI across the software lifecycle.

How do AI agents make this software development lifecycle more difficult, more complex?

Agents thrive on context, and so the first thing that they do is, uh, look at the context they have. But, you know, in order to move software from idea all the way to production, a number of a s a series of tasks have to be executed. And without that shared context, the agent may end up creating more bottlenecks down the road. Agents also require a fair amount of steering and coordination. And so if you're, uh, orchestrating agents across multiple tools and multiple vendors, you're also increasing the overhead required by an engineer to do that steering.

There's also been, um, increased, you know, risk in terms of AI proliferation. As teams look at a variety of different tools that are coming onto market, open source and commercial, and bringing those in without it governance, there's a potential cost explosion associated with those. Cause, you know, AI is powered with tokens, and, you know, they may or may not be on a common contract, um, or under cost control by the organization. And last but not least, you know, trust is a real factor here.

Um, you know, do you really trust all of the vendors and the underlying LLM token, um, providers, um, with your organization's data?

So how do we solve the issues created by AI automation in software development?

It's not just a tool solution, it's not about swapping one tool for another or bringing more tools in. It really is a complex people, process and technology challenge. Rethinking the sequential, process-based handoff of the software development life cycle. It's time to shift to a more continuous innovation loop where the agents are able to do many of those handoffs for us. And also, you know, learn and validate against the initial business requirements and technical requirements that were laid out at the beginning and continue to iterate on those on their own to drive the right level of quality.

Um, before, you know, uh, humans need to get involved. It's also important that we address the context challenge, you know, having fragmented tools, each with their own context, as I said, is not going to lead to the best outcomes. And so this is one way that, um, you know, we're excited here at GitLab because we've always taken a platform approach and given customers many different tools across the lifecycle to do all stages like the GitLab orbit service that we're building right now.

It's also really important, as we think about, you know, solving this, that we think about the execution environment. Uh, oftentimes people say, you know, context is king with agents. The more context, the more well-structured context you can give an agent, the better it's able to reason and and make decisions and take action. And, um, that context, uh, is often difficult to stitch together, uh, through mcp, across multiple vendors. And so having an environment where both the compute or the agents, as well as the data, are co-located, we call that an inner loop architecture, where the agents and the data and context are all within one network and one system.

We believe are going to lead to a higher quality and faster and lower cost agentic outcomes. And then finally, super important, especially with AI, that we trust, uh, you know, the actions that are going on. So everything has to be secured, has to be logged and traced so that there's, uh, an ability to s, you know, see. In retrospect, even the reasoning that was done, the actions that were taken by whom and when, in order to have an audit trail and understand, you know, exactly what was changed and why and by whom, including by the agents and the humans that were steering them, right?

And so, uh, that is also a really key part of the solution. And again, it is dramatically easier to do that within a common environment with that inner loop architecture, where both the compute and data are co-located. And you've got a process where instead of having handoffs between humans and tools, you've got kind of one, um, continuous innovation platform that lets you do that inside one cloud. And that's why many teams have found consolidating and going to a strategic vendor relationship, you know, for example, with GitLab, we've, uh, shared with many of our customers.

There's like a s six month payback period on GitLab Ultimate and 483 ROI over three years. When you adapt, uh, adopt the very best of our platform of GitLab Ultimate, and that's a human-driven productivity improvement. We call our approach to solving this intelligent orchestration, and that's really what we're focused on now as company across the platform, including our new duo agent platform.

Can you elaborate on this concept of intelligent orchestration?

When we talk about intelligent orchestration, we're really talking about three things, workflows, context and guardrails. First, when it comes to workflows, I mentioned earlier, you know, the process of software development's gotta change, not just the tooling. We've got to move from beyond this kind of s stage-based process to a continuous innovation loop. And when it comes to AI, we've got to move beyond just the one-to-one AI chat-based experience that are, you know, pretty common now within, uh, IDEs like cursor or with cloud code.

And we're moving towards this multi-agent kind of agent swarming approach, where those agents are working asynchronously on complex tasks. For everything from, you know, taking, uh, a set of product requirements all the way through coding and testing and reviewing to a merge request, um, to things like, you know, security analysis, ongoing, real-time security reviews, code reviews and static analysis, um, to pipeline creation and pipeline monitoring and remediation as pipelines fail. Now, let's talk about context real quick.

Think about, for example, an agent making a code change. Um, historically, you know, in in current generation AI tools has access to that project's source code, but it may not have access, probably doesn't have access to, uh, for example, the issue or the epic that describes the requirements for that code. Uh, may not have access to the security scans of that code. And, you know, yesterday's run that shows there's actually a security issue in the code block I'm looking at, uh, for other reasons, we may not.

The agent may not know which environment this, um, software's going to be built for or where it's going to be deployed. And all of that context could shape and change and drive a higher quality outcome if the agent could take that context into its, um, you know, context window and reason with that with what we're building with the knowledge graph. Because we index the code semantically and we stitch together all of these other pieces of context related to that code, we can hand it over to the LLM in a very efficient way.

That lets it, um, you know, reason about the related entities of that code and get access to the, uh, just the information it needs to make good decisions and take action. In some of our early testing here with GitLab orbit, we've seen, uh, agents 40 more accurate and actually 25 faster, thanks to that well-structured full life cycle context. And then finally, with guardrails, you know, it's really important that the system take into account the standards the organization has set and the re regulatory and compliance requirements that it's under.

And that the agents take on the same accountability, the same responsibility to act within those guardrails that the human engineers do. And so the ability to apply those in one place and have both the humans and the agents working under the same set of rules, uh, that are policy-driven, that are declarative, that are well documented, is a very important part of the platform that we're building. We also think it's really important that the policy gates apply universally and automatically, uh, meaning whether the human wrote the code or whether the AGI agent wrote the code, the same rules apply, the same auditing, the same, you know, governance, um, is, uh, is applied, uh, any anywhere the software is authored or anywhere

it's flowing. And that's done as part of our unified platform approach.

Would it be accurate to say that it's this combination of the process, the platform, and the context that enables you to overcome the fragmentation that you described earlier?

Exactly, yeah, that's a great way of saying it.

Can you give us some real-world examples?

Even within GitLab, we've been building our platform and using it internally, obviously. And, um, you know, when I think about, for example, um, the workflow portion, we have, uh, been having our engineers use the platform for doing things like accelerated, uh, code development. We see engineers who are medium to heavy users actually contributing two to three times more merge requests than those who are light or no users. There's teams who go through and build software and create MRS often run into issues with pipelines failing, and I've heard stories from my own engineering team about, for example, when they use the built-in fix failed pipelines flow that's part of Duo agent platform.

The ability to respond to a pipeline failure can go from hours to minutes. And so again, having the execution and the data in one place, being able to play apply agents across the software lifecycle really can yield some pretty dramatic improvements, um, across the board to get to that full innovation velocity increase that we're looking for.

What are the common threads that drive success among your customers?

First, I see the full spectrum of customers in terms of their readiness and willingness to experiment with AI and to think about how it might, um, truly accelerate innovation across, uh, their organization. And those who are more forward-leaning early adopters, willing to experiment, not just with the tooling, but especially with, you know, changing the processes around software engineering to actually be able to take advantage of AI. Those are the ones that are seeing the early results.

And then second, and this is probably no surprise, but those who have done more intentional efforts around consolidation and aligning on a platform, they simply have much simpler lives in adopting AI. Because they get the benefits of shared context and a more intentional process already defined as part of their human engineering approach, which leads them to a simplified and more intentional agentic approach to software engineering as well. That can be in harmony as the humans and AI collaborate on the same platform-based approach to software engineering.

So, uh, I think being intentional and also really being forward-leaning on AI and thinking about not just the tools, but the process changes required are key factors to success.

How does all of this change a mid-level developer's job and their role?

I think it shifts for developers to go from writing code to really helping define the requirements for that code, how the code needs to be structured, and even more broadly, how the software needs to be architected as part of the overall business. Um, they also need to shift from pure implementation and caring about the syntax of the language and the lines of code to validating that code. And not just validating it technically, but also validating it against the business requirements that were initially set.

I think their execution shifts from really being so tactical about implementing the work items that were assigned to them, to being really more strategic thinkers about the why. You know, why? Why is the business asking to solve this problem? You know, why does the software need to behave this way? Uh, not just the how. It's really a chance for them to move into a much more strategic and creative individual contributor role, where they're an orchestrator using the superpowers of AI agents and coordinating across systems.

Rather than, um, you know, being so narrowly focused on just pieces of a particular project. I think there'll be a lot greater emphasis on, um, correct code, not necessarily writing the code, but actually the ability to read the code and, uh, validate that it's good quality code and meets the business requirement, as well as well-architected services is going to be an important part of the job. So things that maybe more senior managers and architects would do today become a mid-level, um, person's opportunity.

They essentially get this ability to rise in strategic importance and impact in the organization. The ability to test and drive quality through the agentic outcomes is an important part of the role. Because, again, they're thinking about business outcomes that they're creating, not just, you know, a block of code that gets delivered into a project.

This is a golden opportunity for developers to develop strategic skills, to move closer to understanding business needs, to orchestrate systems, and to guide AI agents. What advice do you have for chief information officers and chief technology officers to put these lessons into practice?

If you're a chief information officer managing the shared infrastructure of the organization, obviously one of your really important responsibilities is security and compliance of the organization. Oftentimes, that also involves analyzing or assessing AI, um, as part of of that responsibility. And it's really important to think about how software engineering is going to transform, as I've shared across people, process and technology boundaries. And to support that dramatic transformation while ensuring the security and compliance of the business is intact.

And so that if that, if that means it's part of your purview to provide that to your software engineering teams, then I think looking at a platform-based approach makes a lot of sense. Um, I think being forward-leaning on AI and and setting, you know, the standards of the organization up front so you can unblock adoption is going to be really important. And if you don't have that as part of your purview, but you partner with, say, a CTO or with, um, other parts of the organization to provide the developer experience into them.

It's really important to understand their challenges, uh, that we've talked about here, and support them in that transformation. Uh, I know my CTO and CIO here at, uh, GitLab are working really closely on exactly those set of things.

Bill, for organizations that want to adopt the lessons you've been describing, what are the first steps?

Step number one is audit the current level of investment and tools where there is fragmentation to consider consolidating. I think it's really important that you're measuring end-to-end flow on software delivery, not just, you know, coding velocity, but how long is it taking to go from idea to production? It's also important, as we talked about, to think about security and governance up front. Uh, even, you know, having buying into a system that has AI governance built in with an audit trail and the ability to identify compliance gaps as part of the solution purchase.

You know, cost we identified as a potential challenge, and so obviously the simplicity is key here. The more you can aggregate and consolidate tooling, the better visibility you're going to have into the true cost of AI and ensure that you're getting an ROI on that. You know, a lot of teams like to do pilots. I think it's a great way to try software and kind of get an evaluation quickly on if it's going to solve your, uh, team's needs or not.

Um, I think pilots are a great thing, but e- especially important as you think about consolidation and you think about teams and agents working across the lifecycle that you consider piloting that, uh, because it will require people, process and technology changes. And and doing that in a safe place, uh, outside of production environments is is going to be key. As you think about autonomy and the ability for organizations to move quickly, I would say defining upfront the criteria that you want, uh, you know, people to hold fast to versus the things that are a little bit more freeform and they can make decisions on, uh, will only accelerate the organization's ability to make decisions and move on the AI opportunity.

So those are probably the first things I would focus on.

And are there a few core metrics that folks can use to evaluate their progress on this journey?

Yeah, it's one of the number one questions customers ask me is like, how do I measure, you know, the real value of AI? I put it in a couple of different buckets, you know, first, end-to-end metrics, as I said, not just like lines of code generated from AI or percent of hours saved coding from an AI, but think about overall cycle time. How long does it take to go from idea to production? When you think about merge requests, this is the ability to take code written by AI agents and actually put it into production.

That cycle is really important. So the time from a merge request or a pull request created until it's actually deployed. That ends up being a new bottleneck for a lot of teams. It's important to to understand deployment frequency, how often is code actually making it into production? Uh, that might be another bottleneck worth paying attention to. And then, uh, change failure rate is also a great one to think about, because, you know, it doesn't matter if you're cranking out way more code and getting it into the hand of, uh, customers in your deployment environment if the bugs are increasing.

Um, and so looking at pipeline failures and looking at bug escape rates also super valuable. Uh, the other last thing it's been, I know, I think on lots of teams' minds for a long time, but keep an eye on mean time to recovery. When things do fail in production, how quickly are you able to get up and running again? And then, of course, governance and quality, you know, looking at things like how many security vulnerabilities by severity and the trends there, your compliance audit readiness, um, and so forth.

As well as the final category, business outcomes, you know, are your developers, uh, satisfaction increasing or decreasing? Because developer satisfaction directly correlates to software innovation and velocity, the time saved, uh, across the lifecycle again, end to end, uh, really important to kind of bake back into the capacity of the organization to innovate. And then cost efficiency, you know, is AI your AI cost scaling with the commensurate value that you're getting out of, uh, the investment?

Bill Staples, CEO of GitLab, thank you so much for giving us the inside scoop on using AI effectively in the software development lifecycle.

Thanks so much, Michael, for having me on.