How Snap Built a Production System Where Agents Write 90% of Code

You know, I joined Atlassian right around the time of the SaaS pocalypse, and people are like, Wait, what do you mean? You're going to a SaaS company? And I just never believed the narrative.

AI agents are real and growing, but where does that leave software and work? Tamar Yehoshua is Atlassian's chief product and AI officer after leading product at Google Search and Slack.

You have to build great products that people love, that agents can work with and that are valuable. So s it that that thesis of building something that's valuable, understanding the value is the same. I think we started when we were building products, um, with AI of ooh, this is really cool technology. Let me see what I can do and let me see how I can, what I can improve with the technology. And now people are getting back to the same principles of product management of build a product that that that is great and that works.

But the new twist is people may b be using your product through a coding agent, through cloud coder, co work, through codex, through mcp. So they not always coming to your site, a human being coming and working with your interface, they might be an agent coming and working with it. So that's an interesting twist. So we have, if if you don't mind, I'll take a step back and say kind of our products. So if people are familiar with Atlassian, we do, um, products for teams.

There. Our mission is to unleash the potential of every team, and now that team includes agents, humans and agents. So for each of our products, take, uh, Jira, where you create an issue, um, you collaborate with your teammates on it. You might wanna access that issue from cloud code or codex, or cursor or copilot. And so we wanna make sure that if you're working with a coding agent, you can read and write to that issue. And so that's a big part of making it work for agents that everything that you can do, you can do so.

Confluence, everything that you can do in creating a document, editing a document, adding a table, adding a whiteboard. Nine. We've gotten to the point where 95% of everything you can do in the UI, you can do through MCP or SCLI so that you can access that. So that's, um, that was a very important goal of ours to get to. And then, um, I, and then the stack of what does that mean? That you use AI, and so how do you build products with AI is also super interesting and and very multifaceted.

But I'll pause there.

You described a whole bunch of different aspects. As you're thinking about product design for a person versus product design in an agent environment, what's different, or what gets overlaid onto traditional product design?

An agent needs the context of an organization, an agent doesn't really know they they think of them. Somebody coming into your organization with no background whatsoever, because an agent has access to the intelligence of the foundation models, but doesn't know your organization. So that's a big, big difference of what it when you're when you're talking about AI, we we think of it as what are you people using AI for? They're using it to accelerate their goals.

You c it unlocks the ability to do things you couldn't do before, and it enables you to do things faster. So it's a, um, we think of acceleration as as in, uh, intelligence plus context, so the intelligence comes from the models. So one part is how do you leverage those models? And so we can talk about that as well as how do you leverage the models? And then how do you give it the context that it needs?

But the right context and the context that is for your organization and not too much context that it confuses the the foundation model. So that's a really delicate balance and that's something that's very unique to the agent. And then you have to know when to call it, you need to know how to evaluate it, and you wanna. You also wanna make it cost effective, cause you wanna make sure that by when I say you wanna know, understand what context is going to the models, that also impacts your cost.

Uh, so so it's a. And we're learning as we go, cause as the models get smarter, so the models will use tools and skills to understand how to get their work done. For the like coding agents and for co work and other agents. So you have to be really smart about what you expose, not just the context, but what tools you expose and how you expose them. So, like our MCP and CLIs have over 500 tools from Atlassian that we expose to the agents,

This episode is brought to you by Gartner IT Symposium Expo Ready to scale agentic AI from pilot to production. Join top CIOs and IT execs this October 19th to 22nd in Orlando, Florida. Over 300 Gartner analyst led sessions will cover top priority shaping IT from AI value, governance and cybersecurity to cost optimization, IT operating models and beyond. Get practical insights and connect with peers tackling the same challenges you are. Secure your spot today at gartner.com us symposium.

When you talk about context and not having too much, not having too little, first off, what do you mean by that context? And then let's dive into how do you make those judgment calls? What works about what will work best, both for the AI as well as for the person who's the recipient?

we have something called the teamwork graph, so the teamwork graph is the context graph that we build for each customer for their organization. And just by using our tools, it builds your graph. And it's something that we've had actually before AI, we were using, uh, in our products, and then as AI became more prevalent, we have significantly increased our investment in the teamwork graph. And so think of it as a map of your organization, of your all the people in your organization, the knowledge that you have, the data, the structure data, the code that you have and the communication that you have.

So it has all of this information in the graph, and it has it from the Atlassian products, but also products outside of Atlassian. So we can read your Google Docs or your Sharepoint docs, your email, your Slack messages, your teams messages, we can. We work with Databricks and Snowflake and and Google to get um data out of your structured data, so all, and we use Workday to get your people data. So this graph will then know if you ask our products, if you ask Grovo about what are all the projects your team is working on, it will know what your team is.

Cause we have the hierarchy, you understand it, we it will know the projects that are linked to that team. Cause if you created a Jira ticket or an epic, uh, who created that? Who's working on it? We see the Google docs that you're working on, so we know who wrote the documents, who collaborated on the documents. So this graph tells us so much rich organization about your enterprise. And it's not just let me use MCP to Google Drive and get all the documents that, uh, that that I have interacted with, cause that'll be really vast and large.

So if I'm asking, what are all the projects that my team worked on? It's much more pointed and it can give you the much more relevant information that you need. And that's why it's so powerful, because it does inferences in the graph already that are pre-computed and make it faster. So those that's used inside our products, that's also used by. We have an agent studio where you can build agents within Atlassian, so you could build an agent yourself.

You could also attach agents to your automations, and then you can use this information through our t uh teamwork graph, MCP and CLI in third party coding agents. So we believe in a very open platform, so we take all this information and we give it to you and make it accessible in whatever way that you want. But this is, um, this is like the cornerstone of making making everything more effective with whatever agent you use. We've done a lot of testing, we've done a lot of AB testing.

And let's say, take, uh, cloud code or codex and use it just with MCP, not using our teamwork graph, and then using it with our teamwork graph. And we've shown that it's better quality, it finds the right documents, um, it's the quality. Increased by forty four, and very importantly, the token usage went down by forty eight. Because you don't have to give as much context to the model and it's much more directed, and so we can, uh, help you use less context in your model.

So that's the uh, that's the our teamwork graph. Does that does that make sense?

Yes, and you've described quite a lot, so let's start to unpack it. And we have a bunch of questions that have come in on LinkedIn and Twitter, and so let's go to some questions. And I just want to encourage you guys ask questions you can. When else will you have the chance to ask Tamar? Pretty much whatever you want? Here's a question from Arsalon Khan on Twitter directly relating to this context issue set of issues. And he says, Should AI agents have some sort of quote un onboarding by giving access to certain data that provides context?

Who human or other agents, define how context can be driven from what systems?

The way that the context works in our products is it's all driven by whatever permissions the human has. So if I'm building an agent, I in Rovo studio, it's determined the agent has access to the information that I have access to. So you think of it as an extension of myself in that it's not gonna have access to. If you have a personal, if Michael has a personal conversation with somebody that I can't see, then my agent won't be able to see that.

Now is it building over time? We have the concept of memory, and so, yes, it they, um, our Rovo product under starts to understand me, and there's a it. It has the memory of what I have listened to and what I have done in the past. And then that builds up over time, and so you get it does get smarter and smarter over time as you as it understands you and and what you're doing. Um, and then there's all kinds of, uh, agents that you can build that you like.

We have to the onboarding question, we have onboarding agents for people. Uh, now we haven't built an onboarding agent for an, uh, agent yet. But I think that peop the agents the first time you use them have access to everything. So just directing them to the right thing for that job has proven, uh, has proven effective.

So, in essence, then you could say, would it be correct to say that? There's a kind of routing that takes place of the data that's out there. You have this large body of data and you need to from various systems, and you need to direct the right data to the right place at the time the question is asked.

You think of it, there's this huge graph, our teamwork graph has over 200 billion, uh, entities in it and there are links between it. But then when I'm asking something, the graph is specific to me and what I, what I have access to, and so that's a really important part from the from the access perspective. And then agents can, as I said, use my identity when I build them. And or if I'm even using a third party agent, using a a coding agent or any kind of agent external to, um, external to Atlassian, then it can.

It calls using my identity, so, uh, that's. And we've built in a lot of safeguards around that.

This episode is brought to you by Gartner IT Symposium Expo Ready to scale agentic AI from pilot to production. Join top CIOs and IT execs this October 19th to 22nd in Orlando, Florida. Over 300 Gartner analyst led sessions will cover top priority shaping IT from AI value, governance and cybersecurity to cost optimization, IT operating models and beyond. Get practical insights and connect with peers tackling the same challenges you are. Secure your spot today at gartner.com us symposium.

Let's go to another question, and this is from on LinkedIn from Smale B. And Smale says he'd like to push on this idea. That context management for agents must treat data provenance as a first class constraint, not an afterthought. And he says, in practice, that means auditable data lineage and explicit human in the loop triggers whenever sensitive decisions are involved, so accountability isn't outsourced.

We have a product called Guard in our enterprise offering that does a whole data sensitivity layer. And so you can put rules in there saying if these are certain types of data, always like exclude them, or always have checks and balances. So there is a product that we that we have that does a lot more. Um, I think on the data provenance side and make sure, and the human in the loop is really up to you in what your, in how you configure the agent.

So you can configure agents in Rovo Studio and you determine how to invoke those. So if you a lot of people use them in our automation, so if people don't know our products like Jira and and Jira Service Management and Confluence have automations built into it that they've always had. We run millions of automations every month and they're automations like when a bug comes in of this type, then do this with it. Um, so it's kind of it's workflows for the organization, so a lot of those workflows call agents and you can easily have a step in that, uh, in that workflow, call the agent.

And then a human needs to evaluate it, so it's really up to the person writing the workflow. Or you automatically go through, but you do have accountability and observability. And as I said, you do have the ability to identify and determine sensitive data and make sure that the agent is not, uh, that not, the agent is not working with that.

That's a really important point, this issue of human in the loop. And so it sounds like you're providing a great deal of flexibility to the user of when to invoke stop points, where a human needs to get involved. But many users don't know, you know, they, they, they're not sure either. They're not sure what to do, or they may be tempted to just press the auto button because the LLM seems to know everything and seems to quote unquote, always be right.

And so how do you provide guidance to users on when they need to put in stop points?

So a couple of things, so one, you're right, and a lot of people don't really know what the best way to build the agents are. So we have templates out of the box saying here's some got, here's some, you know, examples of how you should be using them. We also in our agent studio give you the ability to put in eval data, so you can say you can actually eval it within agent studio. To say, is this doing the right thing?

And um, and we have as part of Atlassian, you can work with, uh, FDE or, uh, technical, uh, uh, technical, uh, SE to help you build what you need. So we try and educate people on what they're going for, but the biggest thing is observability. So your admin can see all the agents that are built and they can also give access to who they want. So some companies really want to control it because, to your point, they're not sure if everybody understands how to build the right agent.

So they can say, here's a group of people in IT or in engineering who are going to build the agents for the rest of the company. Because they're trained, we trust them, and then we're going to b expand. Or you can say, anybody in the company can do it, and we're going to watch it. We're going to see what the token spend is, we're going to see, um, what the the value is that's accrued. So what we like to do is again, be an open platform, be flexible, but give you the controls that you as an enterprise need to be able to, like, turn the spigot on or off.

However, you want,

Yeah, I guess you, as the software vendor, need to be, um, capable of handling all kinds of the many different use cases of which you can't possibly know what they will all be in the future.

And we want people to build stuff that we've never imagined. And so we wanna start with. Here's some things that we see everyone using and these are really effective. So, like, let me give you an example just to ground it of an example of an agent that's built on our platform. So, um, take Mercedes-Benz. Mercedes-Benz has been a customer of the Atlassian suite for over a decade. And they use Jira T on their, uh, when they're testing new cars, testing new cars, a defect comes in, the test engineer puts a ticket in Jira.

Well, as you can imagine, there's a lot of duplicate tickets and there's a lot of tickets that may not have all the information that they need. So what they did is they built a robo agent that takes in all of those tickets and auto triages them, um, removes the duplicates, knows who to assign them to, and just gives all the details that you need. And so that triaging agent reduced the manual work for engineers to triage the bugs by eighty five.

And so that's a great example, just to make it concrete of how people are using robo agents with the information from Jira to improve a workflow.

The whole context and the the graph that you're building, it's it's really fascinating. Okay, let's go to another question. This is from Ricardo Ancon and he and this is on LinkedIn, and Ricardo says, How do you see leaders balancing the urgency to implement AI agents with the need to thoughtfully assess which tasks are truly suited for them? Seems like a tough call to make without rushing into mistakes.

I get this question internally all the time for us and what we're doing internally, this balance of moving fast versus actually getting it right. You have to start with what problem you're trying to solve and what your goals are for that problem. We have seen organizations just say, go forth and just use AI, and then people are just prototyping and they're learning. So in some ways, it's very important to do that so people can understand the boundaries of what AI can and can't do.

Cause it's a new technology, so people don't really understand it, but you have to make sure that you're measuring it. So at Atlassian, for example, we've been running a ton of experiments. Now. This is internally not building our products, this is how we use AI internally, and we've set up more structured experiments of goals for each department, for engineering, for finance, for sales. These are processes that we want to automate, and then we run with a team, an internal team, obviously using our tools built on top of Robo.

But um, there's a there's a good report, uh, that we we put out on, uh, we're the day the zero day customer. So that's, I think, a really important point of what you're trying to achieve, as opposed to just go forth and do anything that you want. And we ask people to think about what are the processes where you think you can get a ten x improvement? So this isn't where you want just an incremental ten to twenty improvement, like the Mercedes-Benz example.

Uh, so what we did, like in RD, we're using obviously coding agents for everything, but that only gets you so much improvement. So you wanted to see around what you're doing. So we gave. There was one team working on confluence slides, and we said, how can you reimagine all of the workflows and what would you do? And I'm gonna give one example. There was I've taught, there was like ten different things that they did in that example, but one of them was.

They took the designs from Figma and then the code and said, where are the deltas between the designs and the code? And they used Figma's MCP to automate that gap and coding agents to automatically fix them. So they fixed fourteen bugs in an hour, which it would have taken days. So that's just, uh, giving the again, the concrete example of there was a task and a goal you gave to a team to achieve something, and then it's amazing what they do.

As opposed to saying, just, you know, just use the tools. And to the person's question, I think it was Ricardo. Um, you have to be careful that if you're not measuring what the value was, then you might be going too fast and not being able to see the value. But I think if you set the goals and then you turn and you and you come back to them, whatever the timeline is, month or quarter, it's. You're gonna see which teams made good use, and you're also gonna learn from the teams that didn't hit their goals.

Okay, this is what we needed and why

To what extent do you see people using agents to drive efficiencies faster, cheaper, versus use using agents to accomplish things that they could not otherwise do? Really? So efficiency versus innovation.

it starts with efficiency. So if you take mobile when mobile came around, um, all of the first apps on mobile were websites just put on the phone. And then people started taking advantage of signals like Uber, being able to have the map in your location. So we're in that same phase. So S it first starts with here's a process and we're just gonna make it more efficient. But um, let's take all another example from Confluence, so Confluence launched Uh Remix with Rovo, where you can take on a page, you can highlight some text and use our Rovo AI to remix it into a new format.

Well, that's something you could just never do before. So what we're seeing now is new examples of things that they built that with AI in a way that they couldn't have. So they got much more efficient in how they built it. And they took it from taking months to taking weeks to do it. They had to eval it in a whole new way, invent new ways of evaling. But then they produced a product that customers could use that they couldn't do before, something that they couldn't do before.

And those kind of things really resonate with customers. So I would say we're more in the early innings of seeing those. And even though if you take the foundation models, they're so powerful, and even though I think there's a lot of power in the foundation models to do more of that. I think organizations haven't quite caught up yet. And they're starting with, you know, they started with chatting, that's just kind of the basics and finding information and chatting.

And now they're starting to really change their workflows. Um, in in a way that's completely different than than what they would have done,

Large companies definitely are tend to be risk averse, and their incentives are designed to continue doing what has worked well in the past. And so there is that discontinuity, uh, between relative to, oh, we have this new suite of capability and we should just apply it. No, they're gonna think it through and plan it out. And

you know? While that's true, I'm finding this technological transformation different. In that, every customer I meet with is like, What are you doing at Atlassian? how are you using AI to transform your, your work? And they really see the potential of big change. And it's partly fear driven because they see their competitors doing it and moving faster, and they see everyone launching faster and moving faster. And so there's there is a desire now. They don't always have the ability, but I'm meeting with more AI transformation leaders inside very large companies who are m blazing forward.

And they're we're we're putting out how you can change and use AI in legal, in marketing, in in addition to R D. And we're seeing I'm I'm seeing a much faster uptick, which is really interesting. And and also, like we had in the beginning, people didn't want to turn on our AI cause of all the things you're saying risk this, that. Um, but we showed that, you know, we are enterprise ready and we have things like zero day retention that are really important.

And now we're like, we have huge uptick in usage

among our guests coming up, we have the chief AI officer of Occidental Petroleum, and we have the vice chair of Deloitte. I mean, separate episodes, and we haven't scheduled yet, but we're about to schedule the chief AI officer of Bank of America. So this would be an excellent time to subscribe to the CXO talk newsletter so we can notify you and you can join us. And you can ask questions of all of these people too, so go to cxotalk.

com and subscribe. Do that now, okay? So let's take another question, and this is from Michael Bilar on LinkedIn, who says, How do you look at your budget in terms of SaaS licenses versus tokens?

value. Dr- Where are we getting the value and making sure that as we increase, we, uh, our budget for tokens, that we are seeing the output. Now. This is a question in the industry in general, everyone is asking, How do you measure AI? So the thing that we look at is especially in most of the the cost for us internally is in engineering, like we are using tokens in departments outside of engineering. It's just the usage is small compared to engineering.

And so we look at PRs deployed, uh, PRs deployed per engineer, we look at features developed, and we've seen a huge uptick in PRs and um, and features developed, our our OKRs, are we meeting our OKRs, and then the value that that's bringing to customers. We also have a product called DX. We acquired them last fall. And they measure, um, AI adoption in engineering teams and look at all of these metrics and they help you measure it.

And they also do survey based, um, of like, our engineers feeling more productive, and so that's a tool that engineering leaders can use to measure the output. But this is what we're looking at again, each company. It's early days trying to figure this out, but working with the CFO to say, well, we're sh- delivering more value. Our products are being deployed faster, so we are able to, um, bring more value to our customers.

You're trying to assign quantifiable value metrics to licenses ver- or or to token usage against the benefit that arises from using these tokens. It sounds

100%, and it's not. I don't think of it as tokens versus SaaS. Like we have a lot of SaaS products we use, like Databricks and and Salesforce, and like those we need for the for whatever purposes we're we're using. Now. We're using their AI as well, their AI capabilities. But um, do you still have a need to the? I? I don't feel like it's a trade off between the two, because if you have a need for those products, those those needs don't go away.

Um, at least we haven't found that any of those needs go away. Now there are constantly new products in the AI realm that we are evaluating. So, uh, my team also runs, uh, corp dev and Atlassian Ventures. And so we are constantly looking at what is new in the industry to make sure that we can stay up to date. It not only for building our products, but things that we might want to use. So, like at Ba, Atlassian Ventures might come across a startup doing something new and cool that we should be using internally.

do you ever have the temptation to look at some product that you subscribe to, some SaaS product, and say, you know, we only need a few of these features. And we can just code this, vibe, code this ourselves and have a smaller scale town version, scale down version that's just perfect for us. And then, uh, we don't even have to pay that cost anymore.

I think that works for small companies, I don't think it works for large companies. S So once once you're a company of like thousands of people or even hundreds of people, if you vibe code something, you gotta support it. And if you have all these customers internally, then your job becomes a software vendor. You can't like vibe code it and put it out there and not support it. Like, internally, if my somebody on my team would vibe code something to be used instead of one of the SaaS products.

So if you're a really small company, you have just fewer users, you have fewer needs. But what I'm seeing in the industry is, oh, I could just vibe code Slack, I could just vibe code Jira. These just don't last, cause the people in the company are doing it to just get underwater. And then how do you deal with data protection and how do you deal with SSO? And how do you do? Like, as you grow, there are just so many demands.

We spend a lot of our engineering resources on enterprise readiness and your vibe coded stuff, it's just not secure, it's not enterprise ready. So no, internally at Atlassian, we don't see that at all. We do see a lot of agents being built to automate parts of your job. So, uh, we we see like, uh, agents built for the sales team to help them prepare for sales calls, to help them get approvals and automate them. We see like the finance team agents to close the books.

We see in engineering, like tons, the most agents are in engineering. I gave you the Figma one, but there's like so many examples of PMS are not no longer writing weekly updates, they use agents to write those weekly updates. So that's where we see and people vibe code whole apps to do things, so s. We see a ton of vibe coding internally to help people do their jobs, but they're not. Instead of large SaaS vendors that we that we buy their software,

I take it you are not a believer in the SaaS pocalypse, where agents are going to destroy software companies.

you know, I joined Atlassian right around the time of the SaaS pocalypse. And people are like, Wait, what do you mean? You're going to a SaaS company? And I just never believed the narrative, partially for what I said. And we are so embedded, we have 370,000 customers. We, um, are in eighty five of the Fortune 500 people run, uh, their their tier zero workflows on Atlassian. That's just you can't, you can't substitute that, and we are AI-ifying everything that we do, so all our products, we are, you know, as I said, we're like talking with agents, making sure we can write to agents, we can read to agents.

Making sure that we have a AI native workflows for all of the workflows that go through Atlassian, so we're helping our customers move to being AI native. We we say that we're like the bridge to helping them become AI native, because we're already embedded there. Now, if we did nothing and didn't introduce AI into our products, I would be worried. But I met, I spent a lot of time with the team at Atlassian before I joined.

And like, it's. They built so much, uh, infrastructure and so many pieces, uh, layers of the stack to make sure that they would be relevant for the AI era, which is why I was never worried.

I totally agree. I just don't see people tearing out their enterprise systems anyway. Let's take some more questions. Uh, okay, so this is a very interesting one from Swami Vijayathan and on LinkedIn. And he says, as agents start chaining into other agents, how does that model hold up when no human is realistically reviewing every intermediate step? And who's accountable when that chain breaks? How do you approach this challenge as a product design principle?

I don't think anybody has figured out the answer to this, so I'm not gonna claim to have figured it out. But I can say the principles to your question, the principles are. Whoever built the agent, the agent that starts, is still accountable. There always has to be a human accountable. You can't, you can't say that there isn't, cause I just don't think it works if there isn't. So the C human needs to understand that there's going to be a chain of of agents that are called and should put in the checks and balances to make sure it's not veering off course.

Uh, I do worry about that in, you know, we've we've seen in the industry how this can go wrong, but that's where I think you have to build safeguards. And I also think that what's gonna happen is agents are going to be putting in guardrails for other agents. So you're gonna see agents being the defensive and you're gonna see agents. So let's say you have an agent that's chaining a a lot of steps, you're gonna have agents verifying each step as well.

And we already do that. We use LLM as a judge, so as we're testing new features, we use LLMs to test the features of the LLMs. And we've been using that in the industry for a while. So I think you're gonna just see a lot more of the LLMs, teach um, testing LLMs, and people will learn where to put the safeguards on. But I think of that person, that individual, always has to, uh, has to be responsible.

And I also think that we as software providers need to, and this goes back to your question earlier on, needs need to help companies figure out how to do that and help tell them. Here's where you gotta put this, the safeguards in. And as the models get stronger and smarter, we're all learning what that's gonna look like. Every new model we test thoroughly, and we learn what it's capable of, and we change our harnesses, and we change how we're calling the models based on that.

Last week on CXO talk, we had the head of engineering for Snap that makes Snapchat, and they use they're huge users of agents, uh, very sophisticated internally. And he said that half of their engineering goes into the engineering itself with the agents. And then half of their investment is setting up the guardrails and ensuring accountability and processes for accountability.

100%, um, so if you think of pre AI, uh, if you think of the the software development life cycle, about approximately twenty of your time was spent coding and you had left of code and you have right of code. And so all of that has, like the the coding has shrunk with agents, but the left of code and the right of code has gotten more substantial and is a lot more. Um, it's a lot more complicated.

As the the head of engineering at Snap said, there's more guardrails that you need to do. So, if you think about the number of incidents you would have per PR, let's just assume that that is the same. But now you have so many more PRs, so you're gonna have a lot more incidents. And you have to make sure that that code is quality. So one of the things that we've been doing is we've been investing tools in left of code and right of code.

So I'll take you through a little bit of that. On the left of code is, uh, we just launched what we call the AI planner, so the AI planner is, uh, is what you would do to identify what goes into the coding agent. So we take, we start from the prompt of what you want to build. And then we work on top of our context graph. So we understand the code that's existed before, we understand all your design docs, all the people involved.

And what we do is we take that, uh, um, all that information from the prompt. We create the spec, then we create the technical document, then we automatically create the work items. And it's a nice collaborative environment where you can put in your teammates for that, uh, and then that's what goes to the coding. Then after that, on the right of code, we have the AI reviewer, which uses um, LLMs again, but we have the context of all of your previous incidents.

So we can look at and say this code had these kind of issues before because they're they're documented in Jira. So make sure to test for those and to look for those in your AI reviewer, look for those type of incidents. So, to the person's point from Snap, it gets a lot more complicated to do the AI reviews, um, to do the reviews for AI code that's generated. And you wanna make sure that you're looking at all of the context of previous code and issues.

So we have an AI reviewer, and then likewise, as you deploy, the AI SRE will look at what are the issues and the incidents that you may have had previously in production. So we try and look at the full life cycle from the idea through to the deployment and use LLMs across every aspect of that on top of the context in order to get the best result.

And this is from Eldad Poston Corin, who says, We spent 20 years designing SaaS around the human user. If agents become users too, what part of today's SaaS product architecture becomes obsolete? First,

Obviously, parts of the UI that were very human centric, but I think you still will have people logging in, but you might have them logging in for different purposes. So, uh, I keep on using Jira as an example, but Jira, you had to go in and fill out like these complicated forms of each thing that whoever, the program manager set up Jira, or your Jira admin set the U. You had to fill out. Well, now with AI, you don't have to do that anymore.

So that's how AI is making the U, the user interface, for humans easier, but that's the same for agents. So agents don't have to go in. And you're not gonna like simulate the forms, you're gonna have a much easier way of inputting the data, which will then transform to, um, to users as well, to make it easier. So I again, I don't think the UIs are gonna go away, but I think they're gonna get simpler. And then, on the flip side, humans wanna see what agents are doing.

So the transformation of SaaS has to be that you have visibility into what all the agents are doing. So how do you visualize when a project used to take ten people, fifty people, and now you have agents, and you may have hundreds, maybe thousands of agents running on something? So the whole concept of how you visualize software for agents instead of just humans, that's gonna change as well, and I'm really excited about what that is gonna be.

We're investing a lot in design because we this might be a contrarian view, but we actually feel that the design of products is gonna become so important. Of how do you understand and observe what's going on? And so I don't think it's gonna go away, I think it's gonna become a differentiator.

sounds like there's a a certain amount of experimentation that you're doing on an ongoing basis because the capabilities are evolving so rapidly.

It's amazing how fast it's going. And so we have our teamwork graph that really is invisible to users, it's agents can call it, our products can call it. But we made a website called teamwork graph dot com so that people could visualize it. So you can go and put, you can log in and you can see your teamwork graph. You can see who you're connected to, how you're connected to it. Because we thought it was important, as people rely on this concept, to be able to visualize it.

So that was really it. It's kind of an example of humans need need some need to see it, and so we, that's how how we don't think it's going away. But agents can be so much more powerful and don't need all those layers, and they can just go slurp up a lot more data without having to visualize every single thing.

Let's jump to a question from Tim Crawford. He's a big time CIO influencer, he's been a guest on CXO talk a number of times. And Tim says it's impressive to hear how Tomorrow and team are thinking about efficiency opportunities. He's curious how this is driving toward deriving insights and potentially greater value opportunities.

Sometimes we have them up front, we say, okay, we're gonna be able to launch this faster or take this presence and make it better. And sometimes things like surprise us in in that you you see it as you're as you're going. So, um, so what are some of the insights? Some of the insights is I believe how information is flowing in organizations is gonna change how organizations work. So if you think, uh, pre AI, information flowed through hierarchies, but now with things like Rovo, any person in the company can say, what's the status of an OKR?

can say, what, uh, what are all the issues that happened? If, like, if there's an outage, what was the outage? And why? It doesn't have to flow through hierarchy anymore. And the people whose jobs it were just to pass information back and forth, they can now work on higher leverage activities. So that's one insight that I've had from watching all of this transform. I, for example, no longer have monthly OKR meetings. I have an agent that runs that gives me the updated OKRs, goes through, looks at all the confluence docs, the Jira tickets, the Slack messages, and then says, these are green or these are red.

And then I can say, Oh, I want a meeting to discuss that one that's red. I'm concerned about it, let's dive into it. But it just un to me, it just unlocks so much that I'm not waiting for information that to come up through the hierarchy. And I'm also not wasting people's time to ask them all these questions. I can just find all the information myself, so I think that's one how hierarchies work. And I think all leaders need need to change how they're thinking about how they run organizations and where their points of leverage are.

It's a very interesting topic for me of how leaders need to change how they lead in the era of AI. And I think if they don't, then their organizations will not see efficiencies and speedups, then it'll be just micro improvements. Um, and and it has to not at the only at the top, but all the way through. We are launching so much faster than I've ever seen in my career. Like, and so you have to empower people to to do things without approvals and then have a, uh, have a culture where they will escalate when they need.

So that's my one insight. I'm sure there are lots more, but

We've had a number of guests on this show recently talk about the importance of senior leaders, senior business leaders, having a very solid, I was gonna say deep, but let's say extremely solid understanding of AI and agents. Because you need that understanding in order to know how to take best advantage of the capabilities.

I couldn't agree more. And, uh, you so our CEO, Mike Cannon-Brooks, uh, when OpenClaw came out, got super pilled on OpenClaw and was like building, uh, agents to run his personal life and still does. But that depth challenged all of us. And he came back and said to the executive team, you all need to implement OpenClaw just so that you can see what everyone's talking about. And that depth, um, really influences how we think about what products we're doing and it and we're building in it.

And it sets a bar for the executive team all the way through that. We need to, we need to keep up with it and we need to make sure that we're playing with it. And and if you're not, then you don't really understand how this is gonna change your teams and how they work. And not every team is is moving forward as fast as others. So we measure that, we look at it, we're we're using the carrot, uh, not the stick, but at some point, maybe that transition.

So one thing I I didn't talk about yet is how are how are we transforming internally our organization? One of the tools that's been the most effective for us is something we called AI Builders Week. So small companies can say we're only gonna hire AI native people. If you're a large company, you got a lot of great people that you want to train them. So once a quarter, we take a week out of everything that everyone's doing in product and design, and we, um, we train them on a subject.

So it might be prototyping or evals, or agent building or coding. And so we give training and then we have them do a project and then they showcase the project, kind of like a demo days. But um, and then we're finding that those projects that are built during AI Builders week, people continue to use them. 70% of what's built during Builders Week actually gets used later. So this is a very effective tool that we have on how we're up leveling our organization.

And it's really important that you don't just tell people to use AI, but you help them along the path and you give them the time and the space. So that's one of the biggest complaints I've heard from people is like, Well, I'm so overworked, I don't have time to learn how to use the tools. And so you, you gotta give them that space, and people are so appreciative of that time and space.

This is from David Pellas, Uh, on LinkedIn. He says, Uh, can rovo agents access the metadata in Atlassian's database of all the projects in Jira? Can Rovo, uh, cross project dependencies?

Yes, as long as the P agent has the access to that. So, as I said before, if I built the agent, if I had access to those projects, then yes.

And this is from Babbel Kamere, and Babbel says, How do you reimagine entire business processes rather than one process at a time? So that as a company, you know strategically where you're going.

You have to start by to ga- gaining the skills, and so you may start by one business process. So I think that's okay that you start that, that people understand what AI can do. And then you have to really go back to the your strategy. What is your strategy as a company? What is your north star? what are your differentiators? Where are you gonna be successful? So we put together our strategy and our the pillars of our strategy, and then you need to ladder up to that.

So get the company comfortable with it and then set ambitious north stars and say, okay, how are you gonna get there? In a way that people might not be able to see initially that they can achieve them, but they can.

That's not really an AI issue, is it? That's really one of organizational dynamics and change that.

But that was kind of my point of, uh, in the beginning, is that AI gives you a tool to do this. And that's why you have to learn. You have to first do the the business processes that you've transformed so that you understand what the capabilities are. So one pro, uh, one issue I think people do is they go too large with AI. I'm gonna transform everything, and those usually fail. That's the boil of the ocean.

So you, you gotta do the steps leading up and then you will get to the bigger, the bigger, uh, prize, I believe. But yeah, the bigger prize is just management, leadership.

Here's a question from Puneet Patwari on LinkedIn, and she says, asking as, uh, or he asking as an Atlassian employee, what would be your future direction direction for ecosystem and forge and its potential for users?

I think for for so forge, for people who don't know, it's our marketplace, it's where people build apps on top of Atlassian products. Um, I think it's so critical in the age of AI to enable our third parties and our partners to use the functionality within Atlassian. Um, so that they can build AI workflows for customers and expand our ecosystem. So if you're, uh, if you're building on Forge, you can use, uh, uh, LLMs through our AI gateway, and you can use our agents, and we're already seeing partners do that.

So I think it's a really, really critical and important piece of our strategy to enable our partners to use AI on our platform.

Can you offer advice to CIOs who enlarge organizations who are just looking at this swirl of agentic AI and tokens and investments and budgets and just trying to map a pathway through?

Understanding what you're trying to achieve and but push the boundaries. Uh, I was meeting with a leader at Atlassian the other day that they have automated so many of their processes that they didn't believe that they could automate before. And they did it so fast. And so just make be ambitious in what you're setting, but understand the tools yourself, as we were describing, cause you don't wanna set a goal of something that the tools really aren't capable of.

But I just think that these tools can do more than what people are using. And we see once people build on our ecosystem that they're amazed, they're like, Wow, this works so much better than I anticipated. There's so much more that I can do. So be ambitious, but be educated, that's, I would, I would say, that combination,

So then there's this combination of understanding the capabilities and being willing to push the envelope around what could be possible. Essentially,

exactly, that's what I would say.

you touched on this whole issue of tokenomics and the increasing cost of AI usage. How does that affect your thinking about product design?

So one of the things that we invest a lot of time into is something called the AI gateway. So we do not use one model in our products, we use a gateway that then routes to the model. That's the most appropriate model for the job. That takes into account quality, latency and cost. And so what we've found is that our investment in the AI gateway has been so important. So that we can optimize and not get locked into one model, but optimize our costs.

And so there's there's tons of things where we don't need the most expensive model by default, we can use a lesser model. So that's what, and and I also believe, I mean, the the prices keep coming down like the the frontier model may stay really expensive, but there's so much that we can do in our tools for the previous generation model. So I want to leverage the frontier model so that we can give our customers much more advanced uh functionality.

So I'm super excited when they come out with more functionality and we implement it very fast. And if people are getting the value, we want them to be able to get that value through our product. But we also then wanna make sure that we're optimizing for cost on the things that don't need that. So we put a lot of energy and effort into this.

Okay, and with that, Tamari, Ohshima, thank you so much for taking time to join us. I'm very grateful to you

Thank you for having me.

and everybody who's watching and you guys who ask such great questions, thank you. Before you go, go to cxotalk. com, subscribe to our newsletter so we can notify you of upcoming shows. We want you to participate.