How decision making will change when AI answers are cheap and (too) easy

Support for the show comes from Comcast Business. With advances in AI, cybersecurity is getting more powerful, but so are attacks. In the past twelve months, Comcast Business detected 24.5 billion phishing attempts designed to exploit user behavior, steal credentials and compromise devices. AI is forcing a shift in how organizations defend themselves, demanding not only faster responses, but continuous intelligence. To survive in the current climate, leaders must build a security first culture to equip their team with the tools they need to stay protected.

With Comcast Business, connectivity, intelligence and response work as one orchestrated system. Get better equipped to handle today's cyber threat landscape with the 2026 Comcast Business Cybersecurity Threat Report. Read it today at Comcast Business dot com slash cybersecurity. Cyber confidence starts with Comcast Business. Black Friday is around the corner, your biggest chance to grow sales this year. Don't wait. Realize is Taboola's advertising platform for businesses, letting you launch campaigns on top premium websites beyond search and social.

Add Realize to your media mix and scale this Q4. And as an exclusive offer for podcast listeners, new customers get $700 credit back. Just search Realize Podcast on Google or use the promo code Podcast twenty six. Don't miss out on this limited time offer. Terms apply. Support for the show comes from ServiceNow. AI is moving fast across the enterprise, but without visibility, it's just chaos. Different tools, different models, different teams using AI in completely different ways.

ServiceNow turns that chaos into control. With the AI Control Tower, you see all your AI across the business in one place. What it's doing, what it's done, and what it's about to do, so you stay in control. To put AI to work for people, visit servicenow dot com.

Hey everybody, it's Nilay. So we just had our second kid and I'm gonna be off on parental leave this summer, which is exciting for a lot of reasons. And one of them is that we're gonna have some really fun guest hosts on Decoder who have invited some really interesting guests of their own. My idea here is that there are some great journalists and creators out there who do very similar work to what I try to do on Decoder.

They just do it in different formats and with different constraints. So I asked a bunch of my friends and colleagues to think of the show as a sandbox to play in and to do things and have conversations they wouldn't ordinarily be able to have. I think it's gonna be a lot of fun. First up is Jon Fortt from CNBC, who is co-anchor of Closing Bell overtime and creator and host of the interview show Fortt Knox. And his first guest, Cassie Kozyrkov, was chief decision scientist at Google.

I think that makes her a pretty perfect decoder guest. Here's John.

Hello and welcome to Decoder. This is Jon Fortt, CNBC journalist, co-host of Closing Bell Overtime and creator and host of the Fortt Knox podcast. As you just heard Nilay, I say, I'm stepping in to guest host a few episodes of Decoder this summer while he's out on parental leave, and I'm very excited for what we've been working on. For my first episode of Decoder, a show about how people make decisions, I wanted to talk to an expert.

So I sat down with Cassie Kozyrkov, the CEO and founder of AI consultancy Kozyr and the former chief decision scientist at Google. For a long time, Cassie has studied the ins and outs of decision making, not just decision frameworks, but also deeply understanding the underlying social dynamics, psychology, and even in some cases, the role the human brain plays in how and why we make certain choices. This is an interdisciplinary field Cassie calls decision intelligence that mixes everything from statistics and data science to machine learning.

Her expertise here landed her as a top advisor inside Google, where she spent nearly a decade helping the company make smarter use of data. In recent years, her work has collided head-on with artificial intelligence. As you'll hear Cassie explain, generative AI systems like ChatGPT are making it easier and cheaper than ever to get advice and analysis. But unless you have a clear vision of what it is you're looking for and what values underlie the decisions you make, all you'll get back from AI systems is a lot of messy data.

So Cassie and I really dug into the science behind decision making, how it intersects with what we're seeing in the modern AI industry, and how her work today in AI consulting is helping companies better understand how to use these tools to make smarter decisions that can't just be outsourced to agents or chatbots. I also wanted to know a little bit about Cassie's own decision-making frameworks and how she made some key decisions of her own, like what to pursue in graduate school, why she left academia for Google, and then why she struck out on her own just as the generative AI boom was really starting to kick off.

This is a fun one. I think you're really gonna enjoy it. Okay, decision scientist Cassie Kozyrkov. Here we go. Cassie Kozyrkov, you are the founder and CEO of Kozyr and the former chief decision scientist at Google. Welcome to Decoder. I'm gonna welcome myself to Decoder too, cause this isn't even really this isn't my podcast. I'm just sort of like having fun punching the buttons, but it's gonna be, it's gonna be a lot of fun. Yeah, it's so great to be here with you, John.

And I guess, uh, two of us friends managed to sneak on and take over this podcast. So I'm really excited for the kind of, uh, mischief we'll cause here. Let the mischief begin. So, chief decision scientist at Google, I think starts to frame what it is that you're good at. And we're gonna get into the AI implications and leadership and technology and all that. But first, let's just start with the basics. What's so hard about making decisions?

Depends on the decision. It can be very easy to make a decision. And one of the things that I actually advise people is unless you're a student of decision making, your number one rule should be to try to match the effort you put into the decision with what's at stake in the decision. So, of course, if you're a student, you can go and agonize over, you know, how would I apply a decision theoretic approach to choosing my sandwich at lunch?

But don't be doing that in real life, right? The slow down and think carefully and consider the hard decisions do your best by them is for the the again important ones that will touch your life or more critically, the lives of thousands, millions, billions of other people, which is something that we see with technology that scales. So, are decisions hard? Not all of them.

It sounds like you're saying in part knowing what's at stake is one of the first tough things about making decisions.

And your priorities. So one of the things that I find really fascinating about what AI in the kind of large language model chat bot sense today is doing is it's making answers really cheap. And when answers become cheap, that means that the question becomes really important. Because what used to happen with decision making for again the big thorny data driven decisions is a decision maker might come up with something. And then they would ask their data science team to work on it.

And then by the time that that team has come back with an answer, it's been well, a week if you're lucky, but you know, could be six weeks, could be six months. And in that time, you actually get the opportunity to think about what you've asked, refine what that means to you, and then maybe re-ask it. Have that shower thought. We're like, oh man, I should not have phrased it that way. But today you can go and have AI make the attempt at an answer for you, and you could get an answer really quickly.

And if you're used to just immediately running in the direction of your answer, you won't think as much as you should about. Well, how do I test this if this is actually what I need and what's good for me? And also, what did I actually ask in the first place? What was the the world model, if you like, what were the assumptions that went into this decision? So, uh, yeah, it's all about priorities, it's all about knowing what's important.

E Even before we get there, though, like staying at the very basic level, how do people learn to make decisions? There's the, I I guess, fundamental idea of if you touch a hot stove, you do it once, and then you sort of know not to do that again. But w how does the wiring in our brains sort of work? That that teaches us to be a decision maker and our own develop our own processes for doing it.

Ooh, I didn't know that you were gonna drag my neuroscience degree into this. Uh, it has been a while. Uh, I apologize to any actual today practicing neuroscientists that I'm about to offend. But um, at least when I was in grad school, the models that we had for this were that you would have your dopaminergic midbrain, which would is a a region that's very important for movement. Executing some of what you would think of as the more instinctive or, um, driven by basic rewards, rewards like, uh, sugar, avoidance of pain, those kinds of rewards.

So you have that what you might think of as evolutionarily older structure. This and isn't it fascinating that movement and decision making be similar, similarly, uh, controlled in the brain is a movement, a decision is taking an action, the same thing as a decision. We can get into that. And then there is another structure that is sort of forward in the the prefrontal cortex. T typically your ventral medial and dorsolateral pre prefrontal cortices will be involved in different kinds of what you would think of as effortful or slowed down decisions.

Things to do with, you know, the the difference between choosing a stock because I don't know, you feel like it, you don't even know why. And sitting down and actually running some numbers, doing some research, integrating all of that and having a good, long think, ponder as to what you should do. So, broadly speaking, different regions, broadly speaking, from different evolutionary stages, the prefrontal cortex is a little newer. And you have these systems sometimes in coordinated action, sometimes a little in conflict, involved in decision making.

But what we also really cared about back in those days was moving away from the the cartoonish take that you get in popular science. That you just have one region and it just does this thing, and it only does this thing. Instead, it's an entire network that is constantly taking in inputs and processing them. So, of course, memory would be involved in decision making, and of course, the ability to imagine, which you would think of more as your visual, occipital cortices, that would definitely be involved in some way or other.

So it's a whole thing, it's a whole network s uh, of activations that are implementing human decisions. And to summarize this for you, John, neuroscientists have no idea how we make decisions. So that's that's the funny conclusion, right? What we can do is we can prod and we can pry, and we can get some sense. But at the end of the day, the actual nitty gritty of how humans do it is a mystery. And what's also really funny is humans think they know how they make the decision, but quite often you can plant a decision.

And then, unbeknownst to your participant, we call them, it's a victim, a victim in the study, unbeknownst to them, the decision was made for them all along. It was primed in some way, uh, some kind of inputs got in there. They thought they made a decision. And then afterwards, you ask them, So why did you pick red, not blue? And they will sing you this beautiful song, explaining how it was their grandmother's favorite color or whatever it is.

Meanwhile, the experimenter implanted that. And if you don't believe me, uh, go see a magic show. It's the same principle, right? Stage magicians will plant decisions in their audiences so reliably, otherwise the show wouldn't work. And I'm always fascinated by how seriously we take our human ability to know and understand ourselves and feel like we've got all this agency side by side with professional stage magicians entertaining crowds every day.

But it's it sounds to me like maybe what really drives decisions, and maybe the this motion and movement region of the brain is part of it, is want, what we want. Like when we're babies, when we're toddlers, decisions are, do I get up? Am I hungry? So do I cry? Do I like? It's it's basic stuff that has to do with with mostly physical things, cause we're not intellectuals yet, I guess. And so you, you need to have a want or a goal in some in order for there to be a decision to be made, right?

So that's whether we understand what our real motivation is or not. That's a key ingredient having some kind of goal in decision making.

Well, it depends how you define it. So with all these terms, when you try to study it in the the social and biological sciences, you'll have to take a word which we casually use, however we feel like, like the word decision. And then you have to give it a a little box that makes that definition more concrete. It's just like saying let x equals, right at the top of your page. When you're doing math, you can say let x equals the speed of light.

Now from now on, I'm whenever I write x, it means the speed of light. And then for some other person's paper, let x equals five, and then whenever they write x, it means five. So similarly, we say let decision equal, and then we define it for the purposes. Typically, what decision analysts will say, um, defines a decision the way that they do their let decision equals at the top of their page is they say that it is an irrevocable allocation of resources.

And then it's up to you to think about again how you wanna define what it means for the allocation to be irrevocable and what it means for them to be allocated. Is this an act that a human must make? Is it an act that, uh, a system downstream of a human might make? And what are resources? Are resources just money, or could you think about time, or could you think about opportunity? Like, if I choose to go through this door, well, in this moment, in this universe, right now, I didn't choose to go through that door, and I can't go back.

So, in that sense, absolutely every movement that we make is an irrevocable allocation of resources.

And in companies, it's, you know, do if you're Google, do I buy YouTube or not? I mean, that was a pretty big decision back then. Do I hire this person or that person? You know, if it's a key employee role, that can have a huge impact on whether your company succeeds or fails. Do I invest in AI? Do I adopt this or don't I at this stage?

right, and you can choose how to frame that to make it definitionally irrevocable. Like, if I hire John right now at this point in time, then I'm maybe giving up doing something else, like eating my sandwich instead of going through all the paperwork of hiring John. So I could think that's irrevocable, or I could think of it as, sorry, John. If I hire John, I might be able to fire John tomorrow. And that feels like, uh, and release whatever resources that I cared more about than than time and current opportunity.

And so then I could treat that as that I'm able to have a a two-way door on this decision. So really, it depends on how you wanna frame it, and then the rest will somewhat follow in the math. A a big piece of how we think about decision making in psychology is to actually separate into what the name of the field there is, and that is judgment and decision making. Judgment is separate from decision making. Judgment is where you do all the effort of deciding how to decide.

What does it actually mean for you to be allocating your resources in a way without take-backsies, right? So it's up to the decision maker to to think about that. What are we measuring, what's important, how might we actually wanna approach this decision? Even saying something like this decision should be taken by gut instinct rather than by effortful calculation is part of that judgment process. And then the decision making process that follows, that is just sort of riding the mathematical consequences of whatever judgment setup you made.

Speaking of setup, give me the typical setup. Why do clients hire you? What what kinds of positions are they in where they're like, Okay, we need we need a decision scientist here?

Well, typically, the the big ones are the ones involving AI systems deployment, right? How would you think about solving a problem with AI, right? That's a big decision. Should I even put this AI system in place? I'm potentially gonna have to gut whatever I'm already using. So if I've got some handcrafted system, some software developers have already written for me, and I'm getting reasonably good results from that. Well, I'm not just going to throw AI in there and, you know, hope for the best.

Actually, in some situations you do, cause you wanna say I'm an AI company, and so you wanna default to putting the AI system in unless you get talked out of it. But quite often it's effortful, it's expensive, and we wanna make sure that it is gonna be good enough and right for that company's situation. So how do we think about measuring that, and how do we think about the realities of building it? So it has all the features that we would require in order to want to proceed.

It's a huge decision, this AI decision.

How much do a a leader's or a company's values matter in that assessment?

Incredibly, yeah, I I think that's something that people really miss when it comes to what looks like data or mathy situations. That once we have that bit of math, it kind of looks objective. It looks like you start here, you end up there, and there was only one right answer. What we forget is that that little, uh, math piece and that data piece and that code piece is a thin layer of objectivity in a big fat subjectivity sandwich.

Where that first layer is what's even important enough to automate, what's important enough to do in the first place? What would I want to improve? Which direction do I want to steer my business in? What matters to me? What matters to my customers? How do I want to change the world? These are things without one right answer, and things which will need to be articulated somewhat clearly in order for the rest to make sense.

The companies tend to articulate those things through a mission statement, and very often, at least in my experience, those mission statements aren't nearly detailed enough to guide the sort of granular, uh, and and deep series of events that AI is gonna lead us down. No,

Absolutely, and this is a really important point that that blossoms into the whole topic of how to think about decision delegation. So the first thing a leader needs to realize is that when they are at the very top of the food chain in their in their organization, they do not have the time to be involved in very granular decisions. In fact, most of the job is figuring out how to delegate decision making to everybody else, choosing whom to trust or what to trust.

If we're going to start to delegate to automated systems, and then letting go of that decision. So you don't want to be asking the CEO about nitty gritty topics around, let's say, the cybersecurity pieces of their new shiny new AI system. But what you need to do as an organization is make sure that somebody in the project is thinking about all the components that need to be thought about and that it's all delegated to the right people.

So part of my role, then, is asking a lot of questions about what's important, who can do this, how do we put it all together, and making sure that we're not operating with any blind spots or missing any components.

how ready are clients typically to provide you with that information? Is that a conversation they're used to having?

We've come a long way, but we have, for the longest time, as a civilization working with data, we've been fascinated by just being able to potentially do a thing. We don't know what it's for. But isn't it cool that we can move this data? Isn't it cool that we can pull patterns out of it? Isn't it cool that we can store it at scale or collect it at scale without actually asking ourselves, well, where are we going and how are we gonna use it?

And we are growing out of that painful teething phase where everyone was like, this is fun and let's do it for theory. Kind of like saying, Well, we've invented a wheel, and now we can invent a better wheel, and we can now make it into a tire, and it can have rubber on it, but maybe it's made from carbon fiber or what. I don't know what tires are made from. Now we are moving into okay. This thing enables movement.

Different investments in this thing enable different, let's say, speeds of movement. But where do I want to go? Because if I want to go two yards over, then I don't actually need the car, and I don't need to be fascinated with it for its own sake. Whereas if what I really need to do is I need to be in the adjacent city tomorrow and I don't currently have a car. Well, then we're also not gonna talk about inventing it from scratch by hiring researchers.

We're not gonna think about building it in house. We're gonna ask who can get you something that will get you there on time and on spec. These conversations are new, but this is where we're going. We have to.

We need to take a quick break. We'll be right back.

Support for the show comes from Vanta. As AI adoption grows, so do your company's security risks and requirements. Most compliance tools promise automation, but you're really stuck doing all the work by hand, chasing screenshots, validating controls, and playing catch-up when audit season comes around again. Vanta works differently. Their agentic trust platform is built to scale with you, not slow you down. With over 1,400 automated tests across more than 400 integrations, Vanta collects evidence and monitors your controls year round.

Plus, you can access the Vanta agent everywhere you work, whether your team is in Vanta every day or living in tools like Claude or Cursor. They work around the clock from full program context, so nothing slips through the cracks. The numbers back it up. Vanta delivers 526% ROI over three years and pays for itself in three months. Over 16,000 fast-growing companies like Ramp, Writer and Harvey save time and resources while scaling fast with Vanta.

Learn more at Vanta dot com slash decoder, spelled V A N T A dot com slash decoder. This is advertiser content from Comcast Business.

Hey everybody, welcome back to another edition of Decoder Sessions. This segment features Andrew Melnitsky, the Verge's general manager, and Ivan Shefrin, the executive director for managed security services at Comcast Business. I hope you enjoy this conversation.

Let's talk about artificial intelligence. I want to understand how has AI, um, you know, changed the cybersecurity landscape for you and your team at the attack level?

The reality is it has had a dramatic impact on everything we do. The bad guys, uh, leverage, uh, leverage AI to for malicious and bad purposes. So the only way to, uh, uh, fight fire is with fire, and we so we now use AI just as the attackers are using it to break in more quickly. We have to pivot and use it to defend more quickly at scale.

Comcast Business specifically detected 25.4 billion phishing attempts and 21.9 billion drive-by compromise events over one year. Uh, talk to me a little bit about that data.

Phishing is so popular because it's so easy to do, and AI has accelerated that. Every business should have, uh, software in place to detect phishing attacks, uh, to train their users, and also that now extends to collaboration software, not just email, such as Zoom, such as Slack, such as Teams, um, because that's becoming increasingly prevalent. Everybody expects to get breached now, and that is not the attitude we use at Comcast. We we are take protecting our customer data extremely seriously.

Support for the show comes from Granola. Have you ever left a meeting thinking that was productive? And then ten minutes later, when it's time to take next steps, you're at a loss. Or worse, you've been heads down the entire meeting, furiously typing notes instead of actually listening, and now your brain is fried. Meetings are a mess. Granola fixes that. Granola is an AI-powered notepad that makes your meeting notes useful. It capture what happens in your meetings and turns it into clean, structured notes with decisions and action items pulled out and made easy to find.

And the best part, Granola works through your device's audio, which means it integrates seamlessly into the video conferencing tools you already use. No setup, no awkward bots. It's just your normal meeting with superpowers. You get to actually listen instead of frantically typing every word, and still walk away knowing exactly what was decided, who's doing what, and what comes next. Once you try it on your first meeting, it's hard to go without. You could try it totally free.

Just head to granola dot ai slash decoder. That's granola dot ai slash decoder to get your time back. Go to granola dot ai slash decoder and try it on your next call.

We're back with Cassie Kozyrkov, discussing how AI fits her work studying the science of decision making. It it sounds like, correct me if I'm wrong here, AI is gonna help us a lot more with giving us facts and options and less with giving us values and goals.

That is a hope, because when you take values and goals from AI, what you're doing is you're taking a sort of average from the internet. Or perhaps in a system that, um, has a little bit more a logic running on top of it to to direct its output, then you might be taking those values and goals from the engineers who designed that system. So I, it's kind of like saying, if I'm going to use AI as my rough draft every time, that rough draft might be a little bit less me and a little bit more the average soup of culture.

And if everyone starts doing that, it's, uh, it's certainly a kind of blanding or averaging of our insights. Perhaps you want that, but I think that there's still a lot of value in having people who are close to their problem areas, who are close to their business, who have individual expertise. Thinking a little bit before they begin and really framing what the question is, rather than taking the question from the AI system. So, John, how this would go for you, let's say, is you might ask an AI system, how do I live the best possible life?

And it's gonna give you an answer, and that answer is not gonna fit you. That's the thing. It's gonna fit the average Joe. What is or who is the average Joe, and how does that apply to you?

For example, it's gonna go to Instagram and it's gonna look at who's got the most likes and followers, decide that those people have the best lives, and then take the attributes of those people, how they look, how they talk, the the level of education that they say they have, whatever. And say, Well, here's what you need to do to be like these people who the data tells us people think have the best lives. Is that kind of a version of what you mean?

Something like that, more convoluted, be because something that is worth realizing is that what machines have over us as an advantage is memory and attention, right? And what I mean by this is if I flash fifty digits on screen right now, a and then I ask you to recall it, you're gonna have no idea. Then I can kind of go back to those fifty and be like, Ah, the machine remembered it for us this whole time.

It is clearly better at memory than John is. Then we flash these things. I say, quick, what's the sum of these digits? Again, difficult for you, easy for a machine. And so anything that kind of fits in our heads as we discuss it, is going to be a shortcut of what's actually possible when you have memory and attention at scale. In other words, we've described this Instagram process that fits in our heads right now. But you should expect that whatever is actually going on with these systems is just too big for us to hold in there.

So, sure, Instagram and sure, some other sources, and probably some websites about how to live a good life and even applied philosophy, all kinds of things, all jumbled together. Something too complicated for us to understand what it is, but the important thing is not tailored to us specifically, not without us putting in quite a lot of effort to feed in the information required for that tailoring, which I encourage that we do. I certainly understanding that advice is cheaper than ever, will frame up whatever is interesting to me, give it to the system, of course, removing the the most confidential things.

But I've asked all kinds of things about how I might, let's say, improve looking at real estate for me, with my particular situation and my particular tastes, very different answer from if I just say, well, how do I invest? And then I've even improved silly things like I discovered that I, uh, tie my shoelaces too tight. I had no idea. Thank you, AI. I now have better technique for feet that are less sore.

Wait, did you discover through AI that you tie your shoelaces too tight?

Yeah, I went, I went debugging. I I wanted to try to figure out why my feet are sore, and so, uh, helped me diagnose this. I gave a lot of information about me, about when my feet are sore, what I'm doing at the time, what shoes I'm wearing, right? And we got through some little debugging process. Okay, first thing we'll try is use a different shoelace tying technique from the one that I have, uh, change which loops and then loosen it a little bit.

I'm like, Wow, now my feet don't hurt. How awesome. So whatever it is that's bugging you, you could go and try to debug it a little bit with the AI system and just see what you get. Maybe it's useful, maybe it isn't. But if you simply give it nothing and say something like, how do I be as healthy as possible? You'll probably not get any information about what to do with your shoelaces, right? You're just gonna get something from very averaged out, smoothed out soup.

So, in order to get something useful, you have to bring something to the table. You have to know what's important to you, you have to know what you're trying to achieve. Sometimes because your feet hurt right now, it's important to you right now, and you're kind of reacting the way that I was. I probably wouldn't ask any proactive questions about my shoelaces, but sometimes what really helps is stepping back and saying, Well, what is there in my life right now that could be better?

And then why not ask for advice? AI makes advice cheaper than ever before. That's the big revolution. It also helps with all kinds of nuanced advice, like pulling out some of your decision framing. Help me frame my ideas, help me ask myself the questions that would be important for getting through some or other decision.

Where are most people making the biggest mistakes, or where do they have the biggest blind spots when it comes to decision making? I is it on asking the right questions? Is it, uh, deciding what they want? What would you say it is?

One is not getting in touch with their priorities again when you're not in touch with your priorities. Anyone's advice, even the best person, can give you advice which will be bad for you. And this is something that also applies to the AI sphere. If we aren't in touch with what we need and want, and we just ask the uh, the soup to give us back some, you know, average first draft, and then we follow it to a t.

What are the chances it actually fits us

Let me put a specific on this, cause I'm I'm a parent of a soon to be seventeen year old, second semester junior in high school, who's getting ready to apply to colleges. And this is one of the first major decisions that young people make, and it's two sided, which is really fraught because you're deciding where to apply and the schools are deciding who to let in. So it's you know, it seems like that applies here too, because some some people are gonna apply to a school because their parents went there, or because it's Ivy League, or because it's not Ivy League.

I don't know. Uh, so through that framing, you talk about the types of mistakes that people make from the perspective of a of a of a high schooler applying to college.

I'm gonna always keep trying to play this game of tying this back a little bit to what we can learn about our own interactions within LLM. Because I think that's helpful for people as well in this brave new world of how do we use these tools? So we have three stages, approximately of you have to figure out what's worth asking, right, and and what's worth doing. Then you need some advice or technical help or something, some execution bit, which might be you, might be the LLM, might be your dad giving you great advice.

And then when you receive the advice, you have some moment where you evaluate, is this actually good for me? Do I follow this? Is it good advice? Is it bad advice? Do I implement it? Do I execute it? Right? These kind of three stages. And so the first one, the least comfortable one, is asking yourself, Well, how do I actually frame what I'm asking? So to apply it specifically to your kid, it would be what is the purpose of college for me?

Why am I even asking this question? What am I imagining? Are some things I might get out of this college versus that college? What would make them different for me? What are my priorities? Why are these priorities my priorities? These are things where if you are not in tune with your answers to them, what you will do is you will receive advice from wherever, from culture, from the internet, from your dad. And you are likely to end up doing what is good for them rather than what's good for you, right?

From not asking yourself enough.

Like the magician scenario, they feed you an answer subconsciously, and you end up spitting that back without even realizing it's not what you really wanted.

Or consciously, you could have, you know, your dad might say, as my dad did, economics is a really interesting and cool thing to study. And this kind of went into my head when I was maybe thirteen and kept knocking around in there. And so that's how I found myself in economics classes and ended up majoring in economics at the University of Chicago. Actually, it's it's not always true that what your parents put in there makes its way out, of course, cause both of my parents were physicists.

And, uh, I very quickly discovered that I wanted nothing to do with physics, um, because of the constant parental, you know, you should do better in physics, and you should take more physics classes. And then, of course, after I rebelled in college, I ended up in grad school taking physics in my neuroscience program. So there you go, comes around full circle. But the point is that you have to know what you want, what's important to you, and really be in touch with this.

So that you're not pushed around by other people's advice. And even the best advice, this is important, even the best advice could be bad for you. So when you think someone is competent and capable, and so I should absolutely take their advice, that's a mistake. Because if what's important to them is not what's important to you, and you haven't communicated clearly to them, or they don't have your best interests at heart, this intelligent advice is going to lead you off a cliff.

And so that brings me to the AI setting, right? With AI, it could be a performant system, but if you haven't given it the context to help you, it's not gonna help you.

AI presents itself as very competent and very certain that it's correct, with very little variation that I've seen based on the out actual output. Like it, it's not saying, eh? I'm not totally sure, but I think this when it's about to hallucinate versus, oh, here's the answer when it's absolutely right. It's it's sure, like almost 100% of the time.

so that's a design choice. Whenever you have actual probabilistic stages in your AI output, you can instead surface something to do with confidence, and this is achievable in many different ways. For some models, even some of the basic models, what happens there is you get a probability first, and then that converts into action or output that the user sees. For other situations, you could, for example, in the back end, you could run that system multiple times, and you could see, let's say you you were to ask, what is two plus two?

And then in the back end, you could run this, let's say, a hundred times, and you discover that ninety nine out of a hundred times the answer comes back with a four in it, right? You could show then some kind of confidence around this being at least what the cultural soup thinks the answer is, right? Let's say, what is the capital of Australia? If the cultural soup says over and over that it's Melbourne, which it isn't, or that it's Sydney, which it also isn't.

For those for whom that's a surprise, it's Canberra is the right answer. But if enough of the cultural soup says Sydney, and we're only sourcing from the cultural soup, we're not kicking in some extra logic to go specifically to Wikipedia and only draw from that. You would get the wrong answer with high confidence, but it would be possible to score that confidence. And in situations where the cultural soup isn't so sure of it, then you would have a variety of different responses coming back, being averaged.

And then you could say, well, the thing I'm showing you right now is only showing up in 20% of cases, or 10% of cases. Or you could even give a breakdown. This is the modal answer, the most common answer, and then these are some answers that also show up. Not to do this is very much a user experience design decision, plus a compute and hardware decision.

It's also a cultural issue, isn't it?

It's a cultural issue.

It seems to me like in the US, and maybe this is true of a lot of Western cultures, we value confidence and we value certainty even more sometimes than we value correctness. Like there's this culture in business where we sort of expect right down to the moment when a company fails for the CEO to say, I'm really confident that we're gonna make this work. And you know, cause people wanna follow somebody who's confident. And then, like the next day, they say, Ah, well, I failed.

It didn't work out. And we kind of accept, oh, well, they gave it their best and they were really comp. Same in sports, right? The team's down, uh, three games to one in a best of seven series, and the t team that's only got one win. They're like, Oh, we're really confident we can win. Well, really, the statistics say you're probably not gonna win, but we know that they have to be confident if they're gonna have any chance.

So we sort of accept that. And in a way, haven't we created that AI in our own image?

We've certainly created the AI in our own image. There's a lot of user experience design that goes into that, but I don't think it's a it's an inevitable thing. Uh, I know that on the one hand, there is this concept of the fluency heuristic. So a system or a person that appears more fluent, less hesitation, less uncertainty, is perceived to be more trustworthy. This is, um, research done, it's it's old old research in psychology. Now you see that the fluency heuristic is absolutely hackable.

Because if you forget that you're dealing with a computer system that has some advantages, like memory, attention, and well, fluency. So you could just rattle off a bunch of nonsense you don't understand very quickly. That lands on the user or the um listener as competence, and so should be more trustworthy. So our fluency heuristic is absolutely hackable by machine systems. It's much harder for me to hack it as a human, though we do have bullshit artists who manage it very well.

But it's very difficult to speak fluently on a topic that you have no idea about, and you don't know how any of the words go together. And that only works if that's the blind leading the blind, where no one else in the room also knows how any of it works. But on the other hand, I'll say, at least for me, I think it has helped me in my career to form a, uh, reputation that, well, I say it like it is, and so I'm not going to pretend I don't know a thing when I don't know it.

I even you asked me about neuroscience, I told you, Ah, it's been a long time since my my graduate degree, you know, maybe we should adjust what I'm saying, right? I'm I do that. That is not for all markets, let's just say. Many would think she has no idea what she's talking about. Maybe we shouldn't, uh, do business with her. But for sure, there is value, and I I've definitely found it's helped me to become a sort of battle tested trustworthy.

Now, that said, when it comes to designing AI systems, that stuttering lack of confidence would not be a great user experience. But similarly, some of the things that I talked about here would be expensive compute-wise. So what I see a lot in the AI industry is that we have business people thinking that something is not technologically possible because it is not being given to users and particularly given to users at scale or even offered to businesses.

Quite often, it is very much technologically possible. It is just not profitable to give that feature. There is no good business case, there's no sign that users will respond to it, that will make it worth it. So when I'm talking about running something a hundred times and then, uh, outputting something like a confidence score, also you would you would have some decision making around. Is it a hundred? is it ten? is it a thousand? And this depends on, um, a slew of factors, which of course we could get into if that's the kind of problem you as a business are solving.

But when you just look at it at on the surface, I'm saying essentially a hundred times more compute, right? Run this thing a hundred times instead of once, and for what? Will the users respond to it? Will the business care about it? So, yeah, frequently, yeah, you'd be amazed at what's already possible. Agents like Operator, the Claude Computer Use, Project Mariner, all these things, they are underperforming relative to where they could be performing on purpose because it is expensive to run them well.

So, uh, yeah, it will be very exciting when, uh, businesses and users are ready to pay more for these capabilities.

We need to take a quick break. We'll be right back.

This holiday, switch to the Simplicity Plan from Verizon. It's one plan, one price, just $30 a line per month for you or the whole family. And on their best five G network. Switching to Verizon and shopping their best deals makes the holidays simple. Head to Verizon dot com or your nearest Verizon store today. Support for the show comes from CrowdStrike. AI is already fundamentally changing how companies build products and run their businesses. If you listen to the show, you probably already know this.

But while AI drives innovation, every new model and agent also creates new cybersecurity vulnerabilities. That's why CrowdStrike built AI detection and response, or AI

IDR. AIDR helps organizations discover where AI is being used, monitors AI activity at runtime, and helps teams detect, investigate and respond to threats involving AI systems. As companies adopt AI, security has to be built in from the start. More than 70% of the Fortune one hundred trust CrowdStrike to protect their businesses. Now, CrowdStrike is helping secure the AI era. You can learn more about AI detection and response at CrowdStrike dot com. Support for the show comes from Outshift, Cisco's incubation engine for frontier technology.

You can find multi-agent AI in every enterprise roadmap right now. But here's the problem everyone's facing. Agents can pass messages, but can't think together. These systems silently underperform through cognitive failures, misreadings, unverified claims, false consensus, and conventional monitoring can't detect them. Outshift by Cisco is building the fix with their new comprehensive resource, the Internet of Cognition. It's an open source guide that's a foundation for building multi-agent systems from design to production. The goal is to give agents shared context, shared memory, and guardrails, all to drive the results we actually expect from AI.

You can lean on this resource to understand agentic AI's evolution and how it shifted from isolated agents to orchestrated superintelligence. You can gain insights for how to future-proof your business by tackling complex challenges with multi-agent systems. You can read the paper, experience the demo, grab the code from Outshift dot com. That's O-U-T-S-H-I-F-T dot com.

We're back with decision scientist Cassie Kozyrkov, switching gears to her time at Google and some of her key career decisions. Back me up now because you left Google about two years ago, a little less than that. You were there for about ten. It's long before the, uh, open AI, chat GPT wave of AI enthusiasm had swept across the globe. But you were working on some of this stuff. So I want to understand both the work at Google and what led you there.

I think you mentioned that your dad first mentioned economics to you when you were thirteen, and that sounds really young, but I think you started college a couple years later, right? So it actually wasn't. You were actually on your way to, uh, to go to studies at the time. What made you decide to go to college that early and what was motivating you?

Yep, yep, I was, yeah.

One of the things we don't, uh, talk about enough is that knowing what motivates someone is that tells you more about them than pretty much anything else could. Because if you're just observing the outcomes and you're having to make your own inferences about how they got there, what they did, why they did it. Particularly with survivorship bias, it might look like they're such total heroes. Then you look at their actual decision process, and that may tell you something very different.

Or you may think someone's not very successful without realizing that they are optimizing for a very different thing from you. Uh, this is all a very long way of saying that I'm glad we're friends, John, cause I'll, I'll go for it. But, uh, but it's always just such a private question. But yeah, why did I go to college so young? Honestly, it was because I had skipped grades in elementary school. The reason I skipped grades in elementary school was I came home.

I was now, you know something, and informed my my mother that I wanted to do this. I cannot remember why. For the life of me, I don't know. I just I was doing something on a nine-year-old whim. And skipping grades wasn't a done thing in South Africa where I was growing up. So my parents had to really battle with the school and even the Department of Education to have it be allowed. So there I was getting to high school at twelve, as one does.

I actually really enjoyed being younger. Okay, you get bullied a little, but but I enjoyed it, and I enjoyed seeing that you could learn a lot.

And I, I wasn't intellectualizing it the way I am right now, but you could learn a lot from people who are older than you, cause they kind of push you. And I, I'm a huge believer in just the act of being surrounded by people who will push you, which is maybe my biggest argument for why college still makes sense in the AI era. Just go be in a place where everyone's on a journey of self-improvement. So I had, in learning this, ended up making friends with twelfth graders when I was thirteen.

And then at fourteen, they are all out already and in college. And I'm s. I had spent most of my time with these older kids, and now I'm stuck and I basically want my friends back. And so that is why I went so young. It was 100% just a a teenager being driven by being a social animal and wanting to be around my peer group. But be be fair to yourself too. It sounds like you just wanted to see how fast the car could go, right?

That's part of what it was at nine is that you realized that you were capable of bigger challenges than the ones you had been given. So you were kind of like, Well, let's see, right? And so, and then you went and you saw, and you were actually able to to handle that, you know, the intellectual part. And people probably said, Oh, but the social part will be hard. Hey, I got friends who are seniors, that part's working too.

Well, let's see if I can actually drive this at college speed. That was part of it, right?

I am so easy to manipulate with the words you can't do X, so easy to manipulate. I'm like, No, let me show you. I love a challenge. Let's let's get this thing done. So, yeah, I think you're right. I think you're right in your assessment.

And so then you you went on, uh, into graduate work, University of Chicago, and then beyond neuroscience, some economics in there. Um,

Uh, so so I actually went to Duke for neuro neuroeconomics, that was the the field neuroeconomics, and, uh, that is, you know, you know, you get macroeconomics and microeconomics. Well, this was like nanopicoeconomics, right? This is how the brain implements decision making. And so, of course, uh, courses involve, uh, around experimental microeconomics were part of it, but this was from the psychology and neuroscience departments. And so, um, it's technically a graduate degree in psychology and neuroscience, with a focus on the neuroscience of decision making, which is called neuroeconomics.

And then I also went to grad school twice, which is definitive proof that I am a bad decision maker. Uh, if anyone was going to think that I I personally am a good one, I've just got the technique, folks, I'll advise you. But I went to grad school twice. No, I'm just kidding. It was good for me to go to grad school twice, and my second one was for mathematical statistics. My undergraduate work was econ and statistics, so then I I went to math stat, where I did a lot of, um, what we called back then machine learning, what we would call today AI.

how many PhDs were involved there?

No PhDs were harmed in the making of this person.

Okay, so, but study on on both of those, uh, disciplines, and then what were you gonna do with it?

So coming back to college, where I was taking courses around decision making, despite having the majors of economics and statistics, I got a taste for this. So I'll tell you why I was in the stats major, the stats major. Because at about age eight or nine, just before this jumping of grades, I discovered the most beautiful thing in the world, which everybody knows is spreadsheets. That was for me the most gorgeous thing. Maybe it's the librarian urge to put, you know, order onto chaos.

So I had this gemstone collection. Its entire purpose is to give me another row for my spreadsheet. That was the whole thing. I get an amethyst. I could be like, Oh, it is purple, and how hard is it? And it's translucent, and ha. I still find, though I have no business doing it, that the act of data entry with a nice glass of wine is just such a soothing thing to do. So I have been playing with data.

I once you start collecting it, you also find that you start manipulating it. You start to have these urges like, Oh, I wonder if I could get the data of all my files on my computer all into a spreadsheet. Well, let me figure out how to do that, and then you learn a little bit of coding. And yeah, so I just got all these data skills for free, and I thought data was really pretty. So I thought stats would be my easy A.

Little did I know that it's actually philosophy, and um, the philosophy bits are are always the bits that should kick your butt, or you're missing the point. But of course, the manipulating the data bits super duper easy. Statistics, as I realized as I began to soak in the philosophy, is the discipline of changing your mind under uncertainty. Economics is the discipline of, uh, scarcity and the allocation of scarce resources, and even if money is not scarce, something is scarce, you know, people are mortal, time is scarce.

So how are you going to make allocations of what you might call decisions? Got in there through economics, changing your mind. What is your mindset to what actions are on the table? What would it take to talk you out of it? Got in there through, uh, uh, what is it? Statistics? Yes, I, I know what I'm saying, and then how does this actually work in the human animal? And how could it work better? Came in through the psychology and neuroscience side of things, so I was studying decision making from every perspective, and I was hoarding.

So here is what did I know, what career I was gonna have. I was actively discouraged from doing this. Now, is at the University of Chicago, even that liberal arts place, my um undergraduate, um college advisor said, I have no idea what job you think you're gonna get with all this stuff. And I said, that's okay, I'm, I'm learning, I'm, I think it's this is the kind of important, and I hadn't articulated back then, what I'll say now, which is data is pretty, but there's no why in data.

The why comes from the decision maker, right? The purpose has to come from people. It's either your own purpose or the purpose of the people that you represent. And that why, that why gives direction to all the rest of it. So just studying data where it feels like there's a right answer, because the professors set the problem up so that there's a right answer. If they had set it up differently, there could have been different answers, and realizing that the setup is infinite choices.

That is what gives data, its why and its meaning, that is the decision piece, and that's the most important thing I think any of us could spend our time on, though we all, uh, we all do spend our time on it and do approach it from different lenses.

So then why Google, and why did you promise yourself you wouldn't work for a company for any more than ten years?

Uh, now we're we're really getting into all the things. So Google is a funny one. And I now I'll, uh, I'll definitely say some things that I don't think I've said on on any podcasts, but the true story of that is that I was in the math stat Ph D program, and what I didn't know was that my advisor, this was at North Carolina State, my advisor had just taken an offer at Berkeley, where he could not bring any of his students along with him.

Uh oh, right, that's pretty bad, pretty bad, uh, mid PhD thing. Now, separate from this going on that I had no idea about. I take Halloween pretty seriously. It's my thing. Um, uh, at Kozyr, it's a a a holiday, so we call it a, so that people can Halloween properly if they want to. And I had come on Halloween morning, dressed as a punch card, as one does with proper Fortran, to print happy Halloween, as one does, and a Googler was giving a talk.

I and I was sitting in that audience, the only person in costume, because everyone else is lame. Um, let that go on the record. My former classmates, you too, should have been in costume, but we can still be friends. And so at nine a m, I'm dressed like this. The Google lady is talking to the head of department. It's like, who's that grad student who's dressed as a punch card? The head of department, not having seen me, still said, Oh, that's probably Cassie.

Last year she was dressed as a sigma field, um, just from measure, theory. So uh, I was being, uh, a huge nerd. The Googler thought culture fit 100%. Um, let's get her application in. And so the application was just for a summer internship, which seemed like a harmless thing to do. Sure, let's try it. It's an adventure. It's Google, as I was signing up for it, my advisor was like, this is a very good thing for you.

You shouldn't even hesitate. You shouldn't, you should, dad, don't be asking me if I want you here doing summer research. Definitely go to Google, you can finish your PhD there, go to Google, and the rest is history. So, um, a much, much better option than, um, having to to restart and refigure out with a new advisor.

How did you end up becoming this translator between the the data people and the decision makers?

The the role that I ended up getting at Google, it's formal at the during the internship, um, the formal internship name was decision support intern, and I thought to myself, we'll figure out the support and we'll figure out the intern, but decision, this is what I've been training for my whole life. The team that I was in was is sort of a a SWAT team for data driven decision making. They were very, very close to the revenue, to Google's primary revenue, so this is a no messing around team of statisticians that calls itself decision support.

It was hardcore statistics, flavored with data science, and it also had a very hardcore engineering group, very big group. I learned a lot in there and I also applied potentially to stay in the same group for a full time role with the the strong prompting of my PhD advisor. I thought I was gonna join that group and then a completely tangential thing happened, which is that I took a a weekend in New York City before going to Mountain View, which is where I'd picked out my apartment.

I thought I was gonna join this group. I was really, really excited to be surrounded by deep experts in what I cared about. These experts were actually working more on the data side of things because it's so regimented and what the decisions are and how we approach them is so regimented in that part of Google. But I took this trip to New York City, and I realized, and this was one of the biggest, like, gut punch moments, decision making moments for me.

I realized I'm making a terrible mistake that if I go there, I will just not enjoy my life as much as if I go to New York City, right? So that that was, there was so much instinct, there was so much. Oh, no, I should actually really re-evaluate what I'm doing. Am I gonna enjoy living in Mountain View? I was just so set on getting the offer that I hadn't done. What I really should have done, which is to evaluate my priorities properly, and so New York City was a massive change for me.

Uh, so, um, the first thing I did was I called the recruiter and I said, whoa whoa, can I get a role in New York City instead? It doesn't matter which team is there something we can find for me to do here. So I joined the New York office instead. Very, very different projects, very, very different group, and there I realized that not all of Google has this regimented approach to decision making. There is so much translation, even at a place like Google, that's necessary for products that are less close to the revenue stream.

So then there has to be a lot more conversation about why and how do we do our resource allocation? And who's even in charge here? Things that you know, when you're, when you're moving billions around at the click of a mouse, you you tend to have those questions answered. But in these other parts of Google, so much more color in how you could approach it, and such a big chasm between the people tasked with that and any of the data or engineering or data science efforts we might have, and so to really try to fill that gap, try to put a bridge on it so that things could be useful.

I would work way more than my formal job said that I should, trying to build infrastructure. I I built early statistical consulting because that wasn't there like you, you couldn't just go ask a statistician who'd sit down with you and and talk through what your project was gonna be. So I convinced people to offer their 20% time. Uh, stats people by specialization to to offer their support on projects that were not their own project, put some structure to this, made resources and courses for decision makers for how to think about dealing with data folk, really like, try to bring these these two areas together.

Eventually it became my job, but for the longest time it wasn't. Sometimes there were questions about, um, what are you? Who are you? Why are you actually doing what you're doing? But just seeing that that things could be more effective, and kinder to the experts who are gonna work on poorly specified problems. If you specify the problems well first, uh, was motivating, so that's why I did it.

Trying to tie this all together, it sounds like that values and goals piece and the philosophy element that you talked about in school, being important coming back into play versus just focusing on the external expectation, like, you're gonna work for. Of course, you're gonna go to Mountain View. That's where the power is. That's where the data people go, and you're smart enough to be with the data people. So if you're gonna run the car as fast as possible, you're gonna go over there, but you made a different kind of decision, perhaps, than you know, the nine year old Cassie made.

And good, what wait a minute, actually, what's gonna be best holistically for me, and how can I work within that, pulling in some of this, some of this other information?

Yeah, for sure, and I think that that something that we can say to your seventeen year old is that it is okay, it's okay if it's difficult when you're young to take stock of what you actually are. You're not formed yet, and maybe maybe it's okay to let the wind take you a little bit, particularly when you have a great dad who's gonna give you great advice. But it would be good if you can mature eventually into more of a habit of saying, well, I'm not the average Joe.

So what do I actually want? And working for what is thought of as, um, I don't wanna offend any internal Googlers, but you know, they did have a reputation for kind of being the top team. So if you wanted to be number one and then number one again and number one some more times, that would have been the way to do it. But again, maybe maybe it's worth having something else that you optimize for in life, and I, as it turns out, am, uh, I'm a theater kid, lifelong theater kid, I'm an absolute nerd of theater.

I'm going to London for just a few days in two weeks, and I'm seeing, like, every evening, uh, matinees, I'm just I'm just gonna hoard as much theater as I can for the soul. And so living in New York City was gonna be just a better fit, not not only for theater, but for so much more that that city has to offer, as I'm sure, you know, John, uh, so Having lived yeah,

in both Silicon Valley and the New York area, I, I promise you that, yes, the theater is far better in New York.

I mean, I did. I go, I went to all the plays in um Silicon Valley as well, and you know, I did my homework, I, I knew what I was getting into or out of, but yeah, it's it. It takes practice and skill to know that some of those questions are even questions worth asking, and I've, I've developed that practice and skill, uh, from originally knowing how to do it to help others, knowing the having studied it formally, being book smart about it.

These are the questions you ask. This is the order you ask them in. It's something else to turn that on yourself and ask yourself the hard questions. The book smartness isn't enough for that.

That's good for all of us, whether we're running businesses or just trying to figure out life. We've all got decisions to make. Cassie, Kozyrkov, founder and CEO of Kozyr, former chief decision scientist at Google, thanks for joining me on this episode of Decoder.

Thanks for having me, John.

I'd like to thank Cassie for taking the time to speak with me and thank you for tuning in. I hope you enjoyed it. If you'd like to let us know what you thought about this show or what else you'd like us to cover, drop us a line. You can email the team at decoder at the verge. They really do read every email, or you can hit me up directly on X or Threads. I'm at Jon Fortt on all platforms.

Decoder also has a TikTok and an Instagram. Check those out at decoderpod. They're a lot of fun. If you like Decoder, please share it with your friends and subscribe wherever you get your podcasts. Decoder is a production of The Verge and is part of the Vox Media Podcast Network. Decoder is produced by Kate Cox and Nick Statt. The show is edited by Ursa Wright. The Decoder music is by Breakmaster Cylinder. See you next time.

As soon as autumn rolls around and you swap your iced lattes for pumpkin spice, that's your signal to start thinking about the holidays, the decor, the gifts, the shopping lists, but more importantly, the holiday treats, cheesecakes, cookies. Is there anything better? And when the recipe calls for cream cheese, there's only one option Philadelphia, the original cream cheese, made with fresh milk and real cream. This holiday season, whether you're going all out or keeping it simple, make it really Philly good.

Find your favorite holiday recipe at cream cheese dot com. Life can be messy and complicated, especially during the holidays. One thing that isn't switching to the simplicity plan from Verizon. It's one plan, one price, just $30 a line per month for you or the whole family. Make the simple choice this season, switch to Verizon and shop their best holiday deals today. Why can't we get AI to production? Our competitors are already. How do we keep data secure?

AWS AI cuts through the noise. Ready to use agents and the broadest set of AI tools. Stop overthinking, start building. AWS AI is how.