The Risks of Too Much AI: Fortune’s Jeremy Kahn

Stay tuned after today's episode to hear Sam and I break down the key points made by our guest.

In what ways are we over reliant on AI technologies? On today's episode, we chat with a journalist who shares some of his fears about AI.

I'm Jeremy Kahn from Fortune, and you're listening to Me, Myself and AI.

Welcome to Me, myself and AI, a podcast on artificial intelligence and business. Each episode, we introduce you to someone innovating with AI. I'm Sam Ransbotham, professor of analytics at Boston College. I'm also the AI and business strategy guest editor at MIT Sloan Management Review.

And I'm Shervin Khodabandeh, senior partner with BCG and one of the leaders of our AI business. Together, MITSMR and BCG have been researching and publishing on AI since 2017, interviewing hundreds of practitioners and surveying thousands of companies on what it takes to build and to deploy and scale AI capabilities and really transform the way organizations operate.

Hey everyone, thanks for joining us today. Shervin and I are excited to be talking with Jeremy Kahn. He's the AI editor at Fortune magazine. At Fortune, he writes its weekly newsletter Eye on AI. He's also the author of a book, Mastering AI, a survival guide to our superpowered future. Jeremy, we're looking forward to talking today.

It's great to be here.

So I'll raise the complaint first. I had planned to just to skim the book and just, you know, try to pull out a highlight or two, but it was interesting. And so it. It unfortunately cost me a lot more hours this this weekend preparing than I than I planned. So that's my that's my angry start to this podcast

That's a good complaint. episode. This is this is what Sam's compliments always sound

Yeah, yeah, I was gonna say, sorry, not sorry on that one.

Well, but maybe go negative here. We heard a ton about the benefits of AI, but let's start with some of these risks that you've foreseen. I thought you did a nice job outlining those. For example, you cautioned against letting AI cognitive prowess diminish our own prowess. Can you walk us through how some powerful tool might diminish our own human abilities?

Yeah, absolutely. It's one of the things I'm concerned about is that this technology is, in some ways, so easy to use and so seductive that we'll sort of overuse it. And as a result, lose some of our important cognitive abilities. W- One I worry about is just critical thinking. I mean, anyone who's used one of these chatbots from OpenAI or Anthropic or any of the others, it gives you a very fluid, plausible pat answer and a very capsule summary of whatever information you want.

And I think it's just really easy to kind of take that answer and not think too hard about it and go off and say, Oh, I've got the answer. And in the way these systems are currently designed, there's no real prompting of the user to think about the provenance of the information that they're receiving. And in fact, in some cases, the way these things are designed, there's no way. The system itself doesn't really understand the provenance of the information it's giving you, so it can't actually tell you how it knows what it knows.

Explain provenance, though, just start with

Yeah, so province, I mean, what was the source of this information? Any particular fact, it's telling you, where did that come from? or its analysis, how did it arrive at that? And, you know, people said some of these things about Google when Google came along. There was some criticism that Google was gonna make us stupid. But one of the good things about Google is at least you have these links, so you have to think about, you know, where's the information coming from?

At least there's something there in the UX that kinda prompts you to think about the source of information. If you wanna know more than just the few lines that the link displays, you have to click the link. You don't have to do any of that with a generative AI based chatbot response, and I worry about that. I worry it's we're just not gonna think too hard about where's this information come from. So that's one of the ones I'm concerned about.

The other is the tendency with these LLMs to think that writing and thinking are completely separate activities. I'll just give the system a few bullet points and it will write the whole essay for me, and it will. It will do all the hard work of thinking through the arguments. And I worry about that because, again, I think writing and thinking are not actually separable activities. That it's really through writing that we refine a lot of our reasoning.

Very often you find holes in your own argument when you actually go down to write something long form that you you haven't thought of when you just outlined a few bullet points. And again, there's some tendencies that were already there with other technologies. Powerpoint most famously had this tendency to reduce things to bullet points, often with some negative repercussions. That's why Jeff Bezos had famously kind of banned PowerPoint presentations at Amazon because he was very worried about the kind of sloppy thinking that came in when you just had people delineate their thoughts in bullet points.

first of all, I agree with both of your points, and I think the first point you made about is it making us stupider. I think we've actually seen that documented. BCG did one of the largest controlled studies with MIT, Harvard and Wharton, where we looked at a thousand strong consultants and they were given chatgpt to work with. And in certain tasks that included broad creative thinking, ideation, they actually did better when they were told to use the tool versus those who weren't.

Cause this expands the realm of possibilities for many in terms of like, how do you create a campaign? or how do you consider this from different angles? But on things that consultants are supposed to excel at, like, you know, business strategy, problem solving, like, what is the best strategy for a company in Southeast Asia that's trying to enter a market? You know, here's the revenues, here's the competitors, et cetera. Gen AI didn't do really as good, and in fact, what we saw is that group, there were sort of a reinforcement of the wrong thing because it was an over reliance, as as you said, on something that doesn't work.

So, for sure, we've seen it will take critical thinking away. But I also feel like the paradigm change is when before all these large language models, AI was a tool that would help us predict better or optimize or do things that humans couldn't do. I think, as you say now, Jeremy, we've entered a phase where AI is doing that which humans could do too, and in many ways better. We think of it as more of a coworker than a tool.

And so now you have a coworker, and just like with a coworker, you need to challenge them. And when you're having a conversation, you know, I find the best outputs that I get from a gen AI chat is when I really engage with it. The same way I would engage like a young associate or a young apprentice. And you challenge them. And and it often it says, Yep, I made a mistake, I'll go deeper, I'll go deeper.

And if you don't take it, as you were saying, at face value, you get to the answer. What sort of worries me in the long term is the first thing you said, which is, are we overusing it? What have humans invented that they haven't overused?

Yeah, I mean, that's a good question. I I think there is that risk. That's why in the book I talk a little bit about why I think there should be very specific design choices to try to prompt people not to overuse it. But of course, the companies developing the product have no incentive really to get you not to use it. So it's a it's a bit of a problem. But, you know, it would be useful if, as part of the design, there was something that sort of prompted you to think about the source of this information.

I wanted to use talk about using it as a kind of a junior colleague, that's kind of interesting. And I actually think one of the ways you can avoid some of these risks is if you don't use it that way. So you do the initial drafting of of the document or the proposal, or, you know, in your example of the strategy for the company, like the consultant would, the human consultant would do that initially. And then ask the system to act as almost like a senior colleague To critique to critique it.

And I think, actually, sometimes you both get the benefits of this large knowledge base. It can draw from. It does sometimes point out things that you've missed or that, you know, you didn't think about. And you don't have the de-skilling risk that you have when you do treat it as the kind of initial drafter of things, as the the one who's gonna do the work of the junior colleague. That's maybe a way that we should train people to use the systems to think about it more as a coach or an editor, um, auditor of our work, as opposed to the initial drafter.

Yeah, that's interesting because I've noticed that my own use of a plain old text editor has gone up versus because I find that, you know, all the spelling correction and grammar correction is a distraction. When I'm trying to think about what I'm thinking about. And so I've gone kind of old school and bare bones here. Maybe I'll be using Vim before too long. So AI is not gonna go away, though. What what can people practically do to avoid this?

I guess you one suggestion is draft it first. Are there other things we could suggest?

I think draft it first. I think training of people about what these systems are actually good at and what they're not. So at least you have that awareness. That should be part of how employees are kind of trained to use these systems as co-pilots, as well as students, as this gets introduced into education that they're taught. What are the strengths and weaknesses of these systems and what can't they do? And, and, you know, show people some examples of how it hallucinates or how it can miss whole lines of argument, maybe because they're not, you know, represented as well in the data set.

I agree with everything you're saying, and because I feel the same Well, that's way, I feel

no fun. Come on. like, I

mean, no, but. But I there was gonna be a. But because I have children in high school and I see the benefits, I see the risk, right? Like my son was saying, I saw him on it. I'm like, What are you doing? You're doing math? But he says, Look, I don't understand how to solve. This problem is teaching me, and now I'm gonna do it on my own, and then it's gonna critique me. I'm like, great, that's fine.

But the point that it can take a significant amount of cognitive load away from humans, and sometimes that's good and sometimes that's dangerous. I think that point is not lost on people. My worry is that in reality, it's not gonna happen because this is very early stages, right? We're talking about something that's two years old that right out of the gate was great, is getting better. It's improved by, like, orders of magnitude, and it has some flaws.

But the other thing I see is, I mean, we're talking about individual use of it, right? But when you think about all of these investments are basically done for the enterprise or being used by the enterprise or being consumed by the enterprise. And that's where, like, billions and trillions continue to go in it. I still think that's in very early stages. And what I worry about there is that the gap between the haves and have-nots is going to increase significantly at the company level, right?

Like we had, you know, Sam, in our research, we talked about the AI gap. And like the 10% who are getting significant value versus most who aren't, and what do they do? I feel like that's even gonna narrow more because of the sort of exponential power that this new generation has. And I think that the challenge that many companies are going to have is also the challenge of rethinking their skill set and their talent. And, you know, now that I could use AI to do things that a thousand people could do, maybe make it fifty more effective in a coworker mode.

What kind of talent do I need? How do I hire them? I think the 10% that we talked about in our prior You're predicting generation of

that it gets smaller.

AI, Sam, right, those who get, I think those 10% are going to be those who, in the long term, figure out the talent play and the skill play. And they become sort of new companies,

yeah, I mean, I mean, I think you you're probably right about that. One of the things I highlight in the book is the risk that thems have got got shall get sort of thing. Those that already are sort of ahead and have the largest pool of data and have figured out how to use this technology well already are more likely to continue to pull ahead of others. I think that's true both sort of within professions and across industries.

And I do take your point about talent, but I think you need both. I think you both need a workforce that is going to be extremely flexible. I think that's the lesson here is that, you know, I'm not a big believer in the idea that we're gonna see mass unemployment from AI. In fact, quite the opposite. But I do think most people's jobs are going to change, and you have to be flexible. And I think the companies that have the most resilient and flexible workforces, in some ways, will do the best.

Because, you know, I'm talking about how these systems work right now, and they might work very differently in twenty years. Absolutely, they will. And if you're just starting your career, I think you have to be prepared to be riding this wave of this evolution of this technology. Today, you know, it would be helpful to you to have quite a lot of prompt engineering skills the way the systems

that's right.

work right now. I don't think that's gonna necessarily be true in ten years, because I think we will have perfected away some of that, optimized away some of the need for that.

When I was listening to you, Jeremy, my mind went into not just the workers being reskilled or retrained, or maybe just new talent coming in, like companies hiring people that they normally wouldn't hire because they have spikes and things that AI doesn't have. But my mind went into actually the younger generation. And I was gonna ask you, Sam, like, as an educator, as a professor, what do you think we could do in schools now for these young people?

You know, when we went to school, if we thought deeply and we memorized a lot of things, you know, read a lot and do a lot of math and exhibit hard work, we pretty much did well. And I think,

spoken like a true engineer.

well, okay, fine for for like science and engineering fields, that, you know, that was the case. And and what do you think this is gonna mean for, you know, high school and college, sort of education disciplines in, like, five to ten years from now?

Yeah, sure. And I have kids approximately the same age, and you know, we deal with university students all the time. And I worry about this advice to be flexible. I mean, not that I disagree with it, but I wonder if if it has about the same amount of substance that buy low and sell high does. You know, I think no one's gonna take the the counterargument that, oh yeah, you should be more rigid and less flexible.

But how how do we get them to do that? I mean, I have some techniques that we use in class, but Jeremy, how do we go beyond? Sort of a yes, be flexible? Is there something that people should do? Someone's listening right now? What should they do tomorrow to be more flexible?

That's a very good question, and I'm not sure I have really great answers for that. Um, except that I think people should have to change fairly frequently. I mean, I think the only way to become adaptable is to actually practice adapting. I then I think some jobs are naturally more like that than others. Some have kind of rote processes or ways of doing things and they don't change very often. Or a person's not asked to change very often what they do.

Or, and I think some others have to kind of learn on the fly. You have to constantly evolve. Maybe you're facing different challenges or questions every day. And I think for students, you could actually force them to learn in different ways at different points, you know, even year to year or term to term. But I think that sort of thing, like building that into the educational system, would be a way to teach people to to change and to adapt.

I don't really know other ways of doing it. I spent some time talking to folks at NASA about how they try to teach astronauts to work alongside automated systems. In the book, I was talking about this issue of about vigilance. If you're constantly just putting the person in this mode where all they have to do is kind of babysit an automated system, humans are really terrible at it. Um, so how does how? And NASA tried all sorts of different ways to get people to be better at it, and they they ultimately couldn't get people to be better at it.

Except by introducing faults at like random intervals so that the people were constantly on guard, which was an interesting insight. But I think maybe we have to do something similar in terms of training students or the workforce, where you kind of artificially introduce disruption and change in order to teach people to get used to disruption and change.

that's an interesting idea. It actually reminds me, I think you had an example in the book about a experiment at Georgia Tech, which I, of course, noted, cause that's my background. But people followed robots who were clearly demonstrating that they were making mistakes over and over again, and yeah, they still followed the robot even after the robot was exhibiting these mistakes. And I I thought that was pretty, pretty fascinating, you know? One of the things you're saying is it's incumbent on an educators in the educational system to introduce doubt in these systems and introduce, you know, the idea that they will fail.

That makes a lot of sense. And if I guess if NASA does it, then I believe it.

I think we could draw a lot of lessons on, you know, from both the space industry and the aviation industry, which have worked alongside these automated systems of various kinds for decades. I think one of the the most undervalued aspects of this is what they call human factor engineering or human computer interaction. I think those things are really gonna come into into prominence.

One of the things about the interacting with the device versus interacting with the people is this empathy question and it, in some ways, interacting with a device makes us less empathetic.

Yeah, I I speak a lot about empathy in the book. It is ultimately, I think, the thing that is uniquely human, the fact that we can relate to one another based on our own lived experience and relate to others lived experience. And the problem with AI technology is it has no lived experience. And chatbots can mimic empathy pretty well. They can sound sympathetic, empathetic, but it's not authentic, it's ultimately kind of a fraud. It the empathy isn't real, and I think we need to keep in mind this distinction between real empathy and its imitation.

And I worry, particularly with people using chatbots as kind of substitute friends, as kind of companions, that this is another kind of disc. And never have to talk to anybody else and never have to deal with real humans who are, you know, much messier creatures to deal with. Particularly, there's some interesting studies about men who have been left alone with Alexa, um, basically the Amazon digital assistant. And, you know, it's an environment where they didn't realize they were being recorded in any way, and they were really abusive to, particularly the female voiced AI Alexa assistant.

Very quickly, things descended into kind of misogynistic language and talk. And then there were some studies of those men and their interactions later with real people, and they carried over sometimes some of those habits of dialogue to real relationships. And I think that's dangerous, and people are very, very worried about that with kids. Actually, it's one of the reasons there was actually some pressure put on Amazon to introduce a mode to Alexa that a parent can turn on, where the Alexa will only respond if the child is polite and says please and thank you.

Again, that's one of those things we might want to consider in in other AI systems, is, you know, are we using these systems to kind of prompt us to, as humans, to be polite in our interactions?

So let me transition here. We've got a series of rapid fire questions. What's the first thing I that comes to your mind? What do you see as the biggest opportunity for AI right now?

Uh, I think drug discovery is maybe the biggest opportunity

Man, we've certainly seen some examples of that with Covid. I hope we don't have another similar opportunity coming up. What's the biggest misconception that people have about artificial intelligence?

that we're all gonna lose our jobs.

Yeah, that whole fear mongering of the job loss. So what was the first career that you wanted?

The first career I wanted, so I I wanted to be an archeologist at one point.

yeah, so that actually that really ties into your desire for data provenance. Then you can make a connection between the roots of data and the roots of anyway. All right, when is there too much artificial intelligence?

I think there's too much if it's doing things where you want human emotion and connection. I think there's a great example with that Google Olympics ad where they had the kid writing the fan letter and the dad had the kid in the ad. Use AI to write the fan letter. And and there was such backlash against that, and I think rightfully so. I mean, that's something where, you know, you want the kid to do that. That's like this connection with their hero, and there's a lot of learning that takes place in the writing of that letter.

And that's one of these, you know, fundamental human experiences is based on connection between two people. You don't want AI doing that.

yeah, I mean, this is why we have the machines to do these other things, not to do the fun parts. Is there one thing you wish that artificial intelligence could do right now that it can't?

Yeah, I would like it to actually do my shopping for me, which it can't quite do. I think it's going to very soon be able to do that. I'm really looking forward to it.

Jeremy, it's been wonderful having you on the show. Thanks for coming on and for inspiring a lot of deep thoughts about the future.

Great, it's been fantastic to be here.

Hey, sure, that was fun. It was a different conversation. We often focus on some of the positives from artificial intelligence, but I think Jeremy had some good risk that didn't go crazy. Existential robots are gonna take over the world types of risk, but some very subtle how AI is changing our behavior types of risk that I think were interesting to think about. What do you think people could learn from what Jeremy said?

I mean, just like any piece of technology, you cannot treat it like a black box. The users have to have a certain level of understanding of what are the limitations, what are the risks, what kinds of things can I not worry about the output. And where are some areas where I need to sort of probe and push and go a bit deeper.

I think a certain level of understanding is a problem, though. What is that certain level? I mean, I don't think we're asking people to understand how, you know, I don't know how silicon transistors work. That doesn't seem necessary for understanding the modern world of AI. On the other hand, we are coming pretty hard against the idea of just accepting everything AI says. So I think finding that that middle seems tough. Do you need to be able to make a model yourself?

Do you need to be able to use a model? Where's the line?

I don't think the understanding that I was talking about is a technical understanding, but it is a tool that has limitations. And just like when you're driving a car, you need to have some understanding that when you're driving thirty miles an hour and you apply the brake, you'll stop much sooner than if you're driving a hundred miles an hour. And

I think if it's wet, that'll change that equation.

and it will change that equation depending on the condition of the road and your tires and all these things. I mean, in reality, some people learn those the hard ways, and that's why they teach these things in driving schools and they teach people you know what to do in case of a skid. I think that's sort of the analog. So while I think, first of all, I don't think the conversation was necessarily dark. I mean, I think there's just so much over exuberance over AI that, like, a little bit of

That's a better way of phrasing like, back to it.

reality, I think you'll be good. And and I think, you know, a better way to paint, maybe even, which is not something Jeremy did, but the most extremely potential negative situation to create that dialectic that you could then start thinking of it the other way. But

Switching to your tire example, I think that's a good one. I mean, to me, I don't know, and I think probably you're the same with your engineering background. You, you're thinking about friction, you're thinking about coefficients of friction that change as the surface changes. I'm in the middle of teaching my our sixteen year old how to drive, and there's so much that you take for granted that I have forgotten about.

yeah, I mean, honestly, so much of my driving, even now, I think about things that I was taught. And like, when the conditions change, most of them, thank God, are things that I haven't needed to do myself, like experientially. But I was told it's raining, there's a thin film of oil, you will not see it or feel it un unless it's too late, so slow down. But that's my point, like, when you're driving, you're not thinking about those things, but you do know that, hey, it started raining, I'm gonna slow down.

Like, that's what most people on the freeway do, like, as soon as it starts raining a little bit, cause they've been told you just have to slow down, cause there's gonna be a film of oil and it's gonna be dangerous. And so

you drive different places than I do, I think.

well, what I was gonna say, though, is that be adaptable, I still think, is a goal and an outcome we drive to. But the question is, how do you become adaptable? And I think it starts with education. And I think that, just like with any tool, just like when they taught people how to use spreadsheets, they said, Look, you can have the totals, but then you need to have some kind of a check. And if you don't, you might have a great spreadsheet and it's gonna make a mistake.

And if you're linking spreadsheets, I do feel like that is the piece that's going to unlock this level of adaptability and caution, et cetera, which is just by demystifying it and by teaching. By having multiple situations of an answer gone wrong, an over reliance creating a real unrealistic outcome from AI. I mean, there are many YouTube videos where there's a conversation with ChatGPT and it's being convinced that two plus two equals five.

Coming back to your education point, I think that was a good one. Because to be able to do that, to be able to work in these ways, is going to require education. And I think most of our discussion has been about individual level interaction. But you know, what do we think about companies and how they should be educating? We've had guests, for example, Katia Walsh at Levi's had a boot camp, Michelle McCracken at Delta, they also had a boot camp.

What kinds of things are people needing to do to promote that education, and what kind of education should it be? So boot camp is one answer, but even then, I'm not sure what the boot camp should have in it.

I don't think the answer is a simple let's everybody have a boot camp. I think the answer is let us segment the types of use, let us segment the context, let us identify the various zones that gen AI is going to be leveraged in our company. There are some no fly zones that strategically it's not gonna happen. There are some places where it has maybe not carte blanche, but it is areas where the risks are so low and and

Fraud detection.

it doesn't really matter. Yeah, you don't wanna use it for fraud detection, right? Exactly right. And then there are areas where it is, as always, it's been human in the loop, human and AI. And now what can go wrong? And then what are the guardrails and what is the proper education for the type of user and the type of situation? I think that's the kind of curricula that's needed.

what I liked about what you said was this pushback against. We hear a lot about boot camps, and I don't want to be negative about boot camps, but they strike me as lacking some of the personalization that the tool offers. That we have a variety of cases, a variety of ways. You've listed a bunch of them that we can use these tools and the monolithic boot camp or the sort of classic corporate education doesn't really feel right here.

You I'm sure you have to do these things in your organization, I have to do them too. These, um, I don't want to complain about them too much. I understand why many of them exist, but they're many of these corporate education tools are not particularly tailored to what you know.

Yeah, but look, I mean, this dates me as a person.

It's okay, you're in a safe space with me,

But like, I do think about spreadsheets, and I do think about the education I went through as a young consultant, learning how to get the best out of spreadsheets. Because

right? You had to miss a total at some point to get that right.

but and how to use a formula, or how do you do it better or faster? And as the tool evolved, then more education and more education, and I do actually think those things help. And

Did you get to be a guru of spreadsheets by by classes or by doing?

by both, but the classes really helped, and and talking to people who were in those classes really helped. And then it depends on what you're using it for. It wasn't the same, you know, for everyone. I mean, I think most consultants, when they start this, they're in the world of pretty heavy, numbers and spreadsheets, and they do that and they check and debug. And now the thing is so advanced that it's making like debugging itself, and, you know, it's telling you, are you sure you wanna do this?

But I do think that right now, this is a black box. And I think there is such a big variance in within an organization in people's understanding of risks and benefits and what it can do and what it can't do. And is it gonna replace me, or is it not gonna replace me? Like, there's just such noise and level of misunderstanding that the only way to break through that is through real, thoughtful design of content and education that's company wide.

And I agree with you that I think, like for most corporations, the way they deal with this right now is by creating

kind of a monolithic package.

right, exactly, well, creating a monolithic package for education. But I don't think we have yet developed the right curricula at scale for the learning and development of the enterprise. I mean, many companies have done it, you know, we have done it at BCG because we use these tools all the time and and we need to know. But for a typical company, for a non digital native company, I don't think these curricula exist. And I think that this is a real big concern for the chief human resource officers of these companies to put the put in place these kinds of curricula.

Yeah, but it seems time because, I mean, by analogy, I liked your spreadsheet analogy, because, you know, spreadsheets are what opened up the world of computers to most people. They were part of the killer app that opened up the use of personal computers for most people. And I I expect that the large language models are gonna have that same sort of effect with artificial intelligence. That's the gateway, the inroad for most people to artificial intelligence. And and you're right, I think most companies don't have good curricula for that.

We've got a bonus episode coming up in a couple weeks talking about exactly this talent in the world of generative AI, so everyone can pay attention to that. Thanks for listening today. Next time, Shervin and I talk with Rebecca Finley, CEO of the Partnership on AI, please join us.

Thanks for listening to me, myself and AI. We believe, like you, that the conversation about AI implementation doesn't start and stop with this podcast. That's why we've created a group on LinkedIn specifically for listeners like you. It's called AI for Leaders, and if you join us, you can chat with show creators and hosts, ask your own questions, share your insights, and gain access to valuable resources about AI implementation from MITSMR and BCG. You can access it by visiting MITSMR dot com forward slash AI for leaders.

We'll put that link in the show notes, and we hope to see you there.