# Tech at Work: What GenAI Means for Companies Right Now

- **Show:** HBR IdeaCast
- **Published:** 2 May 2024
- **Source:** https://hbr.org/podcast/2024/05/tech-at-work-what-genai-means-for-companies-right-now
- **Transcript:** https://podcasts.opennash.com/shows/hbr-ideacast/tech-at-work-what-genai-means-for-companies-right-now-3548c4/
- **Provider:** OpenNash Podcasts (machine-generated)

## In brief

Generative AI is not going away, and the best way for individuals and companies to learn it is to use it widely and openly. Guest Ethan Mollick argues that nobody has an instruction manual, so experimentation by employees matters more than central IT rollouts. Companies that treat AI mainly as a way to cut headcount risk missing a general-purpose technology that could expand what they do.

The tension runs between the large early gains and the hidden costs. A Boston Consulting Group study shows big improvements in speed and quality. But workers hide their AI use, stop checking good-looking output, and risk losing the apprenticeship that builds expertise. The hosts add research showing that AI teammates and algorithmic bosses can reduce motivation and helpfulness among colleagues.

## Key insights

1. **Employees are already using AI, often secretly** Mollick cites a survey in which over 60% of people use AI secretly at least some of the time. Fear of being fired or seen as replaceable gives workers a reason to hide it, so firms must build a culture of sharing.
2. **GPT-4 gave consultants large gains out of the box** In a BCG experiment with 18 business tasks, AI users produced about 40% higher quality and worked about 26% faster. Gains were largest for the weakest performers.
3. **Good AI makes people stop checking its work** Researcher Fabrizio Dell'Acqua calls it falling asleep at the wheel. The better the system looks, and the subtler its errors, the less people verify it, which Mollick expects to produce many small failures rather than one big disaster.
4. **Bloomberg's custom model lost to general GPT-4** Bloomberg spent over $10 million on a finance-specific model, but GPT-4 beat it in almost every category. Mollick concludes the future is probably not training your own AI.
5. **AI teammates and algorithmic bosses can hurt collaboration** In a Super Mario Party experiment, teams did worse and felt less motivated with an AI teammate. In another study, workers managed by an algorithm gave about 20% less advice to peers.

## Questions this episode answers

### Is generative AI actually going to stick around, or is it hype?

Mollick says there is no sign it will disappear and doesn't expect progress to level off soon, even though he admits nobody knows. Juan Martinez compares it to email, which raised similar worries but became too useful to ignore.

### How should I start using AI at work if I'm a manager?

Mollick advises starting to use it publicly, openly sharing where you succeed and fail, and keeping it casual. One example is having AI record a meeting and give advice, then giving it feedback.

### Why are employees hiding their AI use from their bosses?

Mollick says workers fear being fired under unclear rules, being valued less once the boss knows AI did the work, or showing they are replaceable. He says this is a culture and incentive problem for leaders to fix.

### Should my company build its own custom AI model?

Mollick says probably not. Bloomberg spent over $10 million on a finance model, and the general-purpose GPT-4 beat it in almost every category, because larger frontier models tend to be smarter.

### Is it a good idea to use AI to cut staff and keep output the same?

Mollick calls that a shortsighted failure of imagination. He argues that AI is a general-purpose technology like electricity, so firms should use it to expand what they do rather than only cut costs.

## Facts

- **Juan Martinez** [0:15:10]: "we developed eighteen realistic business tasks, and these ranged from analysis to, uh, persuasion to creative tasks."
- **Juan Martinez** [0:05:51]: "I think the best survey we have is over 60% of people use AI secretly at least some of the time."
- **Juan Martinez** [0:11:00]: "that's the explicit goal of OpenAI this year is to release fully working autonomous agents."
- **Juan Martinez** [0:11:36]: "I would suspect that this summer of 2024, we will see GPT five class models that will represent a significant improvement over GPT four."
- **Tom Stackpole** [0:04:21]: "Today we're talking to Ethan Molick, a professor of innovation at the Wharton School at the University of Pennsylvania"
- **Juan Martinez** [0:28:13]: "They were paired in teams of two, and they had to work together to gather fruits and veggies from around the kitchen."
- **Juan Martinez** [0:28:26]: "The researchers asked these teams to play together for six rounds, the two humans. Then they added an AI team member and asked the team to play another six rounds with the AI."
- **Juan Martinez** [0:28:45]: "84% of the people in the experiment said they preferred to play with their human teammates over the AI teammate."
- **Juan Martinez** [0:28:51]: "the second article was published on HBR dot org in February 2024, and the focus is on how working for an algorithm"
- **Juan Martinez** [0:28:55]: "They paid one thousand gig workers from an online labor platform to create slogans for a van rental company's social media marketing campaigns."
- **Juan Martinez** [0:29:14]: "they found that the workers managed by the algorithm offered roughly 20% less advice to their peers than the workers managed by the person."
- **Co-host** [0:35:22]: "The quality of the actual slogans that the two groups came up with didn't differ significantly"
- **Co-host** [0:35:22]: "He's a marketing professor at the Wharton School and co-director of AI at Wharton."

## Transcript

**[0:00:02] Tom Stackpole:** For the last year and a half, we've been hearing that generative AI is going to change everything. In that time, companies have invested huge amounts of time, money, and resources, and most of them are still waiting for the payoff. Even as it seems like everyone else is on the cusp of cracking how to use this technology. If you're a leader, you're probably asking, is this real or is this just hype? And honestly, here at HBR, we're asking ourselves the same thing.

**[0:00:31] Tom Stackpole:** Welcome to Tech at Work, a four-part special series of the HBR IdeaCast. I'm Tom Stackpole.

**[0:00:36] Co-host:** And I'm Juan Martinez. Every other Thursday, we'll bring you research, stories, and advice about the technology that's changing work and how to manage it.

**[0:00:45] Tom Stackpole:** We're both senior editors covering technology here at the Harvard Business Review. And in the last year, we've seen more pitches than we can count on generative AI. Juan, how would you say the questions we're seeing have evolved over the last year, and where are we now?

**[0:00:59] Co-host:** I'd say that in the beginning, everyone was asking what's possible. Now, they want to refine their use. They want to figure out exactly how to use it perfectly for their business case, with their employees, the way that their security structure is set up. It's about getting everyone on board, making sure you can have it work within the organization, and then optimizing it. And so that's where I think we are now, that optimization stage.

**[0:01:23] Tom Stackpole:** Um, I think I'm feeling a little more skeptical than you are about this, but I'm curious, what are the big questions that you have right now, and what would convince you that this is the real thing?

**[0:01:34] Co-host:** Gen AI is not going anywhere. So the question now is, how do I use it? How does my team use it? How does my company use it? There are a lot of questions that that they're gonna have to answer. And, you know, just get in there and start playing around and learn how to converse with AI. It's really important that you do that, whether we're using Gen AI ten years from now or whether we're just using regular AI to, you know, answer our customer service questions.

**[0:01:59] Tom Stackpole:** So I think you make a lot of good points. For me, there are still a few things that give me pause, especially when thinking about, you know, adoption at scale. This could have huge ecological costs, it could really change our media environment. But thinking about the questions that businesses need to be able to answer specifically, one, what do we want to be able to do with these tools? Two, what impact will trying to do that have on our employees and our customers?

**[0:02:23] Tom Stackpole:** Or to say that another way, what are the risks? And three, are we confident enough in the promise of this tech to really invest in it and see what it can do? And maybe I'm just naturally pessimistic, but I see, you know, the high compute costs, I see risk around copyrighted materials, I see trust issues with employees, I see regulation coming down the pike, and I still don't see proven use cases that are gonna make it all worth it.

**[0:02:48] Tom Stackpole:** But tell me why I'm wrong.

**[0:03:33] Co-host:** It's the kind of thing like forty years ago, if I had told you that there was a way for people to send letters to each other that would arrive instantly, you would have said, Wait, hang on a second now. People could intercept those messages and, you

**[0:04:00] Juan Martinez:** know, the it'll it'll turn up compute power because people will be sending so many messages all the time. And these are really good questions that you would have asked about email. But we're using email and we started using it and we had to figure out how to use it because it wasn't going anywhere. It was too convenient and it was too impactful, uh, for businesses to just pretend that it's not there because they're concerned about these questions.

**[0:04:21] Tom Stackpole:** Well, I think our guest today will probably agree with you. Today we're talking to Ethan Molick, a professor of innovation at the Wharton School at the University of Pennsylvania, who has become one of the leading experimenters with these new tools. Way back in 2022, right after OpenAI launched ChatGPT, I called Ethan to ask him to write an article for us because he seemed to have an immediate, intuitive understanding of how to use it. Now, a year and a half later, he's just published a new book, Co-intelligence, Living and Working with AI, about what he's learned about using generative AI, what it can and can't do, and the risk companies face in trying to integrate it into their work.

**[0:04:37] Tom Stackpole:** We start out talking about what Ethan has learned through direct experimentation with these tools.

**[0:04:39] Juan Martinez:** So the crazy thing about the state of AI right now is that nobody knows anything. I talk to all the major AI labs on a regular basis, we have conversations, and, uh, I think people think there's an instruction manual that's hidden somewhere. There is not. There nobody knows anything. There's no information out there. So the best thing I could do is this is the principle of my book, is that you should use AI for everything you legally and ethically can, because that's the way you get the experience with how these systems operate.

**[0:04:47] Juan Martinez:** So for me, I will use it for, uh, research and for writing. I use AI as an experiment for most things that I do to see what it works for and what it doesn't, and the results are often quite surprising. So I think a lot of people are waiting for instructions that are not forthcoming, and you have to sort of take charge and do this yourself to some degree.

**[0:04:55] Tom Stackpole:** For people who are kind of anxious about this, do you have a pitch for why people should start messing around with this, or or even just sort of trying to figure out what it's actually like to use this?

**[0:05:06] Juan Martinez:** I have a few pitches. I mean, the first pitch is I think it's important. I think a lot of people think this might be going away or AI is here and now I got time to adjust. This is a rapidly advancing technology, and I don't think there's any indication in any circumstance that it's going to disappear. And I also don't think we're gonna see the advances plateau that quickly. Maybe a year from now, maybe two years, but it already operates at a very high level.

**[0:05:28] Juan Martinez:** I think we need to get used to a world that has AI in it, and trying to put that off doesn't doesn't help you. Uh, and in fact, knowing how it works will help you adjust as the systems get better. The second reason is that it is helpful. I mean, you know, it's funny because when I actually talk to large groups of of employees and executives, the executives almost always are not using AI, but lots of employees are already automating their job.

**[0:05:51] Juan Martinez:** Right now, the huge advantage comes to you as a user. If you can figure out a way to make your job better, then you use AI to do that. Um, and a lot of people are secretly using AI at work. I think the best survey we have is over 60% of people use AI secretly at least some of the time. Um, and so there's advantages to you. We can talk about what that means for organizations and what that means for leaders in just a bit, but, you know, there is value in doing that.

**[0:06:10] Juan Martinez:** And then I would say the the third reason is once you get over the freakiness, it's super interesting and fun to explore. Like, this system does a lot of things that are really neat, and you can be the first person to figure out what those things are. In my mind, you're like the prompt master. I go to your LinkedIn page, I see prompt after prompt, it's like you're giving good feedback. What research would you cite to talk about the best prompts and the best way to use the answers?

**[0:06:42] Juan Martinez:** And and do you have your own best practices for how to use prompts and how to take answers? So there are two kinda core ways to interact with the AI that we call, uh, conversationalist or interactionist and and structured. And so at a conversational prompt, you're literally just chatting back and forth with the AI, and there's everything matters in this case. There's papers showing that punctuation matters, that if you ask the questions in a dumber way, you get less accurate answers.

**[0:06:59] Juan Martinez:** If you approach it as a debate, the AI will argue with you. If you approach it as I am the teacher, you're the student, and you sort of imply that's what you want, the AI will be much more pliable. If you approach it as you are a dumb machine that just does work, it'll act like a machine that just does work. We don't actually fully understand how to make that happen best, so I just don't worry about it.

**[0:07:19] Juan Martinez:** Like I said, I give it a context, you are X, I am Y, let's work together on this, and that gets you a large part of the way there. Part of the problem is that they're putting too much magic in prompting. There is not a single one of the AI lab people I talk to who think prompts are gonna be that important two years from now, or that, like, prompt crafting or prompt engineering is a skill.

**[0:07:37] Juan Martinez:** It it will be if you're trying to build large-scale enterprise deployments, but for most people's work, the AI already can tell you what to do, and it will only get better at having you prompt things. When I teach my students how to prompt, I typically make them prompt four or five times before producing something, and interestingly enough, by the time you've prompted four or five times, not only is it hard to recognize it's AI writing, but also the AI detectors don't work anymore.

**[0:08:00] Juan Martinez:** So it feels much more like a blend of human and AI work. So this is not a type of query in and get a result back. It's a conversation with an intern, with an employee that you are trying to get to do good work.

**[0:08:12] Tom Stackpole:** In the book, you you talk a lot about how you used AI to write this. Can you tell us a little bit more about kind of, like, what that process was like, what you learned worked well, what didn't work well, how that whole process changed how you were thinking about using these tools?

**[0:08:26] Juan Martinez:** So one of my principles in the book is to be the human in the loop, to figure out what you're good at and do well. For right now, where AI is on the ability curve, you are probably better at it at whatever core task you like to do most, and AI is probably not as good as you at that. So I like to style myself a pretty good writer, and AI is not as good a writer as me.

**[0:08:47] Juan Martinez:** But the AI made writing the book much better because it did a lot of ancillary tasks. So that could be anything from the stuff that typically stops you from writing a a book, which is, you know, give me twenty versions of this sentence, cause I'm stuck. I would ask the AI to just give me variants. Um, you know, summarize these 200 research papers, and I would read them myself, but then I had the summaries available to work from.

**[0:09:06] Juan Martinez:** I'd actually send the summaries to some of my fellow academics, and they they quite liked them, they thought they were they were well done. AI does a pretty good job simulating customers. So I had AI readers read chapters and give me feedback on it. So it wasn't about the writing task itself, it was about removing the friction from all the other stuff that would have stopped me from doing my writing. And I think that's part of what you want to think about when you're using AI is not so much how do I replace the core thing I love to do and that I think I'm better at.

**[0:09:33] Juan Martinez:** Maybe AI could do that, maybe not, but it's about how do I make it so that's what I get to focus on.

**[0:09:37] Tom Stackpole:** So, o- one of the things that you write about that that I think is really useful is you have a great way of thinking about the kinds of work tasks that we should continue to do versus the kinds of tasks that Gen AI might be able to take on. Uh, can you kind of break these down into categories for us?

**[0:09:52] Juan Martinez:** Sure, so there are sort of four categories and then a subcategory, not to make it hard, but I talk about just me tasks, which are tasks that only you as a human can do. And that might be because you want to do them, or it might be because you only a human should do them because a human should remain in the loop of that particular task or idea. You're a professor, what are some of your just me tasks on your day-to-day?

**[0:10:15] Juan Martinez:** So, an interesting just me task, in my case, is letters of recommendation. It's about purposefully setting my time on fire as a signal flare to people that I care about someone. So I'm supposed to spend forty five minutes or an hour doing that, and I'm supposed to struggle with what I write. If I just give the AI the resume of the person, the job they're applying for, and say, I'm Ethan Molick and do a thumbs-up emoji, write a letter of recommendation, I will get a better letter of recommendation than the one I would write in forty five minutes or an hour.

**[0:10:38] Juan Martinez:** So there's an open question to me about whether or not we do that or not. You know, I still do grading by hand, even though I know the AI will do a better job because I feel like that's an obligation as a professor. When I write reviews of academic papers, I do them by hand, but in the principle of I, you know, use AI for everything, I turn them in and then I have the AI then do, uh, review and see where the AI and I differed.

**[0:11:00] Juan Martinez:** Um, so I'm trying to think about these things, but those are the kind of moral lines that may get crossed soon. So those are just me tasks. Then the other two categories are delegated tasks, where we do some of the work with the AI. And then, zooming out again, we finally have automated tasks. You say, go handle this, and it goes and solves your problems for you. And that's the explicit goal of OpenAI this year is to release fully working autonomous agents.

**[0:11:23] Tom Stackpole:** Yeah, I mean, one of the things that that's been interesting with sort of critics of some of these LLMs is people are saying, you know, there there are limits to this architecture. There is gonna be a plateau, and it's gonna be coming sooner than people think. I mean, w- what do you kind of think of that argument?

**[0:11:36] Juan Martinez:** So I think that nobody knows the answer, and I see splits even when I talk to people at OpenAI between we're on the an infinite curve, you know, here to AGI versus this is gonna even off. And I don't think anyone knows, and people don't know until the training is done and the model is shipped, essentially. So I would suspect that this summer of 2024, we will see GPT five class models that will represent a significant improvement over GPT four.

**[0:12:01] Juan Martinez:** After that, that's the next question. Is there a GPT six that's as big a leap over GPT five? Are we seeing the top of the the curve? I don't know the answer. I know how I feel personally about working with AI or having AI work for me, but there's a lot of studies now happening that are talking about AI as a boss, AI as a supervisor. Can you talk a little bit about what that means and how people are are are responding to being controlled or supervised by AI?

**[0:12:36] Juan Martinez:** So we don't actually have a lot of evidence on the current version of AI. Part of the p- problem that happens is our conversations about AI are confusing because prior to ChatGPT, when you would have a podcast or an HBR article about AI, it was about AI as algorithmic approaches that were usually math-based about forecasting. And so when we talk about, um, AI supervision, almost all the studies are about these earlier models that were sort of cold algorithmic control.

**[0:12:57] Juan Martinez:** So what's it mean to be an Uber driver and have the algorithm telling you what to do? And what happens when doctors or managers are getting advice from AI, and how do they feel about that? And I think that we're not seeing the same sort of effect because large language models feel like we're interacting with a person. It feels warmer, it feels more humane, and we don't quite know what the effects are going to be. That being said, I think the dangers are still there of algorithmic control.

**[0:13:20] Juan Martinez:** It's just dressed up in a nice way.

**[0:13:22] Tom Stackpole:** You know, in this context, you you've talked about how important it is to still have expertise. Um, my first magazine job was being a fact checker, and it was, like, brutally slow and surprisingly hard, cause there's facts kind of baked into all kinds of things, and it's not always immediately obvious, you know, what kind of assumptions are being made. So, how how do we make sure that we can still be experts and and do this kind of fact-checking work?

**[0:13:43] Juan Martinez:** Expertise is kind of great because it gives you heuristics and rules. You can glance at something and say, is this good, is this bad, right? And you build expertise through deliberate practice, through trying something over and over again, getting feedback on it. Uh, to me, the biggest risk, actually, is the destruction of deliberate practice inside organizations, because the way we do this is we actually have a medieval apprenticeship system inside white-collar work, right? When my students, uh, graduate from Penn, I think they're awesome, but they are not specialized at working at HBR or Goldman Sachs or McKinsey or whatever name your company of choice.

**[0:14:15] Juan Martinez:** They go there and they spend a couple years learning the ropes. That's how we sort of teach people is like, we get an intern who is very, very smart but inexperienced, and in return for doing some of our work, they learn, right? And they get paid relatively little for their job. And then, if they're good enough, they advance in the organization. That's the basics of how organizations work. Intern work is the most delegatable work to AI.

**[0:14:34] Juan Martinez:** It's so easy to hand off, like, write this deal memo, do this research project, give me the initial, you know, briefing for this interview, create a transcript and and highlight the the key points. And but I think the real danger is that we're gonna destroy expertise building inside organizations because people are just going to have AI do that work for them. Well, you've actually studied this in a real world setting. Can you talk a little bit about the study with BCG?

**[0:15:10] Juan Martinez:** And they were given ChatGPT four to perform some of their work tasks. How did it work? What did they do, and what did you learn from it? So, this is work with a whole set of, uh, co-authors, uh, at Harvard and MIT and Warwick. We went to Boston Consulting Group, so a elite consulting company. They gave us 8% of their global workforce, which was amazing, and we did a couple different experiments. The one of the main ones was we developed eighteen realistic business tasks, and these ranged from analysis to, uh, persuasion to creative tasks.

**[0:15:31] Juan Martinez:** And we asked people to do these tasks. We measured them before doing the tasks without AI, and then in the second set of tasks, half the people were randomized to using AI, half not. What we found was pretty extraordinary, which is 40% improvement in the quality of answers, and about 26% improvement in speed, and about a twelve and a half percent improvement in amount of work done. And that was with GPT four out of the box, without any of the special training that lots of companies are spending their time and money trying to build.

**[0:15:57] Juan Martinez:** We found the impact largest on the bottom performers, not on the top performers, though we're still trying to figure out whether or not that was a result of early use, where people didn't know very well, or whether that's a universal thing, though that's a result people have found before. And so very, very powerful result right out of the box for a very elite set of tasks, which was fascinating. So, if you're an enterprise, how do you sort of take your weakest quote-unquote employees and then give them ChatGPT four and help them become better?

**[0:16:25] Juan Martinez:** I think if you're an enterprise owner, there's a lot to think about as or as a manager, because the incentive right now is for your employees to secretly use these systems, and they're doing that all the time. You can think about the reasons, there's a lot of them, right? In one case, like, what if my rules aren't clear and I get fired for using it? Second is, you guys think I'm a wizard right now, because I might I'm suddenly producing all this amazing work, and you don't know how I'm doing it.

**[0:16:47] Juan Martinez:** If you know it's AI, you might value my l- work less, or maybe you start to see, Hey, I just showed that I'm replaceable by AI. I don't wanna show you that. So, there's a lot of reasons that people don't wanna share. So it starts with a culture problem and incentive problem, right? How do we incentivize people to do this? How do we build a culture that people want to share what they do? So, I find in startups, in nonprofits, in cooperative enterprises, they're really hitting it off with AI because people share, Hey, I figured out a way to do something cool.

**[0:17:14] Juan Martinez:** In large-scale bureaucratic organizations or highly competitive organizations, everyone's hiding their AI use all over the place for all those reasons. Can you give us examples, if you have any, of these secret cyborgs that really mess stuff up because they just copy to paste it, or because they didn't do the work that you suggest people do in order to make the most of gen AI? Uh, the weird thing about it, right, is, like, the secret cyborg people doing at work often are doing it for tasks they know well and are experimenting till it's good because they know what a good task looks like.

**[0:17:44] Juan Martinez:** So I haven't seen huge incidents inside work environments, right? There's some famous cases of lawyers using this, thinking it works kind of like Google and finding sites for themselves that aren't real, it hallucinates sites all the time, and not checking and getting in trouble with judges. That's become increasingly a sort of phenomenon in the legal field. And, again, you know, something that we'll see growing, but it's from kind of inappropriate use. Like, for example, one use I see people putting this to that's completely inappropriate, and I'm is one of the most common things people tell me they use the system for is performance reviews.

**[0:18:15] Juan Martinez:** Of all the use cases, right? Performance reviews suck to do, but they are meaningful by the process of doing them, right? And, you know, when there's only a couple AI users, maybe it doesn't feel so bad, but after everybody starts using it, we have to rethink about how HR works in that case.

**[0:18:29] Tom Stackpole:** I I think the stories about the hazards of how this could be applied by companies are really interesting, and and I wanna look at a different study. This one by Harvard Business School researcher, uh, Fabrizio Del Aqua, and he studied recruiters using AI. Some were given a good AI, some were given, um, a mediocre one. So how does this play out, and and kinda, what does this example tell us about how companies should be careful about how they're starting to use these tools?

**[0:18:55] Juan Martinez:** And Fabrizio is the lead author on the BCG study as well. And he studied in this a phenomenon that we found it writ large in the in the BCG study, which is that when people use an AI system that's good enough, they actually stop paying attention. And he calls it falling asleep at the wheel. And it's the most common when the systems are best. So if you have a bad AI system, you're checking all the work.

**[0:19:18] Juan Martinez:** If it seems like it's really smart and good, you stop checking it. There's an additional factor, which is that the AI's errors actually become very subtle. So it's very hard to even check the facts. So between those two factors, it's fact-checking becomes hard, and we fall asleep at the wheel. It means that you stop using your brain as much when you're using AI work, and it's really hard to figure out a good way around that. But there's no sort of Boeing-style disaster at this stage of somebody turning something in, right?

**[0:19:43] Juan Martinez:** It is much more a bunch of small disasters of people not paying attention. So I think that's the real danger to me is less that we're going to see, you know, a plane fall out of the sky and more that, you know, we see a steady creeping set of indifferent work appearing.

**[0:20:00] Tom Stackpole:** Coming up after the break, we're gonna talk about what organizations that are successfully using generative AI are doing differently, and why these tools can't be managed like other enterprise technology. We'll be right back. So, one of the things that you articulate, and I think really well in this book, is that there's this tension between how easy it is for for individuals to be innovative with these tools, and it's really, really hard for institutions to to do the same thing for a a variety of reasons.

**[0:20:34] Tom Stackpole:** And so, what questions should companies be asking as they start to figure out what to do about generative AI, how they should be thinking about it, um, what it means sort of within the organization?

**[0:20:46] Juan Martinez:** The truth about innovation is it's very expensive to do. It's R and D is expensive, and the reason why R and D is expensive is because the way we learn is trial and error. It's why drug trials are very expensive. It's why, you know, figuring out whether a product a software product is good or bad is very expensive. But R and D is very easy for people at the tasks they do at their own job, right?

**[0:21:07] Juan Martinez:** Where, you know, if we if we try recording this podcast slightly differently every time, if every time you send an email out, you're doing it slightly differently, like, that's pretty costless, and you get a fairly fast feedback, cause you're experimenting in a domain you know well, and you could do it all the time, and that's how we learn. The problem with organizations is that they're viewing this as an IT product that needs to be centrally controlled and implemented.

**[0:21:29] Juan Martinez:** And there's a lot of problems with that kind of central control approach with AI. You're waiting for centralized instruction to tell you how to use it, and it's unclear how a senior management team would be able to tell a line worker how to improve their sales technique using AI, or if they'd listen anyway. Have you come across any examples of companies that have incentivized this use well and have actually brought use cases to their employees and said, Hey, do this, it'll help you?

**[0:21:58] Juan Martinez:** I mean, one of the more, uh, both effective, I think, and also more extreme effects was, uh, Ignite Tech, which is a software holding company. The CEO basically got into the idea of AI very early and gave everybody, uh, GPT four access last summer and said everyone needs to use it. And he has told me that he then fired everybody who didn't put a couple hours in by the end of the month, but he also offered cash prizes to anyone who gave the best prompts.

**[0:22:22] Juan Martinez:** Another organization I know, when they do hiring, they require you to try and automate the person's job with AI before you put the job request out, and then you put out a different job request that's altered for what you think the job's gonna look like in the future. So I think, uh, modeling behavior, incentivizing with rewards, and thinking about the future are the three things you want to be able to do to incentivize organizations properly.

**[0:22:47] Tom Stackpole:** It sounds like you're also saying that companies need to really change how they're thinking about where innovation comes from and and who's responsible for it and who's being rewarded for it, right? I mean, there's also kind of, uh, just sort of a a structural or or kind of even a a just how they're thinking about this that needs to change.

**[0:23:03] Juan Martinez:** I absolutely think that we're not ready for this world that's happening. We've built corporations around the idea that the only control system we have are other humans and that we on the only way to get advice is to escalate things up a a chain. And that's not true anymore. So organizations need to change in lots of ways. The the locus of innovation has always been on the edge, but now it matters more than ever because your only advantage as a large company with AI is that you have more people using it and you can adapt faster.

**[0:23:31] Juan Martinez:** Otherwise, everybody else has the same AI tool you do. And most companies I talk to have worse AIs than every kid has access to in mo- in most of the world because they're so scared about, you know, privacy and other concerns, sometimes rightly, sometimes wrongly, that they don't allow the most advanced version.

**[0:23:46] Tom Stackpole:** Um, you know, we've had this kind of, like, natural experiment where Bloomberg invested in in their own GPT, and now we're looking at how that compares to frontier models, basically for doing the same task. And you have one that's just out there and one that's been trained with all this really valuable proprietary data. What is that sort of telling us? What what have we seen in that that kind of experiment?

**[0:24:08] Juan Martinez:** So, just for people who aren't that familiar, frontier models are the most advanced models, and right now there's a very strong scaling law in AI, which is the bigger your model is, which also means more expensive it is to train, they're just smarter. And the result is that the most advanced frontier models are often much better than specialized models built for specialized tasks. And we don't have all the answers yet, but Bloomberg decided to build a finance GPT, and they spent over $10 million on it, from what I can tell.

**[0:24:34] Juan Martinez:** And they trained it on finance data, and it was supposed to do things like sentiment analysis for stocks and so on. And it did that, and it was pretty solid. But then, in the fall, the team retested that compared to GPT four, uh, which is the advanced model available to everybody all over the world, and GPT four beat it in almost every category. We're seeing the same sort of effects for specialized medical models being beaten by GPT four, which is not built for medicine, or especially law models in the same way.

**[0:25:00] Juan Martinez:** What this tells you as a company, you're going to be kind of thinking about the use cases. If smarts is valuable, you'll need to use a frontier model, but the future is probably not training your own AIs.

**[0:25:10] Tom Stackpole:** One of the things that's kind of surprising is I think a lot of companies are looking at generative AI, and they're saying, This is great, we can cut headcount. You know, we can automate tasks, so, you know, now we can have ten people doing the work that that eighty people used to do. But I'm curious what you think of that instinct of this is a labor-saving tool. Let's cut headcount. Let's keep doing the same stuff with with fewer people and just, you know, count the bonus profits.

**[0:25:35] Juan Martinez:** I I to me, I mean, that is a shortsighted failure of imagination in many different ways. Seriously, if you really think that this is another general purpose technology, like, you know, electricity, steam, or computer, but happening in a very compressed time period, then the worst thing you could do is say, let's make sure to keep productivity the exact same by firing people who might be the people who could leverage this into the next generation of things.

**[0:25:58] Juan Martinez:** And I see that happening, people will reduce headcount to keep performance exact same at the exact same moment that everybody else is getting a performance boost. What I fear is is that companies have gotten into the view that cost-cutting is the highest-value thing they can do. Expansion was hard to do. You do expansion through acquisition. Now you can do expansion through your workers, through everyone becoming more productive, and they need to change their mindsets, or workers will detect that and they're not going to come along as part of the voyage.

**[0:26:23] Juan Martinez:** I mean, I have these rules for organizations I would think about, which is four questions I would ask any company, which is one, what was did you do that was valuable that's no longer valuable? If providing white glove customer service to everybody was your big differentiator, that's about to go away. The second thing, though, is there's something impossible that you could do that you couldn't do before. What is that? So, you know, if you're a consulting company, it used to be, uh, ten hours or twenty hours of someone's work to give you a basic outline of a piece of work.

**[0:26:49] Juan Martinez:** Now you can produce a bunch of stuff with a consultant's mindset in five minutes. What does that let you do? Do I offer individual advice to everybody, to to everybody get their personal consultant? It's very exciting. The third thing is, what can you move up market that you couldn't do before? Like, so now we can provide white glove service to everyone. Your customer service agents know everything about a person and have a great interaction with them.

**[0:27:11] Juan Martinez:** Your salespeople can give personalized sales pitches. And then the fourth thing is, what can you move down market or democratize? I'm I think again about BCG. I think those results showed that it makes consulting much easier to do. They work for high-end Fortune 5,000 companies. What can they do now for small and medium businesses that they couldn't do before? How do they move down market? So I think people who are thinking about these as a strategic shift will be much better off than people who don't.

**[0:27:20] Tom Stackpole:** Uh, two final questions, I think. First, what is your advice for a manager or a team leader about how to get started with generative AI?

**[0:27:20] Juan Martinez:** The advice I would give a manager is to start using it and using it publicly. Model behavior, and the idea is that you're going to show where you fail or succeed. You're gonna model curiosity, that you're trying to figure out from other people how to use it, that you would also be publicly kind of sharing that you don't know what you're doing and working on it. I'd say, Hey, let's try doing AI in this meeting, and let's have it record it and give us advice, and let's give it feedback on on that advice.

**[0:27:28] Juan Martinez:** I think, you know, making it casual and making it interesting, I think is gonna be really important.

**[0:27:29] Tom Stackpole:** And so what about for senior leadership? What should they sort of be doing with this?

**[0:27:29] Juan Martinez:** Uh, absolutely, I think the same things apply, but I also think that they have to realize that this is the real one, right? And if it is, this is enough of an emergency that you should be using it, so you know what it does, and your employees should be using it, and you should be thinking about this, and you need a multi-pronged approach. How do we reorient our organization? We don't have answers yet, like, we don't know what an organization chart looks like with AI included.

**[0:27:46] Juan Martinez:** We don't know what processes need to be changed. This is where leaders and strategy actually matters, and I would love to see more leaders stepping up and saying, I have a vision for how to build a better organization in this time period. You wanna be the visionary leader of the next, you know, next century that everyone looks up to and that there's biographies about you. You figure out how to use AI in a positive way to build the next great enterprise.

**[0:27:54] Juan Martinez:** That's that's how you get famous.

**[0:27:55] Tom Stackpole:** Juan, do you wanna get famous?

**[0:27:55] Juan Martinez:** I I'm already famous, I'm on a podcast with Ethan Molick for Harvard Business Review. My mother never thought I would get this famous. Well, nobody paid attention to me for, like, the last decade, so it's been a very funny rise of, like, I haven't talked about this stuff like this for a while, but okay. Ethan, I learned so much from this and your book was fantastic. Thank you guys so much. This was really interesting.

**[0:28:02] Tom Stackpole:** Thanks for coming on. That was Ethan Molick. He's a professor of management at the Wharton School at the University of Pennsylvania. His new book is Co-intelligence, Living and Working with AI. So, Juan, we touched on this super briefly with Ethan, but before we wrap up, let's talk for a minute about how people feel about working with generative AI.

**[0:28:03] Juan Martinez:** Yeah, this is a really, really important topic, and researchers are exploring it because it's important to understand the challenges that we'll face as AI is integrated into human teams. We've published a few articles on this already at HBR. One that I really love just came out in our May June magazine issue. Full disclosure, I edited it, so you know it's awesome.

**[0:28:09] Tom Stackpole:** Okay, tell us about it.

**[0:28:09] Juan Martinez:** First of all, Tom, have you ever played Super Mario Party?

**[0:28:10] Tom Stackpole:** No, the last the last Super Mario game I played was probably Super Mario three on original NES.

**[0:28:13] Juan Martinez:** All right, so a team of researchers used Super Mario Party to explore how integrating AI into a team affects humans. In the experiment, people were asked to play Super Mario Party together. They were paired in teams of two, and they had to work together to gather fruits and veggies from around the kitchen. They also had to coordinate with other teams to make sure their on-screen characters didn't bump into one another, and they all had to do it really, really fast.

**[0:28:21] Juan Martinez:** So if you're watching the game, you'd see these Mario characters rushing around a kitchen, grabbing tomatoes and lettuce, and maybe Mario is bumping into Goomba, and Toad has a huge stack of teetering plates. You can picture it, right? Very chaotic, very fun.

**[0:28:26] Tom Stackpole:** And how does AI come into this?

**[0:28:26] Juan Martinez:** The researchers asked these teams to play together for six rounds, the two humans. Then they added an AI team member and asked the team to play another six rounds with the AI. And even though the AI is really, really good at Super Mario Party, they found that team performance declined in all kinds of ways when AI was added. They looked at coordination, like how often Mario actually bumped into a Goomba. They looked at how many ingredients the teams gathered each round, and they asked all the people on the teams how motivated they felt when playing with the AI teammate.

**[0:28:35] Juan Martinez:** And all of that was worse with the AI. Here's Bruce Colgate, one of the authors of this paper. He's a strategy professor at Columbia Business School. So we had, you know, initially all humans playing this thing, but then suddenly we put in this change, you know, like, we would take out their best friend, Luigi, and replace it by an AI, you know, algorithm, and they just they did not like it. You had the humans, you know, increasingly getting more and more depressed over time, playing with this, uh, AI driven Luigi or Mario.

**[0:28:45] Tom Stackpole:** So, Juan, what are the takeaways for managers who may be trying to figure out how to integrate generative AI into their teams?

**[0:28:45] Juan Martinez:** 84% of the people in the experiment said they preferred to play with their human teammates over the AI teammate. That led researchers to conclude that when you add AI to the mix, team sociability can fall. That means human team members feel less motivated, less trusting, and they make less effort. So if you're gonna add AI to your team, keep that very real downside in mind.

**[0:28:51] Tom Stackpole:** Okay, so what's up next, Juan?

**[0:28:51] Juan Martinez:** All right, so the second article was published on HBR dot org in February 2024, and the focus is on how working for an algorithm, which is already a reality if you're an Uber driver, changes workplace dynamics.

**[0:28:55] Tom Stackpole:** Okay, so we're talking about algorithmic management, right?

**[0:28:55] Juan Martinez:** Exactly, that's it. They started by surveying workers in the transportation, distribution, and logistics sectors, and they found that workers who were managed algorithmically were less inclined to help or support colleagues. And this was true even when they controlled for factors like the size of the organization, employee turnover, type of job, income, gender, et cetera, et cetera. Next, the researchers did a field experiment. They paid one thousand gig workers from an online labor platform to create slogans for a van rental company's social media marketing campaigns.

**[0:29:14] Juan Martinez:** The workers were randomly divided into two groups. One group was guided and evaluated by an algorithm, and the other group by a human. After the workers had completed the task, the researchers asked them to offer advice to others on how to create effective marketing slogans. And they found that the workers managed by the algorithm offered roughly 20% less advice to their peers than the workers managed by the person.

**[0:33:46] Tom Stackpole:** Okay, but what about the marketing slogans? Was there a difference between the quality of their work?

**[0:35:22] Co-host:** No, that's the thing. The quality of the actual slogans that the two groups came up with didn't differ significantly, which suggests that algorithmic management doesn't necessarily affect workers' task-based performance, but it can decrease their pro-social motivation. And this was especially true when algorithms were monitoring and evaluating employee performance. Here's Stefano Puntone, one of the authors of this paper. He's a marketing professor at the Wharton School and co-director of AI at Wharton. This is his advice for companies using algorithmic management.

**[0:36:33] Co-host:** So the moment that you are f- as a worker, you feel you're being appraised by an algorithm, your performance, your worth, basically, as an employee, is being measured by a machine. We find that then people tend to objectify coworkers and be less helpful to them. Companies really ought to think about the, uh, byproducts for organizational culture, for individual feelings and cognition when you deploy algorithm. It's not just about can the machine do it? The question is, should you let the machine do it?

**[0:36:43] Tom Stackpole:** Yeah, so it sounds like what he's saying is that there's really a cultural element that we need to be thinking about here, because this can really change how people feel about each other, even if they're not necessarily working with a machine coworker, but if the the sort of terms of the system are are being dictated by a machine.

**[0:36:43] Co-host:** Yeah, I mean, this is the early days, right? Nobody knows exactly how to use these tools to get the maximum benefit out of them, and every role within every organization is gonna be a little bit different. So you have to figure out what the right option is for your organization, but then you have to go into use cases and start to figure out how it works for each individual role, and we're definitely not there yet.

**[0:36:49] Tom Stackpole:** Okay, thanks, Juan. Next time on Tech at Work, how will the end of third-party cookies change the internet? It's a big shift that's coming up fast, and it has huge implications for digital advertising and publishing and how all kinds of incentives on the internet really work.

**[0:36:49] Co-host:** We'll talk with a researcher who's studying how ad effectiveness will be affected, and we'll speak with an agency executive who's guiding her clients through this transition. That's in two weeks, right here in the HBR IdeaCast feed.

**[0:36:52] Tom Stackpole:** Did you know HBR has more podcasts to help you manage your team, your organization, and your career, including nearly one thousand episodes of IdeaCast alone? Find them at hbr dot org slash podcasts, or search HBR in Apple Podcasts, Spotify, or wherever you listen. And if you wanna help our show, go to your podcast app and rate the show five stars. It helps more than you may know, and we read every comment.

**[0:36:52] Co-host:** Thanks to our team, senior producer Anne Sanny, senior editor Kurt Nickish, audio product manager Ian Fox, and senior production specialist Rob Eckert. Special thanks to our friends on HBR's video and social teams, Nicole Smith, Ramsey Kabbaz, Kelsey Hansen, Scott Lapierre, and Elena Mata, and much gratitude to our fearless leaders, Maureen Hoke and Ady Gnosis. Thanks for listening to Tech at Work, a special series of the HBR IdeaCast. I'm Juan Martinez.

**[0:37:04] Tom Stackpole:** I'm Tom Stackpole. We'll be back in two weeks.