Solving Real User Problems With Generative AI: Slack’s Jackie Rocca
Me, Myself, and AI · 2 Apr 2024
Source: https://sloanreview.mit.edu/audio/solving-real-user-problems-with-generative-ai-slacks-jackie-rocca
Transcript: https://podcasts.opennash.com/shows/me-myself-and-ai/solving-real-user-problems-with-generative-ai-slacks-jackie-rocca-269936/ (via OpenNash Podcasts, machine-generated)

How can generative AI help us collaborate more effectively at work? Find out on today's episode.

I'm Jackie Rocca from Slack, and you're listening to me, myself and AI.

Welcome to me, myself and AI, a podcast on artificial intelligence and business. Each episode, we introduce you to someone innovating with AI. I'm Sam Ransbotham, professor of analytics at Boston College. I'm also the AI and business strategy guest editor at MIT Sloan Management Review.

And I'm Shervin Khodabandeh, senior partner with BCG and one of the leaders of our AI business. Together, MITSMR and BCG have been researching and publishing on AI since 2017, interviewing hundreds of practitioners and surveying thousands of companies on what it takes to build and to deploy and scale AI capabilities and really transform the way organizations operate.

Hi everyone. Today, Shervin and I are talking with Jackie Rocca, vice president of product at Slack. Jackie, thanks for taking the time to talk with us. Welcome.

I'm so excited to be here.

Let's get started. I think most of our listeners, many of our listeners, likely use Slack, but let's fill everyone in. What is Slack?

Slack is a collaboration platform, so it started out as a really effective way to message and work with your colleagues. And it's really expanded over the years to include a lot of your work tools that are accessible from right within Slack. So it's a great way to work with your team, with your colleagues. We actually have external functionality, you can work with customers or contractors. And we are really looking to make it the hub of all of your AI tools and bring in AI into that conversational interface.

All right, you mentioned AI, let's go. What's gonna be AI related? I heard communication tool. Where's AI in that?

I lead our Slack AI product area. We are building native functionality. We are looking at ways that AI can be used to address top user problems that we've been facing for years. Whether you're overloaded with all the communication and tools that you have in Slack, or if you're just trying to find that knowledge across your enterprise, we are using AI to solve some of those top user pain points. Because we are a collaboration platform, we're also a place that a lot of your AI tools are being built into Slack today.

We actually have more than 10,000 AI powered tools that are available through applications or third party developers.

All right, 10,000. Let's go through them all, right? Starting at the top.

A lot of them as well are private to specific organizations, but some of them are public, available to download today, too.

That's quite interesting. Tell us more about generative AI. I have to imagine they're either doing a lot or thinking about doing a lot with generative AI. Can you tell us a little bit about that?

Yeah, so we are very excited about what's happening in generative AI. First of all, we've been using AI and machine learning for many, many years. If you think about our search functionality or when you're recommended different channels to join. So that foundational element has been there for many, many years, like many products. But as many of you know, the industry in the space with LLMs and generative AI has really taken off in the last few years.

I was on parental leave when a lot of the things happened with ChatGPT, and I was so excited about how some of these capabilities could help us take new approaches to some of the user problems that we've been facing for such a long time. We're really taking the approach of starting with those user pain points and constantly trying to think through. How does new advancements in technology help us take a fresh approach to solving those issues, rather than starting with the technology itself and saying, Oh, generative AI is out there.

Let's throw the spaghetti at the wall and see what sticks. The features that we have launched and available today are around channel recaps, thread summaries, and search answers. And let me talk through a few of those. So on channel recaps and thread summaries, this is really about getting up to speed quickly in channels. So often there's just so much conversation back and forth. And whether you're away from Slack for an hour or a day, or you're coming back from vacation, it can be overwhelming to catch up on that conversation.

So what we are offering and what's available now for customers is the ability to summarize that content as you want it. And then there's the ability to go back to the source as well to validate that, or even dig in deeper if you wanna get more information. So that's really around channel summaries and thread summaries as well. Another great use case that we're seeing is, let's say there is some sort of technical incident, so maybe your service is down or having issues, and every single moment really counts in those situations.

Maybe that subject matter expert just got paged in the middle of the night. Rather than have that person have to spend minutes and minutes catching up on everything that's happened so far, they can request a quick summary and jump into that conversation and contribute to the solution immediately. And also, you can get a summary at the end and say this was the root cause and this is what the resolution was. So it's been super valuable, so many different use cases.

And then the second category is around search answers, especially for companies who've been on Slack for a long time. There's so much conversation in there. People talk about projects, about people, about topics, about company policies. If you think about all of that knowledge that really lives in Slack, how can we make it more accessible and easier to use? So with Slack AI search answers, you can simply ask a question, you know, what is Project Gizmo? I haven't heard about that, or maybe I don't wanna ask my colleague or disrupt their work.

You can just ask Slack AI to get up to speed on different topics. You wanna learn about company acronyms. So many of us have been in meetings at your company and somebody throws something out there, or you're a new hire. And rather than ask your manager these thirty two questions that you have, just ask Slack AI. It's sort of your trusted companion that you can rely on.

super valuable, yes. The ones you mentioned are user facing, as in I'm a Slack user. I've been gone for some time. Or I, you know, on fifty different channels and a lot's going on. And some are incidental to what I care about, or some are really, really critical. And then somebody says, Hey, why aren't you responding to my Slack? And I'm like, I don't know what channel are you or what are you talking about? And so I, I get that it's super valuable.

Are there also use cases where the generative AI creates messages on my behalf?

Yeah, we are focused first on really keeping the human in the loop and making sure that you are always in control. We know that Slack can be a really sensitive place. There are a lot of private conversations, people are in a work context, so they really want to have control and it's really important how they show up. Our experiences are very guided and situational, but we want to make sure that we are not creating and posting things on behalf of our users without them having really the agency to make that choice and send that message.

Yeah, I love that. So, Yep,

Well, you have so many users of your product, how do you go about introducing some features like this?

yeah, and I'd love to talk through some of our core product principles because this has really guided us here. One of our product principles is don't make me think, which in the generative AI space for us means Slack AI should not require our customers to be prompt engineers or experts in the field. It also means it shouldn't require training. So we really wanna have situational and guided experiences. Let's surface some Slack AI answers for you when we think it's appropriate.

You know, this channel is really busy. Let's surface a summary that might be really valuable to you at that moment. We don't require you to figure out how to write that prompt so that you get that great answer. We're doing that work for you. A second principle that we've also really leaned on is prototype the path. So, you know, there's a lot of potential with generative AI, but we know that accuracy and relevancy, and frankly, latency as well, are really important to the user experience.

So we've been prototyping for the last several months so that we can again not make users think. And we can do all that work for you to make sure that we are creating a great experience. And we're shipping those experiences that we have high confidence in, and we're continuing to work on those things that maybe need a little bit more refinement.

How long does that take? Is that a short process, long process? Fifteen minutes, twenty minutes?

More than fifteen or twenty minutes. One of the big challenges as a product leader working in the generative AI space is balancing time to market with getting that experience 100% right. We know that generative AI is not 100% right, so we need to kind of find that middle ground between. We know that there's a ton of customer value here and we feel great about the experience we're shipping, but we're gonna continue to iterate and make it better over time.

So for me personally, that's been. That's been a challenge because I always want to have things as close to perfect as possible, but we know it's also important to give people that value sooner. So it's a balance for us. We have spent many months to try to get that experience the best it can be, both from the features that we've already launched, as well as new things that we are already prototyping to have as fast follows down the line.

and I have to imagine the experimentation loop or the test and learn loop is quite fast with, like, all the users, right? I like your principles on human in the loop and all that. You mentioned. It's a balancing act, and I could imagine there's so many things you could be doing with AI and gen AI, right? And so even if those principles are always followed, you could be incredibly ambitious in so much with gen AI, or mildly ambitious.

And so how do you navigate that path? How far do you go? how fast do you go?

It's a great question, and we debate this all the time as a team when we talk about prioritization and what to build and when to launch. For us, again, we're really trying to focus on what are those top user problems that we think generative AI can help fix, and we're prioritizing those first. There are some table stakes features that we could absolutely and probably will build within Slack AI, so things around helping you compose messages. Slack primarily is a text based tool, although we do have video capabilities through.

We have a meetings product called Huddles, but it's still primarily text based. The top pain points are around how do I create focus and reduce some of the noise? How can I better find information and leverage the knowledge archive that I have in Slack? So we're really trying to put more of our energy towards solving those things that we've heard repeatedly from customers. And hopefully we'll have some opportunistic things as well, especially as we go throughout the year.

Where we can also address things that are really cool and delightful that we certainly want to build into the experience, but maybe don't hit that very first set of features that we've launched.

That's a great answer.

Yeah, the generative part of that, I mean, I think that everyone probably their first blush is gonna say, Hey, can it type some text for me? But I think that really isn't where Slack is competitively different in in some ways. And what I mean by that is that you've talked about the history of being a communication channel, and that when you hear communication channel, you think about typing messages, right? Or, you know, that's the production of messages.

But there's so much organizational communication that's happening in these tools. And you've got that rich history of that. That strikes me as what's critical and differentiating here versus, I mean, there's nothing wrong with generating text, but the other part seems unique.

Well, there might be some things wrong with that. And actually, you can push back, I, you I accept know, I mean, I

that. Go ahead, keep going.

well, I mean, I think that's a point that she's making, right? Cause there is clearly a huge pain point on what's going on in the organization and what should I know? And and all that, which makes a ton of sense. But then there is something personal about Slack. And now you're gonna have to wonder, is it Shervin that's typing it? is it Shervin's agents that's typing it? And maybe at some point, we will be in that world where, you know, we get messages and text and all that.

And you wonder, who wrote this? Is it the person or is it the person's agent? And I don't know. I mean, I think like that could be. What's wrong with this is that at some point, then we just sort of applicate the whole thing and let's AI just do everything, which is not, I think, where we wanna go.

I think I was reacting to how nice the first use cases were, though, because actually, as a, I mean, I'll tell you, it will surprise nobody here that I'm an inbox zero kind of person. I don't like any little red alerts up there. I clear them as soon as I can. And that's actually. I've struggled with Slack because of that, because at any point, there's so much coming through that I can't satisfy my inbox zero needs there without 100% attention.

And what they're offering is a a very different way of solving that pain point for me. So that seems very appealing.

Absolutely, I mean, me personally, I can't imagine my life without some of the Slack AI features. And I have the privilege to be able to also have the things that we're still prototyping and haven't launched yet. But the way that I consume information has changed. We think about this sometimes in different tiers of messages and channels, so today everything has a equal priority. In general, a message is a message, whether you are in that channel with the CEO of your company or it's a social channel with your colleagues.

We do offer channel sections, which is a great feature if folks haven't checked it out. It's a really nice way to organize your channels. But can we do more there? Can we think through what is sort of a tier one channel or conversation? And those are things that you want to read every message for, and you probably won't get a summary unless maybe you're on a parental leave or gone for an extended period of time. And there's just so much there.

But there's a huge group of sort of these tier two conversations that you want to know what's happening and you want to stay in the loop. But you might not need to read every single message, and maybe you don't want it to show up in Slack every single moment of the day. Or it's just a really fast moving. I kind of just want the TLDR of what happened when I was out. So we're really trying to think about things in these different buckets of use cases and prioritization and help make that easier for our users to consume.

Hey, Shervin, I just want you to know you're always tier one for me. I read everything you send, so I would never summarize you,

Well, I don't have to summarize you, cause your your messages are so like terse. And it feels like it's been written by a by a physician, not by a college professor.

all right. So actually, I was thinking about as you were talking about this talent mobility and changing workforce. And that seemed like a a big role here as well. That, you know, as we're having people leave jobs, retire, after having people shift jobs back and forth. How do you reconcile the need to sort of capture all that information with? Does information go with a person? How do you reflect that?

Yeah, one of the early premises of Slack was how it's different than email, so with email, if that person leaves the organization, that knowledge is really lost. With Slack, one of our founding perspectives was really that Slack should be this searchable log of all communication and knowledge. That's what Slack stands for. So we have this persistent place that it's really the organization's knowledge. You post something in a public channel, you happen to leave the company. While still, the knowledge about what you had worked on in the project and the people and things like that can still persist, and you can still get value out of that.

Or if you're a new hire joining the company, you can still learn about what had happened, the learnings, the experts that you should connect with, et cetera. We actually think that building AI on top of that is a really great connection point to make those experiences more valuable. Because we have all that knowledge, all of that insight that we can pull on and just make it more valuable. It all already exists today, but maybe it's not fully tapped to its potential, or it's not easy to make sense of or get to that quick answer that you're looking for.

And we aim to make that experience better.

This all sounds amazing. What's hard about it?

There's a lot that's hard about it. We know that generative AI is not perfect, so we have been spending a lot of time trying to get the best prompts that we can create on behalf of our users. So we get the right detail of information, we're not sharing an answer when we're not certain that we're providing sources back so people can reference the source content easily. So there's a lot of work to be done to make that experience better.

We also are really laser focused about the user experience. So we have played around with how does the summary display itself? What's the correct entry point? How do we make this feature and functionality obvious? So when it's valuable to a user, they know it's there, but not so annoying that we're surfacing it all the time. For example, we talked about those tier one channels, maybe summarization's not appropriate there. So we're continuing to do a lot of work to make sure that that user experience is great, both from the actual how do you find Slack AI and when does it surface to you?

But as well as the experience when you get that response back and make sure that the information is relevant, accurate and well sourced.

You're leading the charge on AI and gen AI in Slack. Tell us about the team. What's the team like?

We actually formed this team through a group of people that was prototyping on the side, really just trying to build experiences in our spare time, a lot of nights and weekends. Just really passionate about how this can help solve these core user problems using new technology. So it started there, and then, as the year progressed and as we got more momentum behind these prototypes and got feedback internally and externally. We've actually put a more sustainable team structure in place and started hiring against the goals that we had set more officially.

But, you know, everyone is very excited about the future of AI. We've been able to attract and retain just an incredible team, everything from, obviously, engineering, machine learning team design. That user experience is so important. Data science is critical to how we think about generative AI, as well as all of our go to market teams. Just really trying to hear customer feedback. How do we bring this to market?

I don't know if you can answer this question, and if not, that's fine. But I'm curious what's under the hood and whether that's a pre trained model that's sort of available, or is it you're training your own models, given the uniqueness of the domain?

So what I will say is that the space is changing very fast and we're open to different options. And again, super laser focused on what's best for our users.

That's a good answer.

So we are evaluating pre trained foundational models. We have things that are machine learning models that we've had in Slack for a long time. We are actually part of Salesforce for folks who might not know, and Salesforce has an amazing AI research team as well. And there's obviously open source models too, so we are evaluating everything. And I do expect this space to change over time, but we have taken a really deliberate approach to how we're thinking about data and trust.

So we have built an architecture where your Slack data does not leave Slack, everything is hosted within our virtual private cloud, our VPC. So that when you're using AI in Slack, you can be confident that your data, your really sensitive company data, is not leaving Slack's infrastructure. And that we still abide by all of the compliance and enterprise grade security capabilities Of course, that we have in Slack today.

What's exciting that's coming next? What are the AI developments that you're pretty fired up about?

There's so much to be excited about, both at Slack and things that are happening outside of Slack with foundational models and other products. I can't wait to see sort of what the next year brings. We are continuing to iterate on different experiences in Slack. We are constantly getting feedback from our customers, trying to stay rooted in user problems to help define our roadmap. But we're also listening to what's happening outside as well, because the space keeps changing, keeps evolving.

And, you know, it's hard to predict one year to the next what's going to be possible, but we're also really trying to stay grounded. I'll be the first to admit that I I don't have that crystal ball of how things will evolve in the next three to five years, but I'm really excited to be here for the ride.

Tell us a little bit about your background. How did you get interested in this? How did you get to be able to do these things?

I've been at Slack for about five and a half years now, and it has been an amazing and fun ride. I led our product led growth team for a long time, so our self service group, and just really, again, was passionate about this space. I've heard the same user problems or user needs be referenced time and time again. And was just s sort of a self learner to say, Okay, we've got this new technology, can we take a fresh approach?

And I also did have the privilege of coming back from parental leave, so I could really kind of clear my head from the day to day and think a little bit bigger about the opportunity space. Before Slack, I spent six plus years at Google, specifically on YouTube, and was fortunate to be on the launch team for YouTube TV. So I was a product manager over on YouTube TV, which was an incredible experience. When I joined, we were, you know, on the whiteboard, trying to figure out should this thing have three tabs or five tabs, and and how do we think about what channels to put into the service?

So it was really exciting being part of that journey to launch YouTube TV and see it scale for a time. Before that, I got my MBA at Stanford, and then also I have my beginnings as a management consultant, so I started my career at Bain in Los Angeles. It's been a ride. I think a lot of people who end up in product management have had this indirect path to get there. I don't think I knew what product management was when I was in college or in my twenties.

So I'm really grateful to have found Slack, to have found product management, and more recently, to help lead through this sort of AI revolution for the team and for the company.

that's an amazing background.

The self learning part of that's really interesting too, cause I think, you know, that's a shift that's happened over the course of even since we started this podcast. Shervin, at the beginning, we were talking about a machine learning model, and very few people, I think, woke up on Saturday morning and played around with some machine learning model. I think practically everybody has gone and screwed around with a lot of the generative tools that are out there.

And I think that's a fundamentally different way that people are learning about these technologies. And I think what you just said echoes a lot of that.

And I think there's really two sets of skill sets to think about. There certainly is the technology side, what's possible, what's unlocked, and it's important to understand. And hopefully, if that's not someone's expertise area, they can connect with others who maybe can work between strengths and weaknesses. But I think there's equally the opportunity to think about how can it be applied. So what would that user experience look like? What specifically are we trying to solve? Because I think it's it's exciting to just try a bunch of things or experiment with generative AI and what's happening with LLMs, which is great.

And again, I'm not gonna discourage people from playing around. But when you're really thinking about, okay, well, what am I going to build? what am I going to ship to customers? The technology is part of it, that's kind of like the backbone. But I think what will really set different products and services apart is what value do they ultimately provide?

Yeah, you need the left brain and the right brain Yeah, and very cross functional team. yeah, So now we have a segment called five yeah, questions. Did tell you about that wonderful. how we like it? So I'm gonna ask Yeah. you five questions. Just give us the first thing that comes Yeah. to your mind in rapid fire style. What do you see as the biggest opportunity for AI right now?

I think the biggest opportunity is finding true product market fit and solving real user problems.

What is the biggest misconception about AI?

Let's come back to that one.

Okay, what was the first career you wanted?

I wanted to be a doctor.

Me too. When is there too much AI?

That's a great question. I think we are very far from too much AI. Perhaps that's when you can't tell if AI or a human made something. And we've kind of gotten so far far down the path and away from that human component that maybe that's a little bit too much AI. But

Aren't we there yet? I mean, already?

I think we are in very, very early days.

What is the one thing you wish AI could do right now that it can't?

I'm excited for AI to get more into the robotics space and actually take care of physical tasks. For me, I am a new-ish mom and just there's so much stuff that burdens my life that I would prefer to have that time to either spend with my family or at work. So I'm excited for some of the physical, maybe robotics opportunities to I love come.

that. A little bit less cerebral, a little bit more physical, please, right? All right. What is the biggest misconception about AI?

it's a tough question. I think the fact that any of us really know what the future looks like, but I think we're all trying to learn and be humble along the way. There's probably a misconception that we can create that twenty thirty product strategy and probably know exactly what the future brings.

Hm, hm. By the way, this is a hard question. Sam, what is your answer to this? The misconception thing, Yeah,

I think my answer would be more sort of people not understanding how limited these tools are. You know, it's sort of like AI can do all these things and LLMs can do all these things. When it's just a Bayesian update of the next probabilistic next word, and there's a lot less thinking behind it than there is just pattern. I I think that would be my misconception. So anyway,

I think it's a hard question because I think there are very divergent perspectives out there today. And so there are so many opinions that it's hard to say that there is this perspective out there. Because I would argue that there isn't a clear, consolidated, agreed upon sense of what the future looks like. So everyone's probably going to be a little bit wrong.

that's a ill posed question to begin with, right? It seems like it, right? Like I w- I would say that it's gonna replace humans like that. That's, for that one is me, like

the most concern. the biggest, Yeah,

biggest misconception. Although I know this is off topic, but like what you said about the probabilistic next letter or next word, it's not obvious that that's not how our brain works, by the way, right? It's a whole bunch we, you know, we of

just could be doing a better job of it, so, you know, we're just a I mean, it's not better trained.

obvious that that's just not how our brain works either. And could be, yeah,

A fun conversation is what kinds of things that humans do could AI not do, presumably do in the future? I think a lot of things around experiences, and Shervin, you'd mentioned, I think more of that human element hopefully makes us unique. And then I think it's that those big visionary things that happen that maybe aren't able to predict, like going to the moon. Or, you know, will AI kind of think through those experiences, or just these big breakthroughs that happen that I wanna say only humans can really dream stroke of up these,

genius, right? yeah, these strokes The of genius, yeah,

but maybe we'll be wrong.

yeah, we're not the probabilistic machines that Shervin thinks we are. I've really enjoyed learning more about Slack today. I mean, I think increasingly people are using communication channels like Slack at work, and these tools like Slack really fit that distributed, asynchronous work world that we're in. But calling it a communication tool is just not right and far too narrow. And I think you've really shown a lot of the the opportunities that we have here. Thank you for taking the time to talk with us today.

Thanks,

Thank you so much, it was a pleasure.

Thank you.

thanks for listening. On our next episode, Shervin and I get some AI fuel fashion advice from Jeff Cooper, senior director of data science at Stitch Fix. Please join us.

Thanks for listening to me, myself and AI. We believe, like you, that the conversation about AI implementation doesn't start and stop with this podcast. That's why we've created a group on LinkedIn specifically for listeners like you. It's called AI for Leaders. And if you join us, you can chat with show creators and hosts, ask your own questions, share your insights, and gain access to valuable resources about AI implementation from MITSMR and BCG. You can access it by visiting MITSMR dot com forward slash AI for leaders.

We'll put that link in the show notes and we hope to see you there.