AI Goes Grocery Shopping: Instacart’s David McIntosh

What if your grocery cart remembered what you forgot before you did? On today's episode, we talk to one tech leader using physical AI to improve the grocery shopping experience.

I'm David McIntosh at Instacart, and you're listening to me, myself and AI.

Welcome to Me, myself and AI, a podcast from MIT Sloan Management Review, exploring the future of artificial intelligence. I'm Sam Ransbotham, professor of analytics at Boston College. I've been researching data analytics and AI at MITSMR since 2014, with research articles, annual industry reports, case studies, and now thirteen seasons of podcast episodes. In each episode, corporate leaders, cutting edge researchers and AI policymakers join us to break down what separates AI hype from AI success. My guest today is David McIntosh, chief connected stores officer at Instacart.

Instacart is a grocery delivery and technology company. David runs the in-store side, the smart carts and software that grocery stores themselves buy and use. Before this, David built Tenor, the Jiff search engine inside iMessage and WhatsApp, the Google bot. So he's gone from helping people find the right GIF to putting cameras and AI on shopping carts. And he argues that grocery is one of the hardest places there is to make AI actually work. Dave, thanks for being here.

Thanks for having me. Great to be here.

I can already tell that I have offended you, though. So let's let's get right down into it. Is it GIF or GIF?

You know, when I was running Tenor, we joked that we would hire both types of people. I can tell you a funny story. Often in parties, I will use the word GIF, because maybe it was a couple months after we launched the product. I was at a dinner party that a friend was hosting, and everybody was going around the table talking about what they were working on. And the person next to me just couldn't quite understand that this was a GIF company, not a gift company.

So he was asking me all kinds of questions around how do you compete with Amazon? I just was sort of puzzled. And then finally it dawned on me. Oh, he was thinking, you know, gift with a T the entire time. And so generally, my rule of thumb is if it's a loud, noisy room, I'll use the GIF. It's very hard to confuse that for gift, but in other settings, I use GIF.

So both ways now now actually as I think about it, we might have more trouble today because GIF might be confusing with peanut butter if we're talking about grocery stores. So I don't know. They're just landmines everywhere. But let's bypass all that and just get down to the real part. What are connected stores and what's a cart?

Certainly. With connected store, what we're really doing is taking the convenience and the personalization that our users love about the online experience and we're bringing it into the store. Increasingly, we were hearing from our retailers that we had helped bring them online. The last decade of Instacart's history, we brought their business online with e commerce solutions, fulfillment solutions, retail media solutions. And they were coming to us saying, Hey, Instacart, you brought my business online in the last decade.

How can you help me in store? I'm trying to drive more omni-channel customer behavior. I'm trying to drive new retail media revenue. Can you help me with these problems that I'm facing in the store? And then on the other side, a lot of our users were coming to us saying, Instacart, I love shopping online, but I also go to the store. And so that was really the key insight that birthed connected stores a couple years ago.

Our vision is that on a five to ten year horizon, customers won't have to choose between shopping in store and online. It'll be one single unified mode, all powered by Instacart. Now, how are we building it? You asked about Caper earlier. The way we think about it is that we wanna digitize all aspects of the store, the shopping experience with Caper, the deli and the prepare or counter with FoodStorm, the software, the shelf tags, the ESL tags that sit on the shelf with our pick-to-light technology that makes it easier for shoppers to find things.

Digitize planning for the store with our in-store mode, digitize inventory in the store with our shelf intelligent solutions, and really connect all of these technologies together. And our approach is that the technologies have to be modular. So we don't go to retailers and say, hey, you should adopt all these six products at once. That wouldn't work, obviously. A retailer can start with, for example, a food store in which digitizes the deli and then ladder their way up to other products.

All of these solutions connect to each other and reinforce each other. So then to your question around caper, Caper was an acquisition we made a number of years ago, and it's a smart shopping cart. Think about it as a traditional shopping cart that all of us have used many, many, many times throughout our life. But with a digital screen, camera sensors, the entire basket's a scale so you can weigh produce in the scale, location sensors, so the cart knows where it's at, side-facing cameras so it can understand what's on the shelf, um, as well as an Nvidia Jetson board on every single cart that's running all of the sensor fusion and other AI algorithms at the edge.

And really, what we're seeing with Caper is that it transforms the shopping experience for users in the store. As you put things in the cart, it keeps track of your running total. You can clip coupons, find deals, very, very popular capabilities, you can bag as you go. When we started connected store a couple years ago, you would hear me talk a lot about with respect to Caper, skip the line as the primary value proposition. And really, what we've been seeing as as we've scaled, we're now, for context, live in more than one hundred cities.

We've tripled year over year. What we're seeing is that the proposition is much more rooted in the screen. Users are spending thirty minutes plus in their shop, and they're highly engaged with the screen. The number one thing I hear from from customers is they love the running total. They love being able to keep track of what they're spending because the vast majority of Americans are shopping on a specific budget. And nobody wants to get to the line, realize they're over budget and have to put something back.

Second, I hear the deals, the coupons. As you're going through the aisle, the location-based system will flag deals and coupons for you. If you put an item in the cart, it'll surface the coupon. It might even say, hey, you'll get an additional dollar off if you get this other related item in the store. And maybe the the final thing I'll say about it is that you can really think about it as an ecosystem with physical AI at the center.

We built a product that serves retailers that are deploying it in the store. It serves consumers, delighting them with deals and other capabilities around value. It's helped CPGs drive discovery of new items, and then finally, store associates. You've got to make the product simple and modular for the store associates to deploy. Like, as an example, as you push the carts into each other or to a charger, they just charge. Sounds obvious, uh, you know, in hindsight, but before us, the industry was really based around manually plugging in carts, which, as you can imagine, nobody wanted to do.

The carts weren't charged, people stopped using them, or you sort of negatively spiraled. Really, what we're doing is carefully balancing these four stakeholders in this ecosystem to make a product that works for everybody. It makes everybody's life easier with that physical AI system running on the Jetson board at the very center of it. There's something kind of interesting there that you said that people tended to underspend to avoid the embarrassment of being at the checkout and having to put something back, which I, you know, now that you say that makes a lot of sense to me.

I kind of thought at first that it might work the other way, that people would see the running total and sort of freak out and say, oh my gosh, I didn't realize I'd spent so much. So do you have data around which way people are tend to make that mistake? Definitely, and this is something that our retailers have now measured quite a bit, as you could imagine. What we've seen is that it really is around these capabilities that help users track their spending and then for retailers, increase consumer spend in the store.

So we now have a number of of retailer partners that, you know, over the last year, two years have done multiple, very in-depth studies where they look at an A B test users who have used it, users who haven't, you know, study the loyalty numbers and study their spend over time. And consistently, what they're finding is there's significant, in some cases, double-digit percentage increases in consumer spend using the Caper carts. Now, I'm not saying all of that is just because of the running total capabilities.

We know, for example, that our recommendations-based capabilities also add to that proposition. So, for example, we launched a capability where as a user was approaching checkout, we surface a screen that says, Did you forget? And so, for example, on that screen, one of the items might be the yogurt, right? And we hear from users all the time saying, Wow, this is amazing. I actually did forget the yogurt. And so that one feature alone drove an incremental 0.9% spend.

We've been able to measure those types of capabilities, recommendations, reminding users to buy things they probably have forgotten. Those things stack on top of the benefit of showing a user the running total, and those all ladder up to the basket size increases that retailers are experiencing, which is one of the big tailwinds behind the adoption that we're seeing of the product in the industry.

So we had a guest from Wendy's on who was involved in their drive-through automation. He would go through the drive-through and just watch all the stuff that they screwed up to figure out what they needed to get right and need to get better. What kind of things have you done wrong that have been learning experiences for you through watching what people do?

Our number one lesson is that success comes from the retailer leaning in and driving these in-store technologies. You cannot be successful without the retailer. And so what does that mean? Well, that means that in many ways, the number one thing we can do to drive more product adoption, more usage in the store, is the retailer grabbing the reins and pushing. And to do that requires balancing that ecosystem that I told you earlier. We cannot be successful without the store associates being engaged and supportive.

And so going back to the example of stackable charging, if the carts have to be plugged in individually, associates will hate that. So the carts will be charged for a day, they'll be charged for day two, day three, maybe the first week after launch. But over time, you'll start to see a real erosion in support for the product. Similarly, associates want all kinds of very easy tools to be able to run the program, monitor the carts, do the audits.

Retail environments are incredibly chaotic. There's a million different edge cases, right? And so as an example, we initially thought that it would be a good idea to enable associates to keep track of the carts via tablets in the store. And indeed, a lot of stores do have tablets that they issue to associates. The problem is the tablets, again, it's a retail environment. It's complex. They're often misplaced. They're often another part of the store. And so imagine there's a customer that's waiting for a verification at the end of their shop, and the tablet's at the end of the store.

They've got to wait three minutes. It's just a terrible experience for everybody, right? And so giving staff the option of conducting those verifications on cart versus on the tablet, that slight tooling change, that slight tooling tweak, uh, is incredibly important. But then I would say, even more broadly, we've seen just how many edge cases you encounter in the real world and how you really have to solve for them. So a ton of stores don't have Wi-Fi that is strong throughout the store.

You often have these gaps as you're walking throughout these big stores, and so many stores don't have good LTE or cell connection, uh, you know, either. And so we had to build the cart to really deal with these big gaps in Wi-Fi throughout the store, which means a lot of these intelligent systems need to be on device at the edge. They also have to be on device and at the edge because users expect a really fast response, right?

They don't want to have to wait.

Measuring, tracking, sensing, there's a lot you can do with that data. But at the same time, I think people are gonna be worried about that. And I think you're, you're sure you're not naive to the idea that people are thinking of this as a surveillance cart. How are you mitigating against that? Or what? What's your way of thinking about that or handling those concerns?

Absolutely, we've really built Caper carts with privacy in mind. The cameras are focused solely on recognizing products, right? And what's going into the cart, what you're taking out of the cart. And similarly, you know, the side-facing cameras are focused on the shelf and helping retailers understand what's in stock, what's out of stock. And, of course, from a data perspective, with respect to our our retailer partners, we're very conscious of the data that we're capturing, the sharing and so forth.

And what I will tell you at a high level is, uh, our our retailers really see the value in using the data to drive better consumer experiences. So retailers see a lot of value in saying, okay, how do we make sure that that user doesn't forget the yogurt on the way home? That's a win for a user, that's a win for the retailer. How do we make sure that we deliver really relevant recommendations for users as they're going through the store?

How do we make sure they can take their shopping list that they built online and bring it in in the cart in the store? And so this is an area that we're super conscious around, but again, we've we've really focused the data capabilities around the products and understanding the products that customers buy so we can ultimately deliver more value to consumers.

Yeah, so one thing I always think about in these scenarios is where does the economic welfare go? By economic welfare, I just mean the total value created when a transaction happens, the value to everybody involved added up. Economists usually split welfare into two pieces. There's consumer surplus, which is the value I get as a buyer beyond what I actually pay. If I'd happily have paid five for something and I get it for three, that surplus is mine.

But there's also this producer side, which is the value that the seller captures above what it costs them to provide it. If we add those together, you get the total welfare, the total pie. So whenever one of these scenarios comes up, the question I'm really asking is this. Are we making that pie bigger, or are we just cutting it into different size slices and handing them to different people? Take your forgotten yogurt example you mentioned. That looks good for everybody.

I wanted the thing, I got it. I didn't have to drive back to the store or spend the gas and time to go get it. That's brand new value that didn't exist before. The pie got bigger and nobody had to come out behind for me to come out ahead. But your CPG example with me buying a brand that I might not have bought otherwise, well, that's moving a product choice from one vendor to another. It changes what vendor captures the surplus, but it doesn't really change my side of things as a consumer.

I was gonna buy something either way. And here's where it gets more tangled in this scenario, because the grocer is a third party in all this. The producer surplus isn't just one lump sum. It's divided amongst the CPG company that makes the product, the grocery chain that sells it, and so on down the line. So something like getting people through the store faster isn't really about my surplus or the product maker's. It's the grocer lowering its own cost and keeping that value for itself.

So I'm having trouble here because the welfare seems to be moving in a whole bunch of different directions at once. What do you think the net is going to be? Is it just an overall improvement in efficiency? And where does value actually land with the shopper, with the vendor, with the CPG company, with the grocer? And where is it coming from?

Yeah, I think it is an overall welfare win for sure, and I'll give a couple examples. Even taking your example, I would argue I get a lot of delight from discovering new things in the grocery store, right? Maybe it's a seasonal mango that's really ripe and delicious, right? And today, often what retailers are doing is they'll put up paper displays, like a local retailer near me puts up like a little cart wheel that spins. If you think about all the displays as you walk through the grocery store, those didn't happen by magic.

That happened because often a CPG actually had to pay somebody from their extended team to print the item. First, bring into the store, set it up, put the items on it, and they've got to go back and monitor it over and over again. There's a lot of friction involved in setting those things up. And so if you can deliver these recommendations to users and they're more relevant to a person, you can help everybody discover that new seasonal mango, for example, that new brand of cereal that they haven't tried yet.

That is a huge welfare improvement. And I think that's what we're seeing with the strength of the recommendations. That's why people are using them and tapping on them and participating in that deal ecosystem. Another example would be a CPG would certainly be willing to reach a customer and give them a discount for an item if they tried it. But often today, they have no vehicle for doing so, right, because most consumers, they don't have the retailer's app on their phone.

So I think recommendations alone, there's an enormous amount of welfare to be generated. And then when you think about the way that CPGs can reach consumers today, again, there's a lot of inefficiency baked into the system. That's where I see a lot of these win-win wins created, where for the first time, we can put together an understanding of what's in the consumer's basket, what they've bought in store, what they've bought online before, and put all those things together with this physical AI system at the center.

With that screen in front of them, you can start to deliver a whole of these new novel shopping experiences. Oh, by the way, I'll tell you, our gamification experiences were actually inspired by a store walk with a grocer. This was a partner of ours, CEO of of Good Food Holdings, Neil Stern. Very early on, we got together with Neil and team, and we were talking about how we could uplevel together their in-store experience. And Neil and Adam, who's a CEO of a couple different banners of of Good Food Holdings, they were recounting the story of how they would put plastic ladybugs throughout the store, and they would let kids find the ladybugs.

And if you found enough ladybugs, you'd you'd win a prize, right? It was, you know, maybe a new seasonal item. And so that was the inspiration for these gamification capabilities, which is we were seeing retailers were already doing it. They were trying to make the shopping experience more delightful. And so once you've got the screen in front of a customer, once it's connected to this physical AI system, you can start to deliver all these brand new, delightful experiences that customers just didn't have access to before.

I think that's another example of how we can overall lift the welfare, to use your words of customers as they're shopping. And by the way, that's the that's the big idea behind connected store. When we think about the five, ten, fifteen year view, when I walk these stores and talk to customers, you know, it's amazing just to see how much they love the product. You know, they're shopping with their kids, their eyes are lighting up as they're putting things in the cart.

Their kids are are saying, No, mom, no, dad, let me scan. I'm gonna scan the item. I'm gonna put the the item in the cart. And, you know, I think back to I went to the grocery store with my parents. That was the thing that we all did together. It was fun, right? And so what we're really doing with this product is we're personalizing the shopping experience. We're making it more delightful by bringing together all of these new inputs, the physical AI system, the screen, in a way that really can make the shopping experience more convenient, more personalized for everybody ultimately around the world.

What's next on the agenda? What kinds of things are you thinking about?

One of the key things I'm thinking about is how you start to make the stores more optimizable with all of this data that we're collecting. So a very basic example is inventory, right? With these carts going up and down the aisle that are building understanding of what's on the shelf, or with Instacart shoppers. We have a product called Store View, where Instacart shoppers that are going in a store to pick an order can also scan the shelf with their mobile phone.

Once you have this understanding of what's on the shelf, you can actually make it easier for retailers to ensure that that out of stock never happened in the first place. And one of the things that we're finding as we're talking with retailers is that it's not that useful for them to know that the item's out of stock or not, right? There's some marginal use to that, but you actually want to try to catch the problem before it even happens.

So you often want to capture while it's low in stock. And then you probably want some sort of agentic workflow built where why put a human in the middle of that? Why not have, for example, an agent communicate with the suppliers in the background? And you probably want that agent, by the way, to be deeply integrated into your point of sale and your catalog system. In some cases, we even have access to, uh, back of, uh, house information.

And so you probably want that agent to look at all of those things and then say, okay, well, maybe I need to email the supplier, right? Or call them or text them and say, Hey, you know what? You need to change the amount you're delivering this week, right? Sort of tune and tweak that. Similarly, you can imagine with all the data we're collecting on where things are getting bought in the stores. So you can imagine now that we have this understanding of where are people actually buying things in the store, that could be optimized.

Now, imagine connecting that to an agentic system where that coordination of where those displays across the store is automated, right? And so as a result, you see the future here, which is we're ultimately making it easier for people to access the food they really love and enjoy in these stores. We're making the experience not just on caper, but also in terms of the selection in the store, more and more personalized for users over time. And then I think secondarily, it's continuing to really connect all these different experiences in the store.

It's not just landing caper in the store, but then it's digitizing the deli, it's digitizing the shelf. It's really connecting all of these experiences together in a highly unified way. And so going back to really our vision, it's that on this five, ten, fifteen year horizon, consumers don't have to choose between shopping in store or online. It'll be one single unified mode, powered by Instacart, powered by this physical AI and agentic AI system that we're building.

Well, and I think back of what we've been talking about, what sort of sticks with me is this idea of measurement is coming up almost everywhere, that measurement's kind of at the heart of almost everything you're talking about. When you were talking about inventory, my first reaction was, well, you've got point of sale, you've got back office inventory, what's magic here? And I think the difference here is getting that measurement at the moment that someone picks it up and picking up on exactly where that's coming from.

And it seems like the hard part then is is less about the technology itself and more about getting the trust of people to use these systems and to think about using them. And the whole problem just seems a lot messier than I first realized. Um, I thought we'd end up talking more about Jiff versus gift, but we didn't. But thanks for getting specific about this very messy problem today.

Wonderful. Thanks for having me. This is great.

Thanks for listening today. On our next episode, Sam speaks with GoFundMe CTO Tim Cadigan. Please join us. Thanks for listening to Me, Myself and AI. Our show is able to continue in large part due to listener support. Your streams and downloads make a big difference. If you have a moment, please consider leaving us an Apple Podcast review or a rating on Spotify and share our show with others you think might find it interesting and helpful.