An Industry Benchmark for Data Fairness: Sony’s Alice Xiang

Why did one large company decide to create a data fairness tool that's free and publicly available? Find out on today's episode. I'm Alice Xiang from Sony, and you're listening to Me, Myself and AI.

Welcome to Me, Myself and AI, a podcast from MIT Sloan Management Review, exploring the future of artificial intelligence. I'm Sam Ransbotham, professor of analytics at Boston College. I've been researching data analytics and AI at MIT SMR since 2014, with research articles, annual industry reports, case studies, and now twelve seasons of podcast episodes. On each episode, corporate leaders, cutting edge researchers, and AI policymakers join us to break down what separates AI hype from AI success. Hi listeners, thanks again to everyone for joining us.

Today I'm talking with Alice Xiang. She's the global head of AI governance at Sony and lead research scientist for AI ethics at Sony AI. She leads a team that guides the establishment of AI governance policies and frameworks across all of Sony's business units. She's been a research scientist and a whole bunch more. Alice, thanks for joining us.

Thank you so much for having me.

Hey, to start, we first talked a few years ago when you were at the Partnership on AI, and I'm curious what you're up to now. Can you tell us about Sony's work on responsible AI ethics and governance?

Yeah, sure. Sony is a large multinational company headquartered in Japan, with a diverse array of businesses around creative entertainment and technology. So we have operating companies focused on music, motion pictures, video games and electronics. And AI became an early focus of ours back in 2018 when we first set up our AI ethics guidelines. As a technology company, we wanted to ensure that this new and emerging technology was being used responsibly across our business units. And indeed, since then, it's only grown in importance for our company.

I have two hats on at Sony, one is as global head of AI governance, where over the past several years, we were one of the early companies to start investing in AI ethics. And when I joined, I established our AI ethics office and our processes in terms of how different AI uses and AI integration into products and services is evaluated for responsible AI. Now we're at the point of not just thinking about AI ethics, but also AI governance.

So how do we establish these frameworks for ensuring the responsible evaluation of these technologies that are increasingly being integrated in every aspect of business? And that's kind of one hat that I have on, kind of policy making, guidance setting, um, so on and so forth. And then the other is leading our AI ethics research team within Sony AI. A lot of the work of my team over the past several years has been looking at what are some of the fundamental gaps and barriers that practitioners face in terms of being able to develop responsible AI in practice.

And one of the major areas there is lack of ethically sourced data, even for pretty basic things like being able to check for bias in models. There's not really great fairness evaluation data sets in many areas like human-centric computer vision. So we've been doing a lot of work there, and that's recently culminated in a publication in Nature of our Fair Human-centric Image Benchmark, also known as FHIBE. And so we really hope that our work can help enable the broader community, both within Sony and outside, to be able to move towards more trustworthy and responsible AI development.

Yeah, I got pretty excited when I saw the FHIBE work. I think it's pretty interesting. It's definitely a problem, and I'm glad there are a few people working on this. Give us some details, like what exactly is involved in FHIBE and what are the pieces? And how do I use it tomorrow if I want to?

It's so interesting, cause when I first got into this AI ethics space around computer vision, I kind of assumed that a lot of my role would be helping people in how exactly to do fairness assessments. Like it's a pretty difficult area in terms of how do you measure fairness, what does fairness mean, what do you do when you have biases? But I realized, actually, the biggest initial barrier folks have is even just being able to evaluate for bias.

And so theoretically, what you want there is you want a data set that's been ethically sourced. So that means there's been appropriate consent and compensation and sourcing throughout the process. So everyone who's participated in that data collection process has been appropriately compensated and has consented to their data being used and has control over how that data is being used. And then also for bias evaluation in particular, you want to have a very diverse, ideally globally diverse data set, because you don't want to be checking for bias.

But, you know, all of your subjects have light skin, for example, there's in that case, you really wouldn't be able to tell whether your model performs well on darker skin tones. And so, you know, it's quite easy to say, you know, what would need to go into this. But when you actually look at the different data sets that are available, it turned out that the standards in the field were quite low. Computer vision is a field that played a major role in terms of the deep learning revolution.

And part of that was these web scraped data sets that were massive and relatively cheap to source. Because they, you know, didn't involve any of these considerations of consent and compensation. And even though it's been many years now of this field progressing and the technology has just gotten better and better, that baseline of relying on problematically sourced data sets hasn't necessarily changed a lot. And so a lot of our work in the past several years has been really thinking deeply of how do we actually do this in practice?

Like, it's very easy to say, yeah, you know, like, please collect data from people around the world. Please ask them for consent, please pay them, you know, and then please, like, make a rigorous benchmark that can be used to check a lot of different types of AI models. But that's much more difficult to do than it is to say, and that's what a lot of our project has been about.

Yeah, I think that's always the case with this, you know? No, I think no one comes out there and says, Hey, you know, I'd really like to be unethical. And let's let's have our company have terrible governance. I mean, it's kind of like data cleaning in some way. I mean, everybody knows that they want clean data, but it's easy to say and hard to do. Let's start with measuring bias. How does that manifest itself in corporate America in terms of mistakes?

Yeah, great question. For the most part, I I think there's a lot more awareness of bias than there is, like, good practices of actually measuring and mitigating it. So when folks don't do this, what ends up happening is you have technologies that are released that maybe don't work well for certain subpopulations. And especially in areas like human-centric computer vision that are used in surveillance contexts, law enforcement contexts, but also just very everyday contexts. Like, you know, unlocking your phone, making payments on your phone, or border control, like, you know, going through border entry as well.

These are all different areas where, if the technology doesn't work well for you, at minimum, it's an inconvenience. Maybe you have to look at your phone several times at different angles before it can recognize you. But this can also lead to much more problematic impacts in terms of anything from financial fraud to folks being wrongfully arrested. So this is an area where I think everyone knows it's a problem, everyone wants to make it better. No one actually wants these technologies to perform poorly on folks.

But if you don't have good ways to measure bias in the first place, then there's no way you're going to be able to then further try to mitigate that.

Yeah, actually, I like your gamut of possible badness. I wouldn't be surprised if we added up the societal cost of all the tiny problems and that there would actually be a big number in terms of, I mean, I don't know, just something silly, like total lost productivity from looking at your phone three times versus one time. When we have these measures, actually, maybe I'll tell a story here. I mean, one of the things I do in class is I have a data set that has, uh, star-bellied sneeches, if you remember star-bellied sneeches from the Dr.

Seuss. And we're not discriminating against stars or non-stars, and what we do is correlate data in that data set and say that even if you ignore those stars, you end up with biased data. So help me with my class exercise. We end up making a bunch of biased models that inadvertently use stars on star bellies. What should students do from there? And if you're listening students, don't copy this.

In terms of how FHIBE can help in this um situation, for example, there's two major components of FHIBE. One is like the ethical sourcing component, which I talked a little bit before of the consent and compensation. The other is making sure that you have a wide variety of diversity and labels for these different sorts of demographic attributes and other attributes as well. And so a few notable aspects of FHIBE were that we did use self-reported demographic information in an effort both to ensure our labels were more accurate and also that we weren't relying on third-party annotators to guess.

For example, you know, is this person from this ancestry or that ancestry, this gender, that gender? You know, it's it gets into a pretty dicey place pretty quickly. So self-reported demographics were really key. And then we had also extensive annotations about the environment, other physical attributes of the person, the cameras being used, all sorts of things. And this really allows you to kind of slice and dice a bit more. What are some of the relevant types of biases that you might be concerned about?

And also, on a more granular level, diagnose what might be some of the underlying causes. So, for example, when we talk to folks in computer vision, something like skin tone, it might be the skin tone itself. It might be, you know, issues of contrast, like with the background and stuff. So there's many different ways in which someone's say race or gender might manifest into visual artifacts that then can make the model perform better or worse. And that's really useful for folks then to be able to figure out how to improve their models.

And so using your example, you know, we want to know which creatures have, like, the stars on their bellies, which don't, so we can see how they're being treated differently. But not just that dimension, we wanna see a lot of other dimensions as well, and then be able to see, okay, where we see those differences in performance for the model. Why is that, like, why is the model having such a hard time on stars versus not stars?

And then go from there in terms of further improvements.

Yeah, I like that because it seems like you're pointing out a couple of different things that one. I think when we talk about these things, it tends to go down a path of, oh, get more data. And certainly there's nothing wrong with getting more data. I think we all would like more data. It's not always practical or possible to do that. And you actually mentioned a whole different category of the ethical issues around the ways that you get it, and that's true as well.

But I mean, one thing you're pointing out there is that the modelers can actually make some technical improvements here. Hey, I don't want us to say, all right, well, we give up, you know, we just take whatever data we have and just let the modelers figure it out. Because that never is a good solution. But you are offering some hope here, and I guess that's some of what FHIBE is about is giving modelers a better data set to work with to do that.

Have you seen people, you know, pick up and and start using this?

Yeah, we've seen a lot of great pickup, like even just in the first couple weeks of FHIBE's release. There were over sixty different institutions downloading it, folks from academic institutions, industry, and government institutions. So it was great to see kind of the wide swath of folks who were interested in using FHIBE. And, you know, really, our hope is that this can be an industry benchmark in terms of both lifting standards around responsible data collection in general.

So, you know, that can be true, regardless of whether the future data sets are collected for training versus evaluation purposes, fairness versus not fairness-oriented purposes. And then also, most immediately, to be able to start checking their models, since fairness evaluations can be very empowering in that it opens up a lot of different avenues of how you could make model improvements. So, like you mentioned, there's possibilities in terms of collecting more data. There's also possibilities in terms of trying to think about loss function of the model.

What is it being optimized for? How to, like, train it to optimize a bit more for different groups and thus have more balanced performance. There's also possibilities on the non-technical side as well. For example, your model does not perform very well in certain lighting conditions, then maybe you don't use that particular model for certain downstream uses. Or maybe you try to have mitigations, like if it's on a device, maybe that device will have like a flashlight or something to illuminate before it does whatever task is carrying out.

So it's always good on the mitigation front to think about these models as being embodied in, you know, real-life systems as well. Cause it's not just a matter of the model itself, it's everything around it that then impacts how ethical the use case is.

I like a lot about that because I think most of the world would love it if if you would come out and say, all right, here's a magical answer to solve bias and to solve fairness. And, you know, you just use this benchmark. And what you're pointing out is a lot of different mitigation strategies that each one, I'm guessing, is imperfect. I mean, a flashlight on a device will probably help, but not completely solve. But, you know, you add enough of these things together and then we're improving.

It's not a oh, we've solved the problem, but it's made some steps towards that. And actually, that makes me think about a a phrase I was reading something that you're talking about, uh, data nihilism. So may maybe talk briefly about what that means for listeners who didn't read that article and can tie that to what you were just saying.

Yeah, great. When we first started FHIBE, you know, it was really like an AI ethics problem, and so we kind of put together all these ethical desiderata, including consent and compensation. And at the time, there wasn't as much discussion, I would say, in the mainstream about ethical sourcing of data for AI models. Nowadays, though, I think that has grown a lot, especially with the whole gen AI revolution. Now everyone's very conscious of the fact that their data and their content is probably being ingested by AI models somewhere somehow.

And I think that can lead to a sense of data nihilism, where folks feel like, okay, either we have these super powerful models, which, you know, maybe at this point the cat's out of the bag there. And we just have to give up on all of our data rights, we have to give up on control there, or we try to reverse things, but maybe that's not possible. And that's what I mean by data nihilism, this sort of like feeling of this dichotomy that we're trapped in, where we really can't have the technology and also have any sort of control over our data.

And I think what's really notable about FHIBE is we really sought to show that you could source data in a more ethical way. Obviously, it's more difficult, it's more expensive, but, you know, FHIBE was kind of the proof of concept that at least on some scale, you're able to do this. And there's so many brilliant minds right now working in the AI space that if these practices are considered important, then, you know, we can figure out ways to try to scale this and change how these technologies are being developed and preserve more data rights in the process.

Yeah, because I get that, that whole idea of, well, it's out there, my images are out there, everything is out there anyway. But, you know, we can all make a bit of progress towards that, and if everybody does a little bit, then it can help. I I like that overall message. How does that reflect back into Sony products?

FHIBE is being used as well across Sony, even before the launch. We tested it out with a number of our business units that are developing computer vision technologies. And so it has become, you know, an important part of enabling them to do fairness assessment more. And that way, we are able to do that sort of diagnosis. See, you know, are there any failure modes of these models? And then work with them in terms of possible mitigation strategies as well.

And I think it's been great to see that because again, like, the release of FHIBE publicly is to try to encourage this happening, you know, everywhere in the industry as well. And we hope this will put more attention on this issue and unblock folks to be able to see. Yeah, like, there maybe are some things that could be improved in these models before they go out, and hopefully eventually that becomes more industry standard as well. Cause one thing that's quite difficult now is, you know, there aren't without necessarily a requirement, so is to to do these sorts of assessments.

It's really on individual business units or companies to decide that they care about this and want to assess for bias.

Yeah, that last point's a huge point, because if you give someone a a questionnaire, do you want to do the right thing? Everyone's going to tick the yes box. But in the end, we all are constrained by time and resources, and I admit I'm lazy. If there's a shortcut, I'm going to say, well, why don't I go down that path? And it certainly has been a shortcut within the community to go for this data that's just sort of out there and lingering.

But I mean, I think one thing you've done there is that moving towards making it easy. And we've had a previous guest, Ziad Obermeyer. He was working with medical data. I mean, it's very hard for normal people to get a bunch of medical data to build models. And his thinking was that Nightingale would collect a bunch of data and then let the solving of the data problem differ from the solving of the data sourcing problem. And, you know, that's a lot of what you've done there.

I mean, you mentioned FHIBE's largely around image information. What about things like voice, sound, other modalities? How should we be thinking about those?

We focused FHIBE on human-centric computer vision, precisely because it is one of the most sensitive areas. Like, there's just so much biometric information, other personally identifiable information that is available in images. And also, you know, the IP rights around images are particularly important as well. And so, you know, I think in other modalities like voice and such, they have similar challenges and similar considerations as well around the consent of the individuals being recorded, for example. And in certain contexts, there might be also copyright considerations as well.

But yeah, I kind of see this as like FHIBE sort of took a bit more of a superset of a lot of the ethical issues that might come up. And like in most other modalities, you'll see a bit more of a subset of that. And so there's obviously more research that would need to be done to kind of apply those. But some of the general concepts in terms of, you know, consent, compensation, privacy, IP, diversity and fairness, all of those kind of at a high level would apply in these different contexts.

So we hope that some parts of this can be recycled as well for folks that are collecting data in other modalities.

Yeah, I like that. I mean, it's it's a principle level, in some sense, it's just a a stream of ones and zeros. So how we interpret that is largely up to us. Maybe I was, maybe I was self-motivated, cause I've never been able to get through a phone tree successfully because nothing recognizes my beautiful Southern accent. But that's a that's a problem I face. I guess we'll have to wait for the next version of FHIBE to help out with that.

Yeah, it's a great example of how important diversity in these evaluation data sets is. And and also, yeah, you can imagine, like some of the challenges, too, in terms of like how to identify folks with different accents and how to classify those and how to source from. I guess, you know, not just around the world, but very specific regions as well, in order to get that kind of diversity and then all of the different languages as well layered on top.

So definitely also a very rich area of research, and we hope that, you know, FHIBE helps inspire more of this as well. Cause I think the reality of ethical data collection is a lot of it is quite difficult, challenging work that requires this level of like operationalization and thinking about real world challenges, too, that often, I guess, researchers shy away from. It's not as glamorous as, you know, coming up with the new way for models to actually learn and such.

And algorithmic improvements usually are much faster to develop and publish, or maybe not always faster to develop, but at least, you know, people get very excited about the technical improvements. That said, you know, when we talk about responsible AI, you know, like what you mentioned, it's really hard to make progress in these areas if we don't actually think about real world. How do you collect data from different folks?

Actually, we had a guest on a few weeks ago from Wendy's who was working with their drive through restaurants. And you know, that's all voice information. And you know, I remember Will was talking about how how many different varieties of the ways that people speak there are. And and they faced a lot of challenges in that process there. But going back to FHIBE, showing that example, that it can be done and that it's a good example, that this is a process that you can follow there.

I want to switch here a little bit and talk about how you got there. You know, when I last talked to you, you were at the Partnership on AI. Tell us a little bit about your career path and how you ended up at Sony and how you got interested in these things. Take us on that path.

I feel like it's sort of a winding path. Overall, my background is very interdisciplinary, so I started out more on the math and economics side, thinking I was gonna work in economic policy, and so that was my main target there. And then my first stint working at a tech company and developing my first ML model really was very illuminating to me. Cause at the time, that's when I first got interested in these issues of algorithmic bias, because I realized the model that I myself was developing was quite biased.

The data was skewed towards certain geographies, what it was learning like, made a lot of sense for people in those geographies, but not elsewhere necessarily. And part of it was also kind of, you know, my personal experience as well. So the data I had access to was primarily from folks on, like, the East Coast and West Coast, and this was job-related data. And for myself, coming from Appalachia, I knew that, you know, it was quite different in terms of the economic opportunities that people had there.

And any sort of model built on this data was not necessarily gonna work the best for folks like those who I grew up with. And that concerned me a lot. I talked to a lot of folks, you know, in the company about this, but there was no such thing at the time as like a field of algorithmic bias. This was like twelve years ago now, but it got me really interested in this area. And how do we think about ethics and justice in the context of these new technologies that we're learning from massive amounts of data?

And that kind of sort of steered my career more into, like, the tech policy space and the AI and algorithmic fairness space in particular. And so after finishing all of my graduate programs, I have graduate degrees in law, statistics and economics. I ended up working a bit at a law firm and then going to the Partnership on AI since I wanted to really focus in on this issue of algorithmic bias. And there I started up a research lab focused on these areas, and pretty quickly, these issues of data availability became quite salient.

If you look at the algorithmic fairness literature, there's a lot of emphasis on like metrics and how do you measure bias, how do you mitigate it? Cause those are kind of the really fun technical problems. But at PAI, we were doing these multi-stakeholder convenings, where consistently convening after convening, all of the companies were saying, Yeah, you know, like, we actually are struggling with that first step of, you know, if we want to collect data that has demographic information in it, our privacy team will just shut that down.

But then if we don't have any demographic data, it's really hard to do any sort of fairness assessment. So kind of going back to the star belly example, if we don't even know who has stars on their bellies and who doesn't, then how are you going to tell if your model works well for creatures with stars on their bellies? And so that became a major focus for me from a research perspective of what can we do in that context?

And when I moved to Sony, I thought that this was a really great opportunity to not just report on this issue and kind of try to impact this more on a policy level, but instead to kind of create a solution to this problem. And that's what inspired FHIBE. Since it's one thing to say, yeah, this is a problem, people need to do something about. It's another thing to create a data set that fills the gap.

I like your bringing up the catch-22, because I think that's something that we don't always appreciate. That, you know, we have to collect a bunch of seemingly invasive information in order to assess whether or not we are making decisions based on seemingly invasive information. And this is an inherent problem. One of the things that we do here is I've got a little quick segment here where I like to ask some rapid-fire questions and just answer the first thing that comes to your mind.

What's moving faster about artificial intelligence than you expected, and what's moving slower?

That's a good question, I guess. What's moving faster is just how much it's being integrated into everyday life, especially in companies. I think it's been interesting to see just in the past couple years how everyone's KPIs have suddenly become about AI adoption. And that would've been hard to predict maybe five to ten years ago. And then in terms of what's moving slower, in general, kind of actual implementation of anything around, like AI ethics in model development.

The field has been around now for several years and there's a huge literature at the moment, but it still often feels very divorced from actual practitioners implementing these different techniques. And, you know, again, that's kind of part of the motivation of FHIBE to create that incentive there. But yeah, it has been something that I haven't seen as much of across the the field.

I have to say, I'm not a bit surprised that that was your answer about about slower, given your background. What's the worst use of AI? How are people using this technology poorly?

Say contexts where there isn't sufficient human expertise or oversight, but there are very important decisions being made. I say it kind of abstractly in a way, because, you know, I think oftentimes folks will point to specific types of use cases, like high risk use cases, like, you know, HR or healthcare or criminal justice. And I think it's a little bit more use case specific than that. Like, you can theoretically have a good use in those domains if you have the proper human oversight, the proper training, both for the individuals and, you know, proper oversight of the technology.

But where it's just kind of being used in a very autonomous fashion, and there's either no one providing oversight or the individuals providing oversight don't have the knowledge or expertise to do so. That's where I would be most worried.

How do you personally use these tools?

Not as much as maybe you'd expect. I feel like I spend a lot of time auditing them, so it's probably that's the most common use case for me.

I'm not sure that counts as a normal use case. Yeah,

Time wise, probably the most common is me trying to audit them and see where they might have failure modes. But I also do use it in my daily life to, you know, check on the health of my plants, for example. So, yeah, I will say, you know, they are quite useful in contexts where maybe you are a novice and you want some starting point for diagnosing things.

one question we used to ask people was what the first career you wanted, and given the variety of your background, let me just ask, what did you want to be when you grew up?

When I was a little kid, I wanted to be an artist. And so, in a way, I feel like my current work is starting to nudge back into that direction with the way AI development has gone. And a lot of these AI ethics issues now actually coinciding a lot with artist rights, which was not like at all the case when I first entered this field. But that is kind of nice in that I have always been really interested in the field of art as well.

I was not expecting that. All right, well, this has been fascinating. I think that the kinds of efforts you're making, which are about making it easier for people to do the right thing, that, you know, FHIBE is one example. But as long as societally we make it easier for people to do the right thing, then people will be more likely to do the right thing. And so I'm thrilled at the amount of effort and and work that you all put into doing this.

Thanks for taking the time to talk with us today.

Yeah, thank you so much for having me. And yeah, hopefully our work can help inspire others also to do the same.

Thanks for listening today. I hope you'll download FHIBE and see if their data can help. On our next episode, I'm joined by Taylor Stockton, my former student and chief innovation officer at the U.S. Department of Labor. Please join us.

Thanks for listening to Me, Myself and AI. Our show is able to continue in large part due to listener support. Your streams and downloads make a big difference. If you have a moment, please consider leaving us an Apple Podcast review or a rating on Spotify and share our show with others you think might find it interesting and helpful.