In brief
Responsible AI is not mainly a data or model problem. Saleema Amershi argues it is a socio-technical problem: AI will always make some errors, so builders should plan for failure. That means designing ways for people to understand the limits, override the system and steer it. Calling these problems 'failures' rather than 'issues' makes teams treat them as showstoppers instead of low-priority items.
The central tension is speed versus care. Amershi says her group is both pressing the brakes and pressing the accelerator. It slows things down to think about who could be harmed, and it builds new methods, such as ways to prototype AI interfaces, so progress can stay fast. The episode also shows how hard this is. AI myths, such as 'it's just math', and industry habits of bolting an interface onto a finished model get in the way.
Questions
What does 'responsible AI' actually mean in practice?
Amershi says it is about how to design, develop and deploy AI that is fair, reliable, safe and trustworthy. She treats it as a socio-technical problem, meaning the people using the system matter as much as the data and models.
Why does she say companies should plan for AI to fail?
Amershi says some errors are built into how AI works, because a model simplifies the world and trades off some mistakes against others. Since they cannot be removed, builders should expect them and design ways for people to notice and override them.
What is the difference between a false positive and a false negative?
In her facial-recognition example, a false positive lets the wrong person in and a false negative locks out the right person. She says the first is usually much costlier, so the system should be tuned with that in mind.
Is a biased AI system just reflecting reality?
Amershi disagrees. She says a model is a simplification and cannot capture every factor in who makes a good hire, so a gender gap in tech does not mean a biased recommendation is reality.
What are the Guidelines for Human-AI Interaction and who are they for?
They are practical design guidelines the group built from roughly 200 starting recommendations, which were grouped, revised and tested with practitioners. Amershi says they are mainly for product teams and also for researchers.
Facts
Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.
Saleema Amershi is a principal researcher in the Adaptive Systems and Interaction Group at Microsoft Research.
“Dr. Saleema Amershi, a principal researcher in the Adaptive Systems and Interaction Group at Microsoft Research, contends that if you plan to fail, you're actually more likely to succeed.”
0:00:27 · Gretchen Huizinga · Amershi's role
Saleema Amershi works at the intersection of human-computer interaction and artificial intelligence, creating tools and techniques for people who build and use AI systems.
“I work at the intersection of human-computer interaction and artificial intelligence. And so I create technologies and techniques to help people both build and use AI systems.”
0:01:57 · Saleema Amershi · Research focus
Saleema Amershi earned her PhD at the University of Washington, near Microsoft's Redmond campus, which made it easy to collaborate with Microsoft Research.
“I did my PhD at the University of Washington, which is really just across the lake from Redmond here. And so that made it really easy to collaborate with MSR.”
0:31:10 · Saleema Amershi · Education
Saleema Amershi completed three internships at Microsoft Research before joining the lab.
“And so I ended up doing three internships here at MSR. I just kept coming back.”
0:31:10 · Saleema Amershi · Career path
Saleema Amershi attended the University of British Columbia for her undergraduate and master's degrees and began as a math major.
“I went to UBC for undergrad and my master's. And I actually started off as a math major.”
0:32:12 · Saleema Amershi · Education
Saleema Amershi worked on intelligent tutoring systems at the Laboratory for Computational Intelligence at the University of British Columbia.
“And then I started working at the Laboratory for Computational Intelligence at UBC, working on intelligent tutoring systems.”
0:32:41 · Saleema Amershi · Early research
Saleema Amershi says early intelligent tutoring systems used giant belief networks hand-tuned by experts for a single system, which were powerful but not scalable, and this pushed her toward machine learning.
“you would create like these giant belief networks that were hand tuned by experts for just one system. And that was like, yes, it was powerful, but it was not scalable.”
0:33:34 · Saleema Amershi · Tutoring systems
Saleema Amershi says she was the first human-computer interaction hire in the machine learning group at Microsoft Research.
“I was the first like HCI hire in the machine learning group.”
0:19:54 · Saleema Amershi · Career at Microsoft
Saleema Amershi explains that an algorithm is tuned to optimize a metric such as precision or recall, which affects how many false positives and false negatives it produces.
“by design an algorithm will be tuned to optimize for some metric so maybe you try to optimize for precision or recall which really like affects the amount of false positives and false negative errors you can have.”
0:07:32 · Saleema Amershi · Model tuning
Saleema Amershi argues that in a facial-recognition access-control system, false positives are costlier than false negatives because someone else could access your account.
“in the facial recognition scenario, false positives are much worse than false negatives, right? If somebody accesses your account, that can be much more costly than if you have trouble accessing it yourself.”
0:08:38 · Saleema Amershi · Facial recognition
Saleema Amershi gives the example of a doctor who overestimates a clinical AI system's abilities and over-relies on it, which can cause harm.
“if a doctor is relying on a system to make clinical recommendations about their patients, if they think that the system is smarter than it actually is, they may over rely on it and that can result in harms.”
0:10:27 · Saleema Amershi · Mismatched expectations
Saleema Amershi says that when an AI hiring system is biased against recommending women for engineering jobs, many people respond that this is just reality.
“maybe your system is biased against recommending women for engineering positions. If you ask people about this fairness and bias issue, a lot of people will come and say, you know, well, this is just reality.”
0:12:58 · Saleema Amershi · Hiring bias
Saleema Amershi argues that humans and AI systems have different regions of error and uncertainty, so ideally they can complement each other.
“I think we should think about these humans and machines as having different error regions, right? So yes, each of these systems will have uncertainty, but they'll be different.”
0:14:26 · Saleema Amershi · Human-AI complementarity
Saleema Amershi says that if a system must explain itself to users, that need dictates the choice of an interpretable model.
“If you need a system that can provide an explanation to a user so that they can make an appropriate decision, you're going to need an interpretable model.”
0:16:13 · Saleema Amershi · Interpretable models
Saleema Amershi deliberately uses the word failures for responsible-AI problems because calling them issues tends to lower their priority.
“I use the word failures really intentionally because we used to call responsible AI issues, you know, issues or problems, but just that term will put it lower in the priority stack, right?”
0:20:47 · Saleema Amershi · Terminology
Saleema Amershi says roughly 20 years of research advances on developing AI systems effectively motivated the creation of the human-AI interaction guidelines.
“there's actually been a lot of research advances over the last like 20 years around how to develop these types of AI systems effectively.”
0:22:40 · Saleema Amershi · Guidelines motivation
Saleema Amershi recalls that the team behind the Guidelines for Human-AI Interaction began with about 200 pieces of guidance collected from many sources.
“We collected guidance and best practice recommendations from a wide variety of sources. I think we had like 200 to begin with.”
0:23:39 · Saleema Amershi · Guidelines process
The team grouped and revised the collected guidance and tested it with real practitioners to see whether people could detect it in interfaces.
“And then we iteratively grouped them, revised them, tested them with real practitioners to understand if they were really something that people could detect and notice in interfaces.”
0:24:08 · Saleema Amershi · Guidelines process
The human-AI guidelines and related studies came out of Microsoft's Human-AI Interaction and Collaboration working group, called the HAKE working group.
“the guidelines work, all came out of this human AI interaction and collaboration working group, which we call the Hake working group.”
0:26:01 · Saleema Amershi · HAKE working group
Saleema Amershi says Wizard of Oz prototyping, a common early-testing technique, is hard to use for AI systems because their varied behavior is hard to mimic.
“prototyping techniques like Wizard of Oz techniques, for example, that's often used to do early prototyping and testing. But that's really hard with AI systems because it's hard to mimic the different ways an AI system will behave.”
0:27:00 · Saleema Amershi · Prototyping methods
Saleema Amershi says AI policy is needed and should be driven jointly by industry and government agencies.
“I think we need policies around this, but that needs to be driven both by industries and government agencies working together, right?”
0:29:40 · Saleema Amershi · AI policy