Person
4 episodes · 71 quoted claims
Ethan Mollick says AI benchmarking company Epoch reported getting two to eighteen weeks of engineering work from a single prompt to Claude for a couple hundred dollars.
“Epoch, which is a AI, uh, benchmarking company, was reporting that they could get regularly sort of between two and eighteen weeks of engineers' work out of a single prompt to Claude for a couple $100, right?”
Ethan Mollick says the company StrongDM ships product to customers with no human involved in writing or reviewing the code.
“I know one company, Strong DM, that has been shipping product to customers, and, uh, there's no human involved in that.”
Ethan Mollick cites an Anthropic study finding that managers were the most successful at first-pass coding with AI because they specify what they want well.
“there's a great study out of Anthropic that shows that the people who are the most successful coders at the first pass of coding are actually managers, not coders”
Ethan Mollick says early studies with colleagues at Harvard and Warwick found that less experienced BCG consultants were worse at using AI.
“we found that, like, BCG consultants less experience were worse at using AI.”
Ethan Mollick says OpenAI moved 20% of its staff off research to harden its systems against security threats.
“OpenAI pulled 20% of their staff off of research to 20% of the resources to hardening their systems.”
Ethan Mollick says leaders of large financial institutions are moving more than half of their security staff onto responding to AI-driven threats.
“I have talked to leaders of large financial institutions who are pulling 50% plus of their CIS- their staff”
Ethan Mollick says the biggest thing he got wrong in the past year was expecting agents to be hard to organize, when swarms of 100,000 agents now self-organize.
“There are 100,000 agent swarms that are self-organizing. I didn't expect that to happen”
Ethan Mollick says he runs projects in which a swarm of a hundred AI agents works on a problem and reports back to him like a manager.
“I literally have projects where I have a swarm of a hundred agents solving a problem for me.”
Ethan Mollick says that in his first startup he spent about eight hours a week doing payroll taxes by hand in Excel before learning a payroll company could do it.
“I would do taxes by hand in Excel for every person at the company. It took, like, eight hours every week before I realized better way.”
Ethan Mollick defines a 'dark factory' as a factory where robots do all the work, so no humans or lights are needed.
“So dark factory is factory where all the work is done by robots.”
Ethan Mollick draws on Harvard's Mike Tushman for the idea of organizational ambidexterity, doing two things at once: exploiting what works today while exploring the future.
“to take an idea from Harvard, from Mike Tushman and company, uh, it it was the idea that you would need ambidexterity.”
Ethan Mollick says the coding profession moved from writing elegant code to systems engineering as AI took over code writing, and AI is now doing some systems engineering too.
“You're hiring coders to be the best at writing elegant lines of code. Then AI could write the code. So then it became about sort of systems engineering.”
Ethan Mollick says the AI labs explicitly aim to replace all human labor, a goal he thinks they may be wrong about and that society shouldn't let happen.
“the AI labs, to be clear, are aiming for the job apocalypse. They want AI to replace all human labor.”
Ethan Mollick says consultants' claimed AI success stories are often based on GPT-4, a model he calls obsolete.
“if they have successful cases, it's for GPT four, a model that's obsolete and no one would use for anything anymore”
Ethan Mollick says one of the first effects of AI that executives see, which also appeared in his research, is that job boundaries blur.
“jobs start to blur. Coders do design work, designers do coding, product managers do everything.”
Ethan Mollick says he has spoken to C-level leaders who used AI to query company information and diagnose problems at plants that no one had raised with them.
“who've actually diagnosed problems at plants that no one brought up to them because they could just literally query all of the information at their company's disposal”
Ethan Mollick says there is ample evidence that AI can regularly do work that would take a human two to eighteen weeks, at reasonable cost and with high success rates.
“AI can regularly do the kind of work that would take a human two to eighteen weeks in a reasonable amount of time, a reasonable amount of money with high success rates.”
Ethan Mollick says GPT-3, the base generation of the technology behind ChatGPT, had existed for over a year before ChatGPT and produced only okay work.
“GPT three, which is, uh, the sort of base generation of this technology has been around for over a year, and it was okay, right?”
Ethan Mollick says the models went from D-minus to B-minus performance with ChatGPT, and an A-minus level with an internet-connected model like Bing.
“Suddenly, they became B minus students in classes, and now, if you use a even more advanced model like Bing that's connected to the internet, now they're A minus students.”
The first productivity results on these tools came from two controlled experiments, one on business writing and one on coding.
“one is a controlled experiment that was done with people writing business writing, the other was a controlled experiment with coding”
Ethan Mollick says an American plant adding steam power in the 1800s raised productivity by about 25%, which he uses as a comparison for the 30 to 50% gains seen with ChatGPT.
“the, uh, change that happened in productivity when a American plant added steam power in the eighteen hundreds was about 25% in productivity”
Ethan Mollick says ChatGPT reached 100 million users faster than any product known in history.
“it was the fastest product to chat GPT to 100 million users of any product we know of in history”
Ethan Mollick says programmers have shifted from Stack Exchange to ChatGPT as their way of deciding what to code next.
“everyone in programming is using this to help with coding, right? So, that's sort of moved from Stack Exchange to ChatGPT, uh, as their method of figuring out what to code next.”
At the time of recording, J.P. Morgan had banned the use of ChatGPT in the office.
“there are a number of organizations, I know, like J. P. Morgan, at at the time we're talking here, banned the use of, uh, ChatGPT in the office”
Ethan Mollick gives the example of using Bing AI to learn Porter's five-forces framework and research the precision agriculture industry, saving about twelve hours of work.
“you could tell it to learn how to do a Porter's five-forces analysis and then look up everything about the precision agriculture industry”
Ethan Mollick says a controlled study found Google's medical AI model gave advice at least equal to doctors' advice for common illnesses.
“some of the fine-tuned models, like Google's medical model, is giving in, uh, you know, another controlled study is giving advice that's at least equal to that of doctors to patients with common illnesses”
Ethan Mollick says ChatGPT achieves top scores on every creativity test available, though he says it is not sentient.
“It maxes out every creativity test we have, even though it's not really sentient, right?”
Ethan Mollick raises as one hiring scenario the idea of a prompt engineering role, where people skilled at getting AI to work are far more productive.
“this idea that there's some prompt engineering role in the future, that some people are just good AI whisperers”
Ethan Mollick explains these AI systems predict the next word in response to a prompt while trying to please the user, which is why they sometimes make things up.
“they're predicting the next word that to to give you, the next token to give you in response to your prompt, and they're doing that to try and maximize your happiness”
Ethan Mollick says he used AI to create a credible deepfake video of himself giving a speech he never gave, in about four minutes.
“I was able to, using AI, create a comp- a a very credible, uh, deep fake video of me giving a speech I never gave in my voice, and it took about four minutes to do, right?”
Ethan Mollick expects OpenAI to release GPT-4, a model much larger than the GPT-3.5 models behind ChatGPT.
“there is going to be elements of what's called GPT four release, which is a model that's much larger than the ChatGPT GPT 3.5 models we've been playing with”
Ethan Mollick says Google has by far the largest AI research team and has been cautiously rolling out products.
“Google has by far the largest research team on AI, and they've been cautiously rolling out products and have specialized products in different industries that might happen soon.”
Ethan Mollick names Anthropic's Claude as another large language model available alongside ChatGPT.
“there's other companies, like Anthropic has Claude, which you can get access to.”
Ethan Mollick says diffusion-based models, used for example in DALL-E, create images from text and may soon do the same for video.
“there are these what are called diffusion-based models, which are often used for creating images.”
Mollick cited a study of a representative sample of American workers showing AI use at work rising from 30% in February to over 40% in April.
“the usage went from 30% in February to 40% of people are using AI at work, uh, a little over 40% as of April.”
A Boston Consulting Group study involving Mollick and academics from Harvard, MIT and Warwick found a 40% quality improvement for people using GPT-4.
“40% improvement in quality for people who used GPT-4 versus not, um, big speed improvements”
Moderna built custom GPTs to help employees prepare their annual performance reviews, which introduced staff to AI.
“they built a whole series of GPTs that help people uncover their own performances, it's improvement needs and what they've done over the year”
Mollick described Manus as a small Chinese company whose AI agent runs on Anthropic's Claude model.
“Manus is Pa- is, uh, is a small Chinese company. Actually, I don't know how big they are, uh, but they're they are, um, they are the, uh, they use Claude as their underlying data, so Anthropic's data.”
The Cybernetic Teammate experiment gave 776 Procter and Gamble employees real work tasks to do alone or in teams, with or without AI.
“Procter and Gamble, where they gave 776 of their employees, and we did real work that they actually do.”
A Wharton Generative AI Labs study found that saying 'please' improved AI accuracy on one math problem, while on another it made results worse.
“So if you do a hundred tests on a particular math problem, it turns out in that math problem, if you say please, you get, uh, more accurate answers than if you don't say please”