Kevin Scott said he had been at Microsoft for almost six and a half years at the time of the recording.
“I've been at Microsoft a little over 6 years now, so almost six and a half years.”
Person
Kevin Scott
1 episodes · 22 quoted claims
Microsoft started building its first AI supercomputer in 2019 and deployed it at the end of that year, and GPT-3 was trained on it.
“we started working on it in 2019, and we deployed it, uh, at the end of that year. And it was the computing environment that GPT-3 was trained on.”
Kevin Scott said NVIDIA has been Microsoft's compute and network partner since NVIDIA acquired Mellanox.
“NVIDIA's been, you know, our compute and network partner since they bought Mellanox”
Kevin Scott said Microsoft was deploying NVIDIA's Hopper GPUs at scale, including features such as eight-bit floating point arithmetic.
“we're deploying, uh, hopper now at scale, and you know, like a bunch of the features of hopper, like, you know, eight bit floating point”
Kevin Scott said that at one point he took control of Microsoft's entire GPU budgets to focus them on evidence-backed paths.
“at one point, I seized the entire GPU budgets for the whole company.”
Kevin Scott said Satya Nadella describes a 'no regrets' investing approach, which Scott applied to the OpenAI partnership.
“Satya, uh, like, has this thing that he talks about, uh, like, no no regrets, uh, investing.”
Kevin Scott described an OpenAI plot showing roughly tenfold annual growth in compute used to train state-of-the-art models from 2012 to about 2018.
“that we're, you know, basically consuming ten times compute, uh, more compute every year for, like, training state-of-the-art models.”
Kevin Scott said he had been experimenting with Databricks' 12-billion-parameter Dolly 2.0 model, which runs on a single machine.
“I was just playing around yesterday with, uh, that 12 billion parameter, uh, Dolly 2.0 model from Databricks”
Microsoft developed an orchestration tool called Semantic Kernel and released it as open source.
“we have, uh, one that we've developed called Semantic Kernel that we've also open sourced.”
Kevin Scott said ChatGPT was built on a ten-month-old model with some reinforcement learning from human feedback (RLHF) added and was released partly to practice before GPT-4.
“ChatGPT was a ten month old model with a little bit of RLHF on top of it.”
Kevin Scott said GitHub Copilot was the first copilot Microsoft built, letting developers describe code in natural language.
“the first thing that we built was GitHub Copilot, which is a coding tool.”
Kevin Scott dated the iPhone launch to 2007 and said the App Store followed a year later.
“when smartphones, uh, you know, came on the market fifteen, sixteen, seventeen years ago now, like, it's, uh, yeah, 2007, I I guess, was iPhone launch”
Kevin Scott praised Sal Khan's TED talk on using AI for individualized tutoring.
“Sal Khan's TED talk, which is just amazing, like, Unbelievably good.”
Kevin Scott argued that the lasting smartphone apps, such as TikTok, Instagram, WhatsApp and DoorDash, were non-obvious hard problems rather than the initial launch apps.
“Like, you know, we, we, we don't even think the the original applications on these platforms”
Kevin Scott said his daughter wants to become a surgeon, which he considers a good job given limited progress in robotics.
“my my daughter, for instance, like, has decided she wants to go be a surgeon.”
Kevin Scott said he and Sam Altman have discussed their suspicion that the next trillion-dollar company would be founded in 2023.
“I suspect that this year, like the next 1 trillion dollar company, gets founded.”
Kevin Scott predicted projects like RedPajama would make good progress on more capable open source models.
“the Red Pajama project, uh, is like this a and?”
Kevin Scott said GPT-4's visual input capability was not yet widely available but would reach wide distribution.
“openAI doesn't have GPT V and wide distribution, but like, it'll get to wide distribution at some point in the not too distant future.”
Kevin Scott said OpenAI CEO Sam Altman testified before the Senate Judiciary Committee on the Tuesday of the week of the recording.
“Sam Altman was testifying in front of the Senate Judiciary Committee on, uh, like, Tuesday of this week.”
Kevin Scott attended a Virginia Governor's school for science and technology, where his computer science teacher was Dr. Tom Morgan.
“I went to a science and technology high school.”
Kevin Scott said Google put many of its compiler specialists on his interview panel, and he joined the new Google New York office as about the tenth person.
“they took what must have been every compiler person in the company at the time and put them on my interview panel.”
Kevin Scott said his team at Google won a Google Founders Award for solving an unglamorous problem using advanced computer science.
“the whole team won a Google Founders Award, which was a big deal”