Topic
29 facts from 2 episodes, in date order. These are what speakers said on the shows, each linked to the moment it was said. They aren't independently verified.
2017
Gomez got an internship with Lukasz Kaiser at Google Brain, which he later learned was meant for PhD students rather than undergraduates.
“So I, I somehow managed to get an internship with Lukasz Kaiser, uh, at at Google Brain, um, and I found out at the end of that internship, I wasn't supposed to have gotten that internship.”
2019
Aidan Gomez founded Cohere in 2019 and, as a Google Brain intern, co-authored the 2017 paper 'Attention Is All You Need'.
“Aidan founded Cohere in 2019, but before that, during his time as an intern at Google Brain, he was a co-author on the landmark 2017 paper.”
2019
Before founding Cohere, Gomez collaborated remotely with Jeff Dean and Sanjay Ghemawat on Google's Pathways, a program to train models across a larger unit of compute built from wired-together supercomputers.
“And then I was also collaborating remotely with Jeff Dean and Sanjay on Pathways, which was like their, you know, bigger than a supercomputer training program.”
2019
After GPT-2 was released, Gomez called his future co-founders Nick and Ivan and proposed building large language models.
“I called up Nick, I called up Ivan, my co-founders, and I said, You know, maybe we should figure out how to build these things.”
2023
Aidan Gomez described the previous year as the year of enterprise proofs of concept for generative AI.
“Last year, it was sort of the year of the POC or the proof of concept.”
Undated
Aidan Gomez said the Transformer architecture was originally built to improve machine translation at Google.
“we were making good progress on translation, which is what we built the transformer for”
Undated
Aidan Gomez said it took about a year after the Transformer paper for the research community to take notice of it.
“So I think it took about a year for the community to take notice.”
Undated
Aidan Gomez said the Transformer's creators did not expect the technology to advance as far as it has in seven years.
“There's no way seven years ago when we created the transformer, any of us thought we'd be here in seven years.”
Undated
Aidan Gomez said his early Transformer language models at Google were trained only on Wikipedia and could write only Wikipedia-style articles.
“It was just trained on Wikipedia, and so the model could only write Wikipedia articles.”
Undated
Aidan Gomez said Cohere had about 350 employees at the time of the interview.
“we're about 350 at the moment, which is insane to me”
Undated
Aidan Gomez said Cohere began with three founders working in a space he described as basically a closet.
“it was just Nick, Ivan and I. We were in like this tiny little. It was basically a closet”
Undated
Aidan Gomez said a single large AI model can contain hundreds of billions to trillions of tunable parameters, which makes it costly to store and run.
“We're talking like hundreds of billions, trillions of tunable parameters inside just a single one of these models.”
Undated
Aidan Gomez said he and cofounder Nick worked under Geoffrey Hinton in Toronto before founding Cohere.
“We were both working for Jeff Hinton in Toronto, and he was the guy who kind of created neural networks.”
Undated
Aidan Gomez said Cohere was founded to give enterprises AI from an independent company not tied to any one cloud provider.
“the world needed Cohere and the ability to adopt this technology from an independent, an organization that wasn't tied to any one cloud, any one hyperscaler.”
Undated
Aidan Gomez said a CTO at a large retailer might spend half a billion to a billion dollars with one cloud provider for compute.
“If you're like a CTO at a large retailer, you're probably spending half $1 billion, $1 billion, on one of the cloud providers for your compute.”
Undated
Aidan Gomez said Cohere's work requires spending hundreds of millions of dollars a year.
“In order to do what we do, you need to spend hundreds of millions of dollars a year.”
Undated
Aidan Gomez said Cohere spends only a fraction of what some competitors spend on compute while serving the needs enterprises actually want.
“we might spend like twenty what some of our competitors spend on compute”
Undated
Aidan Gomez said doctors reportedly spend about 40% of their time on note-taking between patient visits.
“doctors spend 40% of their time taking notes, and this is in between patient visits.”
Undated
Aidan Gomez argued that AI-assisted note-taking that cut documentation from 40% to 20% of doctors' time would create roughly 25% more doctor hours.
“We bring that 40% down to 20%, like overnight, we have 25% more doctor hours.”
Undated
Aidan Gomez said Cohere honors robots.txt and code licenses when collecting training data from the web.
“When we're scraping code, we look at the licenses that are associated with that code.”
Undated
Aidan Gomez said AI makes it possible to create millions of fake accounts that present a false consensus, and he proposed human-verified accounts as a defense.
“you can spin up a million accounts, and you can create a a million fake people that project one idea and present a false consensus”
Undated
Aidan Gomez said the hallucination rates of language models have dropped dramatically over the years.
“hallucination rates, you've you can track them over the years, they've just dropped dramatically, and they've gotten much better”
Undated
Gomez says early language models lacked an inner monologue because internet text shows only the final product of reasoning, not the thinking behind it.
“And the internet is like a set of documents, which are the output of a reasoning process, with the reasoning all hidden.”
Undated
Gomez says synthetic data works well in code and math because answers can be verified, unlike fields such as biology and chemistry.
“These are places where the answer is very verifiable. You, you know, when you're right or you're wrong, and so you can generate tons of synthetic data and just verify whether it's correct or not.”
Undated
Gomez says he believes AGI, defined as useful, generally capable technology, will be built, but he rejects the idea of building a godlike superintelligence.
“We will build AGI, if what you mean is very useful, generally capable technology that can do a lot of the stuff, uh, that humans can do and flex into a lot of different domains.”
Undated
Gomez argues that although neural networks are theoretically universal approximators, practical limits such as enormous size bound what can be realized.
“even if in theory, you know, like a neural net is a universal approximator, it can learn anything to universally approximate. You would need to build a neural net the size of the universe.”
Undated
Gomez says physics can probably be modeled well by sequence models because it consists of states and transition probabilities.
“physics is just like a series of states and, um, transition probabilities, so I I think it's probably quite well modeled by sequence modeling.”
Undated
Gomez says falling model prices reflect price dumping, with companies giving models away for free or at a loss, rather than true commoditization.
“I think what you see is you see price dumping, um, and so you see people giving it out for free, giving it out at a loss, giving it out at zero margin.”
Undated
Gomez says a total technological refactor driven by AI is underway and will last the next ten to fifteen years.
“in reality, the state of the world is there's a total technological refactor that's going on right now and will last the next ten to fifteen years.”
Episodes