Aidan Gomez said the Transformer architecture was originally built to improve machine translation at Google.
“we were making good progress on translation, which is what we built the transformer for”
Person
Aidan Gomez
2 episodes · 47 quoted claims
Aidan Gomez said it took about a year after the Transformer paper for the research community to take notice of it.
“So I think it took about a year for the community to take notice.”
Aidan Gomez said the Transformer's creators did not expect the technology to advance as far as it has in seven years.
“There's no way seven years ago when we created the transformer, any of us thought we'd be here in seven years.”
Aidan Gomez said his early Transformer language models at Google were trained only on Wikipedia and could write only Wikipedia-style articles.
“It was just trained on Wikipedia, and so the model could only write Wikipedia articles.”
Cohere builds AI models and makes them available to enterprises rather than building a consumer chatbot.
“We build models and we make them available for enterprises.”
Aidan Gomez said Cohere had about 350 employees at the time of the interview.
“we're about 350 at the moment, which is insane to me”
Aidan Gomez said Cohere began with three founders working in a space he described as basically a closet.
“it was just Nick, Ivan and I. We were in like this tiny little. It was basically a closet”
Aidan Gomez described the previous year as the year of enterprise proofs of concept for generative AI.
“Last year, it was sort of the year of the POC or the proof of concept.”
Aidan Gomez said a single large AI model can contain hundreds of billions to trillions of tunable parameters, which makes it costly to store and run.
“We're talking like hundreds of billions, trillions of tunable parameters inside just a single one of these models.”
Aidan Gomez said he and cofounder Nick worked under Geoffrey Hinton in Toronto before founding Cohere.
“We were both working for Jeff Hinton in Toronto, and he was the guy who kind of created neural networks.”
Aidan Gomez said Cohere was founded to give enterprises AI from an independent company not tied to any one cloud provider.
“the world needed Cohere and the ability to adopt this technology from an independent, an organization that wasn't tied to any one cloud, any one hyperscaler.”
Aidan Gomez said a CTO at a large retailer might spend half a billion to a billion dollars with one cloud provider for compute.
“If you're like a CTO at a large retailer, you're probably spending half $1 billion, $1 billion, on one of the cloud providers for your compute.”
Aidan Gomez said Cohere's work requires spending hundreds of millions of dollars a year.
“In order to do what we do, you need to spend hundreds of millions of dollars a year.”
Aidan Gomez said Patrick Lewis, whom he credited with inventing retrieval augmented generation (RAG), leads Cohere's RAG effort.
“we have the guy who invented RAG, Patrick Lewis, leading that effort at Cohere”
Aidan Gomez said Cohere spends only a fraction of what some competitors spend on compute while serving the needs enterprises actually want.
“we might spend like twenty what some of our competitors spend on compute”
Aidan Gomez said Cohere's Command R and Command R Plus models were first made available on Microsoft Azure.
“when we released Command R and Command R Plus, our our new models, they were first available on Azure.”
Aidan Gomez cited Databricks and Snowflake as independent database companies that succeeded without belonging to a big cloud provider.
“fantastic companies like Databricks and Snowflake, which are independent”
Aidan Gomez said doctors reportedly spend about 40% of their time on note-taking between patient visits.
“doctors spend 40% of their time taking notes, and this is in between patient visits.”
Aidan Gomez argued that AI-assisted note-taking that cut documentation from 40% to 20% of doctors' time would create roughly 25% more doctor hours.
“We bring that 40% down to 20%, like overnight, we have 25% more doctor hours.”
Aidan Gomez said Cohere honors robots.txt and code licenses when collecting training data from the web.
“When we're scraping code, we look at the licenses that are associated with that code.”
Aidan Gomez said AI makes it possible to create millions of fake accounts that present a false consensus, and he proposed human-verified accounts as a defense.
“you can spin up a million accounts, and you can create a a million fake people that project one idea and present a false consensus”
Aidan Gomez said the hallucination rates of language models have dropped dramatically over the years.
“hallucination rates, you've you can track them over the years, they've just dropped dramatically, and they've gotten much better”
Gomez says Geoffrey Hinton, who taught at the University of Toronto where Gomez studied, recently won the Nobel Prize and is regarded as the godfather of deep learning.
“obviously Jeff recently won the Nobel Prize. He's kind of like, uh, attributed with being the the godfather of of deep learning in U of T, the school where I went.”
Gomez got an internship with Lukasz Kaiser at Google Brain, which he later learned was meant for PhD students rather than undergraduates.
“So I, I somehow managed to get an internship with Lukasz Kaiser, uh, at at Google Brain, um, and I found out at the end of that internship, I wasn't supposed to have gotten that internship.”
Before founding Cohere, Gomez collaborated remotely with Jeff Dean and Sanjay Ghemawat on Google's Pathways, a program to train models across a larger unit of compute built from wired-together supercomputers.
“And then I was also collaborating remotely with Jeff Dean and Sanjay on Pathways, which was like their, you know, bigger than a supercomputer training program.”
After GPT-2 was released, Gomez called his future co-founders Nick and Ivan and proposed building large language models.
“I called up Nick, I called up Ivan, my co-founders, and I said, You know, maybe we should figure out how to build these things.”
Gomez says Cohere will not build a ChatGPT competitor and will instead build a platform and products that help enterprises adopt language models.
“We're not going to build a chat GPT competitor. What we wanna build is a a platform and a series of products to enable enterprises to adopt this technology and and make it valuable.”
Gomez says many enterprise proof-of-concept AI projects in 2023 failed because of unfamiliarity and common implementation errors.
“And so in 2023, there were a lot of these POCs, a lot of people trying to get familiar with the technology, wrap their heads around it, and a lot of those POCs failed.”
Gomez describes a healthcare use case where a model reads a patient's full longitudinal record alongside the reason for the visit and produces a briefing for the doctor.
“but what we can do is we can feed that entire history in alongside the reason they're coming in”
Gomez says a Cohere-built research assistant let an insurance company's actuaries respond to RFPs much faster and win more of them.
“And it dramatically sped up their ability to respond to RFPs, and so it grew their business, uh, cause they were just winning many more of them.”
Gomez says Cohere can deploy on premises, inside a customer's virtual private cloud, or outside it, which helps with regulated and sensitive data.
“flexible to deploy on prem, if you want us in VPC, outside of VPC, literally whatever the customer wants”
Cohere partnered with Fujitsu, described by Gomez as the largest systems integrator in Japan, to build a Japanese language model that required pretraining from scratch.
“we partnered with Fujitsu, um, who's like the largest SI in in Japan, um, to build a Japanese language model.”
Gomez says Cohere recommends customers start with fine-tuning, then move to post-training and, if needed, partial pretraining.
“What we usually recommend to customers is start from the cheapest, easiest thing, which is fine tuning, and then work backwards, and so start with fine tuning.”
Gomez says a model matching GPT-4 on enterprise-relevant tasks can now be built for roughly $10 to 20 million, far less than GPT-4 cost to develop.
“today you can build a model that's as good as GPT-4 in all the things that enterprises might care about. For 10 million, 20 million, like just orders of magnitude less than what was spent to develop that model.”
Gomez says Cohere must pay for a supercomputer that costs hundreds of millions of dollars a year.
“we have to pay for a supercomputer and those things cost hundreds of millions of dollars a year”
Gomez says Cohere's strategy is to stay about six months behind the frontier and sell enterprises models at sensible prices, rather than spend billions a year.
“To be at the front, be six months behind and offer something to market to enterprises that actually fits their needs at a price point that makes sense for them.”
Gomez says fine-tuning open models such as Llama is less effective than building a model yourself because you lack control over the training data.
“Taking those models and trying to, um, fine tune them, it's just it's not as effective as building it yourself, and you have much fewer levers to pull.”
Gomez says OpenAI was first to put reasoning models into production and that Cohere has been working on them for about a year.
“of course, OpenAI was the 1st to put it into production, but Cohere's been working on it, uh, for about a year now.”
Gomez says reasoning models give model builders a second way to improve performance: customers can pay for more inference-time compute instead of waiting for a new training run.
“There's now a 2nd lever, which is you can charge the customer more.”
Gomez says early language models lacked an inner monologue because internet text shows only the final product of reasoning, not the thinking behind it.
“And the internet is like a set of documents, which are the output of a reasoning process, with the reasoning all hidden.”