21 Nov 2024 · 44m · Sarah Guo, Aidan Gomez · 27 quoted facts
Aidan Gomez, CEO of Cohere, argues that businesses can get large value from today's language models without waiting for ever-smarter ones. He says the biggest gains now come from integrating the technology, and that it will take years to roll out across the economy. His company deliberately stays about six months behind the frontier labs and sells to enterprises that care about security and cost.
The central tension is that raw model improvement is flattening while new levers appear. Gomez says easy gains are used up and further progress needs scarce expert data. Reasoning models, which spend more compute at answer time, give a new way to buy intelligence without a new training run. He is also skeptical of superintelligence narratives and of the claim that models are being commoditized.
Gomez says many 2023 pilot projects failed because teams treated models like humans and ignored how prompts, data formatting and retrieved results are presented. Cohere responds by making models more tolerant of such variation and by offering more structured APIs.
Gomez describes a pyramid: every company needs generic assistants, which it can buy, but custom tools that reflect its own business must be built. His example is an insurer whose research assistant for RFPs, the formal requests for proposals it must answer, helped it win more bids.
Gomez says a model as good as GPT-4 for enterprise tasks can now be built for $10 to 20 million, orders of magnitude less than the original. Cohere's strategy is to trail the frontier by six months or a year rather than spend billions a year.
Reasoning models let a customer pay for more thinking time per answer instead of waiting six to twelve months for a bigger training run. Gomez says this turns capital spending into a consumption model and could shift chip and data-center priorities.
Gomez says models handle easy tasks well, and the remaining gains need expert knowledge in fields like biology and chemistry. Synthetic data works in code and math because answers can be checked, but elsewhere you eventually exhaust the humans who know the field.
Gomez says many 2023 proofs of concept failed because teams overestimated the models and ignored quirks like how prompts and retrieved data are formatted. He says Cohere is trying to fix this with more robust models and more structured products.
Gomez recommends a mix: buy generic tools that every company needs, and build only the specialized things no vendor would sell you. He says those custom tools are where a company's competitive advantage comes from.
Gomez says progress is entering a flat part of the curve, especially in specialized science and math, where expert data is scarce. He says it matters little for most businesses because the technology is already good enough, and integrating it will take about half a decade.
It means giving a model more computing effort and time while it answers a question, so it can reason step by step and fix its own mistakes. Gomez says this lets customers pay for a smarter model today instead of waiting months for a new training run.
No. He says falling prices reflect companies giving models away at a loss, not true commoditization. He says only a handful of firms know how to build them, and growth pressures will eventually end the free pricing.
Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.
Cohere was described by the host as valued at more than $5 billion in 2024 and as providing AI language models and solutions for businesses.
“a company valued at more than 5 billion in 2024, which provides AI powered language models and solutions for businesses”
Aidan Gomez founded Cohere in 2019 and, as a Google Brain intern, co-authored the 2017 paper 'Attention Is All You Need'.
“Aidan founded Cohere in 2019, but before that, during his time as an intern at Google Brain, he was a co-author on the landmark 2017 paper.”
Gomez says Geoffrey Hinton, who taught at the University of Toronto where Gomez studied, recently won the Nobel Prize and is regarded as the godfather of deep learning.
“obviously Jeff recently won the Nobel Prize. He's kind of like, uh, attributed with being the the godfather of of deep learning in U of T, the school where I went.”
Gomez got an internship with Lukasz Kaiser at Google Brain, which he later learned was meant for PhD students rather than undergraduates.
“So I, I somehow managed to get an internship with Lukasz Kaiser, uh, at at Google Brain, um, and I found out at the end of that internship, I wasn't supposed to have gotten that internship.”
Before founding Cohere, Gomez collaborated remotely with Jeff Dean and Sanjay Ghemawat on Google's Pathways, a program to train models across a larger unit of compute built from wired-together supercomputers.
“And then I was also collaborating remotely with Jeff Dean and Sanjay on Pathways, which was like their, you know, bigger than a supercomputer training program.”
After GPT-2 was released, Gomez called his future co-founders Nick and Ivan and proposed building large language models.
“I called up Nick, I called up Ivan, my co-founders, and I said, You know, maybe we should figure out how to build these things.”
Gomez says Cohere will not build a ChatGPT competitor and will instead build a platform and products that help enterprises adopt language models.
“We're not going to build a chat GPT competitor. What we wanna build is a a platform and a series of products to enable enterprises to adopt this technology and and make it valuable.”
Gomez says many enterprise proof-of-concept AI projects in 2023 failed because of unfamiliarity and common implementation errors.
“And so in 2023, there were a lot of these POCs, a lot of people trying to get familiar with the technology, wrap their heads around it, and a lot of those POCs failed.”
Gomez describes a healthcare use case where a model reads a patient's full longitudinal record alongside the reason for the visit and produces a briefing for the doctor.
“but what we can do is we can feed that entire history in alongside the reason they're coming in”
Gomez says a Cohere-built research assistant let an insurance company's actuaries respond to RFPs much faster and win more of them.
“And it dramatically sped up their ability to respond to RFPs, and so it grew their business, uh, cause they were just winning many more of them.”
Gomez says Cohere can deploy on premises, inside a customer's virtual private cloud, or outside it, which helps with regulated and sensitive data.
“flexible to deploy on prem, if you want us in VPC, outside of VPC, literally whatever the customer wants”
Cohere partnered with Fujitsu, described by Gomez as the largest systems integrator in Japan, to build a Japanese language model that required pretraining from scratch.
“we partnered with Fujitsu, um, who's like the largest SI in in Japan, um, to build a Japanese language model.”
Gomez says Cohere recommends customers start with fine-tuning, then move to post-training and, if needed, partial pretraining.
“What we usually recommend to customers is start from the cheapest, easiest thing, which is fine tuning, and then work backwards, and so start with fine tuning.”
Gomez says a model matching GPT-4 on enterprise-relevant tasks can now be built for roughly $10 to 20 million, far less than GPT-4 cost to develop.
“today you can build a model that's as good as GPT-4 in all the things that enterprises might care about. For 10 million, 20 million, like just orders of magnitude less than what was spent to develop that model.”
Gomez says Cohere must pay for a supercomputer that costs hundreds of millions of dollars a year.
“we have to pay for a supercomputer and those things cost hundreds of millions of dollars a year”
Gomez says Cohere's strategy is to stay about six months behind the frontier and sell enterprises models at sensible prices, rather than spend billions a year.
“To be at the front, be six months behind and offer something to market to enterprises that actually fits their needs at a price point that makes sense for them.”
Gomez says fine-tuning open models such as Llama is less effective than building a model yourself because you lack control over the training data.
“Taking those models and trying to, um, fine tune them, it's just it's not as effective as building it yourself, and you have much fewer levers to pull.”
Gomez says OpenAI was first to put reasoning models into production and that Cohere has been working on them for about a year.
“of course, OpenAI was the 1st to put it into production, but Cohere's been working on it, uh, for about a year now.”
Gomez says reasoning models give model builders a second way to improve performance: customers can pay for more inference-time compute instead of waiting for a new training run.
“There's now a 2nd lever, which is you can charge the customer more.”
Gomez says early language models lacked an inner monologue because internet text shows only the final product of reasoning, not the thinking behind it.
“And the internet is like a set of documents, which are the output of a reasoning process, with the reasoning all hidden.”
Gomez says synthetic data works well in code and math because answers can be verified, unlike fields such as biology and chemistry.
“These are places where the answer is very verifiable. You, you know, when you're right or you're wrong, and so you can generate tons of synthetic data and just verify whether it's correct or not.”
Gomez says he believes AGI, defined as useful, generally capable technology, will be built, but he rejects the idea of building a godlike superintelligence.
“We will build AGI, if what you mean is very useful, generally capable technology that can do a lot of the stuff, uh, that humans can do and flex into a lot of different domains.”
Gomez argues that although neural networks are theoretically universal approximators, practical limits such as enormous size bound what can be realized.
“even if in theory, you know, like a neural net is a universal approximator, it can learn anything to universally approximate. You would need to build a neural net the size of the universe.”
Gomez says physics can probably be modeled well by sequence models because it consists of states and transition probabilities.
“physics is just like a series of states and, um, transition probabilities, so I I think it's probably quite well modeled by sequence modeling.”
Gomez says falling model prices reflect price dumping, with companies giving models away for free or at a loss, rather than true commoditization.
“I think what you see is you see price dumping, um, and so you see people giving it out for free, giving it out at a loss, giving it out at zero margin.”
Gomez says a total technological refactor driven by AI is underway and will last the next ten to fifteen years.
“in reality, the state of the world is there's a total technological refactor that's going on right now and will last the next ten to fifteen years.”
Gomez said the price of Haiku had quadrupled about two weeks before the recording, as evidence that giveaway pricing is unstable.
“the price of Haiku fourxed, uh, two weeks ago”
Gomez explains how he ended up as an undergraduate intern on the team behind the Transformer. This shows the research setting that shaped his later company.
“I feel like I was raised into AI”
Gomez lays out Cohere's mission of helping organizations adopt language models rather than competing with ChatGPT.
“We're not going to build a chat GPT competitor.”
Gomez explains the repeat mistakes in retrieval systems, and how better models and more structured products can prevent them.
“they overestimate the models, they think they're, they're like humans”
Gomez says companies should buy standard tools and build only what gives them a unique edge, illustrated by an insurance example.
“focus on the stuff that no one's gonna sell to you”
Regulated industries need flexible deployment options, which Gomez calls a differentiator for Cohere.
“I think the big one is trust”
Gomez rejects the view that only AGI labs should pretrain and recommends starting with the cheapest option, fine-tuning.
“that's empirically wrong.”
Gomez argues reasoning models create a second way to improve products, and that the market has not priced it in.
“people haven't really priced in the impact of inference time compute delivering intelligence”
Using a painting analogy, Gomez says finer detail needs scarce expert data, which limits further gains.
“you run out of experts and you run out of that data”
Gomez says AGI is continuous rather than a finish line, rejects the idea of building a godlike intelligence, and disputes commoditization.
“I don't think that models are actually getting commoditized.”
Hi listeners, and welcome to No Priors. Today we're hanging out with Aidan Gomez, co-founder and CEO of Cohere, a company valued at more than 5 billion in 2024, which provides AI powered language models and solutions for businesses. Aidan founded Cohere in 2019, but before that, during his time as an intern at Google Brain, he was a co-author on the landmark 2017 paper. Attention is all you need. Aidan, thanks for coming on today.
Yeah, thank you for having me. Excited to be here.
Maybe we can start, uh, just a little bit with the personal background. Um, how do you go from growing up in the woods in Canada to, um, you know, working on the most important technical paper in the world?
A lot of luck and and chance. Um, but yeah, I happened to go to school at the place where Jeff Hinton, uh, taught, and so, um, obviously Jeff recently won the Nobel Prize. He's kind of like, uh, attributed with being the the godfather of of deep learning in U of T, the school where I went. He was a legend, and pretty much everyone who was in computer science, studying at the school wanted to get into AI.
Uh, and so in some sense, I, I feel like I was raised into AI, like as soon as I stepped out of high school. Um, I was steeped in an environment that really saw the future, uh, and wanted to build it. Um, and then from there, it was a bunch of happy accidents. So I, I somehow managed to get an internship with Lukasz Kaiser, uh, at at Google Brain, um, and I found out at the end of that internship, I wasn't supposed to have gotten that internship.
It was supposed to have been for PhD students. And so they were like throwing a goodbye party for me, the intern, um, and Lukasz was like, Okay, so Aidan, you're going back, how many, how many years have you got left in your PhD? Uh? And I was like, Oh, I'm going back into 3rd year undergrad, uh, and he was like, we don't do undergrad internships. So I think it was a bunch of, like, really lucky mistakes, uh, that led me, led me to that team.
Working on really interesting, important things at Google, what, uh, convinced you that you should start? Cohere?
Yeah, so I bounced around. Like when I was working with Lukash and Noam and the Transformer guys, I was in Mountain View. And then I went back to U of T, uh, started working with Hinton and my co-founder Nick in Toronto, uh, a brain there. And then I started my PhD. And I went to England, um, and I was working with, uh, Jakob, who's another Transformer paper author in Berlin, and
And then we had Jakob collaborating with Jeff on the podcast
D. Oh, nice, yeah, yeah, yeah, okay, fan of the pod, good, good, um, so yeah, I. I was working with Jakob in Berlin. And then I was also collaborating remotely with Jeff Dean and Sanjay on Pathways, which was like their, you know, bigger than a supercomputer training program. Uh, the idea was like wiring together supercomputers to create a new, larger unit of compute that you could train models on. And at that stage, GPT-2 had just come out and it was pretty clear the trajectory of the technology.
Like, we were on a very interesting path. And these models that were ostensibly models of the internet, models of the web, um, were gonna yield some pretty interesting, interesting things. So I, I called up Nick, I called up Ivan, my co-founders, and I said, You know, maybe we should figure out how to build these things. I I think they're gonna be useful.
for anyone who doesn't know yet, can you just describe at the high level, like, what Cohere's mission is, and then what the models and products are?
Yeah, so our our mission, the way that we wanna create value in the world, um, is by enabling other organizations to adopt this technology and make their workforce more productive or transform their product, uh, and the services that they offer. So we're very focused on the enterprise. We're not going to build a chat GPT competitor. What we wanna build is a a platform and a series of products to enable enterprises to adopt this technology and and make it valuable.
And in terms of, like, your north star of how you organize, um, the team and invest, uh, you obviously come from a research background yourself. Like, how much do you think? Um, you know, Cohere's success is dependent on core models versus other, you know, platform and go to market support investments you make.
It's all of the above, like the models are the foundation, and if you're building on a foundation that, um, doesn't meet the customer's needs, then there there's no hope. And so the models are crucial. And um, it's like the heart of the company, but in the enterprise world, things like customer support, reliability, security, these are all key. And so we've heavily invested on both sides. We're not just a modeling organization, we're a modeling and go to market organization.
Um, and increasingly, product is becoming a a priority for Cohere, and so figuring out ways to shorten time to value for our customers. Um, yeah, over the past, like 18 months, since the enterprise world sort of woke up to the technology, we've watched, we've watched folks build with our models. Seen what they're trying to accomplish, seen the common mistakes that they make. That's been helpful, it's been sometimes frustrating, right? Watching the same mistake again and again.
But we think there's a huge opportunity to be able to help enterprises avoid those mistakes and implement things right the 1st time. And so that's really where we're pushing towards.
Yeah, can we make that a little bit more real? Like, what is the mistake that frustrates you most, and how can product go meet that?
Yeah, well, I I think all language models are quite sensitive, uh, to to prompts, to the way that you present data. They all have their own individual quirks. The way that you talk to one might not work for the way that you talk to another. And so when you're building a system like a RAG system, where there's an external database, it really matters how you present the retrieved results to the model. It matters how the data is actually stored in those in those databases.
The formatting counts, and and these small details, they're often lost on people. They, uh, they overestimate the models, they think they're, they're like humans. And that has led to a lot of repeat failures. Uh, people try to implement a rag system, they don't know about these like idiosyncratic elements of implementing one properly, and then it fails. And so in 2023, there were a lot of these POCs, a lot of people trying to get familiar with the technology, wrap their heads around it, and a lot of those POCs failed.
Uh, because of unfamiliarity, because of, um, yeah, these common errors that we've seen. And so moving forward, we have two approaches. One is making the models more robust, so the models should be robust to a lot of different ways that you present data. And the 2nd piece is being more structured about the product that we expose to the user. So instead of just handing a model and saying, you know, prompt it good luck, uh, actually putting more structure around it.
So creating APIs that more rigorously define how you're supposed to use the model, these sorts of pieces, I think, just reduce the chances of failure, uh, and make these systems much more usable for the user.
What are people trying to do? Can you give us a flavor of some of, like, the, um, biggest use cases you see in the enterprise?
It's super broad, uh, so it spans pretty much every vertical. I mean, the common things are like QA, so speaking to a corpus of documents, for instance, if you're a manufacturing company, you might wanna build a QA bot for your engineers or your workers who are on the assembly line. Uh, and plug in all of the the manuals of the different tools and diagnostic manuals for common errors and parts. And then let the user chat to that instead of having to open up a thousand page book and and try to find what they need.
Similarly, QA bots for the average enterprise worker. So plugging in your ITFAQ, your HR docs, all the things about your company. And having a centralized chat interface onto the knowledge of your organization so that they can get their questions answered. Those are some of the common ones. Beyond that, there are kind of specific functions that we power. Um, a good example might be for a healthcare company. They have these longitudinal health records of patients, and that consists of every interaction that that patient has with the the healthcare system.
From visits to a pharmacy to the different labs or tests that they're getting, uh, to doctor's visits. And it can span decades, and so it's a huge, huge record of someone's medical history. And typically, what happens is that patient will call in and they'll ring up the receptionist and be like, my knee hurts, uh, I need an appointment. And the doctor then needs to kind of comb through the past few entries, see has this come up before.
And maybe they missed something that was two years ago, cause they only have 15 min before an appointment. Um, but what we can do is we can feed that entire history in alongside the reason they're coming in. So contextually relevant, right to what they said, they're coming in for, and surface a a briefing for the doctor. Um, and so this tends to be one dramatically faster for the doctor to review, but also often it catches things that a doctor couldn't possibly review before every patient meeting.
They're not going through 20 years of medical history like that is just not possible. Um, but the model can do that, it can do that in under a 2nd. So tho- those are the sorts of functions that we're seeing. Summarization, QA bots, a lot of these, um, you might think of them as mundane, but the impact is immense.
We see tons of startups working on problems such as, um, let's say enterprise search overall, specialized applications to, uh, let's say, like, technical support for a particular vertical. Even looking at health records and reasoning against them and retrieving from them. How do you think about, like, what the end state? um, there's no end state, but what? Some, um, stable equilibrium state is for how enterprises consume from, let's say, specialist AI powered application providers versus custom applications built in house with, uh, AI platforms and and uh, model APIs.
I think it's gonna be a hybrid, I think it's probably. You can imagine, like a pyramid, where the bottom of that pyramid, every organization needs this stuff. And it's like copilot, like a generalist chatbot in the hands of every single employee to answer their questions. And then as you head up the the pyramid, it's more specific to the company itself, or the specific domain or product that they they operate in or offer. Um, and as you push up that, that pyramid, it's much less likely you're gonna find an off-the-shelf solution to address it, uh, and so you're gonna have to build it yourself.
What we've pushed organizations to do is have a strategy that encompasses that full pyramid. Yes, you need the the generalist standard stuff. Maybe there's some industry specific tools that you can go out and buy, but then if you are building, don't build those things that you could buy. Instead, focus on the stuff that no one's gonna sell to you. And that gives you uniquely a competitive advantage. So we we worked with this insurance company, uh, and they they insure, like large industrial development projects.
It turns out, like, I, I know nothing about this space. It turns out what they do is there's like an RFP, uh, put out by a mine or something like whatever the project is for insurance. And they have actuaries jump on that RFP, do tons of research about, you know, the land that it's on, the potential risks, et cetera. And then it's essentially a race to whoever responds 1st usually gets it. And so it's a a time based thing.
How quickly can these actuaries put forward a good, researched proposal? Um, and what we built with them was like a research assistant. So we plugged in all the sources of knowledge that these actuaries go to to do their research, uh, via RAG, and we gave them a chatbot. And it dramatically sped up their ability to respond to RFPs, and so it grew their business, uh, cause they were just winning many more of them. Um, and so it's tough for like, you know, we built horizontal technology.
And LLM is kind of like a a CPU. I don't know all the applications of an LLM, right? It's so broad and really the the deep insight or the the competitive advantage. The thing that puts you ahead, um, is listening to the customer and letting them tell you what would put them ahead. Um, and so that's. That's a lot of what we've been doing is just being a thought partner and helping brainstorm these projects and ideas, uh, that are that are strategic to them.
I'd wager that's, you know, this company is winning. Because the vast majority of their competitors haven't been able to move so quickly to, uh, adopting, you know, and building a like this research assistant product that is helping them. Like, what is the biggest barrier you see to, um, generally, enterprise adoption?
I think the big one is trust, uh, so security is a big one, um, in particular in regulated industries like finance, uh, uh, healthcare. Data is often not in a cloud, or if it is in a cloud, it can't leave their VPC. And so it's very locked down, it's very sensitive, and so that's that's a unique differentiator of Cohere. The fact that we haven't locked ourselves into one ecosystem. And we're. We're flexible to deploy on prem, if you want us in VPC, outside of VPC, literally whatever the customer wants.
We're able to touch more data, even the most sensitive data, uh, and provide something that's that's more useful. So I, I would say security and privacy is probably the biggest one. Beyond that, there's knowledge, right? Like the the knowledge to know how to build these these systems. They're new, uh, it's unfamiliar to folks. Um, you know, the people with the most experience have a few years of experience, um, and so that's the other major piece. Uh, that bit, I think it's honestly just a time game.
Like, eventually, developers will become more familiar with building with this technology. Um, but I I think it's gonna take another two or three years before it really permeates.
Do you think in, like, a traditional hype cycle for enterprise technology, it's probably for most technologies, but in particular enterprise? Um, uh, you know, there's this trough of disillusionment concept of people get very excited about something and ends up being harder to apply or more expensive than they thought. Do we see that in AI?
I'm sure we see some of it for sure, um, but I think, honestly, like, the core technology is still improving at a steady clip and new applications are getting unlocked every few months. So I, I don't think we're in that trough of disillusionment yet. Yeah, it feels like we're super early, it feels like we're really, really early. And if you look at the market, this technology just unlocks an entire new set of things that you couldn't build.
You just fundamentally couldn't build them before, and and now you can. And so there's a resurfacing of technology, products, systems that's underway. Even if we didn't train a single new language model, like, Okay, all the data centers blow up, we can't improve the LLM, we only have what we have today. There's a half decade of work to go integrate this into the economy, to build all these things. To build the, you know, uh, RFP, insurance, RFP response bot, to build the uh, healthcare record summarizer.
Like, there's a a half decade of just resurfacing to go do so. There there's a lot of work ahead of us. I I think we're kind of past that point. There was a question of, oh, is there too much hype? Is this technology actually gonna be useful? But it's in the hands of 100 million people now, or hundreds of millions of people now. It's in production. Uh, there's very clear value. The project is now putting it to work and delivering it, uh, to the world.
in this question of like, integration into the real world, um, some piece of it is, of course, like interfaces and change management, and like figuring out how users are gonna understand the the model outputs and and guardrails and all of that. Um, specifically, when we think about the model and specialization, uh, like, do you have some framework you offer customers or that you use internally around? Um, what version of it they should invest in, right? So we have pre training, post training, fine tuning, retrieval, like in in those sort of traditional sense, like prompting, especially as we get longer contexts, like, how, how do you tell customers to make sense of how to specialize?
It really depends on the application. Like, there's some stuff, for instance, we partnered with Fujitsu, um, who's like the largest SI in in Japan, um, to build a Japanese language model. There's just no way you can do that without intervening on pre training. You can't like, fine tune or post train Japanese into a model effectively, and so you have to start from scratch. Um, on the other side, there's more narrow things like if you want to change the tone of the model, or you want to, um, I, I don't know, change how it formats certain things.
I think you can just do fine tuning, you can take the the end state, um, and so there is this gradient. What we usually recommend to customers is start from the cheapest, easiest thing, which is fine tuning, and then work backwards, and so start with fine tuning. Then go back into post training, right? Like SFT, RLHF, then if you need to, and you know it, it's kind of a journey, right? Like, as you're talking about a production system and the constraints are getting higher and higher, you potentially will need to touch pre training.
Hopefully, not all of pre training, hopefully, it's like 10 of pre training at the very end, or maybe 20 of pre training. But yeah, that's usually how we think about. It is like this journey from the simplest, cheapest thing to the most sophisticated, but most performant.
Moving along the gradient from the cheapest thing makes sense to me. Uh, the idea that any enterprise customer will invest in pre training is, I think, a bit more controversial. I I believe some of the lab leaders would say like, nobody should be touching this. And it doesn't make any sense for people from a scale of compute and data data curation effort required, and just sort of the talent required to do pre training in any sort of competitive way.
Like, W- How would you react to that?
I think if you're building like a if you're a big enterprise and you're sitting on a ton of data, like hundreds of billions of tokens of data. Um, pre training is a real lever that you're able to pull. I think for most, like SMBs and and certainly startups, it makes no sense that you should not be pre training a model. Um, but if you're a large enterprise, I think it's it should be a serious consideration. The question is, how much pre training?
It's not like you have to start from scratch and do a, you know, 50 million training run, but you can do a frag, you could do 5 million training run. That's what we've seen succeed, these sort of continuation pre training efforts. Um, so yeah, that that's one of the offerings that we have. But of course, we don't jump straight into that. You don't need to spend massively if you don't want to. And and usually, uh, the enterprise buying cycle or or technology adoption cycle is quite slow, and so you have time to move back into it.
I would say it's totally at the customer's discretion. Um, but to the folks who say that no one should be pre training,
No one outside of, let's say, AGI labs should be pre training.
that's empirically wrong.
Maybe that's a, uh, like, a good jumping off point into, just like, talking a little bit more about what's going on in the technical landscape, and also what that means for Cohere. Like, what is the what is the bar you set internally for Cohere? You said the model's the foundation. Um, and uh, I believe you've also said like, there's no market for last year's models. Like, how do you square that? With the expense of the capital expense of competition and the rise of open source models now?
Well, I think you have to spend. There's some like minimum threshold that you need to be spending at in order to build a a model that's useful. The things get cheaper, the compute to train the model get cheaper, um, the sources of data. Uh, well, in some directions they get cheaper, in others not. With synthetic data, it's gotten dramatically cheaper, but with expert data, it's getting harder and harder and more expensive. And so what we've seen is today you can build a model that's as good as GPT-4 in all the things that enterprises might care about.
For 10 million, 20 million, like just orders of magnitude less than what was spent to develop that model. And so if you're willing to wait six months or a year to build the technology, you can build it at a fraction of what those frontier labs have paid, uh, to develop it. And so that's been a key part of Cohere's strategy is we don't need to build that thing. 1st, what we'll do is we'll, we'll figure out how to do it dramatically cheaper, and we'll focus on the parts of it that matter to our customers.
So we'll focus on the capabilities that our customers really depend on. Now, at the same time, we still have to spend like relative to a regular startup, we have to pay for a supercomputer and those things cost hundreds of millions of dollars a year. Um, so it is capital hungry, but it's not capital inefficient. Um, it's very clear that we'll be able to build a very profitable business off of what we're building. So that's the the strategy is, don't lead, don't burn, you know, three, five, $7 billion a year.
To be at the front, be six months behind and offer something to market to enterprises that actually fits their needs at a price point that makes sense for them.
Why spend on the supercomputer and the training yourself at all, if you have, um, increasingly open source options?
Well, you don't not really.
Say more.
So for Llama, uh, yeah, you get like the base model, uh. At the end, when it's cooled down and it has zero gradient, you get the, uh, the post trained model at the end, when it's cooled down and has zero gradient. Taking those models and trying to, um, fine tune them, it's just it's not as effective as building it yourself, and you have much fewer levers to pull. Um, than if you actually have access to the data and you can change the data that goes into that process.
And so we feel that by being vertically integrated and by building these models ourselves, we just have dramatically more leverage to offer our customers.
Maybe if we go to, um, projection, and we'll hit on a few things that you've mentioned as well. Um, where are we in scaling laws? Like, how much capability improvement do you expect over the next few years?
We're, we're pretty far along. I would say like we're starting to enter into a sort of flat part of the curve. Um, and we're certainly past the point where if you just interact with a model, you can know how smart it is. Like the the vibe checks, they're losing utility. And so instead, what you need to do is you need to get experts to measure within very specific domains like physics, math, uh, chemistry, biology, um, you need to get experts to actually assess the quality of these models.
Because the average person can't tell the difference at this stage between generations. Yes, like, there's still much more to go do, uh, but those gains are gonna be felt in very specialized areas and have impacts on more researchy, um, more researchy domains. I think for enterprises and the general sorts of tasks that they wanna automate or tools that they wanna build, the technology is already good enough or close enough that a a little bit of customization will get them there.
So that that's sort of the stage that we're at there is. There's a new unlock in terms of the category of problems that you can solve, and that's reasoning, and so online reasoning is something that has been missing. These models, they don't have a, they previously didn't have an internal monologue, right? Like they didn't really think to themselves. You would just ask them a question and then expect them to immediately answer that question. They couldn't reason through it, they couldn't fail, right?
Like, make a mistake, catch that mistake, fix it and try again. And so the fact that we now have reasoning models coming online, of course, OpenAI was the 1st to put it into production, but Cohere's been working on it, uh, for about a year now. Um, this category of tech, I think, is is really interesting. There's a new set of problems that you can go solve, um, and it also changes the it changes the economics. So before, if I had a customer come to me and say, Uh, Aidan, I want your model to be better at X, or I want a smarter model, I would say, Okay, you know, give us six to twelve months.
We need to go, uh, spin up a new training run, train it for longer, train a bigger, bigger model, et cetera, et cetera. Um, now there's that. That was kind of like the only lever we had to pull to improve the performance of our product. There's now a 2nd lever, which is you can charge the customer more. You can say, Okay, let's spend twice as many, uh, you know, tokens, or or let's spend twice as much time at inference time, and you'll get a smarter model.
So there's a a much nicer product experience. Okay, you want a smarter model, you can have it today, you just need to pay this. And so they have that option, they don't need to wait six months. And similarly, for model builders, I don't need to go double the size of my supercomputer to hit a requisite intelligence threshold, I can just double the amount of inference time compute that my customers pay for. So I think that's a really interesting structural change, uh, in how we can go to market and what products we can build and what we can offer to the customer.
I agree. I think it's, um, uh, perhaps undervalued in the ecosystem right now. How, uh, much more appealing it should be to all types of customers. That you can move from a like a capex model of improvement to a consumption model of improvement, right? And it's not like, you know, these are apples and oranges things, but um, but I I think you'll see people invest a lot more in, you know, solving problems. When they don't have to pony up for a a training run and have this delay, as you described,
Yeah, it hasn't been clocked like people haven't really priced in the impact of inference time compute delivering intelligence. Um, there's loads of consequences, even at like the chip layer, right? Like, what sort of chips you wanna build, what you should prioritize for data center construction. Um, if we have a new avenue, which is inference time compute, that doesn't require this densely interconnected supercomputer. It's fine to have nodes, you can do a lot more locally and less distributed.
I I think it has loads of impact up and down this chain, and it it's a new paradigm of, um, what these models can do and and how they do it.
you were dancing around this. But because our, you know, your average person doesn't spend that much time thinking about, like, what is reasoning, right? Do you have any intuition you can offer people for, like, what are the types of problems? This allows us to tackle better?
Yeah, I think any sort of multi-step problem, um, like, there's some multi-step problems you can just memorize, which is what we've been asking models to do so far. Um, like solving a polynomial, right? Like that, really, that should be approached multi-step. That's how humans solve it. We don't just get given a polynomial and then boom. There's a few that maybe we've memorized, right? But by and large, you have to work through those problems, break them down, solve the smaller parts and then compose it into the overall solution.
And that's what we've been lacking, we've really la- and and we've had stuff like chain of thought, which has, um, enabled that. But it's sort of like a retrofitting. It's sort of like we train these models to just memorize input output pairs, and we found a nice little hack to, uh, elicit the behavior that mimics reasoning. I think what's coming now is from scratch. The next generation of models that is being built and delivered will have that reasoning capability burnt into it from scratch.
And it's. It's not surprising that it wasn't there to begin with, because we've been training these models off of the internet. And the internet is like a set of documents, which are the output of a reasoning process, with the reasoning all hidden. It's like a human wrote an article and, you know, spent weeks thinking about this thing and deleting stuff and blah, blah, blah, um, but then posted the final product. And that's what you get to see.
Everything else is implicit, hidden, unobservable, um, and so it makes a lot of sense why the 1st generation of language models lacked this inner monologue. But now what we're doing is we're with human data and with synthetic data, we're explicitly collecting people's inner thoughts. So we're asking them to verbalize it, and we're transcribing that, and we're gonna train on that and model that part of the problem solving process. Um, and so I'm I'm really excited for that.
I think right now it's extremely inefficient and it's quite brittle, similar to the early versions of of language models. But over the next two or three years, it's gonna become incredibly robust and unlock just a whole new set of problems.
What is the basic driver of the slowdown, you know, reaching the flat part of the curve that you you describe with scaling? Is it is it the cost of, you know, increasingly expert data and collecting, as you said, like hitting reasoning traces? That is harder and more expensive than just taking the data on the internet? Is it the difficulty of having evals for, you know, increasingly complex problems? Um, is it just overall, uh, cost of compute?
Like, w- Why do you think that flattening is happening?
When when someone's making an oil painting, um, they do a a back coat and just cover the whole, the whole canvas. And then they they sort of paint in the shapes of, uh, you know, the mountains and the the trees. And and as you, you get more and more detailed, you're bringing out very fine brush strokes. Um, there's a lot more of them that you need to make. Before you could just take a big wedge and just throw paint across the canvas and accomplish the thing that you wanted to accomplish.
But as you start to get more and more targeted, or, um, more and more detailed in what you're trying to accomplish, um, it requires a much more fine instrument. Uh, and so that's what we've seen with language models. We're able to do a lot of the common, simple, easy tasks quite quickly. But as we've approached much more specific, sensitive domains like science, uh, math, that's where we've started to see resistance to improvement. And in some places, we've gotten around that by using synthetic data, like in code and math.
These are places where the answer is very verifiable. You, you know, when you're right or you're wrong, and so you can generate tons of synthetic data and just verify whether it's correct or not. You know, it's correct, okay, let's train on it. Um, in other areas that require testing, uh, and knowledge in the real world, like in biology, like in chemistry, um, there's a. There's a bigger bottleneck to creating that sort of data. And you have to go to experts who know the field, who who have experienced it for decades, and basically distill their knowledge.
Um, but eventually you, you run out of experts and you run out of that data, and you're at the frontier of what humans know about X, Y or Z. There's just increasing friction to fill in these much finer details of this portrait. I think that's a fundamental problem. I, I don't think that there's any shortcuts, uh, around that. Um, you know, at some stage, we're gonna have to give these models the ability to run their own experiments, to to fill in areas of their knowledge that they're they're curious about.
Um, but I I think that's quite, quite a ways away, uh, and it's gonna be tough to scale that. It will take many, many years to do. We will do it, we're gonna get there 100, um, but for the stuff that I care about today. With Cohere, I think there are many applications which this technology is ready for production for. And so the primary focus is getting it to production and ensuring that our our economy adopts this technology and integrates it as quickly as possible, gets that productivity uplift.
And so while that technical question is super interesting about, you know, why is progress slowing down? I I think it should be kind of obvious, right? It's like the models are getting so good, they're hitting, they're running into the thresholds of human knowledge. Um, which is really where they're getting their capability from.
You are so grounded in, you know, getting the capabilities we have and will continue to progress, even if the curve is flattening into production. I I think I know this answer, but how much do you, or how much does Cohere think about, like, AGI and takeoff, and does that matter to you?
Well, AGI means a lot of things to a lot of different people. I think I I believe in us building generally intelligent machines. Like, completely, that it's like, of course, we're gonna do that, um, but AGI has been conflated.
How soon?
We're already there. I. It's not a, you know, it's not a binary, it's not discrete, it's continuous. And we're like, uh, well on our way, we're, we're pretty far down that road.
There's some, uh, definition elsewhere in industry that they're like, you can put a breakpoint at, even if you, even if you have this, um, continuous function, you you can put a breakpoint in. Like it, there's intelligence that replaces, like an educated adult professional in any digital role. Your view is n- there's no really important breakpoint that's happening
That sort of like objective checklist thing. And like, when you've checked all these boxes, then you've got. I think you can always find like a counterexample. You're like, Oh, well, it hasn't actually beaten this one human over here who's doing this like, random, random thing. Um, no, I think it's, I think it's pretty continuous and we're like, quite far, quite far along. Um, but the the AGI that I I really don't subscribe to is this super intelligence takeoff, self improvement, just leading to, uh, the terminator that exterminates us all, or creates abundance.
Unclear yeah, or creates abund- right, right, yeah, um, no, I think we'll be the one to create abundance. We don't need to wait for this God to emerge and do it for us. Let's go do it with the tech that we're we're building, you know, we don't need to depend on that, we can go do it ourselves. We will build AGI, if what you mean is very useful, generally capable technology that can do a lot of the stuff, uh, that humans can do and flex into a lot of different domains.
Um, if what you mean is, you know, are we gonna build God? Uh, no,
What do you think is the driver in that difference of opinion?
I don't know. I I think maybe I'm a little bit more in the weeds of the practical frustrations of the technology. Where it breaks, where it's slow, where it. We start to see things plateau or slow down. Um, and perhaps others are more, maybe they're more optimistic, maybe maybe they see, um, they see a a curve increasing and they just think it goes on forever. Like that will just continue arbitrarily, which I I disagree with. I think there's there's friction points, like there is genuinely friction that enters in.
Like, maybe, even if in theory, you know, like a neural net is a universal approximator, it can learn anything to universally approximate. You would need to build a neural net the size of the universe. And so, like, there's some fundamental barriers to reaching limits that people extrapolate out to. That, I think, will, um, bound the practically realizable, um, forms of this technology.
Are there domains where you, um, just believe LLMs as we have them today are, like, not a good fit for prediction, right? And so an example might be like, are we going to get to physics simulation from sequence to sequence models?
I mean, probably, yeah, like physics is just like a series of states and, um, transition probabilities, so I I think it's probably quite well modeled by sequence modeling. But are there areas where it's poorly suited? Um, I'm sure, I'm sure that there are better models for certain things, more efficient models. Like you can, you can take it. If you zoom into a specific domain, you can take advantage of structure in that domain to carve off some of the unnecessary generalities of the transformer.
Um, or of these, this category of architectures, um, and get a more efficient model. That's definitely true when you, when you zoom in,
And it doesn't sound like you think it's like a at its core, like a a representation issue where it's just not gonna work.
there's irreducible uncertainty in the world. There, like, there's things that you genuinely cannot know, and like, building a bigger model will not help you know this genuinely random, uh, or unobservable thing, uh. And so those things will will never be able to model effectively until we learn how to observe them. Or, uh, you know, I think the transformer and this category of model can do much more than people give it credit for. It's a very general architecture.
Many, many things can be phrased as a as a sequence, and these models are just sequence models, uh, and so if you can phrase it as a sequence, the transformer can do a fairly good job at picking up any regularity in it. But I, I'm, I'm certain that there are examples that I'm just not able to think of right now. Where sequence modeling is, uh, super inefficient. Like you can do it with sequences, you can phrase a graph as a sequence, um, but it's just like the wrong model.
And you would pay dramatically less compute if you approached it from a a different angle.
Okay, one last question for you. So you, uh, concluded earlier that, um, scaling compute at inference time, like, oh, people have noticed, but it's not really priced in, like how big of a change this is. Is there anything else you think is not priced in by the market right now? That, like, Cohere thinks about that, you think about
Yeah, I think there's this idea of like, commoditization of models. I don't really think that's true. I don't think that models are actually getting commoditized. I I think what you see is you see price dumping, um, and so you see people giving it out for free, giving it out at a loss, giving it out at zero margin. Um, and so they see the prices coming down and they assume prices coming down means commoditization. I think in reality, the state of the world is there's a total technological refactor that's going on right now and will last the next ten to fifteen years.
And it's kind of like, we have to, we have to repave every road on the planet. And there's like four or five companies that know how to make concrete, uh, okay, and like, maybe today, some of them give their concrete, create away for free. Um, but over time, there's very. There's a very small number of parties that know how to do this thing. And a huge job in front of us and pressures to drive growth, to show return on investment.
It, it's an unstable present state to be operating at a loss or giving away very expensive technology for free. Um, so growth pressures of the market will push things in a certain direction. And yeah, I, you know, the price of Haiku fourxed, uh, two weeks ago.
Aidan? This has been super fun. Thank you so much for doing this with us.
Yeah, my pleasure, my pleasure. It was super fun, great seeing you.
Find us on Twitter at No Priors Pod, subscribe to our YouTube channel. If you wanna see our faces, follow the show on Apple Podcasts, Spotify, or wherever you listen. That way, you get a new episode every week, and sign up for emails or find transcripts for every episode at no dash priors dot com.