OpenNashPodcasts

Latent Space: The AI Engineer Podcast

FlashAttention 2: making Transformers 800% faster w/o approximation - with Tri Dao of Together AI

26 Jul 2023 · 54m · Tri Dao, Alessio Fanelli · 24 quoted facts

▶ Play episodeOriginal source ↗

In brief

Tri Dao argues that FlashAttention made Transformers faster by changing how memory is used, not by approximating the math. Attention still computes exactly the same result, but it moves less data between slow and fast GPU memory. He says speed on real hardware depends on memory traffic as much as on the number of arithmetic operations. He also argues that open models and open data matter for the field.

The main tension is between what is easy to run on today's hardware and what might be better in principle. Dao says GPUs, software and Transformers have shaped one another, so alternatives such as recurrent networks and state-space models struggle to compete. He remains hopeful about those alternatives for long sequences and high-volume generation, while still making attention as fast as possible. A similar balance shows up in his views on academia and industry, and on open and closed models.

Key insights

  1. Memory traffic, not arithmetic, limits attention speed

    Dao found that approximate attention methods do fewer calculations but often aren't faster in practice. Most of the time goes to reading and writing memory, so FlashAttention focused on cutting that traffic.

  2. FlashAttention stays exact while saving memory

    It computes the same attention result as before, with memory that grows linearly with sequence length instead of quadratically. Compute time is still quadratic, but the hardware-friendly design gives roughly 2-4x faster runs.

  3. Combining systems and ML ideas made it work

    Kernel fusion and tiling came from systems work, and the online softmax trick came from machine learning. Softmax had blocked fusion, and the trick lets it be split into pieces and rescaled.

  4. Fixed-size RNN state could raise generation throughput

    Transformers keep a growing KV cache of past tokens, which limits how many requests fit in memory at once. Dao argues RNNs summarize history in a fixed-size state, allowing bigger batches, though he says this is not yet proven at scale.

  5. Hardware and software lock in the Transformer

    Dao says years of engineering effort, libraries and hardware design have been tuned for attention. That makes alternatives such as LSTMs and GRUs hard to run efficiently, which he calls a feedback loop.

Questions

What does FlashAttention actually do?

It speeds up the attention step in Transformer models without approximating the result. Tri Dao says it does this by cutting the reading and writing of data in GPU memory, which gives roughly 2-4x faster runs.

Why was FlashAttention a surprise to researchers who focused on approximation?

Approximate methods perform fewer calculations, so people expected them to be faster. Dao says they often weren't faster in practice because memory reading and writing dominates the time.

What are HBM and SRAM, and why do they matter here?

HBM is the GPU's large main memory, and SRAM is a much smaller but faster memory next to the compute units, like a CPU cache. Dao says keeping work in SRAM and avoiding trips to HBM is where the speedup comes from.

Will Transformers be replaced by something else?

Dao says alternatives like state-space models and RNNs have matched Transformers on language modeling in some results. He is bullish that they could win on very long inputs and high-volume generation, but says this hasn't been validated at scale.

Why does Dao think Llama 2 matters for businesses?

He says it is less restrictive than Llama 1 and competitive with GPT-3.5 on benchmarks, so many companies can fine-tune and host it themselves. He argues that this reduces reliance on a few companies' APIs.

Facts

Stated by the speakers. Each quote matches the transcript, which shows what was said, not that it is true.

Tri Dao became chief scientist at Together, the company behind RedPajama, in the week of the recording.

“you're a chief scientist at Together as well, which is the company behind the RedPajama.”

0:00:39 · Alessio Fanelli · Together AI · 2023

Tri Dao was due to become an assistant professor of computer science at Princeton the following year.

“And now you're going to be an assistant professor in CS at Princeton next year.”

0:00:39 · Alessio Fanelli · Career · 2023

Tri Dao started college intending to study economics but switched to a math major during his first week as a Stanford undergraduate.

“the first week I was at Stanford undergrad. I took a few math classes and I immediately decided that I was going to be a math major”

0:01:09 · Tri Dao · Education

FlashAttention uses memory that grows linearly with sequence length, while its computation remains quadratic.

“the memory is linear in sequence length. In terms of computation, it's still quadratic, but we managed to make it much more hardware friendly.”

0:02:50 · Tri Dao · FlashAttention · 2022

FlashAttention gave wall-clock speedups of roughly two to four times without approximating attention, according to Tri Dao.

“we do get walk-lock speed up on the order of two to four X”

0:02:50 · Tri Dao · FlashAttention speedup · 2022

Tri Dao says approximate attention methods perform fewer operations but tend not to run faster in wall-clock time.

“these approximation methods, even though they perform fewer computation, they tend to not be faster in walk-on-tap.”

0:04:11 · Tri Dao · Approximate attention

Tri Dao says most of the time in attention goes to reading and writing memory, which is why FlashAttention targeted memory traffic.

“we ended up focusing a lot more on memory reading and writing because that turned out to be the majority of time when you're doing attention is reading and writing memory.”

0:05:16 · Tri Dao · Memory bottleneck

The online softmax trick, which lets softmax be computed in pieces, appeared in a 2018 paper by NVIDIA researchers, according to Tri Dao.

“So this online softmax trick has been around for a while. I think it was a paper from NVIDIA folks back in 2018 about this.”

0:07:41 · Tri Dao · Online softmax · 2018

A late-2021 paper from Google researchers used the online softmax trick to break attention into smaller pieces, as Tri Dao tells it.

“there was a paper from Google. So Marcus Stroup and Statz wrote a paper late 2021 on using this online softmax trick to break attention up into smaller pieces.”

0:08:13 · Tri Dao · Online softmax · 2021

Alessio Fanelli cites GPU memory throughput of about 1.5 terabytes per second for HBM and about 19 terabytes per second for SRAM.

“the throughput on card is like 1.5 terabytes a second for HBM and like 19 terabytes a second for Astram”

0:13:00 · Alessio Fanelli · GPU memory

Tri Dao says memory-hierarchy-aware algorithm design dates to 1980s sorting problems, such as banks sorting transactions overnight.

“the motivating example was banks were trying to sort their transactions, and that needs to happen overnight so that the next day they can be ready.”

0:12:25 · Tri Dao · Memory hierarchy · 1980s

Tri Dao says the first mention of attention in machine learning was in a 2014 paper involving Yoshua Bengio, which came from academia.

“I think the first mention was in 2014, a paper from Banado and others and Yoshua Bengio, which is coming from academia.”

0:20:47 · Tri Dao · Attention history · 2014

Tri Dao says OpenAI committed to scaling after GPT-2, which had 1.5 billion parameters and showed strong capabilities.

“That was the bet that they made after, I think, GPT-2. So they saw that scaling these things up to, back then was 1.5 billion parameter, seemed to give you amazing capabilities.”

0:20:47 · Tri Dao · Scaling · 2019

The Stanford Center for Research on Foundation Models, led by Percy Liang, created the HELM benchmark, which Tri Dao describes as holistic.

“They have this benchmark called HELM, which is this holistic benchmark.”

0:21:55 · Tri Dao · Benchmarks

Researchers at Berkeley set up Chatbot Arena to benchmark chatbot models, according to Tri Dao.

“They set up this kind of chatbot arena to essentially benchmark different models.”

0:26:11 · Tri Dao · Chatbot Arena · 2023

FlashAttention 2 is about twice as fast as the original and, per Tri Dao, close to the efficiency of matrix multiplication.

“we managed to make it 2x faster. And now it's pretty close to probably the efficiency of things like Matrix Multiply”

0:28:13 · Tri Dao · FlashAttention 2 · 2023

AMD developers implemented a version of FlashAttention for their GPUs, according to Tri Dao.

“the AMD folks did implement a version of Flash Attention, I think last year as well, and that's also available.”

0:28:56 · Tri Dao · Hardware support · 2022

Tri Dao cites Tesla's Dojo supercomputer as a hardware bet that emphasizes fast on-chip memory.

“I think Tesla also has this Dojo supercomputer where they try to have as, essentially as fast on-chip memory as possible”

0:33:07 · Tri Dao · AI hardware

State-space sequence models from Albert Gu, Chris Ré and colleagues appeared in 2021 as a Transformer alternative that scales sub-quadratically with sequence length.

“state-space methods that came out in 2021 from Albert Gu and Curran and Chris Ray”

0:38:25 · Tri Dao · State-space models · 2021

Tri Dao says RWKV, a recurrent architecture, has a 14-billion-parameter model that appears competitive with Transformers.

“I think RWKV scale up to, they have a model at 14 billion that seems pretty competitive with Transformers.”

0:39:35 · Tri Dao · RWKV · 2023

Transformers keep earlier tokens in a KV cache during generation, which uses large amounts of memory and limits batch size, according to Tri Dao.

“It's called the KV cache. And that could take a significant amount of memory. So you can't really batch too much because you run out of memory.”

0:41:33 · Tri Dao · KV cache

Tri Dao says Meta made Llama 2 much less restrictive than Llama 1 so that many businesses can use it.

“for LLaMA 2 they kind of make it much less restrictive compared to LLaMA 1s where now you can use it for businesses unless you are a monthly active user or something like that.”

0:44:40 · Tri Dao · Llama 2 license · 2023

Llama 1's context length was limited to 2K tokens, and open-source developers found simple methods to extend it to 8K, according to Tri Dao.

“for LLaMA 1, the context lane was limited to 2K. A bunch of folks figured out some really simple methods to scale up to 8K.”

0:46:48 · Tri Dao · Context length · 2023

Cerebras deduplicated and cleaned the RedPajama dataset and released it as SlimPajama, per Tri Dao.

“I think Cerebus then worked on that and deduplicate and clean it up and release Slim Pajama and so on.”

0:48:18 · Tri Dao · Open datasets · 2023

Chapters

  1. 0:02:17Exact attention instead of approximation

    Dao explains why attention is costly on long inputs and how FlashAttention keeps the exact result while saving memory.

    “attention scales quadratically in sequence length on both runtime and memory”
  2. 0:06:38Kernel fusion and tiling

    This section explains the systems ideas behind the speedup and why softmax made them hard to apply to attention.

    “You just load it once, perform two operations, and then write it down again.”
  3. 0:14:16Why SRAM stays small and HBM grows

    Dao gives the physical reason fast on-chip memory won't grow much, so memory-aware algorithms will matter more over time.

    “SRAM probably won't get that much larger because you don't have that much area”
  4. 0:19:07How Hazy Research works

    Dao describes the mix of theory, systems and application researchers at Hazy Research, and the advice to learn fundamentals.

    “a lot of research ideas come from understanding the old literature”
  5. 0:21:55Academia versus industry

    Dao lays out complementary roles: industry scales models, while academia evaluates, explains and takes riskier bets.

    “They have the luxury to slow down trying to understand stuff.”
  6. 0:27:47FlashAttention 2 and the hardware lottery

    Dao describes how a library refactor turned into a 2x faster kernel, then discusses how hardware and software favor Transformers.

    “I'm going to refactor my code, and use this library.”
  7. 0:41:33Transformer alternatives

    Dao weighs state-space models and RNNs against attention for very long sequences and high-throughput generation.

    “I am personally bullish on RNNs.”
  8. 0:46:16Open source AI and Llama 2

    The conversation covers licensing, open weights versus open data, and why Dao thinks Llama 2 shifts power away from a few companies.

    “we don't want just a concentration of decision-making power in the hands of a few companies”
  9. 0:53:22Lightning round

    Dao names jokes as a surprise and reasoning as an open problem, and urges people to learn both algorithms and systems.

    “try to understand both the algorithm and the systems that these algorithms run on”

Transcript

0:00:09
Alessio Fanelli

Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO and residents at Decibel Partners. Today we have no swyx because he's in Singapore. So it's a one-on-one discussion with Tri Dao. Welcome. Hi everyone, I'm Tri Dao. Excited to be here. Tri just completed his PhD at Stanford a month ago. You might not remember his name, but he's one of the main authors in the Flash Attention paper, which is one of the seminal work in the Transformers era.

0:00:39

He's got a lot of interest from efficient Transformer training and inference, long range sequence model, a lot of interesting stuff. And now you're going to be an assistant professor in CS at Princeton next year. Yeah, that's right. And in the meantime, just to get a low pressure thing, you're a chief scientist at Together as well, which is the company behind the RedPajama. Yeah, yeah. So I just joined this week, actually, and it's been really exciting.

0:01:04

So what's something that is not on the Internet that people should know about you?

0:01:09
Tri Dao

Let's see. When I started college, I was going to be an economist. So I was fully on board. I was going to major in economics. but the first week I was at Stanford undergrad. I took a few math classes and I immediately decided that I was going to be a math major and that kind of changed the course of my career. So now I'm doing math, computer science, AI research.

0:01:32
Alessio Fanelli

I had a similar thing. I started with physics and then I took like a programming course and I was like, I got to do computer science. I don't want to do physics. So FlashAttention is definitely, everybody's using this. Everybody loves it. You just released FlashAttention too. last week. Yeah, early this week

0:01:49
Tri Dao

on Monday, yeah.

0:01:50
Alessio Fanelli

AI time. Yeah, things move fast. Four days ago is one week ago in AI. So maybe let's run through some of the flash attention highlights, some of the innovation there, and then we can dive into flash attention too. So the core improvement in flash attention is that traditional attention is a quadratic sequence length, and to the two, flash attention is linear, which obviously helps with scaling some of these models?

0:02:17
Tri Dao

There are two factors there. So, of course, the goal has been to make attention go faster or more memory efficient. And ever since attention became popular in 2017 with the Transformer paper, lots and lots of folks have been working on this. And a lot of approaches have been focusing on approximating attention. The goal is you want to scale to longer sequences. There are tons of applications where you want to do that. But scaling to longer sequences is difficult because attention scales quadratically in sequence length on both runtime and memory, as you mentioned.

0:02:50

So instead of trying to approximate attention, we were trying to figure out, can we do the same computation and maybe be more memory efficient? So in the end, we ended up being, the memory is linear in sequence length. In terms of computation, it's still quadratic, but we managed to make it much more hardware friendly. And as a result, we do get walk-lock speed up on the order of two to four X, which really helps because that just means that you'll be able to train with 2-4x longer sequence length for the same cost without doing any approximations.

0:03:21

As a result, lots of folks have been using this. It's available in a lot of libraries that do language model training or fine-tuning.

0:03:32
Alessio Fanelli

The approximation thing is important because this is an exact thing versus a sparse. Maybe explain a little bit the difference there.

0:03:40
Tri Dao

For sure, for sure. Yeah. Tension, essentially you compute pairwise similarity between every single element in a sequence against each other. So there's been other approaches where instead of doing all that kind of pairwise computation, you only compute similarity for some pairs of elements in the sequence. So you don't do kind of quadratic number of comparison. And this can be seen as some form of sparsity. Essentially, you're ignoring some of the elements. When you write down the matrix, you essentially say, okay, I'm going to pretend they're zero.

0:04:11

That has some benefits in terms of runtime and memory. But the trade-off is that it tends to do worse in terms of quality because you're essentially approximating or ignoring some elements. And I personally have worked on this as well for a few years. But when we talk to practitioners who actually train models, especially at large scale, they say, tend not to use these approximate attention methods. Because it turns out, this was surprising to me at the time, was that these approximation methods, even though they perform fewer computation, they tend to not be faster in walk-on-tap.

0:04:49

So this was pretty surprising because back then I was, I think my background was more on the theoretical side. So I was thinking of, oh, how many flops or floating point operations are you performing? And hopefully that correlates well with wall clock time. But I realized that I was missing a bunch of ideas from the system side where flops or floating point operations don't necessarily correlate with run time. There are other factors like memory reading and writing, parallelism and so on.

0:05:16

So I learned a ton from just talking to systems people because they kind of figured this stuff out a while ago. So that was really eye-opening. And then we ended up focusing a lot more on memory reading and writing because that turned out to be the majority of time when you're doing attention is reading and writing memory.

0:05:34
Alessio Fanelli

Yeah, the IO awareness is probably like one of the biggest innovation here. and the idea behind it is like you mentioned the the flops growth of the cars have been going up but like the the memory bandwidth not as much so i think like maybe that was one of the assumptions that the original attention paper ad yeah so talk a bit about how that came to be as an idea you know it's one of those things that like an insight it's like obviously why are we like rewriting to like hbm every time yes you know and like once you once you change it it's clear but what was that

0:06:06
Tri Dao

discovery process. Yeah, in hindsight, a lot of the ideas have already been there in the literature. And I would say it was somehow at the intersection of both machine learning and systems, and you kind of needed ideas from both sides. So on one hand, on the system side, so lots of systems folks have known that kernel fusion is great. Kernel fusion just means that instead of performing, you know, loading the same element, instead of performing an operation, write it down, load it back up, and perform the second operation.

0:06:38

You just load it once, perform two operations, and then write it down again. So that saves you memory read and write in the middle there. So kernel fusion has been classic. There's been other techniques from the system side, like tiling, where you perform computations in block, again, so that you can load it into really fast memory. Think of it as a cache. And this is, again, classical computer science ideas. You want to use the cache. So the system source have been thinking about these ideas for a long time, and they apply to attention as well.

0:07:12

But there were certain things in attention that made it difficult to do in a complete cone of fusion, one of which is there is this softmax operation in the middle, which requires you to essentially sum across the row of the attention matrix. So it makes it difficult to kind of break it because there's this dependency, so it makes it difficult to break things into a block. So on the system side, people have been thinking about these ideas, but it's been difficult to kind of do kernel fusion for the entire operation.

0:07:41

On the machine learning side, people have been thinking more algorithmically. They say, okay, either we can change, we can approximate attention, or there's this trick called the online Softmax trick, which says that you can, because of Softmax, the way it's written mathematically, you can actually break it up into smaller pieces, do some rescaling, and still get the right answer. answer. So this online softmax trick has been around for a while. I think it was a paper from NVIDIA folks back in 2018 about this.

0:08:13

And then there was a paper from Google. So Marcus Stroup and Statz wrote a paper late 2021 on using this online softmax trick to break attention up into smaller pieces. So a lot of the ideas were already there. But it turns out you kind of need to combine ideas from both sides. So you need to understand that, hey, we want to do kernel fusion to reduce memory written writes. But we also need this online softmax trick to be able to break the softmax into smaller pieces so that a lot of the systems tricks kind of carry through.

0:08:49

We saw that and it was kind of a natural idea that we ended up using ideas from both sides and it ended up

0:08:55
Alessio Fanelli

working pretty well. Yeah. Are there any downsides to kernel fusion? If I think about databases and the reasons why we have atomic operations. You know, it's like you have observability and fallback in between them. How does that work with attention? Like, is there anything that we lose by fusing the operations?

0:09:13
Tri Dao

Yeah, I think mostly on the practical side is that you lose a little bit of flexibility in the sense that, hey, now you have, for example, is flash attention is just a subroutine that you would call to do attention. But as a researcher, let's say you don't want that exact thing, right? You don't want just attention. Let's say you want some modification to attention. You want to do, hey, I'm going to multiply the query and key, but then I'm going to do this extra thing before I carry on.

0:09:42

And so kernel fusion just means that, okay, we have a subroutine that does the entire thing. But if you want to experiment with things, you won't be able to use that fused kernel. And the answer is, can we have a compiler that then automatically does a lot of this kernel fusion. Lots of compiler folks are thinking about this, either with a new language or you can embed it in PyTorch. So the PyTorch folks have been working on this as well.

0:10:11

So if you write just your code in PyTorch and they can capture the graph, can they generate code that will kind of fuse everything together? And that's still ongoing. And it works for some cases, but for attention because of this kind of softmax rewriting stuff, it's been a little bit more difficult. So maybe in a year or two, we'll have compilers that are able to do a lot of these optimizations for you. And you don't have to, for example, spend a couple of months writing CUDA to get this stuff to work.

0:10:41

Awesome.

0:10:41
Alessio Fanelli

And just to make it clear for listeners, when we say we're not writing it to memory, we are storing it just in a faster memory. So instead of the HBM, we're putting it in the SRAM. Yeah, maybe explain just a little bit the difference there.

0:10:56
Tri Dao

Yeah, for sure. This is kind of a caricature of how you think about accelerators or GPUs in particular, is that they have a large pool of memory, usually called HBM, a high bandwidth memory. So this is what you think of as GPU memory. So, you know, you're using A100 and you list the GPU memory is like 40 gigs or 80 gigs. So that's the HBM. And then when you perform any operation, you need to move data from the HBM to the compute unit.

0:11:25

So the actual hardware unit that does the computation. And next to these compute units, there are on-chip memory or SRAM, which are much, much smaller than HBM, but much faster. So the analogy there is if you're familiar with, say, CPU and RAM and so on, so you have a large pool of RAM, and then you have the CPU performing the computation, but next to the CPU you have L1 cache and L2 cache, which are much smaller than DRAM, but much faster.

0:11:54

So you can think of SRAM as a small and fast cache that stays close to the compute unit. Physically, it's closer. There is some kind of asymmetry here. So HBM is much larger and SRAM is much smaller, but much faster. One way of thinking about it is how can we design algorithms that take advantage of this asymmetric memory hierarchy? And of course, lots of folks have been thinking about this. These ideas are pretty old. So I think back in the 1980s, the primary concerns were sorting.

0:12:25

How can we sort numbers as efficiently as possible? And the motivating example was banks were trying to sort their transactions, and that needs to happen overnight so that the next day they can be ready. And so the same idea applied, which is that they have slow memory, which was disk, like hard disk, and they have fast memory, which was DRAP. And people had to design sorting algorithms that kind of take advantage of this asymmetry. And turns out, you know, these same ideas can apply today, which is different kinds of memory.

0:13:00
Alessio Fanelli

Yeah. In your paper, you have kind of like the pyramid of memory. And just to give people an idea, when he says smaller, it's like HBM is like 40 gig and then Astram is like 20 megabytes. So it's not like a little smaller, it's like much smaller. But the throughput on card is like 1.5 terabytes a second for HBM and like 19 terabytes a second for Astram, which is a lot larger. how do you think that evolved?

0:13:24

So TSMC said they hit the scaling limits for SRAM they just cannot grow that much more HBM keeps growing HBM3 is going to be 2x faster than HBM2 I think the latest NVIDIA thing as a HBM3 How do you think about the future of flash attention? Do you think HBM is going to get faster enough when maybe it's not as useful to use the SRAM?

0:13:49
Tri Dao

That's right, I think it comes down to physics when you design hardware. Literally SRAM stays very close to compute units, and so you don't have that much area to essentially put the transistors. And you can't shrink these things too much. So just physics, in terms of area, you don't have that much area for the SRAM. The HBM is off-chip, so there is some kind of bus that essentially transfers data from HBM to the compute unit.

0:14:16

So you have more area to essentially put these memory units. And so, yeah, I think in the future, SRAM probably won't get that much larger because you don't have that much area. HRAM will get larger and faster. And so I think it becomes more important to design algorithms that take advantage of this memory asymmetry. It's the same thing in CPU where the cache is really small. The DRAM is growing larger and larger. DRAM could get to, I don't know, 2 terabytes, 6 terabytes or something.

0:14:45

whereas the cache stay at like, I don't know, 15 megabytes or something like that. I think maybe the algorithm design becomes more and more important. There's still ways to take advantage of this. I think so in the future, I think flash attention right now is being used. I don't know if in the next couple of years some new architecture will come in and whatnot, but attention seems to be still important. For the next couple of years, I still expect some of these ideas to be useful, not necessarily the exact code that's out there.

0:15:16

But I think these ideas have kind of stood the test of time. The ideas like IO awareness from back in the 1980s, ideas like kernel fusions, tiling, these are classical ideas that have stood the test of time. And so I think in the future, these ideas will become more and more important as we scale models to be larger, as we have more kinds of devices where performance and efficiency become much, much more important.

0:15:40
Alessio Fanelli

Yeah. And we had a Jonathan Frankel on the podcast. And if you go to isattentionallyouneed.com, he has an outstanding bet and he does believe that attention will be the state of the art architecture still in a few years. Did you think flash attention would be this popular? Like, I'm always curious on the research side, you know, you publish a paper and obviously, you know, it's great work, but sometimes it just kind of falls flat in the industry.

0:16:06

Could you see everybody just like just starting to use this or was that a surprise to you?

0:16:10
Tri Dao

Certainly, I didn't anticipate the level of popularity. Of course, we're extremely happy to have people using this stuff and giving us feedback and so on and help us improve things. I think when we were writing the paper, I remember sending an email to one of my advisors. I'm like, hey, I'm excited about this paper, but I think the most important thing will be the artifact, which is the code. So I knew that the code will be valuable.

0:16:37

so we kind of focus a lot on the code and make sure that the code is usable and as fast as can be of course the idea, the paper presents the ideas and explain it and have experiments that validates the idea but I knew that the artifact or the code was also pretty important and that turned out to be the right focus which is we put out the paper, we release the code and continue working on the code so it's a team effort with my co-authors as well

0:17:06
Alessio Fanelli

So we mentioned Hazy Research a bunch of times on the podcast before. I would love for you to spend five minutes just talking about how does the group work? You know, how do people get together? Like, how do you like bounce ideas off of each other?

0:17:20
Tri Dao

Yeah. Yeah. So Hazy Research is a research group at Stanford led by one of my advisors, Chris Ray. I love the people. That is one of the best experience I had. They've made my PhD so much more enjoyable. And I think there are a couple of ways that the group has been working pretty well. So one is, I think there's a diverse pool of people who either, some of them focus on algorithms and theory. Some of them focus on building systems.

0:17:51

Some of them focus on applications. And as a result, there is this flow of idea. So as an example, some of us were working on more algorithms and theory. and then we can talk to the folks building system and say, hey, let's try it out and let's put it in the systems and see how it is. And there you will get feedback from systems folks. It will say, hey, we implemented this or we tried this and this is where it doesn't work, something like that.

0:18:17

And once we put it in the systems, the application folks can use the algorithm or new methods or new models. And we again get great feedback from them because the application folks, for example, some of my good friends, they focus on medical imaging or seizure detection. And that is the problem they care about, right? And if your method doesn't work on the task they care about, they will tell you. Whereas I think a lot of people in machine learning, they're a little bit more flexible.

0:18:43

So they will be like, hey, it doesn't work on seizure detection. Let's try some other task, right? But having that direct feedback of like, hey, it doesn't work there. Let's figure out why. I think that that feedback allows us to do better work. And I think that kind of process of exchanging ideas, just validating it in a real system so that applications folks can try it out and give you feedback. I think that cycle has been very, very useful.

0:19:07

And so that's one, having a diverse group of people. The other one is, and this is something I really appreciate from advice from Chris, was try to understand the fundamental, right? And he's happy letting me go off and read some textbooks and playing with things because I think a lot of research ideas come from understanding the old literature and see how it fits with the new landscape. And so if you just new archive papers every day, that's great, but you also need to read textbooks.

0:19:39

And that's one advice I got from Chris, which is understand the fundamentals. And I think that allows to do more impactful work.

0:19:46
Alessio Fanelli

How do you think about academia versus industry? I feel like AI machine learning has been an area where up until three, four years ago, most of the cutting edge work was being done in academia and now there's all these big industry research labs um you're obviously going to to princeton so you're a academia believer uh how should people think about where to go say i'm like you know i'm doing my master's uh i have to decide between doing a phd and like going into

0:20:13
Tri Dao

open ai anthropic how should i decide i think they kind of play a complementary role in my opinion And of course, I also was considering different paths as well. So I think right now, scaling matters a lot, especially when you talk about language models and generative AI and so on. Scaling matters a lot. And that means that you need compute resources and you need infrastructure and you need engineers to have. And so industry tends to have an advantage when it comes to scaling things.

0:20:47

But a lot of the ideas actually came from academia. So let's take Attention, which got popular with the Transformer in 2017. Attention actually has been around for a while. So I think the first mention was in 2014, a paper from Banado and others and Yoshua Bengio, which is coming from academia. a lot of ideas did come from academia. And scaling things up, of course, I think OpenAI has been great at scaling things up. That was the bet that they made after, I think, GPT-2.

0:21:23

So they saw that scaling these things up to, back then was 1.5 billion parameter, seemed to give you amazing capabilities. So they really committed to that. They really committed to scaling things. And that turned out to be, has been a pretty successful bet. I think for academia, we're still trying to figure out exactly what we're doing in this shifting landscape. And so lots of folks have been focusing on, for example, evaluation. So I know the Stanford Center for Foundation Model led by Percy.

0:21:55

They have this benchmark called HELM, which is this holistic benchmark. So trying to figure out, okay, characterizing the landscape of different kinds of models, what people should evaluate, what people should measure and things like that. So evaluation is one role. The other one is understanding. So this has happened historically where there's been some development in the industry and academia can play a role in explaining, understanding. They have the luxury to slow down trying to understand stuff.

0:22:24

So lots of paper on understanding what's really going on, probing these models and so on. And I think I'm not as familiar with the NLP literature, but my impression is there's a lot of that going on in the kind of NLP conferences, which is understanding what these models are doing, what capabilities they have, and so on. And the third one I could see is that the academia can take more risky bets in the sense that we can work on stuff that's quite different from industry.

0:22:52

I think industry, my impression is you have some objective. You're trying to, hey, for this quarter, we want to scale the model in this particular way. Next quarter, we want the model to have these capabilities. You're trying to get objectives that maybe, I don't know, 70 % that will work out because it's important for the company's direction. I think for academia, the way things work is you have many, many researchers or PhD students, and they're kind of pursuing independent directions.

0:23:21

and they have a little bit more flexibility on, hey, I'm going to try out this seemingly crazy idea and see, let's say there's a 30 % chance of success or something. And however you define success, for academia, a lot of the time success just means like, hey, we found something interesting. That could eventually go into industry through collaboration and so on. So I do see academia and industry kind of playing complementary roles. And as for someone choosing a career, I think just more and more generally, industry would be probably better in terms of compensation, in terms of probably work-life balance.

0:24:00

But my bias perspective is that maybe academia gives you a little bit more freedom to think and understand things. So, you know, it probably comes down to personal choice. I might end up choosing to be a professor next year at Princeton. But of course, I want to maintain a relationship with industry folks. I think industry folks can provide very valuable feedback to what we're doing in academia so that we understand where the field is moving. Because some of the directions are very much influenced by what, for example, OpenAI or Google is doing.

0:24:35

So we want to understand where the field is moving. What are some promising applications? And try to anticipate, okay, if the field is moving like this, these applications are going to be popular. what problems will be important in two, three years. And then we try to start thinking about those problems so that hopefully in two, three years, we have some of the answers to some of these problems in two, three years. Sometimes it works out, sometimes it doesn't, but as long as we do interesting things in academia, that's the goal.

0:25:03

Yeah.

0:25:04
Alessio Fanelli

And you mentioned the eval side. So we did a Benchmarks 101 episode. And one of the things we were seeing, it's like sometimes the benchmarks really influence the model development, you know, because obviously if you don't score well in the benchmarks, you're not going to get published and you're not going to get funded. How do you think about that? Like, how do you think that's going to change now that a lot of the applications of these models, again, is in more like narrow industry use cases?

0:25:32

Like, do you think the goal of like the academia evals is to like to be very broad and then industry can do their own evals or what's the relationship there?

0:25:40
Tri Dao

Yeah, so I think evaluation is important and often a little bit underrated. So it's not as flashy as, oh, we have a new model that can do such and such. But I think evaluation, what you don't measure, you can't make progress on essentially. So I think industry folks, of course, they have specific use cases that their model needs to do well on. And that's what they care about, not just academia, but other groups as well. People do understand what are some of the emerging use cases.

0:26:11

So, for example, now one of the most popular use cases is chatbot. And then I think folks from Berkeley, some of them are from Berkeley, MLSYS. They set up this kind of chatbot arena to essentially benchmark different models. So people do understand what are some of the emerging use cases. People do contribute to evaluation and measurement. And as a whole, I think people try to contribute to the field and move the field forward, albeit that maybe slightly different directions.

0:26:39

But we're making progress. And definitely evaluation and measurement is one of the ways you make progress. So I think going forward, there's still going to be just more models, more evaluation. will just have better understanding of what these models are doing and what capabilities they have.

0:26:56
Alessio Fanelli

I like that your work has been focused on not making benchmarks better, but it's like, let's just make everything faster. Like, let's, so it's very horizontal. So Flash Attention 2, you just released that on Monday. I read in the blog post that a lot of the work was like also related to like some of the NVIDIA library updates. Yeah, maybe run a bit, run us through some of those changes and some of the innovations there.

0:27:18
Tri Dao

Yeah, yeah, for sure. So Flash Attention 2 is something I've been working on for the past couple of months. So the story is the NVIDIA Cutlass team, they release a new version of their library, which contains all these primitives to allow you to do matrix multiply or memory loading on GPU efficiently. So it's a great library, and I built on that. So they released their version 3 back in January, and I got really excited, and I wanted to play with that library.

0:27:47

So as an excuse, I was just like, okay, I'm going to refactor my code, and use this library. So that was kind of the start of the project. By the end, I just ended up working with the code a whole lot more, and I realized that, hey, there are these inefficiencies still in Flash attention. We could change this way or that way and make it, in the end, twice as fast. But of course, building on the library that the NVIDIA folks released.

0:28:13

So that was kind of a really fun exercise. I was saying, I started out as just an excuse for myself to play with the new library. What ended up was several months of improving flash attention, discovering new ideas. And in the end, we managed to make it 2x faster. And now it's pretty close to probably the efficiency of things like Matrix Multiply, which probably is the most optimized subroutine on the planet. So we're really happy about it.

0:28:40

The NVIDIA Cutlass team has been very supportive. And hopefully in the future, we're going to collaborate more.

0:28:46
Alessio Fanelli

Yeah. And since it's an NVIDIA library, can you only run this on like CUDA runtime? So like, could you use this and then run it like an AMD GPU?

0:28:56
Tri Dao

Yeah. Yeah. So it's an NVIDIA library. So right now, kind of the code we release runs on NVIDIA GPUs, which is what most people are using to train models. Of course, they're emerging other hardware as well. So the AMD folks did implement a version of Flash Attention, I think last year as well, and that's also available. I think there's some implementation on CPU as well. For example, there's this library, GGML, where they implemented the same idea running on Mac and CPU.

0:29:24

So I think kind of broadly the idea would apply. The current implementation ended up using NVIDIA's library or primitives, but I expect these ideas to be broadly applicable to different hardware. I think the main idea is you have asymmetry in memory hierarchy, which tend to be everywhere in a lot of accelerators. Yeah, it kind of reminds me of Sarah Hooker's post,

0:29:49
Alessio Fanelli

like the hardware lottery. Yes. There could be all these things that are much better, like architecture that are better, but they're not better on NVIDIA. So we're never going to know if they're actually improved. How does that play into some of the research that you all do too?

0:30:04
Tri Dao

Yeah, so absolutely. Yeah, I think Sarah Hooker, she wrote this piece on hardware lottery And I think she captured really well of what a lot of people have been thinking about this. And I certainly think about hardware lottery quite a bit, given that I do some of the work that's kind of really low level at the level of, hey, we're optimizing for GPUs or NVIDIA GPUs and optimizing for attention itself. And at the same time, I also work on algorithms and methods and transformer alternatives.

0:30:33

And we do see this effect in play, not just hardware lottery, but also kind of software framework lottery. You know, attention has been popular for six years now, and so many kind of engineer hours has been spent on making it as easy and efficient as possible to run Transformer, right? And there's libraries to do all kind of tensor parallel, pipeline parallel if you use Transformer. Let's say someone else develop alternatives, or let's just take recurrent neural nets like LSDM, GRU.

0:31:05

If we want to do that and run that efficiently on current hardware with current software framework, that's quite a bit harder. So in some sense, there is this feedback loop where somehow the model architectures that take advantage of hardware become popular. And the hardware will also kind of evolve to optimize a little bit for that kind of architecture. and software framework will also evolve to optimize for that particular architecture. Right now, Transformer is the dominant architecture.

0:31:40

So yeah, I'm not sure if there is a good way out of this. Of course, there's a lot of development, things like I think compilers will play a role because compilers allow you to maybe still be much more efficient across different kinds of hardware because essentially you write the same code and compiler will be able to make it run efficiently. different kinds of hardware. So for example, there's this Language Mojo. They're compiler experts, right? And their bet is, AI models will be running on different kinds of devices.

0:32:11

So let's make sure that we have really good compilers with a good language that then the compiler can do a good job optimizing for all kinds of devices. So that's maybe one way that you can get out of this cycle. But yeah, I'm not sure of a good way. In my own research, I have to think about both the new algorithms new model and how it maps to hardware. So they're crazy ideas that seem really good, but will be really, really difficult to run efficiently.

0:32:40

And so as a result, for example, we can't really scale some of the architectures up simply because they're not hardware friendly. I have to think about both sides when I'm working on new models. Yeah.

0:32:51
Alessio Fanelli

Have you spent any time looking at some of the new kind of like AI chips companies, so to speak, like the Cerebras of the world? like one of their innovations, like, you know, colocating everything on the chip. So you kind of remove some of this, like a memory bandwidth issue. How do you think about that?

0:33:07
Tri Dao

Yeah, I think that's an interesting bet. I think Tesla also has this Dojo supercomputer where they try to have as, essentially as fast on-chip memory as possible and removing some of these data transfer back and forth. I think that's a promising direction. The issues I could see, you know, I'm definitely not a hardware expert. One issue is the on-chip memory tend to be really expensive to manufacture, much more expensive per gigabyte compared to off-chip memory.

0:33:34

So I talked to some of my friends at Cerebras, and they have their own stack and compiler and so on, and they can make it work. The other kind of obstacle is, again, with compiler and software framework and so on. For example, if you can run PyTorch on this stuff, lots of people will be using it, but supporting all the operations in PyTorch will take a long time to implement. Of course, people are working on this.

0:34:02

So I think, yeah, we kind of need these different bets on the hardware side as well. Hardware, my understanding, has kind of a longer time scale. So you need to design hardware, you need to manufacture it maybe on the order of three to five years or something like that. So people are taking different bets, but the AI landscape is changing so fast that it's hard to predict, okay, what kind of models will be dominant in, let's say, three or five years.

0:34:29

Or thinking back five years ago, would we have known that Transformer would have been the dominant architecture? Maybe, maybe not, right? And so different people will make different bets on the hardware side.

0:34:39
Alessio Fanelli

Yeah, that's the pace of the industry and the research also influence the PhD research itself. So like, for example, in your case, you're working on improving attention. It probably took you quite a while to write the paper and everything, But like in the meantime, you could have had a new model architecture come out and then it's like nobody cares about attention anymore. How do people balance that? Yeah, so I think it's tough.

0:35:04
Tri Dao

It's definitely tough for PhD students, for researchers, given the field is moving really, really fast. I think it comes down to understanding fundamentals. Because that's essentially, for example, what the PhD allows you to do. It's been a couple of years understanding the fundamentals. So, for example, when I started my PhD, I was working on understanding matrix vector multiply, which is a very, it's been a concept that's been around for hundreds of hundreds of years. We were trying to characterize what kind of matrices would have theoretically fast multiplication algorithm.

0:35:37

That seems to have nothing to do with, you know, AI or anything. But I think that was a time when I developed kind of mathematical maturity and research taste and research skill. The research topic at that point didn't have to be super trendy or anything. As long as I'm developing skills as a researcher, I'm making progress. And eventually, I've gotten quite a bit better in terms of research skills. And that allows, for example, PhD students later in their career to quickly develop solutions to whatever problems they're facing.

0:36:10

So I think that's just the natural arc of how you're being trained as a researcher. For a lot of PhD students, I think given the pace is so fast, maybe it's harder to justify spending a lot of time on the fundamental. And, you know, it's tough. What is kind of explore, exploit kind of dilemma? And I don't think there's a universal answer. So I personally spend some time doing this kind of exploration, reading random textbooks or lecture notes.

0:36:39

and I spend some time just keeping up with the latest architecture or methods and so on. I don't know if there's a right balance. It depends on, it varies from person to person. But if you only spend 100 % on one, either you only do exploration or only do exploitation, I think it probably won't work in the long term. It's probably going to have to be a mix and you have to just experiment and kind of be introspective and say, hey, I tried this kind of mixture of, I don't know, one exploration paper and one exploitation paper.

0:37:12

How did that work out for me? Should I, you know, having conversation with, for example, my advisor about like, hey, did that work out? You know, should I shift, I focus more on one or the other? I think quickly adjusting and focusing on the process, I think that's probably the right way. I don't have like a specific recommendation that, hey, you focus, I don't know, 60 % on lecture notes and 40 % on archive papers or anything like that.

0:37:37
Alessio Fanelli

Let's talk about some transformer alternatives. Say Jonathan Frankel loses his bet and Transformer is not the state-of-the-art architecture. What are some of the candidates to take over?

0:37:49
Tri Dao

Yeah, so this bet is quite fun. So my understanding is this bet between Jonathan Frankel and Sasha Rush, right? I've talked to Sasha a bunch and I think he recently gave an excellent tutorial on transformer alternatives as well. So I would recommend that. So just to quickly recap, I think there's been quite a bit of development more recently about transformer alternatives. So architectures that are not transformer, right? And the question is, can they do well on, for example, language modeling, which is kind of the application that a lot of people care about these days.

0:38:25

So there are methods based on state-space methods that came out in 2021 from Albert Gu and Curran and Chris Ray that presumably could do much better in terms of capturing long-range information while not scaling quadratically. They scale sub-quadratically in terms of sequence length. So potentially you could have a much more efficient architecture when sequence length gets really long. The other one has been focusing more on recurrent neural nets, which is, again, an old idea, but adapting to the new landscape.

0:39:03

So things like RWKV, I've also personally worked in this space as well. So there's been some promising results. So there's been some results here and there that show that, hey, these alternatives, either RNN or state-based methods, can match the performance of Transformer on language modeling. So that's really exciting. And we're starting to understand, on the academic research side, we want to understand, do we really need attention? I think that's a valuable intellectual thing to understand.

0:39:35

And maybe we do, maybe we don't. If we want to know, we need to spend serious effort on trying the alternatives. And there's been folks pushing on this direction. I think RWKV scale up to, they have a model at 14 billion that seems pretty competitive with Transformers. So that's really exciting. You know, that's kind of an intellectual thing. We want to figure out if attention is necessary. So that's one motivation. The other motivation is Transformer alternative could have an advantage in practice in some of the use cases.

0:40:04

So one use case is really long sequences. The other is really high throughput of generation. So for really long sequences, when you train with transformer, with flash attention and so on, the computation is still quadratic in the sequence length. So if your sequence length is on the order of, I don't know, 16K, 32K, 100K or something, which some of these models have sequence length 100K, then you do get significantly slower in terms of training, also in terms of inference.

0:40:34

So maybe these alternative architectures could scale better in terms of sequence length. I haven't seen actual validation on this. It's in like, let's say, an RN model release with context length, I don't know, 100k or something. I haven't really seen that. But the hope could be that as we scale to long sequences, these alternative architecture could be more well suited. Not just text, but things like high resolution images, audio, video, and so on, which are emerging applications.

0:41:02

So that's one, long sequences. Number two is a high throughput generation, where I can imagine scenarios where the application isn't like an interactive chatbot, but let's say a company wants to batch as many requests as possible on their server, or like they're doing offline processing, they're generating stuff based on their internal documents that you need to process in batch, right? And the issue with transformers is that during generation, it essentially needs to keep around all the previous history.

0:41:33

It's called the KV cache. And that could take a significant amount of memory. So you can't really batch too much because you run out of memory. I am personally bullish on RNNs. I think RNNs, they essentially summarize the past into a state vector. They have fixed size. So the size doesn't grow with the history. So that means that you don't need as much memory to keep around all the previous tokens. And as a result, I think you can scale to much higher batch sizes.

0:42:02

And as a result, you can make much more efficient use of the GPUs or the accelerator, and you could have much higher generation throughput. Now, this has, I don't think, has been validated at scale. So as a researcher, I'm bullish on this stuff because I think in the next couple of years, These are use cases where these alternatives could have an advantage. We'll just kind of have to wait and see to see if these things will happen.

0:42:30

I am personally bullish on this stuff. At the same time, I also spend a bunch of time making attention as fast as possible. So maybe hatching and playing both sides. Ultimately, we want to understand as researchers, we want to understand what works. Why do the models have these capabilities? And one way is let's push attention to be as efficient as possible. On the other hand, let's push other alternatives to be as efficient as we can scale as big as possible and so that we can kind of compare them and understand.

0:43:00
Alessio Fanelli

Yeah, awesome. And I think as long as all of this work happens in the open, it's a net positive for everybody to explore all the paths. Yeah, let's talk about open source AI. Obviously, together, when RedPajama came out, which was an open clone of the Llama 1 pre-training data set, it was a big thing in the industry. Llama 2 came out on Tuesday. I forget. And this week, there's been a lot of things going on, which they call open source, but it's not really open source.

0:43:32

I actually wrote a post about it that was on the front page of Accurate News before this podcast. I was frantically responding. How do you think about what open source AI really is? In my mind, in open source software, we have different levels of open. So there's free software that's like the GPL license. There's open source, which is Apache, MIT. And then there's restricted open source, which is the SSPL and some of these other licenses. In AI, you have the open models.

0:44:02

So RedPajama is an open model because you have the pre-training data set, you have the training runs and everything. and then there's obviously random lengths that doesn't make it one-to-one if you retrain it. Then you have the open weights model that's kind of like stable LM where the weights are open but the data side is not open. And then you have LLaMA 2 which is the data side is not open the weights are restricted.

0:44:23

It's kind of like not really open source but open enough. I think it's net positive because it's like $3 million of flops donated to the public. How do you think about that and also as you work with together what is your philosophy with open source AI?

0:44:40
Tri Dao

I think that's a great question and I think about it on maybe more practical terms so of course Meta has done an amazing job training LLaMA 1, LLaMA 2 and for LLaMA 2 they kind of make it much less restrictive compared to LLaMA 1s where now you can use it for businesses unless you are a monthly active user or something like that. I think just this change will have a very significant impact in the kind of landscape of open source AI, where now lots of businesses, lots of companies will be using, I expect will be using things like LLaMA 2.

0:45:19

They will fine tune on their own data set. They will be serving variants or derivatives of LLaMA 2. Whereas before, with LLaMA 1, it's also a really good model, but your business companies weren't allowed to do that. So I think on a more practical term, it's kind of shifting the balance between a closed-source model like OpenAI and Anthropic and Google, where you're making API calls. And maybe you don't understand as much of what the model is doing, how the model is changing, and so on.

0:45:51

Versus now, we have a model with open weight that is pretty competitive from what I've seen in terms of benchmarks, pretty competitive with GPT 3.5. And if you fine-tune it on your own data, maybe it's more well-suited for your own data. And I do see that's going to shift the balance. More and more folks are going to be using, let's say, derivatives of LLaMA 2. More and more folks are going to fine-tune and serve their own model instead of calling an API.

0:46:16

So that shifting of balance is important because in one way, we don't want just a concentration of decision-making power in the hands of a few companies. So I think that's a really positive development development from Meta. Of course, training the model takes a couple millions of dollars, but engineers have, and I'm sure they spend tons of time trying many, many different things. So the actual cost is probably way more than that, and they make the weights available, and they allow, probably a lot of companies are going to be using this.

0:46:48

So I think that's a really positive development. And we've also seen amazing progress on the open source community where they would take these models, and they either fine-tune on different kinds of data sets or even make changes to the model. So as an example, I think for LLaMA 1, the context lane was limited to 2K. A bunch of folks figured out some really simple methods to scale up to 8K. Yeah, like the rope. Yes. I think the open source community is very creative, right?

0:47:16

And lots of people. LLaMA 2 will, again, kind of accelerate this where more people will try it out, more people will make tweaks to it and make a contribution and then so on. So overall I see that as still a very positive development for the field. And there's been lots of libraries that will allow you to host or fine-tune these models, even with quantization and so on. Just a couple hours after LLaMA 2 was released, tons of companies announcing that, hey, it's on our API or hosting, and then someone together did the same.

0:47:47

So it's a very fast-paced development and just kind of a model with available weights that businesses are allowed to use. I think that alone is already very positive development. At the same time, yeah, we can do much better in terms of releasing data set. I think data set tend to be, somehow people are not incentivized to release data set. Philosophically, yeah, you want to be as open as possible. But on practical term, I think it's a little bit harder for companies to release data set, legal issues.

0:48:18

The data set release tends to be not as eye-catchy as the model release. So maybe people are less incentivized to do that. We've seen quite a few companies releasing this together, released RedPajama Dataset. I think Cerebus then worked on that and deduplicate and clean it up and release Slim Pajama and so on. So we're also seeing positive development on that front, kind of on the pre-training dataset. So I do expect that to continue.

0:48:46

And then on the fine-tuning dataset or instruction-tuning dataset, I think we now have quite a few open datasets on instruction tuning and fine tuning. But these companies still, they do pay for human labelers who annotate these instruction tuning data set. And that is expensive. And maybe they will see that as their competitive advantage. And so it's harder to incentivize these companies to release these data set. So I think on practical terms, we're still going to make a lot of progress on open source AI, on both the model development, on both model hosting, on Tri training data set and fine tuning data set.

0:49:21

But right now, maybe we don't have the perfect open source model since all the data sets are available. Maybe we don't have such a thing yet, but we've seen very fast development on the open source side. I think just maybe this time last year, there weren't as many models that are competitive with, let's say, ChatGPT.

0:49:44
Alessio Fanelli

Yeah, I think the open data sets, they have so much more impact, the open models. If you think about Eluser and the work that they've done, GPT-J was great, and the PTA models are great, but the pile and the stack, everybody uses them. So hopefully we get more people to contribute time to work on datasets instead of doing the 100th open model that performs worse than the other one, but they want to say they released the model.

0:50:14
Tri Dao

Yeah. Maybe the question is, how do we figure out an incentive structure so that companies are willing to release data sets. And for example, it could be like, I think some of the organizations are now doing this where they are kind of asking volunteers to annotate and so on. And maybe the Wikipedia model of data set, especially for instruction tuning, could be interesting where people actually volunteer their time and instead of editing Wikipedia, add annotation. And somehow they acknowledge and feel incentivized to do so.

0:50:48

So hopefully we get to that kind of level of, in terms of data, it would be kind of like Wikipedia. And in terms of model development, it's kind of like Linux, where people are contributing patches and improving the model in some way. I don't know exactly how that's going to happen, but based on history, I think there is a way to get there.

0:51:05
Alessio Fanelli

Yeah, I think the Dolly 15K data set is like a good example of a company saying, hey, let's do this smaller thing. Just make sure we make it open. It came out very, we have Mike Conover from Databricks on the podcast, and he was like, people just bought into it and leadership was bought into it. You have companies out there with 200 ,000, 300 ,000 employees. It's like, just put some of them to label some data.

0:51:29

It's going to be helpful. So I'm curious to see how that evolves. What made you decide to join Together?

0:51:35
Tri Dao

For Together, the focus has been focusing a lot on open source model and I think that aligns quite well with what I care about, of course. I also know a bunch of people there that I know and trust and I'm excited to work with them. Philosophically, the way they've been really open with like data set and model release, I like that a lot. Personally, for the stuff, for example, the research that I've developed, we also try to make code available, free to use and modify and so on, contributing to the community.

0:52:06

That has given us really valuable feedback from the community in improving our work. So philosophically, I like the way Together has been focusing on open source model, And the nice thing is we're also going to be at the forefront of research and the kind of research areas that I'm really excited about. Things like efficient training and inference aligns quite well with what the company is doing. We'll try our best to make things open and available to everyone.

0:52:34

Yeah, but it's going to be fun being at the company, leading a team doing research on the topic that I really care about. And hopefully we'll make things open to benefit the community. Awesome.

0:52:45
Alessio Fanelli

Let's jump into the lightning round. We usually have two questions. So one is on acceleration, one on exploration, and then a takeaway. So the first one is, what's something that already happened in AI machine learning that you thought would take much longer than it has?

0:53:02
Tri Dao

I think understanding jokes. I didn't expect that to happen. But it turns out scaling model up and training lots of data, the model is going to understand jokes. Maybe it's a small thing, but that was amazing to me.

0:53:16
Alessio Fanelli

What about the exploration side? What are some of the most interesting unsolved questions in the space?

0:53:22
Tri Dao

I would say reasoning in the broad term. We don't really know how these models are doing. Essentially, they do something that looks like reasoning. We don't know how they're doing it. We have some ideas. And in the future, I think we will need to design architecture that kind of explicitly have some kind of reasoning module in it. if we want to have much more capable models. What's one message you want everyone to remember today? I would say try to understand both the algorithm and the systems that these algorithms run on.

0:53:54

I think at the intersection of machine learning system has been really exciting and there's been a lot of amazing results at this intersection. And then when you scale models to large scale, both the machine learning side and the system side really matter. Awesome. Well, thank you so much for coming on, Tri. This was great. Yeah, this has been really fun. Thank you.