Person
Tri Dao
Tri Dao is an AI researcher of Together AI who appeared on the Latent Space podcast to discuss FlashAttention 2, a faster way to run Transformers, the models behind modern chatbots. The episode also covered how memory limits speed, alternatives to Transformers, and open models.
Tri Dao said FlashAttention gave roughly two to four times faster real-world speeds without approximating attention. He said most attention time goes to reading and writing memory. He also said FlashAttention 2 is about twice as fast as the original and close to the efficiency of matrix multiplication. He said he started college planning to study economics before switching to math.
1 episodes · 21 quoted claims
Tri Dao started college intending to study economics but switched to a math major during his first week as a Stanford undergraduate.
“the first week I was at Stanford undergrad. I took a few math classes and I immediately decided that I was going to be a math major”
Latent Space: The AI Engineer Podcast · 0:01:09
FlashAttention uses memory that grows linearly with sequence length, while its computation remains quadratic.
“the memory is linear in sequence length. In terms of computation, it's still quadratic, but we managed to make it much more hardware friendly.”
Latent Space: The AI Engineer Podcast · 0:02:50
FlashAttention gave wall-clock speedups of roughly two to four times without approximating attention, according to Tri Dao.
“we do get walk-lock speed up on the order of two to four X”
Latent Space: The AI Engineer Podcast · 0:02:50
Tri Dao says approximate attention methods perform fewer operations but tend not to run faster in wall-clock time.
“these approximation methods, even though they perform fewer computation, they tend to not be faster in walk-on-tap.”
Latent Space: The AI Engineer Podcast · 0:04:11
Tri Dao says most of the time in attention goes to reading and writing memory, which is why FlashAttention targeted memory traffic.
“we ended up focusing a lot more on memory reading and writing because that turned out to be the majority of time when you're doing attention is reading and writing memory.”
Latent Space: The AI Engineer Podcast · 0:05:16
The online softmax trick, which lets softmax be computed in pieces, appeared in a 2018 paper by NVIDIA researchers, according to Tri Dao.
“So this online softmax trick has been around for a while. I think it was a paper from NVIDIA folks back in 2018 about this.”
Latent Space: The AI Engineer Podcast · 0:07:41
A late-2021 paper from Google researchers used the online softmax trick to break attention into smaller pieces, as Tri Dao tells it.
“there was a paper from Google. So Marcus Stroup and Statz wrote a paper late 2021 on using this online softmax trick to break attention up into smaller pieces.”
Latent Space: The AI Engineer Podcast · 0:08:13
Tri Dao says memory-hierarchy-aware algorithm design dates to 1980s sorting problems, such as banks sorting transactions overnight.
“the motivating example was banks were trying to sort their transactions, and that needs to happen overnight so that the next day they can be ready.”
Latent Space: The AI Engineer Podcast · 0:12:25
Tri Dao says the first mention of attention in machine learning was in a 2014 paper involving Yoshua Bengio, which came from academia.
“I think the first mention was in 2014, a paper from Banado and others and Yoshua Bengio, which is coming from academia.”
Latent Space: The AI Engineer Podcast · 0:20:47
Tri Dao says OpenAI committed to scaling after GPT-2, which had 1.5 billion parameters and showed strong capabilities.
“That was the bet that they made after, I think, GPT-2. So they saw that scaling these things up to, back then was 1.5 billion parameter, seemed to give you amazing capabilities.”
Latent Space: The AI Engineer Podcast · 0:20:47
The Stanford Center for Research on Foundation Models, led by Percy Liang, created the HELM benchmark, which Tri Dao describes as holistic.
“They have this benchmark called HELM, which is this holistic benchmark.”
Latent Space: The AI Engineer Podcast · 0:21:55
Researchers at Berkeley set up Chatbot Arena to benchmark chatbot models, according to Tri Dao.
“They set up this kind of chatbot arena to essentially benchmark different models.”
Latent Space: The AI Engineer Podcast · 0:26:11
FlashAttention 2 is about twice as fast as the original and, per Tri Dao, close to the efficiency of matrix multiplication.
“we managed to make it 2x faster. And now it's pretty close to probably the efficiency of things like Matrix Multiply”
Latent Space: The AI Engineer Podcast · 0:28:13
AMD developers implemented a version of FlashAttention for their GPUs, according to Tri Dao.
“the AMD folks did implement a version of Flash Attention, I think last year as well, and that's also available.”
Latent Space: The AI Engineer Podcast · 0:28:56
Tri Dao cites Tesla's Dojo supercomputer as a hardware bet that emphasizes fast on-chip memory.
“I think Tesla also has this Dojo supercomputer where they try to have as, essentially as fast on-chip memory as possible”
Latent Space: The AI Engineer Podcast · 0:33:07
State-space sequence models from Albert Gu, Chris Ré and colleagues appeared in 2021 as a Transformer alternative that scales sub-quadratically with sequence length.
“state-space methods that came out in 2021 from Albert Gu and Curran and Chris Ray”
Latent Space: The AI Engineer Podcast · 0:38:25
Tri Dao says RWKV, a recurrent architecture, has a 14-billion-parameter model that appears competitive with Transformers.
“I think RWKV scale up to, they have a model at 14 billion that seems pretty competitive with Transformers.”
Latent Space: The AI Engineer Podcast · 0:39:35
Transformers keep earlier tokens in a KV cache during generation, which uses large amounts of memory and limits batch size, according to Tri Dao.
“It's called the KV cache. And that could take a significant amount of memory. So you can't really batch too much because you run out of memory.”
Latent Space: The AI Engineer Podcast · 0:41:33
Tri Dao says Meta made Llama 2 much less restrictive than Llama 1 so that many businesses can use it.
“for LLaMA 2 they kind of make it much less restrictive compared to LLaMA 1s where now you can use it for businesses unless you are a monthly active user or something like that.”
Latent Space: The AI Engineer Podcast · 0:44:40
Llama 1's context length was limited to 2K tokens, and open-source developers found simple methods to extend it to 8K, according to Tri Dao.
“for LLaMA 1, the context lane was limited to 2K. A bunch of folks figured out some really simple methods to scale up to 8K.”
Latent Space: The AI Engineer Podcast · 0:46:48
Cerebras deduplicated and cleaned the RedPajama dataset and released it as SlimPajama, per Tri Dao.
“I think Cerebus then worked on that and deduplicate and clean it up and release Slim Pajama and so on.”
Latent Space: The AI Engineer Podcast · 0:48:18