George Hotz says tinygrad can run all ML models with about 25 basic operations, compared with about 250 in XLA or PrimTorch.
“And yeah, it can run all ML models with basically 25 instead of the 250 of XLA or PrimTorch.”
Person
George Hotz is a speaker on the Latent Space: The AI Engineer Podcast, where he discussed running AI models on cheaper hardware through his project tinygrad, along with GPUs, compute clusters and robots. He said he got into AI through the Hutter Prize in 2014.
Hotz said tinygrad runs all machine learning models with about 25 basic operations, and that on NVIDIA GPUs it is about 5x slower than PyTorch, down from about 100x. He also said comma's new compute cluster delivers about 30 petaflops and cost about half a million dollars.
1 episodes · 19 quoted claims
George Hotz says tinygrad can run all ML models with about 25 basic operations, compared with about 250 in XLA or PrimTorch.
“And yeah, it can run all ML models with basically 25 instead of the 250 of XLA or PrimTorch.”
George Hotz says Midjourney is trained on Google TPUs and that many startups train on TPUs.
“So MidJourney is trained on TPU, right? Like a lot of startups do actually train on TPUs.”
George Hotz describes a systolic array as a grid of multiply-accumulate units that repeatedly multiply and shift data, a power-efficient way to do matrix multiplication.
“Think of a grid of mullax, and then the grid can multiply and then shift, multiply, then shift, multiply, then shift.”
George Hotz says the XLA-to-TPU compiler is a closed-source 32 megabyte binary called libTPU on Google's cloud instances.
“a 32 megabyte binary blob called libTPU on Google's cloud instances. It's all closed source.”
Tinygrad had a hard limit of a thousand lines of code for a long time, which Hotz later removed because it became code-golfy.
“For a long time, TinyGrad had a hard limit at a thousand lines of code.”
George Hotz says tinygrad passes more ONNX compliance tests than Apple's Core ML but fewer than ONNX Runtime.
“We have better support for Onyx than Core ML does.”
George Hotz says tinygrad on NVIDIA GPUs is about 5x slower than PyTorch, down from about 100x slower earlier.
“But on NVIDIA, it's about 5x slower than PyTorch right now.”
George Hotz says tinygrad runs the model in comma's OpenPilot on Qualcomm GPUs, has been in production for about six months, and is about 2x faster than Qualcomm's own library.
“And TinyGrad is about 2x faster on the GPU than Qualcomm's library.”
George Hotz says Intel GPUs have a stable kernel driver and documented hardware registers, which made them attractive after his AMD problems.
“Intel GPUs have a stable kernel driver and they have all their hardware documented.”
After George Hotz emailed AMD CEO Lisa Su, she responded and AMD sent him a pre-release version of ROCm 5.6 that fixed his kernel panic.
“But no, Lisa Sue reached out, connected with a whole bunch of different people. They sent me a pre-release version of Rock M 5.6.”
George Hotz says NVLink-connected NVIDIA A100 GPUs reach about 600 gigabytes per second of interconnect bandwidth, versus about 60 for PCIe 4.
“NV-linked A100s are going to have, and then you can even get like full fabric and NVIDIA really pushes on that stuff, 600 gigabytes per second.”
George Hotz says the practical limit for training models on a tiny box is about 7 billion parameters.
“If you're training on a tiny box, your limit's going to be about 7 billion.”
George Hotz compares a tiny box's 60 gigabytes per second of interconnect bandwidth with the internet's 125 megabytes per second to argue federated training is impractical.
“The tiny box has 60 gigabytes per second. And then your internet has 125 megabytes per second.”
George Hotz says comma's new compute cluster delivers about 30 petaflops, and that comma paid about double the theoretical cost per flop.
“A person is 20 petaflops. It's about 30 petaflops. We built out a little compute cluster. And we paid double what you theoretically could per flop.”
George Hotz says comma spent about half a million dollars on its roughly one-and-a-half-person compute cluster.
“Well, it's a lot cheaper now. So like I said, Comma spent about half a million on our person and a half.”
George Hotz asserts that GPT-4 is an eight-way mixture of experts with about 220 billion parameters each.
“Well, okay, so GPT-4 is 220 billion in each head, and then it's an eight-way mixture model.”
George Hotz says he got into AI through the Hutter Prize in 2014, while trying to build compressive models of Wikipedia.
“I got into AI through the Hutter Prize. 2014, I'm trying to build compressive models of Wikipedia.”
George Hotz changed tinygrad's unary operations from natural log and exp to log2 and exp2 because hardware has accelerators for log2 and exp2.
“I changed them to be log2 and exp2 because hardware has log2 and exp2 accelerators.”
George Hotz says comma's body robot uses two actuators, whereas Tesla's humanoid robot uses 20.
“Except, of course, we made the comma version of it. Tesla uses 20 actuators. We use two.”