# After LLMs: Spatial Intelligence and World Models — Fei-Fei Li & Justin Johnson, World Labs

- **Show:** Latent Space: The AI Engineer Podcast
- **Published:** 25 Nov 2025
- **Source:** https://lspace.beehiiv.com/podcast/s/latent_space_the_ai_engineer_podcast/after_llms_spatial_intelligence_and_world_models_fei-fei_li_justin_johnson_world_labs
- **Transcript:** https://podcasts.opennash.com/shows/latent-space/after-llms-spatial-intelligence-and-world-models-fei-fei-li-justin-johnson-world-117104/
- **Provider:** OpenNash Podcasts (machine-generated)

## In brief

Fei-Fei Li and Justin Johnson argue that AI has to move beyond language into spatial intelligence: understanding, generating and interacting with 3D worlds. They say language is a lossy, heavily abstracted channel compared with the embodied experience of moving through space. Their company, World Labs, launched Marble, a generative model of editable 3D worlds, as a first step toward that goal.

The central tension is between a grand research vision and a product that is useful now. Marble produces Gaussian splats (tiny semi-transparent 3D particles) that run on phones and VR headsets, and it is aimed at film, gaming and design. The hosts keep asking whether such models understand physics or only fit patterns, and the founders admit they do not yet understand it in a human sense.

## Key insights

1. **Scaled compute makes world models newly feasible** Johnson says per-card performance is about a thousand times higher than in the AlexNet era, and total compute available to one model is roughly a million times larger. Visual and spatial data need far more processing than text, so this new compute has something to be spent on.
2. **Marble outputs Gaussian splats, not video frames** Marble builds a scene from many tiny semi-transparent particles that render in real time on phones and headsets. That is why users get exact camera placement instead of prompting a video model with vague directions.
3. **Pattern-fitting models may not learn physical laws** Li says there is no sign that latent modeling yields causal laws of space and dynamics, and she doubts that feeding planetary data to a model would produce F equals MA. Johnson counters that such laws might emerge from a different learning loop of constant prediction and correction.
4. **Academia should chase wacky ideas, not giant models** Johnson says state-of-the-art training now needs far more than a couple of GPUs, so labs should explore new algorithms, architectures and hardware primitives instead. Li adds that the real problem is under-resourced academia, not open versus closed work.
5. **Transformers handle sets, so they can fit 3D** Johnson explains that a transformer treats its inputs as an unordered set of tokens and only gets sequence order from positional embeddings. That means the architecture does not need replacing for 3D and spatial data.

## Questions this episode answers

### What is Marble, and what can I do with it today?

Marble is a World Labs model that generates 3D worlds from text or images, lets you edit them (for example, change a bottle's colour or remove a table), and exports them in several formats. The founders say early uses are in gaming, VFX, film and interior design.

### What is spatial intelligence, and how is it different from what chatbots do?

Li defines it as the ability to reason, understand, move and interact in space, such as picking up a mug or working out the shape of DNA. She sees it as complementary to language intelligence, which Johnson says jumps straight to abstract reasoning and loses the embodied experience.

### What are Gaussian splats and why does Marble use them?

Each splat is a tiny semi-transparent particle with a position and orientation in 3D space, and a scene is built from very many of them. Johnson says they render efficiently in real time on devices like iPhones, which enables precise camera control.

### Does Marble actually understand physics?

Li says no, not in the human sense, because the model learns patterns from data. Johnson says that is fine for a film backdrop but would matter for an architect designing a real building.

### Why might world models matter for robotics?

Li says robot training lacks high-quality real-world data, and internet video lacks controllability. Marble could generate synthetic simulated worlds where embodied agents can be trained, which she calls an important middle ground.

## Facts

- **Justin Johnson** [0:01:57]: "the quarter that I joined your lab was the same quarter that AlexNet came out."
- **Justin Johnson** [0:04:02]: "But even from AlexNet to today, we're getting about a thousand times more performance per card than we had in AlexNet days."
- **Justin Johnson** [0:04:29]: "So the amount of compute that we can marshal today on a single model is, you know, about a million fold more than we could have even at the start of my PhD."
- **Fei-Fei Li** [0:05:00]: "my Stanford lab just announced an open dataset and benchmark called Behavior, which is for benchmarking robotic learning in simulated environments."
- **Fei-Fei Li** [0:07:22]: "We work with the first Trump administration on this bill called National AI Resource Resource, NAYER bill, which is scoping out a national AI compute cloud as well as data repository."
- **Justin Johnson** [0:12:59]: "even going from Hopper to Blackwell, like the performance per watt is about the same."
- **Fei-Fei Li** [0:15:30]: "We thought we were the first people doing it. It turned out that Google at that time was also simultaneously doing it."
- **Fei-Fei Li** [0:16:02]: "it was John Markoff from New York Times, was breaking the Google story. But he by accident heard about us."
- **Justin Johnson** [0:20:12]: "the model was running on a server in California. So it was like actually able to stream like all the way from California down to Santiago."
- **Justin Johnson** [0:22:41]: "maybe it's not super efficient to like, you know, render your text to an image and then feed that to a vision model. That's exactly what DeepSeek did."
- **Alessio Fanelli** [0:22:49]: "it was a Harvard paper where they fed a lot of like orbital patterns into an LLM. And then they asked the LLM to predict the orbit of a planet around the sun"
- **Justin Johnson** [0:31:42]: "Like I could generate this scene and then say, I don't like the water bottle, make it blue instead."
- **Fei-Fei Li** [0:33:54]: "you cannot say pan 63 degrees to the north, right? You just don't have that control. Whereas in marble, you have precise control in terms of placing a camera."
- **Justin Johnson** [0:34:28]: "each one is a tiny tiny particle that's semi-transparent has a position orientation in 3d space"
- **Justin Johnson** [0:34:50]: "our recent RTFM model that does generate frames one at a time."
- **Justin Johnson** [0:37:58]: "30 to 60 fps on like an iphone from four years ago then you are a bit limited in like the number of splats that you can handle"
- **Justin Johnson** [0:41:21]: "Just like take two images of your kitchen, like reconstruct it in Marble, and then use the editing features"
- **Fei-Fei Li** [0:42:36]: "There is a psychologist, I think his name is Howard Gardner, in the 1960s, actually literally called multiple intelligence to describe human intelligence."
- **Fei-Fei Li** [0:48:43]: "in nature, you know, it took 540 million years to optimize perception and spatial intelligence and language."
- **Fei-Fei Li** [0:48:43]: "The most generous estimation of language development is probably half a million years."
- **swyx** [0:45:10]: "it roughly rounds up to about 215 ,000 tokens per day"
- **Justin Johnson** [0:57:12]: "A transformer is actually a model of a set of tokens, right? The only thing that gives, that injects the order into it, in the standard transformer architecture"

## Transcript

**[0:00:00] Justin Johnson:** I think the whole history of deep learning is in some sense the history of scaling up compute.

**[0:00:03] Fei-Fei Li:** When I graduated from grad school, I really thought the rest of my entire career would be towards solving that single problem, which is... A lot of AI as a field, as a discipline, is inspired by human intelligence. We thought we were the first people doing it. It turned out that was also simultaneously doing it.

**[0:00:26] Justin Johnson:** So Marble, like, basically, one way of looking at it, it's the system, it's a generative model of 3D worlds, right? So you can input things like text or image or multiple images, and it will generate for you a 3D world that kind of matches those inputs. So while Marble is simultaneously a world model that is building towards this vision of spatial intelligence, it was also very intentionally designed to be a thing that people could find useful today.

**[0:00:48] Justin Johnson:** and we're starting to see emerging use cases in gaming, in VFX, in film, where I think there's a lot of really interesting stuff that Marble can do today as a product and then also set a foundation for the grand world models that we want to build going into the future.

**[0:01:11] Alessio Fanelli:** Hey everyone, welcome to the Latent Space podcast. This is Alessio, founder of Kernel Labs, and I'm joined by Swix, editor of Latent Space.

**[0:01:18] swyx:** And we are so excited to be in the studio with Fei-Fei and Justin of World Labs. Welcome.

**[0:01:24] Fei-Fei Li:** We're excited too.

**[0:01:25] swyx:** I nearly said marble. Yeah, thanks for having us. I think there's a lot of interest in world models and you've done a little bit of publicity around spatial intelligence and all that. I guess maybe one of the part of the story that is a rare opportunity for you to tell is how you two came together to start building World Labs.

**[0:01:42] Fei-Fei Li:** That's very easy because Justin was my former student. So Justin came to my, you know, in my, the other hat I wear is a professor of computer science at Stanford. Justin joined my lab when? Which year?

**[0:01:57] Justin Johnson:** 2012. Actually, the semester that I, the quarter that I joined your lab was the same quarter that AlexNet came out.

**[0:02:03] Fei-Fei Li:** Yeah. Yeah. So Justin is my first.

**[0:02:05] Justin Johnson:** Were you involved in the whole announcement drama? No, no, not at all. But I was sort of watching all the ImageNet excitement around AlexNet at that quarter.

**[0:02:13] Fei-Fei Li:** So he was one of my very best students. And then he went on to have a very successful early career as a professor in Michigan, University of Michigan, Ann Arbor in Meta. And then when we, I think around more than two years ago, for sure, I think both independently, both of us have been looking at the development of the large models and thinking about what's beyond language models. And this idea of building world models, spatial intelligence, really was natural for us.

**[0:02:51] Fei-Fei Li:** So we started talking and decided that we should just put all the eggs in one basket and focus on solving this problem and started World Labs together.

**[0:03:02] Justin Johnson:** Yeah, pretty much. I mean, like, after that, seeing that kind of ImageNet era during my PhD, I had the sense that the next sort of decade of computer vision was going to be about getting AI out of the data center and out into the world. So a lot of my interests post-PhD kind of shifted into 3D vision, a little bit more into computer graphics, more into generative modeling. And I thought I was kind of drifting away from my advisor post-PhD, but then when we reunited a couple years later, it turned out she was thinking of very similar things.

**[0:03:31] Alessio Fanelli:** So if you think about AlexNet, the core pieces of it were obviously ImageNet. It was the move to GPUs and neural networks. How do you think about the AlexNet equivalent model for world models? In a way, it's an idea that has been out there, right? There's been, you know, Young Lagoon is maybe like the most, the biggest proponent, most prominent of it. What have you seen in the last two years that you were like, hey, now's the time to do this.

**[0:03:52] Alessio Fanelli:** And what are maybe the things fundamentally that you want to build as far as data and kind of like maybe different types of algorithms or approaches to compute to make world models

**[0:04:02] Justin Johnson:** really come to life? Yeah, I think one is just there is a lot more data in compute generally available. I think the whole history of deep learning is in some sense the history of scaling up compute. And if you think about, you know, AlexNet required this jump from CPUs to GPUs, but even from AlexNet to today, we're getting about a thousand times more performance per card than we had in AlexNet days. And now it's common to train models, not just on one GPU, but on hundreds or thousands or tens of thousands or even more.

**[0:04:29] Justin Johnson:** So the amount of compute that we can marshal today on a single model is, you know, about a million fold more than we could have even at the start of my PhD. So I think language was one of the really interesting things that started to work quite well the last couple of years. But as we think about moving towards visual data and spatial data and world data, you just need to process a lot more. And I think that's going to be a good way to soak up this new compute that's coming online more and more.

**[0:04:53] swyx:** Does the model of having a public challenge still work or should it be centralized inside of a lab?

**[0:05:00] Fei-Fei Li:** I think open science still is important. You know, AI obviously compared to the image that Alex in that time has really evolved. That was such a niche computer science discipline. Now it's just like civilizational technology. But I'll give you an example, right? Recently, my Stanford lab just announced an open dataset and benchmark called Behavior, which is for benchmarking robotic learning in simulated environments. And that is a very clear effort in still keeping up this open science model of doing things, especially in academia.

**[0:05:46] Fei-Fei Li:** But I think it's important to recognize the ecosystem is a mixture, right? I think a lot of the very focused work in industry, some of them are more seeing the daylight in the form of a product rather than an open challenge per se.

**[0:06:05] swyx:** Yeah. And that's just a matter of the funding in the business model. You have to see some ROI from it.

**[0:06:11] Fei-Fei Li:** I think it's just a matter of the diversity of the ecosystem. Even during the so-called Alexa ImageNet time, I mean, there were closed models, there were proprietary models, there were open models. Or you think about iOS versus Android, right? They're different business models. I wouldn't say it's just a matter of funding per se. It's just how the market is. They're different plays.

**[0:06:40] Alessio Fanelli:** Yeah. But do you feel like you could redo ImageNet today with the commercial pressure that some of these labs has? I mean, to me, that's like the biggest question, right? It's like, what can you open versus what should you keep inside? Like, you know, if I put myself in your shoes, right? It's like, you raise a lot of money, you're building all of this. If you had the best data set for this, what incentives do you really have to publish it?

**[0:07:03] Alessio Fanelli:** And it feels like the people at the labs are getting more and more pulled in the PhD programs are getting pulled earlier and earlier into these labs. So I'm curious if you think there's like an issue right now with like how much money is at stake and how much pressure it puts on like the more academia open research space? Or if you feel like that's not really a concern?

**[0:07:22] Fei-Fei Li:** I do have concerns about less about the pressure. It's more about the resourcing and the imbalance, the resourcing of academia. This is a little bit of a different conversation from World Labs. You know, I have been the past few years advocating for resourcing the healthy ecosystem, As the founding director, co-director of Stanford's Institute for Human-Centered AI of Stanford High, I've been working with policymakers about resourcing public sector and academic AI work. We work with the first Trump administration on this bill called National AI Resource Resource, NAYER bill, which is scoping out a national AI compute cloud as well as data repository.

**[0:08:18] Fei-Fei Li:** And I also think that open source, open data sets continue to be important part of the ecosystem. Like I said, right now in my Stanford lab, we are doing the open data set, open benchmark on robotic learning called behavior. And many of my colleagues are still doing that. I think that's part of the ecosystem. I think what the industry is doing, some startups are doing, are running fast with models, creating products is also a good thing.

**[0:08:50] Fei-Fei Li:** For example, when Justin was a PhD student with me, none of the computer vision programs work that well, right? We could write beautiful papers. Justin has some beautiful—

**[0:09:01] Justin Johnson:** I mean, actually, even before grad school, I wanted to do computer vision, and I reached out to a team at Google and wanted to potentially go and try to do computer vision out of undergrad. And they told me, like, what are you talking about? Like, you can't do that. Go to a PhD first and come back.

**[0:09:14] swyx:** What was the motivation that got you so interested?

**[0:09:17] Justin Johnson:** Oh, I had done some computer vision research in during my undergrad with with actually Feifei's PhD advisor.

**[0:09:23] Fei-Fei Li:** There's a lineage here. There's a lineage here.

**[0:09:25] Justin Johnson:** So I had done some computer vision even as an undergrad and I thought it was really cool and I wanted to keep doing it. So then I was sort of faced with this sort of industry academia choice even coming out of undergrad that I think a lot of people in the research community are facing now. But but to your question, I think like the role of academia, especially in AI, has shifted quite a lot in the last decade.

**[0:09:45] Justin Johnson:** And it's not a bad thing. it's a sense of it's because the technology has grown and emerged, right? Like five or 10 years ago, you really could train state-of-the-art models in the lab, even with just a couple GPUs. But, you know, because that technology was so successful and scaled up so much, then you can't train state-of-the-art models with a couple GPUs anymore. And that's not a bad thing. It's a good thing.

**[0:10:05] Justin Johnson:** It means the technology actually worked. But that means the expectations around what we should be doing as academics shifts a little bit. And it shouldn't be about trying to train the biggest model and scaling up the biggest thing. It should be about trying wacky ideas and new ideas and crazy ideas, most of which won't work. And I think there's a lot to be done there. If anything, I'm worried that too many people in academia are hyper-focused on this notion of trying to pretend like we can train the biggest models or treating it as almost a vocational training program to then graduate and go to a big lab and then be able to play with all the GPUs.

**[0:10:37] Justin Johnson:** I think there's just so much crazy stuff you can do around new algorithms, new architectures, like new systems that, you know, there's a lot you can do as one person.

**[0:10:45] Fei-Fei Li:** And also just academia has a role to play in understanding the theoretical underpinning of these large models. We still know so little about this or extend to the interdisciplinary, you know, Justin calls wacky ideas. There's a lot of basic science ideas. There's a lot of blue sky problems. So I agree. I don't think the problem is open versus closed, productization versus open sourcing. I think the problem right now is that academia by itself is severely under-resourced so that, you know, the researchers and the students do not have enough resources to try these ideas.

**[0:11:30] swyx:** Yeah. Just for people to nerd snipe, what's a wacky idea that comes to mind when you talk about wacky ideas?

**[0:11:35] Justin Johnson:** Oh, like I had this idea that I kept pitching to my students at Michigan, which is that I really like hardware and I really like like new kinds of hardware coming online. And in some sense, the emergence of the neural networks that we use today and transformers are really based around matrix multiplication because matrix multiplication fits really well with GPUs. But if we think about how GPUs are going to scale, how hardware is likely to scale in the future, I don't think the current system that we have, like the GPU hardware design, is going to scale infinitely.

**[0:12:02] Justin Johnson:** And we start to see that even now, that the unit of compute is not the single device anymore. It's this whole cluster of devices. So if you imagine... Yeah, it's a whole node or a whole cluster. But the way we talk about neural networks is still as if they are a monolithic thing that could be coded in one GPU in PyTorch. But then in practice, they could distribute over thousands of devices. So are there like, just as transformers are based around MatMol, and MatMol is sort of the primitive that works really well on GPUs.

**[0:12:29] Justin Johnson:** As you imagine, hardware scaling out, are there other primitives that make more sense for large-scale distributed systems that we could build our neural networks on? And I think it's possible that there could be drastically different architectures that fit with the next generation or the hardware that's going to come 10 or 20 years down the line. And we could start imagining that today.

**[0:12:45] swyx:** It's really hard to make those kinds of bets because there's also the concept of the hardware lottery, where let's just say, you know, NVIDIA has won and we should just, you know, scale that out in infinitely and write software to patch up any gaps we have in the mix, right?

**[0:12:59] Justin Johnson:** I mean, yes and no. If you look at the numbers, like even going from Hopper to Blackwell, like the performance per watt is about the same. They mostly make the number of transistors go up and they make the chip size go up and they make the power usage go up. But even from Hopper to Blackwell, we're kind of already seeing like a scaling limit in terms of what is the what is the performance per watt that we can get.

**[0:13:19] Justin Johnson:** So I think I think there are there is room to do something new. And I don't know exactly what it is. And I don't think you can get it done like in a three month cycle as a startup. But I think that's the kind of idea that if you sit down and sit with for a couple of years, like maybe you could come up with some breakthroughs. And I think that's the kind of long range stuff that is a perfect match for academia.

**[0:13:35] swyx:** Yeah. Coming back to a little bit of background in history, we have this sort of research note on the scene storytelling work that you did or neuro image captioning that you did with Andre. And I just wanted to hear you guys tell that story about, you know, you were like sort of embarking on that for your PhD and Fei-Fei, you like having that reaction that you had.

**[0:13:55] Fei-Fei Li:** Yeah. So I think that line of work started between me and Andre and then Justin joined. Right. So Andrei started his PhD. He and I were looking at what is beyond ImageNet object recognition. And at that time, we, you know, the convolutional neural network was, has proven some power in ImageNet tasks. So, so ConvNet is a great way to represent images. images. In the meantime, I think in the language space, an early sequential model is called LSTM was also being experimented.

**[0:14:33] Fei-Fei Li:** So Andrea and I were just talking about, this has been a long-term dream of mine. I thought it would take 100 years to solve, which is telling the story of images. When I graduated from grad school, I really thought the rest of my entire career would be towards solving that single problem, which is given a picture or given a scene, tell the story in natural language. But things evolved so fast. When Andre and I, when Andre started, we're like maybe combining the representation of convolutional neural network as well as the language sequential model of LSTM.

**[0:15:13] Fei-Fei Li:** We might be able to learn through training to match caption with images. So that's when we started that line of work. And I don't remember, it was 2014 or 2015?

**[0:15:27] Justin Johnson:** It was a CVPR 2015 was the captioning paper.

**[0:15:30] Fei-Fei Li:** So it was our first paper that Andre got it to work that was, you know, given an image. The image is represented with ConvNet. The language model is the LSTM model. And then we combine it and it's able to generate one sentence. And that was one of the first time. It was pretty, I think I wrote it in my book. We thought we were the first people doing it. It turned out that Google at that time was also simultaneously doing it.

**[0:16:02] Fei-Fei Li:** And a reporter, it was John Markoff from New York Times, was breaking the Google story. But he by accident heard about us. And then he realized that we really independently got there together at the same time. So he wrote the story of both the Google research as well as Andre and my research. But after that, I think Justin was already in the lab at that time.

**[0:16:26] Justin Johnson:** Yeah, yeah. I remember the group meeting where Andre was presenting some of those results and explaining this new thing called LSTMs and RNNs that I had never heard of before. And I thought like, wow, this is really amazing stuff. I want to work on that. So then he had the paper at 20 CVPR 2015 on the first image captioning results. Then after that, we started working together and we did it. We first, we did a paper actually just on language modeling back in 2015.

**[0:16:50] Justin Johnson:** I clear 2015. Yeah. I should have stuck with language modeling. That turned out pretty lucrative in retrospect. But we did this language modeling paper together on me and Andre in 2015, where it was like really cool. We trained these little art, these little RNN language models that could, you know, spit out a couple sentences at a time and poke at them and try to understand what the neurons inside the

**[0:17:09] Fei-Fei Li:** neural network inside the things we're doing. You guys were doing analysis on the different like memory and yeah yeah it was it was really it was really cool and even at that time we had

**[0:17:17] Justin Johnson:** these results where you could like look inside the LSTM and say like oh this thing is reading code so one of the like one of the data sets that we that we trained on for this one was the the um the Linux source code right because the whole the whole the whole thing is you know open source and you could just download this so we trained an RNN on this on this data set and then as the network is trying to predict the tokens there, then try to correlate the kinds of predictions that it's making with the kind of internal structures in the RNN.

**[0:17:43] Justin Johnson:** And there we were able to find some correlations between, oh, like this unit in this layer of the LSTM fires when there's an open paren and then like turns off when there's a closed paren and try to do some empirical stuff like that to figure it out. So that was pretty cool. And that was just like, that was kind of like cutting out the CNN from this language modeling part and just looking at the language models in isolation.

**[0:18:01] Fei-Fei Li:** But then we wanted to extend the image captioning work. And remember, at that time, we even have a sense of space because we feel like captioning does not capture different parts of the image. So I was talking to Justin and Andrea about, can you go what we end up calling dense captioning, which is, you know, describe the scene in greater details, especially different parts of the scene.

**[0:18:29] Justin Johnson:** So then we built this system. It was me and Andre and Feifei on a paper the following year, CVPR, so in 2016, where we built this system that did dense captioning. So you input a single image, and then it would draw boxes around all the interesting stuff in the image and then write a short snippet about each of them. It's like, oh, it's a green water bottle on the table. It's a person wearing a black shirt.

**[0:18:46] Justin Johnson:** And this was a really complicated neural network because that was built on a lot of advancements that had been made in object detection around that time, which was a major topic in computer vision for a long time. And then it was actually like one joint neural network that was both, you know, learning to look at individual images. Because it actually had like then three different representations inside this network. One was the representation of the whole image to kind of get the gestalt of what's going on.

**[0:19:09] Justin Johnson:** Then it would propose individual regions that it wants to focus on and then look at, you know, represent each region independently. And then once you look at the region, then you need to spit out text for each region. So that was a pretty complicated neural network architecture. This was all pre-PyTorch. And does it do it in one pass? Yeah, yeah. So it was a single forward pass that did all of it.

**[0:19:25] Fei-Fei Li:** Not only was doing it in OnePass, you also optimized inference. You're doing it on a webcam, I remember.

**[0:19:32] Justin Johnson:** Yeah, so I had built this crazy real-time demo where I had the network running on a server at Stanford and then a web front end that would stream from a webcam and then send the image back to the server. The server would run the model and stream the predictions back. So I was just walking around the lab with this laptop that would just show people this network running in real-time.

**[0:19:53] swyx:** Identification and labeling as well.

**[0:19:56] Fei-Fei Li:** It was pretty impressive because most of my graduate students would be satisfied if they can publish the paper, right? They package the research, put it in a paper. But Justin went a step further. He's like, I want to do this real-time web demo.

**[0:20:12] Justin Johnson:** Actually, I don't know if I told you this story. But then there was a conference that year in Santiago at ICCV. It was ICCV 15. And then I had a paper at that conference for something different. But I had my laptop. I was like walking around the conference with my laptop, showing everybody this like real-time captioning demo. And the model was running on a server in California. So it was like actually able to stream like all the way from California down to Santiago.

**[0:20:35] Justin Johnson:** Well, latency, it was terrible. It was like one FPS, but the fact that it worked at all was pretty, it was pretty amazing.

**[0:20:41] swyx:** I was going to briefly quip that, you know, maybe vision and language modeling are not that different. You know, DeepSQL CR recently tried the crazy thing of let's language, let's model text from pixels and just like train on that. And it might be the future. I don't know. I don't know if you guys have any takes on whether language is actually necessary at all.

**[0:21:01] Fei-Fei Li:** I just wrote a whole manifesto.

**[0:21:05] swyx:** This is my segue into this. Yes.

**[0:21:08] Fei-Fei Li:** I think they are different. I do think the architecture of these generative models will share a lot of shareable components. But I think the deeply 3D, 4D spatial world has a level of structure that is fundamentally different from a purely generative signal that is one-dimensional.

**[0:21:34] Justin Johnson:** Yeah, I think there's something to be said for pixel maximalism, right? Like there's this notion that language is this different thing, but we see language with our eyes and our eyes are just like, you know, basically pixels, right? Like we've got sort of biological pixels in the back of our eyes that are processing these things. And, you know, we see text and we think of it as this discrete thing, but that really only exists in our minds.

**[0:21:54] Justin Johnson:** Like the physical manifestation of text and language in our world are, you know, physical objects that are printed on things in the world

**[0:22:01] Fei-Fei Li:** and we see it with our eyes. Well, you can also think it's sound, but even sound, you can translate into a corellogram, which is a 2D signal. Right. And then like you actually lose something if you

**[0:22:12] Justin Johnson:** translate to this like purely tokenized representations that we use in llms right like you lose the font you lose you lose the line breaks you lose sort of the 2d arrangement on the page um and for a lot of cases for a lot of things maybe that doesn't matter um but for some things it does um and i think pixels are this sort of more more lossless representation of what's going on in the world and in some ways a more general general representation that more matches what what we what we humans see as we as we navigate the world so so like there's an efficiency argument to be made.

**[0:22:41] Justin Johnson:** Like maybe it's not super efficient to like, you know, render your text to an image and then feed that to a vision model. That's exactly what DeepSeek did. It was like, kind of

**[0:22:49] Alessio Fanelli:** worked. I think this ties into the whole world model. Like one of the, my favorite papers that I saw this year was about inductive bias to probe for world models. So it was a Harvard paper where they fed a lot of like orbital patterns into an LLM. And then they asked the LLM to predict the orbit of a planet around the sun and the model generated look good but then if you asked it to draw the force vectors it would be all wacky you know it wouldn't actually follow it so how do you think about what's embedded into the data that you get and we can talk about maybe it's organizing for 3d world models like what are like the

**[0:23:26] Alessio Fanelli:** dimensions of information there's the visual but like how much of like the underlying hidden forces so to speak you need to extract out of this data and like what are some of the challenges there yeah i think there's different ways you

**[0:23:39] Justin Johnson:** could approach that problem um one is like you could try to be explicit about it and say like oh i want to you know measure all the forces and feed those as training data to your model right then you could like sort of run a traditional physics simulation and you know then know all the forces in the scene and then use those as training data to train a model that's now going to hopefully predict those or you could hope that something emerges more latently right that you kind of train on something end-to-end and then on a more general problem, and then hope that somewhere, something in the internals of the model must learn to model something like physics

**[0:24:10] Justin Johnson:** in order to make the proper predictions. And those are kind of the two big paradigms that we have more

**[0:24:14] Fei-Fei Li:** generally. But there's no indication that those latent modeling will get you to a causal law of space and dynamics, right? That's where today's deep learning and human intelligence actually start to bifurcate because fundamentally the deep learning is still fitting patterns

**[0:24:35] Justin Johnson:** there you sort of get philosophical and you say that like we're trying to fit patterns too but maybe we're trying to fit you know a more broad array of patterns like over a with a longer time horizon a different reward function um but but like basically the paper you mentioned is sort of you know that problem that it learns to fit the specific patterns of orbits but then it doesn't actually generalize in the way that you'd like it doesn't have a sort of causal model of gravity

**[0:24:55] Alessio Fanelli:** right because even in marble you know i was trying it and it generates this beautiful sceneries and there's like arches in them. But does the model actually understand how, you know, the arch is actually, you know, drawing on the center kind of like stone and like, you know, the actual physical structure of it. And the other question is like, does it matter that it does understand it as long as they always render something that would fit the physical model that we imagine?

**[0:25:21] Fei-Fei Li:** If you use the word understand the way you understand, I'm pretty sure the model doesn't understand it. The model is learning from the data, learning from the pattern. Yeah. Does it matter, especially for the use cases? It's a good question, right? For now, I don't think it matters because it renders out what you need, assuming it's perfect. Yeah. I mean, it depends on the

**[0:25:47] Justin Johnson:** use case. If the use case is I want to generate sort of a backdrop for virtual film or production or something like that, all you need is something that looks plausible. And in that case, probably it doesn't matter. But if you're going to use this to like, you know, if you're an architect and you're going to use this to design a building that you're then going to go build in the real world, then yeah, it does matter that you model the forces correctly because you don't want the

**[0:26:06] Fei-Fei Li:** thing to break when to actually build it. But even there, right, like even if your model has the semantics in it, let's say, I still don't think the understanding of the signal or the output on the model part and the understanding on the human part is a different word. But this gets, again, philosophical.

**[0:26:29] Justin Johnson:** Yeah, I mean, there's this trick with understanding, right? Like these models are a very different kind of intelligence than human intelligence. And human intelligence is interesting because, you know, I think that I understand things because I can introspect my own thought process to some extent. And then I believe that my thought process probably works similar to other people's so that when I observe someone else's behavior, then I infer that their internal mental state is probably similar to my own internal mental state that I've observed.

**[0:26:54] Justin Johnson:** And therefore, I know that I understand things. So there, I assume that you understand something. But these models are sort of like this alien form of intelligence, where they can do really interesting things, they can exhibit really interesting behavior. But whatever kind of internal, the equivalent of internal cognition or internal self-reflection that they have, if it exists at all, is totally different from what we do.

**[0:27:13] Fei-Fei Li:** So it doesn't have the self-awareness. Right.

**[0:27:15] Justin Johnson:** But what that means is that when we observe seemingly interesting or intelligent behavior out of these systems, we can't necessarily infer other things about them because they're their model of the world and the way they think is so different from us.

**[0:27:27] Alessio Fanelli:** So would you need two different models to do the visual one and the architectural generation, you think? Eventually, like there's not anything fundamental about the approach that you've taken on the model building. It's more about scaling the model and the capabilities of it. Or like, is there something about being very visual that prohibits you from actually learning the physics behind this, so to speak, so that you could trust it to generate a CAD design that then is actually going to work in the real world?

**[0:27:57] Fei-Fei Li:** I think this is a matter of scaling data and bettering model. I don't think there's anything fundamental that separates these two.

**[0:28:05] Justin Johnson:** Yeah, I would like it to be one model. But I think like the big problem in deep learning in some sense is how do you get emergent capabilities beyond your training data? Are you going to get something that understands the forces while it wasn't trained to predict the forces, but it's going to learn them implicitly internally? And I think a lot of what we've seen in other large models is that a lot of this emergent behavior does happen at scale.

**[0:28:25] Justin Johnson:** And will that transfer to other modalities and other use cases and other tasks? I hope so. But that'll be a process that we need to play out over time and see.

**[0:28:32] swyx:** Is there a temptation to rely on physics engines that already exist out there that are, you know, basically the gaming industry has saved you a lot of this work? Or do we have to reinvent things for some fundamental mismatch?

**[0:28:46] Justin Johnson:** I think that's sort of like climbing the ladder of technology, right? Like in some sense, the reason that you want to build these things at all is because maybe traditional physics engines don't work in some situations. If a physics engine was perfect, we would have sort of no need to build models because the problem would have already been solved. So in some sense, the reason why we want to do this is because classical physics engines don't solve problems in the generality that we want.

**[0:29:07] Justin Johnson:** But that doesn't mean we need to throw them away and start everything from scratch. We can use traditional physics engines to generate data that we then train our models on. And then you're sort of distilling the physics engine into the weights of the neural network that you're training.

**[0:29:19] swyx:** I think that's a lot of what, if you compare the work of other labs, people are speculating that Sora had a little bit of that. Genie 3 had a bit of that. and GD3 is explicitly like a video game. Like you have controls to walk around in. And I always think like, it's really funny how the things that we invent for fun actually does eventually make it into serious work.

**[0:29:43] Fei-Fei Li:** Yeah. The whole AI revolution started by graphics chips, partially.

**[0:29:50] swyx:** Misusing the GPU for generating a lot of triangles into generating a lot of everything else, basically. We touched on Marble a little bit. I think you guys chose Marble as I kind of feel like you're sort of a little bit coming out of stealth moment, if you can call it that. Maybe we can get a concise explanation from you on what people should take away, because everyone here can try Marble, but I don't think they might be able to link it to the differences between what your vision is versus other, I guess, generative worlds they may have seen from other labs.

**[0:30:19] Fei-Fei Li:** So Marble is a glimpse into our model, right? We are a model of spatial intelligence model company. We believe spatial intelligence is the next frontier. In order to make spatially intelligent models, the model has to be very powerful in terms of its ability to understand, reason, generate in very multimodal fashion of worlds, as well as allow the level of interactivity that we eventually hope to be as complex as how humans can interact with the world. So that's the grand vision of spatial intelligence, as well as the kind of world models we see.

**[0:30:59] Fei-Fei Li:** Marble is the first glimpse into that. It's the first part of that journey. It's the first in-class model in the world that generates 3D worlds in this level of fidelity that is in the hands of the public. It's the starting point, right? We actually wrote this tech blog. Justin spent a lot of time writing that tech blog. I don't know if you had time to browse it. I mean, Justin really broke it down into what are the inputs we can, multimodal inputs of Marble, what are the kind of editability, which is, you know, allows user to be interactive with the model.

**[0:31:39] Fei-Fei Li:** And what are the kind of outputs we can have?

**[0:31:42] Justin Johnson:** Yeah, so Marble, like basically one way of looking at it, it's a generative model of 3D worlds, right? So you can input things like text or image or multiple images, and it will generate for you a 3D world that kind of matches those inputs. And it's also interactive in the sense that you can interactively edit scenes. Like I could generate this scene and then say, I don't like the water bottle, make it blue instead. Like take out the table, like change these microphones around.

**[0:32:05] Justin Johnson:** And then you can generate new worlds based on these interactive edits and export in a variety of formats. And with Marble, we were actually trying to do sort of two things simultaneously. And I think we managed to pull off the balance pretty well. One is actually build a model that goes towards the grand vision of spatial intelligence. and models need to be able to understand lots of different kinds of inputs, need to be able to model worlds in a lot of situations, need to be able to model counterfactuals of how they could change over time.

**[0:32:29] Justin Johnson:** So we wanted to start to build models that have these capabilities and Marble today does already have hints of all of these. But at the same time, we're a company, we're a business. We were really trying not to have this be a science project, but also build a product that would be useful to people in the real world today. So while Marble is simultaneously a world model that is building towards this vision of spatial intelligence, it was also very intentionally designed to be a thing that people could find useful today.

**[0:32:57] Justin Johnson:** And we're starting to see emerging use cases in gaming, in VFX, in film, where I think there's a lot of really interesting stuff that Marble can do today as a product and then also set a foundation for the grand world models that we want to build going into the future.

**[0:33:11] swyx:** Yeah, I noticed one tool that was very interesting because you can record your scene inside. Yes.

**[0:33:16] Fei-Fei Li:** It's very important. The ability to record means a very precise control of camera placement. In order to have precise camera placement, it means you have to have a sense of 3D space. Otherwise, you don't know how to orient your camera, right? And how to move your camera. So that is a natural consequence of this kind of model. And this is why this is just one of the examples.

**[0:33:43] swyx:** Yeah, I find when I play with video generative models, I'm having to learn the language of being a director because I have to move them out, like pan, you know, like dolly out.

**[0:33:54] Fei-Fei Li:** But even there, you cannot say pan 63 degrees to the north, right? You just don't have that control. Whereas in marble, you have precise control in terms of placing a camera.

**[0:34:05] Alessio Fanelli:** Yeah, I think that's one of the first things people need to understand. It's like, it's not, you're not generating frame by frame, which is like what a lot of the other models are. what are you know people understand that nllm generates one token what are like the atomic units there's kind of like you know the meshes there's like the splats the voxels there's a lot of pieces in a 3d world what should be the mental model that people have of like your generations

**[0:34:28] Justin Johnson:** yeah i think there's like what exists today and what could exist in the future um so what exists today is the model natively output splats um so gaussian splats are these like you know each one is a tiny tiny particle that's semi-transparent has a position orientation in 3d space um and the scene is built up from a large number of these Gaussian splats. And Gaussian splats are really cool because you can render them in real time really efficiently.

**[0:34:50] Justin Johnson:** So you can render on your iPhone, render everything. And that's how we get that sort of precise camera control because the splats can be rendered real time on pretty much any client-side device that we want. So for a lot of the scenes that we're generating today, that kind of atomic unit is that individual splat. But I don't think that's fundamental. I could imagine other approaches in the future that would be interesting. So there are other approaches that even we've worked on at World Labs, like our recent RTFM model that does generate frames one at a time.

**[0:35:18] Justin Johnson:** And there the atomic unit is generating frames one at a time as the user interacts with the system. Or you could imagine other architectures in the future where the atomic unit is a token, where that token now represents some chunk of the 3D world. And I think there's a lot of different architectures that we can experiment with here over time.

**[0:35:33] swyx:** I do want to press on, double click on this a little bit. My version of what Alessio was going to say was like, what is the fundamental data structure of a world model? Because exactly like you said, like it's either a Gaussian splat or it's like the frame or what have you. You also in the previous statements focus a lot on the physics and the forces, which is something over time, which is loosely. I don't see that in marble.

**[0:35:56] swyx:** I presume it's not there yet. Maybe if there was like a marble too, you would have movement or is there a modification to Gaussian splats that makes sense or would it be something completely different?

**[0:36:07] Justin Johnson:** Yeah, I think there's a couple of modifications that make sense. And there's actually a lot of interesting ways to integrate things here, which is another nice place of working in this space. Then there's actually been a lot of research work on this. Like when you talk about wacky ideas, like there's actually been a lot of really interesting academic work on different ways to imbue physics.

**[0:36:21] Fei-Fei Li:** We call it the wacky ideas in industry.

**[0:36:25] Justin Johnson:** All right, but then it's like Gaussian splats are themselves little particles. There's been a lot of approaches where you basically attach physical properties to those splats and say that each one has a mass or like maybe you treat each one as being coupled with some kind of virtual spring to nearby neighbors. And now you can start to do sort of physics simulation on top of splats. So one kind of avenue for adding physics or dynamics or interaction to these things would be to, you know, predict physical properties associated with each of your splat particles and then simulate those downstream, either using classical physics or something learned.

**[0:36:56] Justin Johnson:** Or, you know, the kind of the beauty of working in 3D is things compose and you can inject logic in different places. So one way is sort of like we're generating a 3D scene. We're going to predict 3D properties of everything in the scene. Then we use a classical physics engine to simulate the interaction. Or you could do something where like as a result of a user action, the model is now going to regenerate the entire scene in splats or some other representation.

**[0:37:19] Justin Johnson:** And that could potentially be a lot more general because then you're not bound to whatever sort of physical properties you know how to model already. But that's also a lot more computationally demanding because then you need to regenerate the whole scene in response to user actions. But I think this was a really interesting area for future work and for adding on into potential marble too, as you say.

**[0:37:40] Fei-Fei Li:** Yeah, there's opportunity for dynamics, right?

**[0:37:43] Alessio Fanelli:** What's the state of splats density, I guess? Can we render enough to have very high resolution when we zoom in? Are we limited by the amount that you can generate, the amount that we can render? How are these going to get super high fidelity, so to speak?

**[0:37:58] Justin Johnson:** you have some limitations but depending on your target use case so like one of the one of the big constraints that we have on our scenes is we wanted things to render cleanly on mobile and we wanted things to render cleanly in vr headsets so those are those devices have a lot less compute than your used than you have in a lot of other situations and like if you want to get a splat file to render at high resolution high like 30 to 60 fps on like an iphone from four years ago then you are a bit limited in like the number of splats that you can handle But if you're allowed to work on a recent, even this year's

**[0:38:30] Justin Johnson:** iPhone or a recent MacBook, or even if you have a local GPU, or if you don't need that 60 FPS, 1080p, then you can relax the constraints and get away with more splats and that lets you get higher resolution in your scenes.

**[0:38:43] swyx:** One use case I was expecting but didn't hear from you was embodied use cases. You're just focusing on virtual for now?

**[0:38:50] Fei-Fei Li:** If you go to WorldLab's homepage, there is a particular page called MarbleLabs. There we showcase different use cases. And we actually organize them in more visual effect use cases or gaming use cases, as well as simulation use cases. And in that, we actually show this is a technology that can help a lot in robotic training, right? This goes back to what I was talking about earlier in speaking of data starvation, robotic training really lack data. You know, high fidelity real world data is absolutely very critical, but you're just not going to get a ton of that.

**[0:39:32] Fei-Fei Li:** Of course, the other extreme is just purely Internet video data, but then you lack a lot of the controllability that you want to train your embodied agents with. So simulation and synthetic data is actually a very important middle ground for that. I've been working in this space for many years. One of the biggest pain point is where do you get the synthetic simulated data? You have to curate assets and build these, compose these complex situations. And in robotics, you want a lot of different states.

**[0:40:06] Fei-Fei Li:** You want the embodied agent to interact in the synthetic environment. Marble actually is a really potential for helping to generate these synthetic simulated worlds for embodied agent training.

**[0:40:22] swyx:** Obviously, that's on the homepage. It'll be there. I was trying to make the link to, as you said, you also have to build a business model. The market for robotics, obviously, is very huge. Maybe you don't need that, or maybe we need to build up and solve the virtual worlds first before we go to embodied. And obviously, I think that's a big stepping stone.

**[0:40:42] Fei-Fei Li:** That is to be decided. I do think that...

**[0:40:46] swyx:** Because everyone else is going straight there, right?

**[0:40:49] Fei-Fei Li:** Not everyone else, but there is an excitement, I would say. But, you know, I think the world is big enough to have different approaches.

**[0:41:01] Justin Johnson:** Yeah, I mean, and we always view this as a pretty horizontal technology that should be able to touch a lot of different industries over time. And, you know, Marble is a little bit more focused on creative industries for now. But I think the technology that powers it should be applicable to a lot of different things over time. And robotics is one that, you know, is maybe going to happen sooner than later.

**[0:41:18] Fei-Fei Li:** Also design, right? It's very adjacent to creative.

**[0:41:21] Justin Johnson:** Oh, yeah, definitely. Like, I think it's like the architecture stuff. Yes. Yeah, I mean, I was joking online. I posted this video on Slack of like, oh, who wants to use Marble to plan your next kitchen remodel? It actually works great for this already. Just like take two images of your kitchen, like reconstruct it in Marble, and then use the editing features to see what would that space look like if you change the countertops or change the floors or change the cabinets.

**[0:41:41] Justin Johnson:** And this is something that's, you know, we didn't necessarily build anything specific for this use case. But because it's a powerful horizontal technology, you kind of get these emergent use cases that just fall out of the model.

**[0:41:51] Fei-Fei Li:** We have early beta users using an API key that is already building for interior design use case.

**[0:42:01] Alessio Fanelli:** I just did my garage. I should have known about this.

**[0:42:06] Fei-Fei Li:** Next time you remodel, we can be of help.

**[0:42:09] Alessio Fanelli:** Kitchen is next, I'm sure. I'm curious about the whole spatial intelligence space. I think we should dig more into that. One, how do you define it? And what are the gaps between traditional intelligence that people might think about an LLMS when Dario says we have a data center full of Einsteins? That's like traditional intelligence. It's not spatial intelligence. What is required to be spatial intelligent?

**[0:42:36] Fei-Fei Li:** First of all, I don't understand that sentence, a data center full of Einsteins. I just don't understand that. It's not a deep... It's an analogy. Well, so a lot of AI as a field, as a discipline is inspired by human intelligence, right? Because we are the most intelligent animal we know in the universe for now. And if you look at human intelligence, it's very multi-intelligent, right? There is a psychologist, I think his name is Howard Gardner, in the 1960s, actually literally called multiple intelligence to describe human intelligence.

**[0:43:15] Fei-Fei Li:** And there is linguistic intelligence, there's spatial intelligence, there is logical intelligence and emotional intelligence. So for me, when I think about spatial intelligence, I see it as complementary to language intelligence. So I personally would not say it's spatial versus traditional because I don't know tradition means what does that mean. I do think spatial is complementary to linguistic. And how do we define spatial intelligence? It's the capability that allows you to reason, understand, move, and interact in space.

**[0:43:57] Fei-Fei Li:** And I use this example of the deduction of DNA structure, right? And of course, I'm simplifying this story, but a lot of that had to do with the spatial reasoning of the molecules and the chemical bonds in a 3D space to eventually conjecture a double helix. And that ability that humans or Francis Crick and Watson had done, it is very, very hard to reduce that process into pure language. And that's a pinnacle of a civilizational moment.

**[0:44:36] Fei-Fei Li:** But every day, right, I'm here trying to grasp a mug. This whole process of seeing the mug, seeing the context where it is, seeing my own hand, opening of my hand that geometrically would match the mug and touching the right affordance points, all this is deeply, deeply spatial. It's very hard. I'm trying to use language to narrate it. but on the other hand that narrative language itself cannot get you to pick up a mug yeah bandwidth constraint yes

**[0:45:10] swyx:** I did some math recently on like if you just spoke all day every day for 24 hours a day how many tokens do you generate at the average speaking rate of like 150 words per minute it roughly rounds up to about 215 ,000 tokens per day and like your world that you live in is so much higher bandwidth than that

**[0:45:29] Alessio Fanelli:** Yeah. Well, I think that is true. But if I think about Sir Isaac Newton, right, it's like you have things like gravity at the time that have not been formalized in language that people inherently spatially understand that things fall. Right. But then it's helpful to formalize that in some way or like, you know, all these different rules that we use language to like really capture something that empirically and spatially you can also understand. but it's easier to like describe in a way.

**[0:45:59] Alessio Fanelli:** So I'm curious, like the interplay of like spatial and like linguistic intelligence, which is like, okay, you need to understand some rules are easier to write in language for then the spatial intelligence to understand, but you cannot, you know, you cannot write, put your hand like this and put it down this amount. So I'm always curious about how you leverage each other together.

**[0:46:20] Justin Johnson:** I mean, if anything, like the example of Newton, like Newton only thinks to write down those laws is because he's had a lot of embodied experience in the world. Right. Yeah, exactly. And actually it's useful to distinguish between the theory building that you're mentioning versus like the embodied, like the daily experience of being embedded in the three-dimensional world. Right. So, so, so to me, spatial intelligence is sort of encapsulating that embodied experience of being there in 3D space, moving through it, seeing it, actioning it.

**[0:46:45] Justin Johnson:** And as Fei-Fei said, you can narrate those things, but it's a very lossy channel. It's just like the notion of, you know, being in the world and doing things in it is a very different modality from trying to describe it. But because we as humans are animals who have evolved interacting in space all the time, like we don't even think that that's a hard thing, right? And then we sort of naturally leap to language and then theory building as mechanisms to abstract above that sort of native spatial understanding.

**[0:47:11] Justin Johnson:** And in some sense, LLMs have just like jumped all the way to those highest forms of abstracted reasoning, which is very interesting and very useful. But spatial intelligence is almost like opening up that black box again and saying maybe we've lost something by going straight to that fully abstracted form of language and reasoning and communication.

**[0:47:26] Fei-Fei Li:** You know, it's funny as a vision scientist, right? I always find that vision is underappreciated because it's effortless for humans. You open your eyes. As a baby, you start to see your world. We're somehow born with it. We're almost born with it. But you have to put effort in learning language, including learning how to write, how to do grammar, how to express. And that makes it feel hard. Whereas something that nature spends way more time actually optimizing, which is perception and spatial intelligence, is underappreciated by humans.

**[0:48:02] swyx:** Is there proof that we are born with it? You said almost born. So it sounds like we actually do learn after we're born.

**[0:48:09] Fei-Fei Li:** When we are born, our visual acuity is less. And our perceptual ability does increase. But we are, most humans are born with the ability to see. And most humans are born with the ability to link perception with motor movements, right? I mean, the motor movement itself takes a while to refine. And then animals are incredible, right? Like I was just in Africa earlier this summer. These little animals, they're born and within minutes they have to get going.

**[0:48:43] Fei-Fei Li:** And otherwise, you know, the lions will get them. And in nature, you know, it took 540 million years to optimize perception and spatial intelligence and language. The most generous estimation of language development is probably half a million years.

**[0:49:02] swyx:** Wow. Yeah. That's longer than I would have got to say.

**[0:49:05] Fei-Fei Li:** I'm being very generous. Yeah.

**[0:49:08] swyx:** Yeah, no, I was, you know, sort of going through your book and I was realizing that one of the interesting links to something that we covered on the podcast is language model benchmarks. And how do Winogrand actually put in all these sort of physical impossibilities that require spatial intelligence, right? Like A is on top of B, therefore A cannot fall through B is obvious to us. But to a language model, it could happen. I don't know.

**[0:49:31] swyx:** Maybe it's like a part of the, you know, the next token prediction.

**[0:49:34] Justin Johnson:** And that's sort of what I mean about like unwrapping this abstraction, right? Like if your whole model of the world is just like saying sequences of words after each other, it's really kind of hard to like, why not? It's actually unfair. Right. But then the reason it's obvious to us is because we are internally mapping it back to some three-dimensional representation of the world that we're familiar with.

**[0:49:52] swyx:** The question is, I guess, how long is it going to take us to distill from, I use the word distill, I don't know if you agree with that, to distill from your world models into a language model? Because we do want our models to have social intelligence, right? And do we have to throw the language model out completely in order to do that? No, right? Yeah, I don't think so.

**[0:50:13] Fei-Fei Li:** I think they're multimodal. I mean, even our model Marble today takes language as an input. Right. Right. So it's deeply multimodal. And I think in many use cases, these models will work together. Maybe one day we'll have a universal model.

**[0:50:28] Justin Johnson:** I mean, even if you do, like there's sort of a pragmatic thing where people use language and people want to interact with systems using language. Even pragmatically, it's useful to build systems and build products and build models that let people talk to them. So I don't see that going away. I think there's a sort of intellectual curiosity of saying how like intellectually, how much could you build a model that that only uses vision or only uses spatial intelligence?

**[0:50:51] Justin Johnson:** I don't know that that would be practically useful, but I think it'd be an interesting intellectual or academic exercise to see how far you could push that.

**[0:50:56] Alessio Fanelli:** I think, I mean, not to bring it back to physics, but I'm curious, like if you had a highly precise world model and you didn't give it any notion of like our current understanding of the standard model of physics, how much of it it will be able to come up with and like recreate from scratch and what level of like language understanding it would need. because we have so many notations that we kind of use that we created, but maybe we'll come up with a very different model of it and still be accurate.

**[0:51:24] Alessio Fanelli:** And I wonder how much we're kind of limited, but you know how people say humanoids need to be like humans because the world is built for humans. And in a way, it's like the way we build language constrains some of the outputs that we can get from these other modalities as well. So I'm super excited to follow your work.

**[0:51:42] Justin Johnson:** Yeah, I mean, there's another engine. You actually don't even need to be doing AI to answer that question. You could discover aliens and see what kind of physics they have. Right. Right. And they might have a totally different.

**[0:51:49] Alessio Fanelli:** Well, Fei-Fei said, we are so far the smartest animal in the universe.

**[0:51:54] Justin Johnson:** So, so far. So, so what do you, so, I mean, but that is a really interesting question, right? Like, is our knowledge of the universe and our understanding of physics, is it constrained in some way by our own cognition or by the path dependence of our own technological evolution? And one way to sort of, and like do an experiment, like you almost want to do an experiment and say, like, if we were to rerun human civilization again, would we come up with the same physics in the same order?

**[0:52:13] Justin Johnson:** And I don't think that's a very practical experiment to run.

**[0:52:17] Fei-Fei Li:** You know, one experiment I wonder if people could run is that we have plenty of astrophysical data now on the planet or celestial body movements. Just feed the data into a model and see if Newtonian law emerges.

**[0:52:34] Justin Johnson:** My guess is it probably won't.

**[0:52:36] Fei-Fei Li:** That's my guess. It's not. The abstraction level of Newtonian law is at a different level from what these language LLMs represents. So I wouldn't be surprised that giving enough celestial movement data, an LLM would actually predict pretty accurate movement trajectories. Let's say I invent a planet surrounding a star and give enough data. My model would tell you on day one where it is, day two where it is. I wouldn't be surprised. But F equals MA or action equals reaction, that's just a whole different abstraction level.

**[0:53:20] Fei-Fei Li:** That's beyond just today's LL.

**[0:53:23] Alessio Fanelli:** Okay, what model would you need to not have it be a geocentric? model because if i'm training just on visual data it makes sense that you think the sun rotates around the earth right but obviously that's not the case so how would it learn that like i'm curious about all these like you know forces that we talk about it's like sometimes maybe you don't need them because as long as it looks right it's right but like as you make the jump to like trying to use these models to do more high level tasks how much can we rely on them i think you can need kind

**[0:53:55] Justin Johnson:** of a different learning paradigm, right? So like, you know, there's a bit of conflation here happening where saying, is it LLMs and language and symbols versus, you know, human theory building and human physics. And they're very different because an LL, like the human objective function is to understand the world and thrive in your life. And the way that you do that is by, you know, sometimes you observe data and then you think about it and then you try to do something in the world and it doesn't match your expectations.

**[0:54:21] Justin Johnson:** And then you want to go and update sort of your, your understanding of the world online. And people do this all the time, constantly, like whether it's, you know, I think my keys are downstairs. So I go downstairs and I look for them and I don't see them. And oh, no, they're actually up in my bedroom. So we're like, because we're constantly interacting with the world, we're constantly having to build theories about what's happening in the world around us and then falsify or add evidence to those theories.

**[0:54:45] Justin Johnson:** And I think that that kind of process writ large and scaled up is what gives us F equals MA in Newtonian physics. And I think that's a little orthogonal to, you know, the modality of model that we're training, whether it's language or spatially. The way I put it is almost like this

**[0:54:58] swyx:** is almost more efficient learning because you have a hypothesis of here are the different possible worlds that are granted by my available data. And then you do experiments to eliminate the worlds that are not possible. And then you resolve to the one that's right. To me, that's also how I also have theory of mind, which is like, I have a few thesis of what you're thinking, what you're thinking. And I try to create actions to resolve that or check my intuition as to what you're thinking.

**[0:55:26] swyx:** And obviously, all of them don't do any of these.

**[0:55:31] Fei-Fei Li:** A theory of mind possibly also will break into even emotional intelligence, which today's AI is really not touching at all.

**[0:55:40] swyx:** And we really, really need it. You know, people are starting to depend on these things probably too much. And that's a whole topic. of other debate. I do have to ask because a lot of people have said this to us. How much do we have to get rid of? Is sequence-to-sequence modeling out the window? Is attention out the window? How much are we re-questioning everything? I think you stick with stuff that works, right?

**[0:56:05] Justin Johnson:** So attention is still there. I think attention is still there. I think there's a lot. You don't need to fix things that aren't broken. There's a lot of hard problems in the world to solve, but let's focus on one at a time. I think it is pretty interesting to think about new architectures or new paradigms or drastically different ways to learn. But you don't need to throw away everything just because you're working on new modalities.

**[0:56:26] Fei-Fei Li:** I think sequence to sequence is actually, in world models, I think we are going to see algorithm or architecture beyond sequence to sequence.

**[0:56:38] Justin Johnson:** Oh, but here actually, I think there's a little bit of technological confusion and Transformers already solved that for us, right? Like transformers are actually not a model of sequences. A transformer is natively a model of sets. And that's very powerful. But because a lot of the transformers grew out of earlier architectures based around recurrent neural networks, and RNNs definitely do have like a built-in architectural, like they do model one-dimensional sequences. But transformers are just models of sets and they can model a lot of, those sets could be, you know, 1D sequences.

**[0:57:07] Justin Johnson:** They could be other things as well.

**[0:57:08] swyx:** Do you literally mean set theory? Like Galois, whatever?

**[0:57:12] Justin Johnson:** Yeah, yeah, yeah. Yeah, so a transformer is actually not a model of a sequence of tokens. A transformer is actually a model of a set of tokens, right? The only thing that gives, that injects the order into it, in the standard transformer architecture, the only thing that differentiates the order of the things is the positional embedding that you give the tokens, right? So if you choose to give a sort of 1D positional embedding, that's the only mechanism that the model has to know that it's a 1D sequence.

**[0:57:35] Justin Johnson:** But all the operators that happen inside a transformer block are either token-wise, right? So that either you have an FFN, you have QKV projections, like you have per token normalization. All of those happen independently per token. Then you have interactions between tokens through the attention mechanism. But that's also sort of, it's permutation equivariant. So if I permute my tokens, then the tension operator gets a permuted output in exactly the same way. So it's actually natively an architecture of sets of tokens.

**[0:58:01] Justin Johnson:** Literally a transform.

**[0:58:03] swyx:** Yeah. In a math term.

**[0:58:05] Alessio Fanelli:** I know we're out of time, but we just want to give you the floor for some call to action either on people that would enjoy working at WorldLabs. What kind of people should apply? What research people should be doing outside of WorldLabs that would be helpful to you or anything else on your mind?

**[0:58:20] Fei-Fei Li:** I do think it's a very exciting time to be looking beyond just language models and think about the boundless possibilities of spatial intelligence. So we are actually hungry for talent, ranging from very deep researchers, right, thinking about the problems like Justin just described, you know, training large models of world models. We are hungry for engineers, good engineers building systems, you know, from training optimization to inference to product. And we're also hungry for good business product thinkers and go-to-market and business talents.

**[0:59:03] Fei-Fei Li:** So we are hungry for talent, especially now that we have exposed the model to the world through marble. I think we have a great opportunity to work with even a bigger pool of talent to solve both the model problem as well as deliver the best product to the world.

**[0:59:24] Justin Johnson:** Yeah, I think I'm also excited for people to try Marble and do a lot of cool stuff with it. I think it has a lot of really cool capabilities, a lot of really cool features that fit together really nicely.

**[0:59:32] Fei-Fei Li:** In the car coming here, Justin and I were saying people have not totally discovered that. Okay, it's only 24 hours. I have not only discovered some of the advanced mode of editing, right? Like turn on the advanced mode. You can, like Justin said, change the color of the bottle, change your floor and change the trees.

**[0:59:52] swyx:** Well, I actually tried to get there, but when it says create, it just makes me create a completely different world. You need to click on the advanced mode. It's a good UI issue.

**[1:00:01] Fei-Fei Li:** We can improve on our UI UI, but remember to click on it.

**[1:00:05] Justin Johnson:** Yeah, we need to hire people who work on the product.

**[1:00:08] swyx:** but one thing we got that was clear from you guys are looking for is also intellectual fearlessness which is something that I think you guys hold as principle

**[1:00:15] Fei-Fei Li:** yeah I mean we are literally the first people who are trying this both on the model side as well as on the product side

**[1:00:24] Alessio Fanelli:** thank you so much for joining us this was fun thank you guys thanks for having us

**[1:00:27] Fei-Fei Li:** yeah you
