i'm not so interested in LM anymore that's one of the bold statements Yan made at NVIDIA GTC 2025 and no surprise it's blowing up on Twitter why because let's face it in the AI world right now large language models LLMs are stealing the spotlight everyone's talking about them before we go any further smash that like button and subscribe to the channel for all the juiciest updates in AI and tech now if you're wondering who Yan Lakhan is here's the deal he's not just anyone he's one of the godfathers of AI research we're talking years and
years of deep expertise so when he drops a statement like this people sit up and listen this isn't some random hot take it's coming from someone with decades of knowledge and that leaves everyone asking is he on to something huge stick around as we break down his four main focuses it's about to get mind-blowing so um Yan there's been a lot of interesting things going on in the last year in AI what has been the most exciting development in in your opinion over the past year uh too many to count but I tell you one
thing which may surprise a few of you um I'm not so interested in LLMs anymore you know they're kind of the last thing they are in the hands of you know industry product people kind of you know improving at the margin uh trying to get you know more data more compute generating synthetic data um I think there are more interesting questions in uh four four things how do you get machines to understand the physical world and Jensen talked about this this morning in this keynote how do you get get them to have persistent memory which
not too many people talk about and then the last two are how do you get them to reason and plan and there is some effort of course to get you know LLMs to reason but in my opinion it's a very kind of simplistic way of viewing um viewing reasoning i think there are probably kind of more you know better way of of doing this so um so I'm excited about things that a lot of people in this community in the tech community might get excited about five years from now um but right now doesn't look
so exciting because it's some obscure academic paper here where Yan gets real about world models he's saying something gamechanging relying only on text as your world model just doesn't cut it if you want to achieve true general intelligence gi why because predicting the next token works great for text but let's be honest real life isn't just words on a screen humans operate in the physical world doing physical things and that's where textbased models hit their limits if we want AI to truly understand and interact with our world we need to think way beyond just text
but if it's not an LLM that's reasoning about the physical world and having persistent memory and planning what is it what is the underlying model going to be um so a lot of people are working on world models right so what is a world model world model is uh we all have world models in our in our mind uh this is what allows us to um kind of you know manipulate thoughts essentially so you know we have a model of the of the current world you you know that if I if I push on this
on this bottle here from the top it's probably going to flip but if I push on it at the bottom it's it's going to slide um and you know if I press on it too hard it's might pop so we have models of the physical world that we acquire in the first few months of life and that's what allows us to deal with the real world and it's much more difficult to deal with the real world than to deal with language and so the the type of architectures that I think we need for systems that
really can deal with the real world is completely different from the ones that we deal with at the moment right lms predict tokens right but tokens could be anything i mean so our you know autonomous vehicle model uses tokens tokens from the sensors and it produces tokens that drive and in some sense it's reasoning about the physical world at least where it's safe to drive and you won't run into poles um why why aren't tokens the right way to represent the physical world tokens are discrete okay so when we talk about token generally we talk
about uh a a a finite set of possibilities in a typical LLM the number of possible tokens is on the order of 100 thousand or something like that right um so when you train a system to predict tokens you can never train it to predict the exact token that's going to follow a sequence in text for example but you can produce a probability distribution of all the possible tokens in your dictionary you know it's just a long vector of 100,000 numbers between zero and one that sum to one we know how to do this we
don't know how to do this with u with video with what know natural data that is highdimensional and continuous and every attempt at trying to get system to understand the world or build mental models of the world by being trained to predict videos at the pixel level basically have failed um even at the even to train a system like a a neural net of some kind to learn good representation of images every technique that works by reconstructing an image from a corrupted or transformed version of it basically has failed not completely failed they kind of
work but they don't work as well as alternative architectures that we call joint embedding which essentially don't attempt to reconstruct at the pixel level they try to learn a representation an abstract representation of the image or the video or the natural uh signal that is is being trained on so that you can make prediction in that abstract representation space um the example I use very often is that if I take a video of this room and I kind of pan a camera and I stop here and I ask the system to predict you know what's
the continuation of that video it's probably going to predict it's a room and there's people sitting blah blah blah it can there's no way it can predict what every single one of you looks like right that's completely unpredictable from the initial segment of the video and so there's a lot of things in the world that are just not predictable and if you train a system to predict at a pixel level it spends all of its resources trying to come up with details that it just cannot invent and so that's just a complete waste of resources
and every attempt that we've tried and I've been working on this for 20 years uh of training a system using self-s supervised learning by predicting video doesn't work it only works if you do it at a representation level and what that means is that those architectures are not are not generated so if you're basically saying that that a transformer So here's the deal yan is dropping some truth bombs he's saying that using a transformer to predict the physical world yeah it just doesn't work and he's got some solid reasons why think about it when you're
predicting the next token in text your brain already has this insane understanding of the physical world stuff you don't even think about like gravity motion and how objects interact that's all implicit knowledge we take for granted but AI not so much it's not wired that way and honestly Yan's point hits hard it doesn't work now before you think I'm just nodding along let me clarify i'm not blindly agreeing with Yan here there's actual research backing this up researchers in China basically said the same thing these architectures aren't really predicting the physical world they're more like
mimicking it copying patterns without truly understanding them it's mind-blowing stuff but here's the million-doll question if these models don't work then what's the solution enter Yan's game-changing VGPA architecture rumor has it version 2 is dropping soon and from what we've heard it's showing some of the most promising results we've seen yet you know Jensen is absolutely right that uh you get ultimately more power in a system that that can sort of you know reason i I disagree with the fact that the proper way to do reasoning is the way you know current NLMs that that
have are augmented by by reasoning abilities you're saying it works but it's not the right way it's not the right way i think uh you know when when we reason when we think we do this in some sort of abstract mental mental uh state that has nothing to do with language like kicking the tokens out you want to be reasoning in your um latent space and so that's right right i mean if I if I tell you you know imagine a cube floating in front of you and I rotate that cube by 90 degrees around
a vertical axis okay you can do this mentally it has nothing to do with language um you know a cat could do this uh we can't specify the problem to a cat obviously through language but you know cats do things that are much more complex than this when they plan like you know some trajectories to jump on a piece of furniture right they do things that are much more complex than that and um that is not related to language it's certainly not done in so you know token space which would be kind of actions it's
done in sort of abstract mental space so that's uh that's kind of the challenge of the next few years um which is to figure out new architectures that allow this type of of things that's what I've been working on for the last so so is there a new model we should be expecting that allows us to do reasoning in this abstract space um it's called we call it JPA uh or JEPA world models um and we've you know um my my colleagues and I have kind of put out a bunch of uh papers on this
kind of you know first steps towards towards this over the last few years so jetpack means joint emitting predictive architecture this is those world models that learn abstract representations that are capable of sort of manipulating those representation uh and and and perhaps reason and produce sequences of actions to you know arrive at a particular goal i think I think that's the that's the future i wrote a long paper about this that explains how this might work about about what does this famous VJA architecture actually look like meta dropped some insights into this last year and
trust me it's worth taking a closer look this isn't just another model it's a potential gamecher for how AI understands and interacts with the real world today machines require thousands of examples and hours of training to learn a single concept the goal with JEPAS which means joint embedding predictive architectures is to create highly intelligent machines that can learn as efficiently as humans vj is pre-trained on video data allowing it to efficiently learn concepts about the physical world similar to how a baby learns by observing its parents it's able to learn new concepts and solve new
tasks using only a few examples without full fine-tuning vja is a non-generative model that learns by predicting missing or masked parts of a video in an abstract representation space unlike generative approaches that try and fill in every missing pixel VJPA has the flexibility to discard irrelevant information which leads to more efficient training to allow our fellow researchers to build upon this work we're publicly releasing VJeppa we believe this work is another important step in the journey towards AI that's able to understand the world plan reason predict and accomplish complex tasks the alternative that we have
now is a project called VJPA and we are at getting close to version two where basically it's one of those joint emitting predictive architecture so it does prediction on on video but at the representation level and it seems to work really well we have an example of this the first version of this is trained on very short videos just 16 frames and it's trained to uh basically predict the representation of full video from a version of of a partially masked one and that system apparently is able to tell you whether a particular video is physically
possible or not at least in restricted cases and it gives you a binary output this is feasible this is not or maybe well no it's simpler than this you you measure the prediction error that the system produces so you take a sliding window of those 16 frames on a video and you look at you know can you predict like the next few frames and you you measure the prediction error and when something really strange happens in the video like an object disappears or changes shape or or you know something like that or sponty appears or
doesn't obey physics so it's physically realistic just by observing videos yeah these are you know I mean you train it on natural videos and then you test it on synthetic video where something really weird happens right so if you trained it on videos where really weird things happen that would become normal and it wouldn't uh Yeah that's right detect those as being odd so you don't do that now here where Yan gets into the human brain secret sauce system one and system two thinking as humans we've got two modes of thought system one is fast
reactive and instinctive like dodging a falling object system two that's the deep slow thoughtful processing where we solve complex problems or plan for the future here's the kicker llms are just now starting to scratch the surface of this kind of thinking but as Yan points out AI systems are still missing some of those intuitive capabilities that make human thought so powerful if we want to get anywhere close to general artificial intelligence AGI we need systems that can think like us both reactively and reflectively this is where the real magic happens and it it connects with
something that we're all very familiar with right so psychologists talk about system one and system two system one is tasks that you can accomplish without really sort of thinking about them they've become you become used to them and you you can accomplish them without thinking too much about them so if you are an experienced driver you can drive even without driving assistance you can drive without thinking about it much you know you can talk to someone at the same time you can you know um etc but if you are a a if you drive for
the first time for the first few hours you are you are the wheel you have to really focus on what you're doing right and you're planning all kinds of catastrophe scenarios and stuff like that imagine all kind of kind of things so that's system two you're recruiting your entire prefoto cortex to your your world model your internal world model to uh figure out the you know what's what's going to happen and then plan action so that good things happen um whereas when you're familiar with this you you can just use a system one and sort
of do this automatically so this this idea that you start by uh you know using your world model and you're able to accomplish a task even a task that you've never encountered before zero shot right you don't have to be trained to solve that task you can just so you can just accomplish that task without learning anything just on the basis of your understanding of of the world and your planning abilities that's what's missing in current systems but if you accomplish that task multiple times then eventually it gets compiled into what's called a policy right
so a sort of reactive system that allows you to just just accomplish that task without without planning so the first thing this reasoning is system two the sort of automatic subconscious reactive policy that's system one um can do system one and are trying to inching their way towards system two but ultimately I think we need a different architecture for system two yan doesn't stop there he goes even deeper he's saying straight up we're not going to reach AGI just by scaling up LLMs and honestly I think he's on to something the future of AGI probably
won't be one single model it'll likely be a hybrid system combining all these different capabilities into one powerhouse and guess what we're already seeing hints of this companies like Google are moving toward Omni models blending multiple AI strengths into one unified system it's fascinating to hear Yen talk about this because honestly he's not far off the mark the road to AGI is going to be wild unpredictable and full of hybrid innovations and I can't wait to see where it takes us next uh but the real world is just much more complicated like okay here's something
that you some of you may have heard me say in the past current LLM are trained typically with something like on the order of 30 trillion tokens right tokens typically is about three bytes so that's 0.9 10^ the 13 bytes let's say 10^ the 14 bytes um that would take any of us over 400,000 years to read through that because that's kind of the totality of all the text available on the internet right now a the psychologists tell us that a four-year-old has been awake a total of 16,000 hours and we have about two megabytes
going to our visual cortex through our optic nerve um every second two megabytes per second roughly multiply this by 16,000 hours times 3600 it's about 10^ the 14 bytes in four years through vision mhm you see as much data as text that would take you 400,000 years to read i mean that tells you we're never going to get to AGI whatever you mean by this um by just training from text it's just not happening
