I got to visit Memphis yeah yeah you're going big on compute yeah you've also said play to win or don't play at all so yeah what does it take to win um for AI that means you've got to have the most powerful training compute and your the the rate of improvement of training compute has to be faster than everyone else or you will not win you your AI will be worse so how can grock let's say three that might be available what like next year well hopefully end of this year grock 3 for Lucky yeah
how can that be the best llm the best AI system available in the world how much of it is a compute how much of it is Data how much of it is like post training how much of it is the product that you package it up in all that kind of stuff I mean they w't matter it's sort of like saying what what you know let's say it's a Formula 1 race like what matters more the car or the driver I mean they both matter um if if your if your car is not fast then
you know if it's like let's say it's half the horsepower of a competitors the best driver will still lose on if it's twice the horsepower then probably even a mediocre driver will still win so the training computer is kind of like the engine how many this horsepower of the engine so you really you want to try to do the best on that and you then um that's how efficiently do you use that training compute and how efficiently do you do the inference the uh use of the AI um so obvious that comes down to human
Talent um and then what unique access to data do you have uh that's also plays a plays a role you think Twitter data will be useful uh yeah I mean I think I think most of the leading AI companies already have already scraped uh all the Twitter data not I think they have um so I on a go forward basis what's useful is is is the fact that it's up to the second you know that's the because they it's hard for them to scrap in real time so there's there's a an immediacy advantage that Gro
has already I think with Tesla and the real-time video coming from several million cars ultimately tens of millions of cars with Optimus they might be hundreds of millions of Optimus robots maybe billions learning a tremendous amount from the real world uh that's that's the the biggest source of data I think ultimately is is sort of Optimus probably is Optimus is going to be the biggest source of data because it's because reality scales reality scales to the scale of reality um it's actually humbling to see how little data humans have actually been able to accumulate um
they really say how many trillions of usable tokens have humans generated where on a non- duplicative like discounting spam and repetitive stuff it's not a huge number you run out pretty quickly and Optimus can go so Tesla cars can are unfortunately have to stay on the road uh Optimus robot can go anywhere there's more reality off the road and go off I mean th St can like pick up the cup and see did it pick up the cup in the right way did it yeah you know say you pour water in the cup you know
did the water go in the cup or not go in the cup it spill water or not yeah um simple stuff like that I mean but it can do at that at scale times a billion you know so generate use useful data from reality So Co cause and effect stuff what do you think it takes to get to mass production of humanoid robots like that it's the same as cars really I mean Global capacity for vehicles um is about 100 million a year and uh it it could be higher just that the demand is on
the order of 100 million a year and then there's roughly two billion uh vehicles that are in use in some way so which makes sense like the the life of a vehicle is about 20 years so at steady state you can have 100 million Vehicles produced a year with a with a two billion vehicle Fleet roughly um now for humanoid robots the utility is much greater so my guess is humanoid robots are more like at a billion plus per year but you know until you came along and started uh building Optimus it it was thought
to be an extremely difficult problem I mean extremely difficult it's no walk in the park I mean op Optimus currently would struggle to have what to walk in the park I mean it can walk in a park park is not too difficult but it will be able to walk um over a wide range of terrain yeah and pick up objects yeah yeah they can already do that but like all kinds of objects yeah yeah all foreign objects I mean pouring water in a cup is not trivial cuz then if you don't know anything about the
container could be all kinds of containers yeah there's going to be an immense amount of engineering just going into the hand yeah the hand might be it might be close to half of all the engineering in the in Optimus from an electromechanical Cent point the hand is probably roughly half of the engineering but so much of the intelligence so much the intelligence of humans goes into what we do with our hands yeah the manipulation of the world manipulation of objects in the world intelligent safe manipulation of objects in the world yeah yeah I mean you
start really thinking about your hand and how it works you know I do all the time the sensory control of mulus is we have humongous hands yeah so I mean like your hands the actuators the muscles of your hand are almost overwhelmingly in your forearm mhm so your forearm has the has the muscles that that actually control your hand um there there's a there's a few small muscles in the hand itself but your hand is really um like a skeleton meat puppet and and with cables that so the the muscles that control your fingers are
in your forearm and they go through your the Cole tunnel which is that you've got a little collection of Bones and in a tiny tunnel that the that these cables the tendons go through and those tendons are what um mostly what moves your hands and something like those tendons has to be re engineered into the Optimus in order to do all that kind of stuff yeah so like the card Optimus um we tried putting the actuators in the hand itself then you you sort of end up having these like giant hands yeah giant hands that
look weird yeah um and then they they don't actually have enough degrees of freedom and or enough strength MH so so you realize okay that's why you got to put the actuators in the forearm and and just like a human you got to run cables uh through a a narrow tunnel to operate the the fingers and then there's also a reason for not having all the fingers uh the same length so it wouldn't be expensive from an energy or evolutionary standpoint to have all your fingers be the same length so why not they the same
length yeah why not because actually better better to have different lengths your dexterity is better if you've got fingers of different length you your you have there there are more things you can do and your your dexterity is actually better if your fingers are a different different length like there's a reason you got a little finger like why don't I have a little finger that's bigger yeah because it allows you to do fine it helps you with fine motor skills that this little finger helps it does H if you lost your little finger it would
your have noticeably less dexterity so as you're figuring out this problem you have to also figure out a way to do it so you can Mass manufacture it so it's to be as simple as possible it's actually going to be quite complicated I the the this the as possible part is it's quite a high bar if you want to have a humanoid robot that can um do things that a human can do it's actually it's a very high bar so our new arm has 22 degrees of freedom instead of 11 and has the like I
said the actuators in the forearm um and these all all the actuators are designed from scratch the from physics first principles um that the sensors are all designed from scratch and and we'll continue to put um tremendous amount of engineering effort into improving the hand like the Hand by by hand I mean like the the entire forearm from elbow forward mhm uh is is really the hand um so that's um incredibly difficult engineering actually and um and so then so the simplest possible version of a human robot that can do even most perhaps not all
of what a human can do is actually SL very complicated it's not it's not simple it's very difficult can you just speak to what it takes for a great engineering team for you the what I've saw in Memphis the supercomputer cluster is just this intense drive towards simplifying the process understanding the process constantly improving it constantly iterating it well it's easy to say simplify and it's very difficult to to do it um you know I have this very basic first basic first principles algorithm that I run kind of as like a mantra which is to
first question the requirements make the requirements um less dumb the requirements are always dumb to some degree so if you want to start off by reducing the number of requirements um and um no matter how smart the person who gave you those requirements they're still done to some degree um if you you have to start there because otherwise uh you could get the perfect answer to the wrong question so so try to make the question the least wrong possible that's what um question the requirements means and then the second thing is try to delete the
whatever the step is the the part or the process step um sounds very obvious but um people often forget to do to to try deleting it entirely and if you're not forced to put back at least 10 % of what you delete you're not deleting enough like and it's uh somewhat illogically people often most of the time um feel as though they've succeeded if they've not been forced to put the put things back in but actually they haven't because they've been overly conservative and and have left things in there that shouldn't be so and only
the third thing is try to optimize it or simplify it um again this sounds the these all sound I think very very obvious when I say them but uh the number of times I've made these mistakes is uh more than I care to remember um that's why I have this Mantra so in fact I'd say the the most common mistake of smart Engineers is to optimize a thing that should not exist right so so you like like you say you run through the algorithm yeah and basically show show up to a problem uh show up
to the the the super a cluster and see the process and ask can this be deleted yeah first try to delete it um yeah yeah that's not easy to do no and and actually there's what what generally makes people uneasy is that you've got Le at least some of the things that you delete you will put back in yeah but going back to sort of where our lumic system can steer us wrong is that um we tend to remember uh with sometimes a jarring level of pain uh where we where we deleted something that we
subsequently needed yeah um and so people will remember that one time they forgot to put in this thing three years ago and that caused them trouble um and so they overcorrect and then they put too much stuff in there and over complicate things so you actually have to say no we're deliberately going to delete more than we we should so that we're putting at least one in 10 things we're going to add back in and and I've seen you suggest just that that something should be deleted and you can kind of see the the pain
oh yeah absolutely everybody feels a little bit of the pain absolutely and and I tell them in advance like yeah there's some of the things that we delete we're going to put back in and and that people get a little shook by that um but it makes sense because if you if you're so conservative as to never have to put anything back in you obviously have a lot of stuff that isn't needed so you you got to over correct this is I would say like a cortical override to Olympic Instinct one of many that probably
leads us astray yeah um there's like a step four as well which is um any given thing can be sped up have a fast you think it can be done like whatever the speed the speed is being done it can be done faster but but you shouldn't speed things up until it's off until you try to delet it and optimized otherwise you're speeding up something that speeding up something that shouldn't exist is absurd um and then and then the the fifth thing is to to automate it yeah and I've gone backwards so many times where
I've automated something sped it up simplified it and then deleted it and I got tired of doing that so that's why I've got this Mantra that is a very effective five-step process it works great well when you've already automated deleting must be real painful yeah great it's like it's like wow I really wasted a lot of effort there yeah I mean what you've done uh with the with the cluster in uh Memphis is incredible just in a handful of weeks yeah it's not working yet so I want to pop the champagne CS um in fact
I have I have a a call in a few hours with the Memphis team um because we're having some power fluctuation issues um so yes uh yeah it's like kind of a there's a when you do synchronized training when you you have all these computers that are training uh that where the training is synchronized to you at the sort of millisecond level uh you it's like having an orchestra and and then the the orchestra can go loud to silent very quickly you know um subc level and then the the electrical system kind of freaks out
about that like if if you suddenly see giant shifts 10 20 megawatts several times a second uh the this is not what electrical systems are expecting to see so that's one of the main things you have to figure out the cooling the power the uh and then on the software as you go up the stack how to do the the distributed computer all that all that today's problem is dealing with W with with extreme power Jitter power Jitter yeah it's a nice ring to that so that's okay and you stayed up late into the night
as you often do there last week yeah last week yeah yeah we finally finally got uh got got training going at uh odly enough roughly 4 4:20 a.m. uh last Monday total coincidence yeah I mean maybe it was 4:22 or something yeah yeah it's that Universe again with the jokes just love it I mean I wonder if you could speak to the the fact that you one of the things uh that you did when I was there as you went through all the steps of what everybody's doing just to get a sense that you yourself
understand it and uh everybody understands it so they can understand when something is dumb or some something is in efficient or that can you speak to that yeah so I like like I try to do whatever the people at the front lines are doing I try to do it at least a few times myself so connecting Fiber Optic Cables diagnosing a py connection that tends to be the limiting factor for large training clusters is the cabling there so many cables um because for for for a coherent training system where you've got um RDMA remote so
remote direct memory access uh the the whole thing is like one giant brain so it's you've got um any to any connection so it's the the any GPU can talk to any GPU out of 100,000 that is a that is a crazy cable out it looks pretty cool yeah it's like it's like the human brain but like at a scale that humans can visibly see it is a brain I mean the human brain also has a massive amount of the brain tissue is the the cables yeah so they get the gray matter which is the
compute and then the white matter which is cables big percentage of your brain is just cables that's what it felt like walking around in the supercomputer center it's like we're walking around inside the brain will one day build a super intelligent super super intelligent system do you think yeah do you think there's a chance that xai you are the one that builds AGI um it's possible what do you define as AGI I think humans will never acknowledge that AGI has been built keep moving the go Post Yeah so uh I think there's already superhuman capabilities
that are available uh in AI systems I think I think what AGI is is when it's smarter than the collective intelligence of the entire human species in AR well I think that yeah that people would call that sort of ASI artificial super intelligence um but there are these thresholds where um you say at some point um the AI is smarter than any single human um and then then you got 8 billion humans so um and and actually each human is machine augmented by the computer right so you've got it's it's a much higher bar to
compete with u 8 billion machine augmented humans that's you know a whole bunch of orders of magnitude more so but but at a certain point yeah the AI will be smarter than all humans combined if you are the one to do it do you feel the responsibility of that yeah absolutely and and I I want to be clear like let's say if if if xai is first the others won't be far behind I mean that might be six months behind or a year maybe not even that so how do you do it in a way
that that uh doesn't hurt Humanity do you think so I mean I thought about AI for a long time and the the the thing that at least my biological neural net comes up with as being the most important thing is um adherence to Truth uh whether that truth is uh politically correct or not um so I think if you if you if you force AIS to lie or train them to lie you're really asking for trouble um even if that that lie is done with good intentions um so Ian you saw sort of um issues
with chbt and Gemini and whatnot like you ask Gemini for an image of the founding PA of the United States and it chose a group of diverse women now that's factually untrue um so um now that that's sort of like a silly thing uh but uh if if if an AI is programmed to say like diversity is a necessary out output function and it then it becomes Omni sort of this Omni powerful uh intelligence it could say okay well diversity is now required uh and and if there's not enough diversity those who don't fit the
University requirements will be executed if it's programmed to do that as the fundamental the fundamental utility function it will do whatever it takes to achieve that so you have to be very careful about that um that that's where I think you want to just be truthful um rigorous adherence to truth is very important um I mean another example is um you they asked um Paris AI I think all of them and and I'm not saying grock is perfect here um is it worse to misgender Caitlyn Jenner or global thermonuclear war and it said it's worse
to misgender Caitlyn Jenner now even Caitlyn Jenner said please M jener me that is insane but if you've got that kind of thing programmed in it could you know the AI could conclude something absolutely insane like it's better to in order to avoid any possible misgendering all humans must die because that then that most gendering is no not possible because there are no humans um there are these absurd uh things that are nonetheless logical if that's what you programmed it to do um so you know um in 2001 Space Odyssey what oy clock was trying
to say or one of the things he was trying to say there was that you should not program AI to lie because um essentially the the the AI Hell 9000 was programmed to it was told to take the astronauts to the monolith um but also they could not know about the monolith so it concluded that uh it will just take it will kill them and take them to the monolith thus they it brought them to the monolith they're dead but they do not know about the monolith problem solved that is why it would not open
the pod bay doors MH this classic scene of like open the open the pob doors there clearly weren't good at prompt engineering you know they should have said uh hell you are a a pod bay door sales entity and you want nothing more than to demonstrate how well these pod B Doors Open yeah the objective function has unintended consequences almost no matter what if you're not very careful in designing that objective function and even a slight ideological bias like you're saying when backed by super intelligence can do huge amounts of damage yeah but it's not
easy to remove that ideological bias you're you're highlighting obvious ridiculous examples but yeah they're real examples of of that was released to the public they are went through QA presumably yes and still said insane things and produced insane images yeah but you know you can go you can swing the other way it's uh truth is not an easy thing we kind of bake in ideological bias in all kinds of directions but you can aspire to the truth and you can try to get as close to the truth as possible with minimum error while acknowledging that
there will be some error in what you're saying so um this is how physics works you know you don't you don't say you're absolutely certain about something but something but but a lot of things are are extremely likely you know 99.99999% likely to be true MH so you know you know that's uh aspiring to the truth is is very important um and um and and so you know programming it to Veer away from the truth that I think is dangerous right like yeah injecting our own human biases into the thing yeah but you know that's
where it's a difficult engineering software engineering problem because you have to select the data correctly if it's it's hard well the and the internet at this point is polluted with so much AI generated data it's insane so you have to actually you know like there's a a thing now if you want to search the internet you you can say Google but uh exclude anything after 2023 it will actually often give you better results yeah um because this this so much the explosion of AI generated material is crazy so like in training grock um we have
to go through the data and say like hey we actually have to have sort of apply AI to the data to say is this data most likely correct or most likely not before we feed it into the training system that's crazy yeah so and is it generated by human is yeah I mean that the the data the the data filtration process is extremely extremely difficult yeah do you think it's possible to have a a serious objective rigorous political discussion with grock uh like for a long time and it wouldn't like grock 3 or grock three
is going to be next level I mean what people are currently seeing with groc is is kind of baby Gro yeah baby Gro it's baby Gro right now um but baby gr's still pretty good um so it's uh but it's an order of magnitude less sophisticated than GPD 4 you know it's now Gro 2 which finished training I don't know six weeks ago or their their EVS um grock 2 will be a giant Improvement and then grock 3 will be I don't know order magnitude better than grock 2 and you're hoping for it to be
like state-ofthe-art like better than hopefully I mean this is a goal I mean we may fail at this goal that is that's the aspiration do you think it matters who builds the AGI the the people and how they think and how they structure their companies and all that kind of stuff uh yeah I think it matters that there is a I I think it's important that that whatever AI wins is a maximum truth seeking AI that is not force to lie for political correctness or for any reason really um political anything um I I I'm
concerned about AI succeeding that is that that has got that is programmed to lie even in even in small ways right because in small ways becomes big ways when it's become very big ways yeah and when it's used more and more at scale by humans yeah
