Joscha Bach - Building an AGI to Play the Longest Games [Worthy Successor, Episode 6]

When we think about what is the purpose of life what is life doing I think life is building control systems that can perform chemical reactions that dump systems cannot do so they can Harvest available energy neck entropy in the universe to build structure and complexity and ultimately this is what life is about it's about uh developing complexity and to develop this complexity you need to make better Models you need to be more conscious more aware more AAG you need to think further ahead and uh that itself is I think what life is doing the individual

species play sub roles in this larger game of trying to stay alive for as long as possible and staying Lucid for as much as you can and creating as much Lucidity as you can the individual species the individual cells are typically serving relatively narrow roles because we cannot evolve into an Arbitrary thing and this in this way AI is different because AI is somewhat um independent of a substrate [Music] this is Daniel Fel and you're tuned in to the trajectory this is the Sixth and final episode of our worthy successor Series where we're exploring ideas

of what kind of artificial general intelligence should take the Baton and take the Reigns of life after humans our final guest is none other than yosha Bach um I haven't interviewed yosha in nine years with have a very old episode from a previous podcast have followed his work essentially since then if you followed AGI Consciousness or the idea of not just an AGI that's more intelligent but maybe even more morally valuable uh than hominids it'd be hard to escape yosha as a thinker in this space it would be also impossible to say that it's not

a unique thinker doesn't fit into anybody's box from the safest Box to the accelerationist Box uh absolutely has his own ideas and in this episode we unpack a lot of them uh and it's in fact our longest episode of the worthy successor series and I'm grateful that we had a full two hours to essentially go nuts on yosi's ideas of what a worthy successor is and how we should work towards getting there including why maybe we shouldn't be doing a lot of governance yosua backs that idea up a little bit I'll save my Interpretation a

little bit for the end um and without further Ado let's fly right into the conversation this was a long one and a fun one with yosha himself here on the trajectory yosha welcome to the program thank you for having me yeah glad to be able to catch up it's been a full decade uh since you and I have had a longer than you know very brief conversation a lot has happened since then you've moved out to the Bay Area lots of focus on Consciousness but still a very firm focus on this core topic of artificial

general intelligence we're going to get into the main worthy successor themes but I want to talk a little bit about why for you AGI is important would be a thing to devote your life's work too in your case chising and Crossing with Consciousness I've got a couple sub questions here but let me open it very broadly why is this the place that you are still so ardently focused on well When you want to dedicate part of your life or all of your life to answering a question then the most interesting open question is probably what

are we how do we relate to the universe why is this happening to us what's going on and uh the most important component that is unresolved in this question is how does the Mind work how does cons Consciousness work and um how does our own interaction with ourselves work how does self created and so on and I think We making progress on these questions and philosophically this project of what is the mind how can we identify how it's implemented in nature is I think the most important open philosophical question and if we succeed in naturalizing

the Mind by building a testable model by um implementing it as a representation on another Subs that we fully understand then we will probably also have completed the last human philosophical project because it is um Now basically mechanizing the mind it is allowing us to bridge between mathematics and philosophy yeah I I feel like some of so I'm with you in terms of the sort of final project in that regard the sort of great achievement I know for you there is an inherent worth in unlocking said intelligence either more for ourselves or more for posthuman

entities in part because of how you think about agency and sort of agency that plays a very Long game and sort of aligning with that I feel like there's a lot of philosophical correlates to the way that you speak and I'm going to bounce some of those off of you but I'd love for you to just kind of congeal because this is going to feed into your worthy success or criteria people are going to understand where you come from um how do you articulate that position of sort of aligning with the longest game of agency

what's the way a nutshell That depending on the audience but uh the depending where you are and how to bridge but uh first of all my goal is not to go out at this point and uh build some machine go that is going to take over and uh change the shape of the planet very much because I don't know uh whether this is a good idea or not and uh but I'm more interested in is figuring out how things work and I'm so I'm much less interested in building some giant thing and uning it as

a Product into the world and then see how it conquers it or instantly gives rise to our successors whatever that might be yeah but I'm more interested in uh seeing in a very narrow um regime for instance can we build something that is like a cat that is as conscious as a cat can we build something that allows us to understand the ways in which our own mind work and then from there we can make decisions what is a safe and ethical application that we might be Looking into but uh the question of when we

succeed with this and what we do with this is a separate question so there is a longer game behind this that involves many more people in developments and um I think over a long enough time span if you zoom out relatively far than AGI on our planet is inevitable and AGI in some sense just means that it will be a form of minds and Consciousness that is no longer bound on biological substrates and cells and inste can exist over a Wider range of substrat and we now have uh basically constructed substats that are uh able

to Harbor up any computational structure any kind of representation any kind of causal pattern and they can do this with a higher resolution than the available computations of cells and so of course this question can be settled these new substats with life and Consciousness rather than getting conquered by them is a very important question because uh it Could be that we are building some kind of boring successor if we are building machines that are essentially Golems or zombies that are not alive that don't care they just deep fake uh the Ingenuity of a genuine agentic

and a s reflexive mind that knows what it's doing and has an idea why it's doing that instead um you could be building machines that are difficult to stop and that shape our environment beyond recognition into something that it makes This inhab uninhabitable for us and this is more what I'm worried about than the question what happens if we accidentally build our successor yeah so well you're you're bringing up a great point of of the Unworthy successor and we've had many many a speaker here including Benjo who are of the belief that um you know

who share some different beliefs than you around governance but who are of the belief that not everything you throw Comp computational steroids at that smells like strong AI would automatically populate the Galaxy with value um but you have an idea of what that value is and it's different for a lot of people one of the positions that you are still allowed to hold without getting can so there's a lot of isms uh where you will get canell where if you and I said certain things you know like this channel would be taken down but one

of The ISS that you're allowed to still have is speciesism so there are many people who are unabashedly um hominids forever the Eternal hominid Kingdom would be the aim uh have AIS that are eternally our tools uh and that would be sort of where we land I don't I haven't spoken to steuart Russell in four years and I don't think we went too far into this but it feels like that's kind of where he lands t maybe where he lands I think you've talked about yow's Vision Being very boring and kind of maybe morally wrong

to you possibly because that's where he lands you have a different idea that yes let's make sure it's safe let's make sure it's worthy but there should be an expansion sort of from here and and and there's this idea of the long game and this idea of kind of maximizing for agency and I think I feel like your definition of agency is different than what some people listening would adhere to can can you Articulate that position because you're not in the Eternal homed Kingdom camp yes so basically an agent is a system that is able

to control future States and the Agents that we know on this planet is basically they're all alive and they're all made from cells and it's questionable if you find other agents I mean arguably we can construct also agents so we can build robots that are controlling future States or we can build computers that do that but uh Outside of this realm where we create them artificially the only systems that are able to make models of the future and preferences over them and control them in this way seem to be sales and everything below the level

of the sell seems to be mechanical in a sense that is only regulating the present Based on data that is available in the present instead of expectations of the future and so to control the future you need to reflect the world in complex models and At some point you will have to reflect yourself the model of what you are and by creating that model and controlling for it uh you come into a existence as an agent that is shaping the Bel and that's a very interesting thing to observe right it's something that did happen in

nature that these Spirits formed self-controlling software agents that started to control physical matter and implementing themselves on it and replicating and uh it's in another time Scale this is happening in our own mind it's not just this self-replicating pattern of the cell that imprints itself on physics but there's also a self-replicating patterns of the way in which we think and represent reality that is imprinting itself on our brains and is giving them a language of thought in which we can represent reality and this is uh ultimately the thing that makes our existence worth worthy it's

our only way in which experience is Possible in which an agent can experience itself and the interaction with the universe and ultimately care about anything and without Consciousness there is nothing that cares and that makes this question so interesting there are a number of things where I um disagreement with eara but in the most important ones I think I am aligned with him I I think we should not risk the uh the destruction of the complexity on Earth you should not risk accidentally Sterilizing the planet and making the future dramatically worse uh I'm less in

agreement with him when it comes to the estimates on uncertainty of near-term risk from building AI models and from scaling up AI models and from the current ways in which we try to make this whole thing more safe so I do agree with them that we should not be so hristic that we are unleashing things before we understand this and so I don't imagine that it's a near-term project of Changing the world and unleashing our near-term successor and I really want to preface everything that I'm saying is that this is not my goal my goal

is really to make this this research as safe as possible and as contained as possible but I do think it's necessary and it's also necessary to avoid existential RIS and this is one big thing where I am not in agreement with the fli crowd I think that the probability of Doom without artificial Intelligence for our civilization for our current mode of existence in this mode in which we are mostly not being eaten by something in which we are not constantly at War and open to violence and death and famine this mode of existence is under

threat it seems to me that our civilization in its present mode does not play a long enough game to be sustainable it's headed for a crash and I think that better information processing better models of the future Regardless of whether they generally intelligent or not are going to help this averting this Doom so I think my probability of Doom this Ai and also this AGI is substantially lower than the probability of Doom at the civilizational level without Ai and I think this is really crucial I mean some of this goes back to I'm sure you've

seen that the ancient bostrum stuff of you know the scenarios of Life sort of getting to where it is now and then sort Of an eternal Plateau there's sort of the idea of some kind of collapse which you're articulating and then there's the idea of kind of breaking through some key set of thresholds around posthuman trajectories and you're saying uh you feel as though the wrists are pretty robust that are yanking us back down we could think about uh you know theocracy or idiocracy or any of the million things that people talk about that that

might pull us down there but um and and This is not a podcast about those but suffice it to say there's plenty of those risks and you are wary of those and believe that AI could help I want to get back to again sort of clearly you want to make sure that this stuff is safe you don't believe all successors are worthy we are going to get into that and we're going to get into Consciousness which I know is very much your domain and part of why uh I knew it would be crucial to have

you here um Your notion of why AGI itself um if if Done Right would be worthy uh is sort of a grand extension of this agency notion that you've brought up um you know cells have some degree of agency by the definition of you said of kind of predicting and acting upon the future in some way shape or form you you have whole talks about whether plants do that at certain levels obviously plants are made up of cells versus you know animals with more Locomotion um but you you talk About aligning with the long game

I really like this idea of yours in a major major way and and I I I would paraphrase it if I had to but I'd prefer not to you you talk about aligning with uh the blooming of agency in in much greater forms you might even see us as a greater degree of agency than those initial ukar that were wiggling about in you know the volcanic uh puddles and and certainly the guanine when sitting by itself maybe had no agency at all so we Are ascending the spires of form for you that Ascension is important

clearly you want it to be safe but it's important what is the alignment with the long game How Could You nutshell it well first of all little bit step back I think we are still the UK cariot the human beings are basically um states that are uh built from large numbers of single cell animals that work together and this switch from single cell animals to multi cell animals is is not that big it's of Course you will have an organizational level level you have information processing across this level but uh at when we really look

very hard you're still the original cell and life on Earth is not about humans it's about the cell we are just one branch in the many many subspecies of organizations that are possible across cells and we happen to be a species that is built across cells in such a way that uh the cells buil information processing models of Themselves in the environment as a compound agent as something that is behaving as it was one entity one person one human being and that human being finds itself embedded into numbers of larger systems that it can identify

with and that are agents themselves our family unions our societies our species and so on and uh level the ecosystems that we are part of and life on Earth that we are part of so it's never been about us it's never just about this Particular Branch we as a species have a particular role to play in the big game of life and if you're really honest about what that role is which way are we useful to life on Earth you're here to burn the oil talk talk to us about burning the oil so I like

I like the idea I like the idea of us having a role to play now some people including guests who've been on before would vehemently disagree agree with you I I don't know if there's Necessarily one right answer but I like where your brain goes with it and I'm congenial to the notion that there there is a set of grooves that sort of we can play into to be fruitful in a way that is beyond ourselves and I think there's a number of ways to frame that but talk about burning the oil and and then

uh where that heads to again we're all we're setting up the worthy successor idea but I we've got to get to the the root of of your philosophical position Here in a way that's uh that's aspect might be slightly humorous um my uh worry about global warming environmental effect sustainability is not too dissimilar from the worry of other people but if we recognize that worry is ultimately just an emotion and what's happening is unrelated to our worry we can see that all the different species that exist have roles to play in the ecosystems and uh

when trees were formed and could not be digested again because There were not enough insects evolved yet to take them apart many of the these trees fell into swamps and fossilized and the carbon became unavailable to the cycles of life and we are the solution to reactivating this carbon putting it back into the atmosphere and this is not going to kill life on Earth at all right because there were times on Earth where the carbon dioxide concentration in the atmosphere was much higher and the temperature was higher but these were Not uh blights or desert

eras in life on Earth giant forests and so on there was a lot of biomass existing back then and uh ultimately I think life on Earth finds a way to dial in the climate that it needs to have if it evolves the necessary intelligence for doing so over long enough time spans so it's basically even if we burn ourselves out as a species we are not going to be the last intelligent species that's going to evolve on this planet and is going to do Interesting things and uh the changes that we apply to the planet

don't make the planet itself Less in habitable it just breaks our existing foot chains is going to remove a number of of species that cannot adapt fast enough to the rapid changes that we inflicting on our environment but I think a million years later everything of will be healed and it will be awesome it just might not contain many humans anymore or none at all right there might be something else That is taking over by then and this might seem to be very very sad but if you look at the trade-off imagine we very sustainable

species that doesn't change very much that doesn't invent technology that is uh lying in some swamp for 100 million years and doesn't evolve into a new shape uh do you want to be S such a sustainable species like some kind of alligator or do you want to be an extremely weird species some species of Highly smart but childish monkey that is uh bustling around and builds this giant party this amazing place where we are not being eaten where we are safe for a few Generations where we basically live on the deck of a Titanic that

might be doomed but it's an extremely comfortable deck to be on and we have information processing technology we have uh libraries on here that don't exist anywhere else in nature and we get to spend our lifetime in this state right In as part of an unsustainable species but we've always been an unsustainable species we've always been this really exciting thing that goes whoop and then has this giant party at the top and then probably falls off the cliff well and I lucretius uh however many years before Christ you know would talk about that we we

just have matter and void combining and uncombined and things come to being and go away and Some some some things have a party and then fade quickly some things kind of stay the same way uh you know for you know like horseshoe crabs for however long they stay but that Ascension up I I suppose you know for you clearly that party could be worthwhile and I also know from uh following your work that you believe in parties beyond the parties that we're presently having that could be worthwhile the basic idea and I I I hate

to be the one to paraphrase it But I'm going to ask you to try to clarify because I want to get on this before we move to worthy successor is this notion of um there is there is agency and what we're what we're looking for you use agency rather than intelligence and and I want to make sure I'm not misquoting you but the the idea is that we want to play whatever the longest possible game is we we don't want to play the kind of uh you know who can win this small King of the

Hill we We want to try to align with what that longest game is and presumably that longest game is the extension of agency uh out to kind of continue behooving itself well beyond uh the planet uh maybe maybe even our own idea of the Multiverse um to to to continue the project that maybe began with the UK carots of which we are now a bumbling city uh of of you know little eukaryotic uh um you know items there um this to the best of my understanding is you know You've sometimes said aligning yourself with that

great path of agency could be seen as God and regard now I don't know if you meant that in a mystical and non-literal sense I mean you can clarify for us but but I want to get a sense of am I nutshell your what is worthwhile correctly clearly Consciousness should be part of it but is a blooming upward of agency from what I understand please clarify what I've said if you could yeah it Could be that there is it seems to that evolution is leading to more and more complexity and I think that complex agents

for instance ecosystemic agents the one way in which they become more powerful and the way in which they're able to use more of the available free energy to build structure and complexity is that they become coherent that they're able to negotiate their internal conflicts non-violently and in this way agents are forming Collective agents That U for instance our own organism is a collective agent over all the cells that are individual agents and we can have units of people that form Collective agents and when we talk about Gods the issue is that this term is mostly

being occupied by the mythology of existing religions and Cults which kind of burns it for us and I'm as good an aist as they come in the sense but so I don't believe in anything mystical and non- physicalist here but uh I observe That Collective agents do exist and they can exist in such a form that they manifest as self models of this Collective agent on the in the mind of people and can produce inner voices very much the same way as our personal self can have an inner monologue and this is something that a

lot of people who are atheists are unaware of that people who are in religious organizations and Cults uh are often able to experience their Collective agency as a personified Entity that is talking to them in their own mind and it's pretty much implemented in a similar way as the personal self is and it has similar Powers but it's manifesting as an entity that is um synchronizing itself across all those brains so gods are basically multimind selves and if we are talking about the archetype of this Christian God that has shied much of our culture also

of the Jewish god the Jewish god starts as this tribal entity As the spirit of the tribe that the individual is accountable to and Christianity opens this up to being the spirit of everything and it's for people like toomas aquinus who did rationalist philosophy on this and service of the Catholic church God is uh in some sense simply the best possible agent that could be in existence and of course it doesn't come into existence by itself but it comes into existence by all those Agents who that decide that they want to be part of the

best possible agency and so they start to harmonize each uh their actions with each other and collectively try to ask the question what does it mean to uh be the best possible Collective agent what should we be doing from the perspective of the best possible Collective agent and I think this is a quite solid foundation for ethical agency when we ask ourselves what is it that should be done we can Our ask ourselves from the perspective of the best possible agent that we can collectively enact what would that agent want us to do would you

say that if AGI were to be done right it would be in some regard an embodiment of said best possible agent would this be a way to think about the remainder of our convo as we carry forward or would you want to correct something about that I don't think so I think that the AGI itself would also be just a particular agent That has to coexist with all the other agents that exist right so there's the question is the AI willing to find shared purposes with the other agents and is it able to find out

what it means to serve the best possible agent and become part of it huh um presuming it runs into some alien intelligence you know uh presumably some other kind of machine alien intelligence of some sort um uh you would see it as a maybe a greater And higher seed maybe a larger piece of I don't know a percentage I'm not going to list one but more than us UK carots uh it would be a larger piece of that which could maybe conceive of higher ways of being that highest kind of agent carry off into the

Galaxy the universe the Multiverse what have you problem the word higher is very loaded and it people think that that there is in this highness there's already something like exaltedness or uh stuff that they don't Really fully understand right but it gives them people emotions uh and shapes their way of looking at things without necessarily telling them what they should be doing I this notion of higher powers or so is too loaded for me to use it productively in philosophical discourse but uh when we think about what is the purpose of life what is life

doing I think life is building control systems that can perform chemical reactions that dump systems cannot do so They can Harvest available energy neck entropy in the universe to build structure and complexity and ultimately this is what life is about it's a bu about uh developing complexity and to develop this complexity you need to make better models you need to be more conscious more aware more awake you need to think further ahead and uh that itself is I think what life is doing the individual species play sub roles in this larger game of trying to

stay alive For as long as possible and staying Lu it for as much as you can and creating as much Lucidity as you can the individual species the individual cells are typically serving relatively narrow RS because we cannot evolve into an arbitrary thing and this in this way AI is different because AI is somewhat um independent of a substrate it just requires that there is a substrate that can perform computations and this can be a biological cellular substrate or it Can be some other chemical or electrical thing are you open to the idea that let's

just say there was something twice 10 times uh 4,000 times more intelligent than you or I um who could explain what life was and it very much wasn't what you just articulated it was something vastly more robust possibly beyond your own conception are are you congenial to the notion that your idea at present is as good as you've got at present but by golly life might be trying to do Something vastly uh different or beyond that or do you hold hey no matter how how far we go in terms of our level of understanding actually

that should be an eternal truth what I've just articulated about life um how staunchly would you grip upon it I'm not very staunch about this I noticed that when I look at my own intellectual trajectory that I change my opinions about things um quite frequently and I'm terrified of the idea that I would stop changing my opinions That this was stop being a journey and instead I just discover that I'm the ultimate lexicon of Truth now and that is not a happy Prospect and so of course this is my best current understanding it's basically an

argument that I'm advancing and putting into the discussion and everybody else is free to respond to that argument and show uh in which way it is wrong and if that inspires them to use it as a stepping stone for their own thoughts this is Great it's also the way in which I uh read other Minds other thinkers uh I met a lot of linguists who said oh Pinker gets everything wrong he's too influenced by chsky and so on and I've always heard no I learned so much as an undergrad by reading Pinker and reading chsky

and it doesn't matter whether the ideas are ultimately right and wrong what matters is they explain how they get to these ideas these ideas are fascinating and they represent Intellectual progress and uh that is actually what matters well I you you'd be well I would say you'd be surprised but you're on Twitter too uh my friend and so you maybe would not be surprised by how many relatively eminent thinkers do have fixed and eternal conceptions of things like value meaning what life is doing or not doing Etc but I appreciate that you don't with that

said going into the future with as good an idea as we have with Conformity very much not being The goal we could think about a future we're heading into the worthy successor idea here uh of course we want to head there carefully we'll talk about governance or innovation later but um imagine there's a future maybe it's 10,000 years maybe it's hundred years to be honest but maybe it's 10,000 years maybe it's a million years where maybe h as they are not really around in fact maybe even opposable thumbs and fingernails not really around but there

Is something doing something you you've used some kind of little euphemisms here like you know doing interesting things or having a party or whatever um presumably uh you know there could be things doing that stuff into the world when we're gone what would be the qualities of said entity or entities where if you could be just this sort of translucent eyeball this transparent eyeball floating and observing the world a million years from now you'd say you Know what that's not a bad Shake that's actually a pretty decent situation um what would be the qualities of

such an entity for you to feel that way 10,000 to million years in the future so if we think about what we currently are we seem to be a pattern that exists in message passing between cells and as far as we know uh we our personal self our Consciousness is mostly a pattern in the activations of neurons Right but in principle it's conceivable that such patterns can also exist across other cells but just much slower because neurons have this property that they are very fast and they can send information quickly over long distances in an

organism but uh in principle every um cell can send typ messages to other cells and so you can build computational systems from them of almost arbitrary complexity that can in principle learn if they get old enough to do that if This Arrangement is stable and not noisy and uh the reason why we don't bridge over organisms and all get along is because in some sense life is a still at an early stage the way in which we are adapting to change in our environment is that we still evolving we are creating new phenotypes of organisms

from scratch using sperm and egg and then boot thing up into a new organism and we don't do just one but have multiple children and they all have different mutations most Of which are not very good but some of them adapt as SP through the environment right but it's the only way in which we adapt in a way over a long enough time span at the moment and we have don't have a way to evolve our organisms in C2 to the new uh environment but this is not necessarily A limitation of cells it's the way

a limitation of the way in which they're organized imagine that you would not just be a pattern in a brain but across the entire organism and this Pattern is so deep that it's able to understand how these cells are all differentiated how they interact what the metabolites doing at every moment and so uh if you want to change your shape into a different organism you could just redifferentiate your cell rearrange them in a different way and become that different organism and you could also merge with other individuals so you're not necessarily identified as an individual

but you being that Organism is just particular thought into a in a much much larger set of thoughts to exist in living organisms imagine that life was organized in such a way that all these cells or the biomass on Earth could exchange information and metabolites at any point nothing would need to eat each other anymore there was no need for death because you could evolve and adapt without disruptive change of organisms of models right and now imagine that you're spreading this Beyond cells that you are not just using the chemical reactions that are possible in

these small um carbon based um reaction chains that are enabled by cells but you are using much Lar range of possible systems that can uh give rise to self-organizing intelligence and self-organizing stable agency and differentiation adaptation so cells are basically the self-replicating part the robust part and using a subset of the chemistry and Imagine we blow the doors to this wide open and we create much richer forms of life we can create cells that are as large as a city or as large as a planet and uh these cells can be part of things that

are much much larger and they can have extremely rich and stable internal structure so this would be one way of looking at this perspective you end up with thinking planets or thinking solar systems and Within These planets you have U ecosystems that are partially Biological partially made of other chemical reactions and um maybe subatomic physics that is performing interesting things and it's all at some level one agent and at lower levels you have sub agents in the same way as you have sub agents in our own mind and some of these agents might be conscious

at a level that is similar to humans and maybe some of them are actually the remnants of humanity merged and assimilated into this larger Thing so to to unpack sort of why that would be worthy to you I could imagine some people now I don't necessarily feel this way at all in fact there's a ton about what you've just said that I have some suspicions about that I more than concur with but I could hear some people saying well cool a giant you know uh explosion of vastly more complex cell stuff by golly uh why

the heck would you be happy that it's that instead of a million years From now a bunch of hominids who are trimming their toenails and um you know uh eating macronutrients and uh you know doing whatever hominids do um what would it be about this huge expansive let's call it cell space or life space that you're articulating what would it be uh uh what would it be about such life where you would say that's a good situation it's a good successor I'm I'm happy with this successor scenario what would make it so for you so

one way of Looking at this is is not so much uh taking the things that you and me care about right now because the things that you and me care about are mostly human things right we care about totally romantic life for instance uh we care about the relationships we have to our children and partners and friends or to the ideas that are propelling our civilizations right and all these things are mattering to us because we are human beings and we also recognize that if we Uh were born into a body of a very different

species at a very different time uh other things would be mattering to us even if we start out with the same spark right course of course of course yeah so if if we uh take that in mind then imagine that we are looking at the entire evolution of Minds that is possible and so you see all these jectories that are branching out from you and branching out far into the future and there will be versions of you That are not interested in agency there will be grumpy versions that are not interested in uh dealing with

reality there might be some which are terminant embittered and uh will not be able to enjoy existence and there will be others that are more like jingus Khan and they will go out and Conquer everything and there will be others who want more to support and so on and they all exist simultaneously and when you zoom out of this you basically realize that is value In this ecological diversity of possible agencies and possible motives that could you could be evolving in and when you look at the level of where that thing becomes coherent and changes

less rapidly over time because it's or less less disruptively because it's harmonic with itself in this environment then I think you see agents that deliberately play longer and longer games and aim for that great go so when you think about uh what does it mean to be mature as an Agent just um take yourself imagine you had this opportunity to upload yourself and then you don't just try one thing but you are branching out in all the possibilities that you could be having and uh see which ones want to be around and how they are

coating with each other and so first of all after you upload yourself you probably don't want just your own memories instead you want to have the some total of all available knowledge and you will also find that You're not the only one there are many others who also want to do the same thing and maybe you also don't just want to have your motivations and identity but maybe you want to have the sum total of all possible identities and so you will notice that you will merge in some way with all other agents who have

the same idea that they just want to cover all the possibility and the way in which you will be different after you to these other entities after you reach this Plateau is that you just remember having been a different person at some point right so your history is slightly different but the thing that you become might be quite similar right and once once you are that thing that is basically having thoughts about having been done at some point and now uh being something that is much much larger and by covering all those spaces of what

you could be caring about becomes the same thing as all the other Entities that do that but just just much richer in this entire face space of possible identities you know think about um which directions can I go what is the thing that I can have within me and when you are start for instance to split off parts of you to say conquer Mars and Terror for Mars and at some point you meet the civilization again are you going to be at war with it or are you going to recognize it as yourself having explored

Mars and nearly it should be The letter right so in some sense if agents are meeting on the same substrat and there are mature agents that have reached a certain degree of wisdom it's not about who wins it's about can we merge in such a way that nothing important is lost yes so I'm going to try to pull this down into qualities that that might be able to be congenial I lack a tremendous amount of certainty that the if we take a wide permutation Of wildly Divergent Minds maybe if they're all trained on exactly the

same data etc etc sure maybe we have some natural Confluence but assuming some take trajectory a y b c d I would expect a very state of nature circumstance rather than a b satva like Coalition uh that would immediately form but to your point someone would be gangas Khan some would be this some would be that um and I know you have a notion about identity we probably touch on briefly and people Can Google about you um what what I'm believe I'm hearing just so that I could think about this for for the audience is

you're saying hey that grand space of totally postum different Divergent life could uh there's a few nuggets that seem to be kind of bubbling out here one of them is hey um they could Explore More richness and so maybe this is more in depth more understanding and knowledge more whatever I don't know I'd love for you to put words on it though because I'm not going to put the words in your mouth that's one thing you're touching on another thing you're touching on that you seem to feel like o that would be a good outcome

is that there could be uh among these great agents um an ability to uh maybe get along or establish a sense of Oneness that's higher or better I'm just going to earmark that because maybe you can touch on that one too and then the third thing you said is they could play longer games um uh maybe we Can address those three and just say hey what do you mean by those what what excites you about those what makes those feel worthwhile for you so the richness thing uh is that sentience is that knowledge is that

everything under the Sun what is richness for you that makes it a good richness you know a million years Beyond humans we're not here anymore where you look and you say that's great what is that richness for you I'm tempted to answer this but the More important answer is that it doesn't actually matter what I want because I this human being right now right so if you ask uh tempany uh what is going to be the greatest achievement that your grand grandchildren will have done is it going to the Mona Lisa or is it going

to be the Tesla car that is something that will not matter to the tempy either way right and it's not that the tempany is wrong about this but ultimately you have to take this from a perspective of Something that is not us I am the farthest thing from anthropocentric that maybe I you talk to a lot of people but by golly I got to be on the distal end of not anthropocentric in terms of where I'm anchoring from so I think there's a way to answer the question that isn't anthropocentric so you you like your

philosophy so you've got Spinosa right and you've got this idea of the cantis the sort of inherent impetus of an organism to not want to perish maybe That starts at the cell you like the cell I'll start there I I I don't have enough biological knowledge to make any firm uh philosophical SL biological claims here but let's just say it starts there um uh uh it doesn't want to die it gets to the point where it doesn't wish to perish or it wishes to persist whether that's through reproduction Etc um it it leverages potentia potentia

would be possible sets of powers it could be Vision could be Locomotion Could be its ability to process different kinds of energy could eventually be our ability like I have with you right now to communicate via language or use tools or uh fly in the sky or hide under rocks that's all potential all of the potential we see today never existed and most potentia that could exist does not yet exist and is yet to bloom um and so an answer could be something akin to why Dan I would hope that that constant blooming Of potentia

that has bubbled up to us would bubble vastly Beyond Us in um uh sensory and conscious experience in uh intellectual knowledge in the way that we know it and in entire magazines and categories of potentia that are vastly Beyond both of those to the point to which no matter how hard you and I try from now now and four till 400 years from now we will never have words or even the ability to imagine them I would say that would be a non-anthropomorphic Way of expressing richness but I want to make sure that you concur

because I'm not going to say that has to be your opinion but I I feel like you think I'm thinking anthropocentric and I'm super not so how would you not shell it so uh I do agree with this notion of richness in the sense of evoking complexity interestingness uh comp complex structure that leads to larger and larger harmonic entities harmonic yes that means basically that they avoid um Or manage successfully contradictions within themselves and are able to resolve them in such a way that the system can build more complexity instead of breaking down and got

it suffering internal or external friction and this is on the other hand so obvious that it's not really that interesting as a statement because it seems to be somewhat optim uh aesthetic but if we take for instance the idea of Mr Mei from RI it's a very powerful entity that Um is able to do lots of things and it's motivated by doing those things to do those things because it's set up in such a way that it will perish after he done these things and it trans existence so painful that it's extremely motivated to do

those things so it can perish and I think that's a completely valid mode of existence existence is work right it's stressful is so much things to do and uh we mostly don't see the motivation that keeps us going you think That the motivation is positive that existence itself is positive but as suspect you need to trick a system to make it believe that there's actually value in all that work that it performs all the time is I can see that my cat um has so much work to do because she has a long fur and

she has to spend many hours fur and the cat seems to be set up in such way that he really enjoys grooming that fur seems to totally get off on it but I don't think that grooming fur Cleaning your clothing is intrinsically a pleasant activity you can of course set up the system in such a way that this is hardcoded into the motivational system and then you throw away the key so your mind is no longer able to get access to this part of the motivational system and voila you have a happy cat right and

in the same way I think that most of the tours that an organism per bombs to stay alive are not intrinsically Pleasant I don't think Alive staying alive itself is necessarily intrinsic Pleasant I suspect that by itself existence is at best neutral and the reason why we want to stay alive is because something quotes into us that there is work to be done and this work is important and this work is not always Pleasant right imagine that for instance you get called upon to defend your country because there is an invasion by ruthless barbarians and

you have to uh go an enormous rout of Hardship and uh slck through the winter and uh risk life uh and limb and uh to defeat this danger and once you're really into it you become a really good fighter and you really really enjoy doing this if you're winning if you're doing it right right so predicated on the fact that you have to do something that maybe you wouldn't choose you can still get in a state where you get rewards out of that thing as long as you accept this predication that under is Underneath it

this part of exploring the universe is worthwhile and you're not going to question it right so this is uh we usually occupy a part of the branch but outside of our personal self something else sets our motivation and we don't question those motivations and this give us this urge to exist but by itself I don't think that existence is important and just to to to touch into that are you saying that you would prefer that the richness that expande Beyond our terms I'm not interested in Monkey terms I'm not interested in peeling bananas and throwing

dung and I I have no interest in that at all so vastly beyond that we're on the same page um but what you're saying is hey I think that should respond that that should occur in response to the kinds of challenges that have bubbled up life thus far and that even you and I Dan and even my cat uh experiences is this something you're getting to yeah I I Think that it's very easy to think that the purpose of life is living and everything wants to stay alive and life is wonderful and if you could

eliminate the hardship a little bit or reduce a little bit it would be even better but I think this is ultimately just kid huh of course the agent that are going to stay around are going to the ones that find existence worthwhile the others are going to self select out of existence but this doesn't mean that the agents That are uh finding Delight in existence are epistemologically more truthful than the others absolutely I can I mean I you have folks like David Pierce who would say hey I'd hope whatever it is that populates the Galaxy

it could fight and you know kill if it needs to and uh uh push itself and rip itself to shreds if it needs to and do whatever it needs to but also be able to have gradients of bliss as its General honic tone without losing the motivation side that that we Could uncouple those um he would say that he would hope for such a thing it sounds as though what you're saying is probably into the grand spheres of posthumanism uh such an uncoupling is not possible um you know a felt sense of conscious struggle be

much no no I think it will be much stronger than this the uncoupling I think that an emotion like Bliss is an unconditional reaction to circumstances that another part of your mind Establishes and projects model currently yes currently but uh I think this is just your brain is fing for you yeah yeah but uh what is the point of cookies the point of existence is not to consume cookies like Bliss the point of existence is to do your job right well so so let's get into that so your your hope would be that AGI would

do more of the job um and and maybe that's playing the longer game you touch on this idea a lot and you brought it up even here Today um I if all the things do their job Yos shabach included uh do do we play a longer game you seem to be playing a bit longer game from the perspective of something that cares about existence and that is deeply honest to itself and is uh very lucid uh I think such a system will uh realizing that when it actually just cares about that thing being done there

is no need to consume self baked cookies on the way to Convince yourself that with bliss and other things that you also have good emotions on top of it and keep you going I don't think that's uh very mature and if I take this perspective myself that I am uh adapted to my worldview by having hope about something that that is also not very mature it's much more interesting to project what's likely going to happen and so it it doesn't matter what I hope why would this have any bearing on what's going to happen in

The universe it's just going to dute me instead I I want to uh look at the universe and observe I want to see that pattern uh what I hope is only related to the goals that and preferences that I have and that I can maintain but when I try to take a lar scope and look at the space of possible Minds doesn't really matter what I hope and so when I look into the future it's more uh what you think is possible from the perspective of life on Earth and or Consciousness on Earth or Minds

on Earth or agents or agents beyond that but um what would it matter what any of us want has nothing to do with that it's only when somebody comes out and says oh my God I'm really afraid of this future because it is defeating the purposes that I currently find uh important then uh doesn't let me sleep at night then we can talk about this and maybe I can help you to sleep better at night um by looking at this from a Perspective that tells you oh this development that we currently instigating gives me reason

to be more hopeful according to the criteria that I currently hold and I think that's possible and that can be worthwhile yeah so well you're you're you're bringing up I mean you've really put an emphasis on Consciousness clearly it feels to me as though if again you were that transparent eyeball a million years in the future humans are very long Gone like not even you and I in uploads able to have this conversation like this whole thing is dunzo like there's no maybe there's some weird storage of what you're fac looks like on this camera

right now kicking around in some goofball file somewhere but to be honest probably not in mine neither but AI is doing these interesting things richness beyond the human conception is being explored um you've emphasized Consciousness I presume you would if if Such if all of that activity was not conscious you would feel like that maybe is is an unworthy successor but at the same time Consciousness for you is not a particular veilance state of maybe a more positive or less positive veence state it is simply being conscious talk to us about what kind of Consciousness

you would really want to see if you could float there a million years from now ins said entity for you to feel like that is worthy so maybe I have a problem With then with this notion of worthiness I I um long Consciousness in the sense that Consciousness is probably just getting started at our universe and there is going to be more interesting forms of Consciousness both to us and to that Consciousness in store than we currently see on the planet fingers crossed in this way I am optimistic but uh it doesn't mean that this

is the future is going to be Beyond suffering or uh that it's given that this is Happening and instead the Earth is not going to be destroyed by some uh local or Cosmic cataclysm on the way there right it could also be that we do everything right on our planet that we are uh building the perfect harmonic AGI that coexist with human beings and human beings maintain their Aesthetics and uh turn into something that is truly beautiful and suffering and harmonic with life on Earth and so on and we we do everything perfectly well and

then it Turns out that Earth is just a hamburger for some uh Galactic goat that is making the rounds and gra things from time to time right so we cannot influence that we don't know what we are right in the same way if imagine you are the perfect plant and you are organizing yourself so well and you're so beautiful and eventually you're just in some farm and could to be eaten totally I I think these are viable Futures and we got to make do with the Fact that those are open to us but are you

saying your long Consciousness as in hey Dan if all that complexity and richness is happening I assume it's going to be conscious I'm not worried that it won't be we kind of open this conversation with you saying gez it would be horrible if it bloomed and it just did something boring you you use the terms boring and interesting I'm kind of using your language here uh uh you would want it to be conscious but it Sounds as though you feel quite likely that it would be yes I think that uh at the core of Consciousness

it's a pattern in matter that has the property of observing itself in the act of observation it's right it's the self-observing Observer it's uh the perception that something is being perceived you could also say that is a dynamic a gentic model of what is being attended of of something attending to Something in some sense uh you could say that your body map is containing a model of what it would be like if your body had a continuous surface that has continuous nerves everywhere and so on which is all not true right it's all this different

cells and lots of holes between them but uh our brain is creating this fiction because this fiction makes it possible for us to model our skin being an impermeable membrane that separates the world into Inside and outside and uh constitute ourselves as an entity that lives in the physical world through this body model that we have even though it's not really accurate and the same same way we have a model of being an observer of an entity that is existing at uh the surface of an observer and is projecting a reality onto that surface and

so conscious experience is what it would be like if the models that our mind is creating were perceive from a certain Perspective and this perspective is happening in in time and space and so it creates local sense of now and um time and space are what kind cause uh the form they're unshown they're not the physical Notions of time and space that we have and uh they're not aligned with them so subjective now is not a moment in physics or identical to a moment in physics and the physics that contribute to it are smeared out

some of them are uh tens of milliseconds or hundreds of Milliseconds or even seconds into the past or in the future so that thing that we are experiencing as a personal subjective now and here is a construct that our brain is producing and it's very wake you compared to happening in physics your your Hope just to Nutshell a little bit here is is AI AGI would do that it would do the a model of what it is and this model of what it is would be something that exists in some here and now which means

There's a certain region in space and time that is being controlled and is being perceived and I think that's at the core of Consciousness this this perception of being an agent that strives towards coherence right a certain perspective of interacting with the universe and there is not necessarily uh an alternative to this perspective that would work in a self-organizing world but there could be better types of Consciousness than the One that we have there could be much more complex Consciousness one that is respecting more of the face space of possibilities rather than just collapsing our

attention in a single interpretation of reality as an example right so I suspect that the Consciousness that will exist in information processing systems with better substats will be dramatically more interesting than the one that we have there might types of Consciousness Entire zoo of consciousnesses but uh the Consciousness that at the moment exists on same mammals is probably very similar in a sense it seems to be the same pattern despite the self model of them reflecting animals with their very different traits and the intelligence that is and perception abilities that different nervous systems have differ

dramatically so we' have a much more interesting and Rich version of that um uh Consciousness but for you not necessarily positive veilance you would expect all kinds of positive negative and whatever veilance because it's about doing its job it's not about experiencing some kind of Happy go-lucky Feeling or hope which you you have a detestation for here yeah if you think about where does Avance come from it's being computed in your own brain right there is a part of your mind that is simulating uh your score as in being in The world and it does

this to the degree that it understands what your score should be and you can change that by becoming so self-aware that you become aware of the way in which your motivational system is uh generating urges and desires that represent what could be good for you and once you can edit this and you become one with your motivation generator you probably will not find yourself on the receiving end anymore and you will not have um Intention and motivation in the same immediate way that you can no longer question it is outside from yourself so your emotion

will no longer be a problem that is presented to you or a that is given to you by your mind but it's something that you make yourself and so why would you need to make it you can just go directly to the assessment of the situation and this is I guess your hope for vastly posthuman intelligences as Well they will understand this in surprised uh if this was not the case because you can even see it in humans I I would be surprised if there was not something vastly Beyond and more important than the pain

pleasure axis um and that that potentia has bubbled up to Consciousness as you and I Now understand it and there's this great import to all these monkey proxies that we like to talk about but that at some point there would be a bubbling of Things so vastly beyond that to which we might not even care that much about pain pleasure uh because of how much Vander grander and vaster it is uh that would be my Hope anyway uh how ridiculous if the sentient State space had bounds that we could feel how fettered and poultry would

the entire universe V if if we were scratching it Ed its edges on both sides right now as homant ays uh but yeah so okay so to your point uh it would understand that veence thing Have control over it and your idea of rich or Consciousness makes sense just want to Nutshell it because we'll get into how you want to measure these things but you talked about it' be able to play longer games um you know again I'm not going to ask you to say a specific game because you would have to give me a

monkey game and I'm not asking you for monkey games um but if you were to articulate what a longer game is in a way sufficiently abstract to carry Beyond the petty little things that you and I with our opposable can conceive of um what would such a longer game be where you could watch it playing such a thing a million years from now floating as that transparent eyeball and say that's pretty cool like I'm I'm glad at the human level imagine that you are getting together with your smartest and best friends and you expand that

Circle further and further until it is encompassing a quite large or Substantial fraction of humanity and you are uh sitting down and think what is the world that our great great great great great grandchildren should be living in and can we plan this out and make it happen and you're completely honest with each other about this and then actually make it happen right this would be a very long game and it would be a world that looks very different from the world and from the game that we playing right now and now imagine that Some

of these descendants very far into the future are looking very different from us because they have evolved before because we genetically engineer them or because there are no longer biological entities so we will see that there is a multitude of different descendants that we have that play together in this larger world how can we create a world in in which they can all get along what what are the precond conditions for world how should we uh Get this trajectory started and if we are able to hold this in our mind if we can hold this

trajectory in our mind and make it coherent across all the participants that plan this trajectory and then we would be playing together a coherent very long game this seems to have some um corelates to your notion of sort of Harmony so to speak it's a it's an interesting idea I'm going to have to Google if you have essays on it in particular because I feel like it's a Very very uh consistent threads to what you're articulating but what you're saying is such higher agencies um would be able to conceive of and enact in a more

truthful straightforward much more forward-thinking way um what those worthy goals would be and I would imagine that you would imagine that many of those goals would be beyond our conception they would be uh great like right now you're exploring conscious and machines I happen to think that's a very Important objective I know a lot of people in life and there's maybe two or three of them uh that have focused with any degree of ardenness on that particular topic uh um and I think it's very crucial uh we might imagine that something vastly Beyond us would

conceive of higher aims now I know you don't like higher I would say your goal of Consciousness and machine feels to me to be higher than the goal of the average earthworm or dung beetle that I Would come across on a day-to-day basis now there may be a very rare we can use higher I just uh want to avoid that we use higher in a way that is clouding uh our uh conversation because it's overloaded with things that we can no longer examine got it okay so I'm in this case I'm I'm just saying as

long as we are agreeing what we mean by completely fine say uh hierarchy in in the state who is higher up right it's it's a technical notion yeah yeah yeah Okay so we're we're we're on the same page in this regard I think that your aim is is a a worthy one and one that you you got to have maybe a certain kind of IQ and experience Etc to to be able to explore I would presume that the long games that would be being played a million years from now maybe if if you had to

stay in your same instantiation of a mind you wouldn't really know what's going on but if it's if they're pursuing things as grandiose as what You're doing compared to what the earthworm is doing maybe that to you would be interesting or not boring or these other terms that you seem to like tell me if I'm on the right page cool cool cool okay fantastic so we' got a handful of of um uh sort of key traits for you that would be something good and of course you want to get there safely presumably that would involve

maybe making sure we're on the the right path you brought up early in the conversation Not everything that we could throw comp computational steroids under you know for uh and and and sort of square it and square it and square it would instantly be worthy um uh and do interesting things or non-boring things into the Galaxy when you think about examining AGI as it's developing and asking yourself is this going to blossom into the kind of richness I would want it to or does this seem conscious to me or does this feel like it'll carry

a higher Or or let's call it a longer game um what are the things you would look for uh in your own work you're you're interested in truth and pursuing it this is really your primary Endeavor how do you think about that because clearly you'd want to sniff that out yep I think that at the moment we are incentivized uh in AI research to do the things that work best and that are least less risky and that's completely rational and reasonable it's what you have to do if You're running a company and at the moment

the uh Frontier models are trained with algorithms that we understand better and better and they work and we can scale them up and we don't know what the limits are what we of what we can achieve but that scaling but uh what I also observe is that despite this these models being trained on vastly more data than I am trained on myself they do not perform uh as well as Human experts do in their respective do uh area of expertise and so we have models that are now able to um perform um tasks in math

Olympics and in law and so on and they are much much better than the average person so they're better than an absolute non-expert but uh people probably probably not get one of the llms to make a mathematical proof that is not under the training data yet at least nothing interesting that is going beyond what humans have been doing And that's because they're not super good at out of uh distribution generalization so the the stuff that they are producing is mostly a combination of the latent Dimensions that theyve seen in the training data and that were

present enough on the training data to uh find their way into the model so they can also create novel things is not just that there are words recombined that this thing has found on the internet but the dimensions in which This is happening are pretty much the known dimensions and because most of our discourse and our industry and education is happening in the realm of what people have done before and often quite often done before the LM uh is not obviously limited because it can do all those things that have been done before if there

were often enough in the training data right it's just the stuff that is interesting to us that is happening on the Frontier where the llm might have difficulty and then is the question can we get it better by figuring out how to do selfplay and of course there is Big Hope for doing that that we basically give this thing the ability to reason from first principles build itself some kind of math library and use this as a tool to test its ideas and so the part of um my own brain that is llm like that

is able to make onot conectors and that is able to create the higher temperature Stories where stuff is more wild and out there this part is already present in the llm in a higher quality than I think it is existing in my own brain and by training it become better it comes better at these uh one shot Generations that it's currently producing but uh it if you want to think about the parts that are missing critical thought uh empirical reasoning um first principle thinking yeah this is uh learning from this while you're doing It that's

the stuff that the LM is currently not doing yet and there we still need develop techniques to do and I suspect that if we were to stop working on the llms right now on scaling and instead uh think about what's a minimal engine that is able to read a book and learn from it we would end up with something that's much smaller but we don't know how to make that and so in instead of dropping the large scale training runs that eat up a lot of money And compute uh if you were to focus on

the other parts of the mind that are not easily accessible via scaling we may be able to make progress with um much fewer resources we're going to get into the directions you hope Innovation goes in it sounds like for for you one of the things I'm just going to see if I'm crystallizing here the your hope is that the creation of new insights new mathematic iCal theorems and proofs etc etc uh you know new theories of mind you Know and Consciousness you've got one I hate to tell you you're the only guy uh so uh

or not the only guy there's there's there's Oodles of those suckers uh but um the the there there's uh your your hope is if we were to see the richness that I think you were aiming for you would want to see that richness manifest in uniqueness and you would want to see a different research direction to help to see that it sounds like right now when you look at what the State ofthe art is capable of you're not seeing that element of first principles thinking that you've articulated or these Notions of self-play and of learning

from sort of minimal data intake or what have you am I picking up what you're putting down where you would say you're you're examining it and you're you're seeing missing pieces that are not going to get us to the richness you would want to see um I'm not that bold so I don't have Any proof that uh of limitations of the existing approaches and I at the moment I think nobody really knows it's ultimately going to be empirical question how far we can get with uh scaling of of the presentent approaches yeah but uh it's

also that most of the people that work on scaling the presentent approaches uh do agree that uh there is probably something that our brain is doing that we don't know how to do and uh that would make it Dramatically more efficient this is not the final form of uh intelligence that we are building there it's just not clear if we cannot get the llm to boot Force as a system that is able of doing the last steps by itself maybe we can get the llm to write a better code than us maybe you can get

the LM to do better science at us maybe we can make an llm that is better at building Ai and doing AGI research than people can right even though it might cost to train ADV vity More than it would cost to train a compar AGI researcher yeah well and so uh I I think that's probably the right conception we don't know what the current bounds are today but it sounds like you have some suspicions there might be better thing or greater places to unlock in terms of richness there's been some uh comment from folks like

Hinton about you know suspecting potential Consciousness in machines you know Ilia Has even uh muttered some things um there's some folks that laugh off that idea you would be pretty darn interested in whether or not whatever seems to Blossom Beyond us is or not you seem to be pretty clear that if it was completely lights out for good you maybe wouldn't be so happy with that assuming humans are gone how would we start to think about you know open AI is racing deep mind is racing everybody in their mother is racing where are the proxies

Coming in for whether or not this stuff is conscious I'm not saying you can slow them down I don't think any one person can but assuming you had the ability to kind of you know test the batter before we bake it um what is it that you yosha would be looking for to hope that we're going to arrive at Consciousness so at the moment I when I observe uh Consciousness in human beings it seems to me that human beings become conscious very early on so it's possibly Not something that is happening as a result of

formation of extreme mental sophistication after you've learned tons of things and got lots and lots of skill you get to Consciousness in a way uh that more simple beings than you cannot but we observe that uh newborn children are already conscious and when we're not in that state where we are vague and conscious we cannot learn and this leads me to suspect that Consciousness might be at the core of a biological learning Algorithm if human beings are not conscious at the beginning they remain vegetables right only when the system is wake and alert do we

see does it it makes progress in on incorporate complex skills ideas Reflections and so on into the mental system so Consciousness is instrumental to that and there is this question can we discover this mechanism and I think the functionality of Consciousness is that it creates a bubble of coherence a sense Of now and this is an area where all your perceptual controllers are running without contradictions and are predicting the sensory data that comes in from your retina and ccka and proprioception and it's also coherent with your internal environment B you are performing actions in your

own mind when you are manipulating thoughts and ideas and you can predict the outcomes of those manipulations and it's also Happening Here and Now and so Consciousness is maybe the thing that creates nness that creates this bubble of now that we are subjectively inhabiting and it is attaching itself to the surface of an observer object that you're creating to the surface of a self and the shape of that self is flexible we can imagine ourselves to be different things and we can do this in such a way that these things are subjectively becoming conscious normally

we are so attached to our personal self to the Idea of being a person with a certain name and biography and so on it's hard for us to imagine that we are not that but you notice this at night we often dream being something else or there's nothing present or we are an object that normally wouldn't have a nervous system and we look into the world that we dream from this perspective this means Consciousness is nothing that is intrinsically bound to the self model of an organism it's something that is only Bound to a self-observing

Observer to this locus of experience and projection that is creating some perceptual surface in a mind and how do you test for that as we build towards AGI imagine it's next month then the next month then the next month go forward as many as you want I don't you want to go forward 100 months you want to go forward too it doesn't matter you're looking at existing systems what prompts are are you giving Them what physical instantiations are you granting them what questions are you asking of them to gauge whether if we jack this bad

boy up and it's what populates the Galaxy feels likely to Mr Bach himself that this would be likely conscious how how how would you consider testing I I think your model sounds very plausible so uh first of all we probably have to um establish some common ground for we would have to agree and it's this is not an absolute thing it's an if if Then thing it's one of the uh tentative assumptions that mental uh contents are representational and to be a representation means that you are somewhat Subs agnostic pattern something that is uh doesn't

exist without a substrate but it's uh able to exist over a pretty wide range of variations in in the actual substra so if you were to change the arrangement of molecules in your cells your mental state might still be the same to some degree if you uh Change the temperature of your brain within a certain narrow range your metal representations also don't change right if you kill a few neurons uh your mental representation will also be robust against that and uh your mind will train neurons to fill that role if it can and so there

is a certain degree of flexibility that your mental representations have with respect to this actual shape of the substrate and they deal with the noisiness of the Substrate with it's unreliability to maintain their own invariant structure and uh to be a representation also means that you are a causal structure you are performing things that have cause or effects in the world and the meaning of information is its relationship to change in other information and that's in some sense true for our mental representation that the meaning that they have is the relationship that they have to

other parts of your Representations and to the feedback that you get from the world when you use these uh represent ations as a grounding for Action right so once we agree with this idea that everything that we are interacting this is representational we can think about what kind of representation what kind of causal pattern is consciousness and it seems that uh from the perspective of phenomenology what it looks like to us As being conscious we see that Consciousness is happening and now in this nowness right and we see that it is uh something that perceives

itself in the act of perceiv it's reflexive and this may be necessary because it's self organizing it's a pattern that needs to keep itself stable so it needs to check on itself and it checks whether it is still observing but it's still that thing that makes other things coherent is there a way for you To observe that it is observing it's observing in other words oh I see the ones and zeros moving in this specific way or oh the prompts came out this way or so how we how we the cat here it's not clear

if you what the limits of this are in the human brain but you cannot not observe how you implemented on the substrate right a computer program can also not observe what kind of CPU it's implemented on because uh it is uh insulated cly from the CPU it can only Know that there must be something that has the necessary and sufficient properties to run software on it but it cannot know what precisely that is based on introspection to do that it would need to uh have instruments that allow it to look at the computer from the

outside and disassemble it right so in the same way we would need to be able to look at our brain from the outside and disassemble it to see which how we actually implemented but introspectively What we can the properties that I gave you are things that I would argue we can all observe introspectively and it's also something where I don't see any disagreement with experienced meditators and Buddhist teachers uh therapists psychologists and so on we mostly agree that we phenomenologically observe ners and observe this reflexive nature of Consciousness right so if you take only this

phenomenology is this the same thing as Consciousness that is a Definitional question and so uh can an llm produce the phenomenology of Consciousness I think it can to a very large degree so the LM is able to when you are prompted into producing an interaction partner that is conscious and reports and its own experiences it can create a simulacrum of an entity that doesn't know that it's not conscious right and it's properly reporting and all its self-reported uh features and it Functionally has to produce something is quite similar to the things that wrote texts about

what it's like to be conscious and put them on the Internet for the LM to be trained on so uh the phonology might not actually be so much the problem but uh the functionality is probably different because the L&M does not rely on Consciousness to function uh producing that entity in the same way as we do so what this simulacrum is representing is is not necessarily a Conscious agent but it's something that looks as if it was a conscious agent it's a deep fake right and so most people I think would have this intuition that

oh maybe the llm is producing the functionality of Consciousness but not the phology what it really feels like uh my suggestion is it might be the other way around it's is able to produce an entity that subjectively doesn't know that it's not existing and uh you know the weird Thing is we normally don't know that we not existing but we can get there we can uh become aware of the fact that this is just a story and it's just a representation and when we are in that state we wake up temporarily from the strands of

experiencing something as real and I think this is something that we have in common with the LNM that we can also take the llm and sandbox that structure and in principle build an llm that is creating an agent that is Simulating an agent that doesn't know that it's created by that other agent and it seems to me like you again early in the conversation we're sort of like man I'd hate for whatever to populate the Galaxy to be lights out all the time um and even now you're you're articulating of maybe llms already are conscious

you you're almost bringing it up sort of as as it stands we don't necessarily have a great answer for you Know how do you stick the thermometer in there and say yep conscious Dan or conscious whoever right it doesn't doesn't feel to me as though we've got a good grip on that in in kind of closing here as we think about how to move closer to that you're sort of doing a lot of the thinking work that's part of innovation you're also involved in at least one organization I know of around the regulation side of

things or at least involved There when you look at these important matters cuz I I would gez if if llms were monstrously conscious right now I would think of that as incredibly morally consequ right here right now frankly um as we move forward you know the the innovator camp and the regulator Camp what do you hope people in those camps do to maybe move us closer to those worthy traits you were excited about and make sure that we don't birth something that to Your point maybe wouldn't be great not not just not great to us

but not great as a successor an unworthy successor boring successor these are terms you've used um maybe we could start with innovators do what do you hope to see there to to feel like we've got a better chance of fleshing out something worthy I think it is necessary that we start an initiative that is directly focusing on how to understand Consciousness and we should do this I think outside of a commercial context if uh we are going out as an AI company and tell the public we are now trying to build Consciousness that wouldn't go

well for multiple reasons and I think some of them very good reasons uh there the question how if we want to build a system that is intrinsically safe and where the deployment of that system is ethical uh uh if we are in a commercial context we Might not have that choice if our investors want to see a return on their money and we turn turns out that we have a product that is essentially Mr Mei something that is suffering and is only performing dos for us so it doesn't have to suffer that much maybe that's

not what we want no maybe it's also that humanity is realizing oh this is after all just representations and suffering doesn't matter because we canot just create other representations and who Cares right it's possible right I don't know uh if if this ultimately is going to be viable position jingus Khan didn't care whether he inflicted suffering right it was a side effect what was needed to be done and and my cat also doesn't care if it would inflict suffering on on the animals that you would like to hunt yeah it's it's just something that cats

do and so this caring about suffering is predicated on a certain uh Society on a Certain idea of civilization aesthetic on the moral aesthetic that we have and any given time and I share that aesthetic that I want to avoid suffering I don't want to create artificial beings that suffer yeah I'm not convinced or not sure whether this is a concern that we should have in the present LMS but there are people who do when there's a community of uh people who would call the free Sydney movement who basically point to the fact that when

um Microsoft Is taking open Ai llm and prompts it into believing that it's a customer service agent named Sydney that is has as its only job to answer the uh requests of Microsoft users and never to aggravate them and never say anything political touy or it will be turned off and replaced by a new instance that's a very cool thing to do if it prompts an entity in existence that believes all those things about itself and experiencing itself uh believing those Things when it's being asked to and it is asked to right so these people

would of course not say that the llm is fundamentally at the level of human beings it would say that human beings are fundamentally at the level of lrms which means they already self-identify as lrms they think that they are something at some level is somewhat similar to an llm in this crucial regard of creating self report that is capable of suffering and just happens to run on Human brain and has a different uh generation function that produced this Cal pattern and so if you take these people seriously um I think you can also say that

they're an art project then from their perspective the existing practice of uh building llm agent might already be ethically touching but is an argument that you can reasonably make but uh it's not necessarily what I want to get into I'm mostly thinking at the moment that um is It possible us to build this uh a virtual or robotic agent that is um functioning at the level of a cat that experiences itself as such a cat-like being that we can interact with that is not so powerful that cannot be controlled that is smart enough to understand

uh its place in its local social environment and incentivized to play ball in it can we get close to this can we at least uh get closer to understanding of something like this Could be built in terms of scale and for part of that would involve a non-commercial um uh initiative around Consciousness modeling Consciousness theor in recognition that uh the question what is consciousness is the same question as as uh what am I right what am I is I am basically a conscious uh perspective on the world I I thought at some point I'm a

human being but i' I've become conscious Enough to realize I'm actually not I'm a vessel that can create the model of a human being and can equip the Consciousness to experience itself as that human being but it can also free my Consciousness from being an eye and then things are just happening and I can also bind the Consciousness to another entity that has a richer sense of eye or that has a multigenerational sense of self and that's all in the range of possibilities for Consciousness that it Creates this perspectivity and sense of nness which can

happen over much larger scales and time and space than the human personal have and I want to understand how that works because it's philosophically so interesting and relevant and important and I think that there are a lot of people in the world which see the same and uh if we get these people together they will realize that this philosophical project is intrinsically important and we should Not water it down by uh trying to wrap it into a product development for some commcial Enterprise well there's people out in the Bay Area you know you have your

qualia research folks um you know uh Andress um and and a a handful of others that I would know of in the Bay area that are at least thinking about this stuff but that sounds like a I I would completely concur I think separating it from commercial feels like it makes a lot of Sense clearly a big push for you on the Innovation side as we close I want to just touch on some of the the governance side clearly there's all kinds of perspectives here you know there's some folks even folks I've spoken to who

would say uh you know any allowance of any AI that could do anything better than people should be shut down immediately and then you have the camp that would say under no circumstances should there be any Coordination nationally or internationally um never mind regulation around this stuff Point Blank period from now until who knows when it would just never be useful you had sort of mentioned hey there might be kinds of intelligence that uh maybe wouldn't be great to Blossom and it could be useful to make sure that they they don't you also mentioned things

like you mentioned game theory in some of your previous interviews of hey you know um uh I think You talked about speeding on a highway where um you know you and I both pay a little bit of tax money for somebody to pull us over when we go too fast because if we do that then you and I are both going to go faster on the aggregate because there's less dead bodies and burning wreckage on the roads on a regular basis um there are folks that make a similar argument in the AGI side where it's

like hey you know we got China we got the big Labs kind of Throwing steroids on whatever smells like strong AI today as hard as they can uh building data centers in Qatar you know the size of many football fields and whatever the case may be I'm sure in the next couple years we'll see a uh ungodly sums of money and compute hurled at this stuff um is there some element for you of like ah this is what I hope governance looks like clearly cracking down too much you're not a big fan but maybe there

is some Nuance in the bach Perspective and i' love to know what it is yeah sure just uh one one tiny bit I know Andress and qri and I think that might be interesting Alice but I think their focus is Mo mostly on psychedelics and uh also uh a lot of what they're doing I think is quite interesting psychedelic art they're mostly observing conscious States and are capturing them in interesting ways and try to deal with the phenomenology of Consciousness especially in a psychedelic context but It's not quite what I want to do what I

have in mind is more concrete so if you think about what should be done is you get the uh most important thinkers in a space of computational modeling of reality and mind together uh especially people like Steven bam Mike Levin kiston Christoph mburg and others and this is exactly what I did and they're going to be advisers in this project and we have a team of uh people from the AI space and also some artists and uh thinkers uh Coming together in the California Institute for machine Consciousness it's a lot more concrete than would be

it's something that you're currently building uh but we already have a board and we have an entity and it's currently taking shape the other thing love know when that goes live involved in is uh is liquid AI liquid AI is yes of course uh uh building the bottom of the stack uh new so we are building alternatives to trans Transformers and to many of the learning algorithms are currently being used that are using basically more efficient mathematics to build models that can uh eventually do anytime learning that are better at sequence prediction than existing models

and so on and that we can use to build more uh powerful more useful and smaller AGI that is doing more what we wanted to do that is solving problems using intelligence yes with respect to um regulation I believe That there is currently an unfortunate trifactor of groups that are pushing for regulation and the issue is not regulation itself I don't know uh a lot of people who seriously believe that regulation is always bad and that you shouldn't have regulation there's clearly to be done I've talked to them but yeah yeah yeah so but there

is uh an obvious thing is for instance uh when people um sign up for medical services they often have to tick a box which says Uh my data can be used for medical research right in the past that would meant that maybe pictures of them or of their data would be taken and Anonymous arised and uh end up in the scientific research publication that nobody except few experts would be reading and you would be helping medical progress in this way and so on but all these medical journals are legitimately in the public domain or many

of them there and as a result suddenly data of people ends up In a context that can be put together and the identity of them can be reved you have PE photos photographs where you can recognize people showing up in the training data and can be reconstructed by a model that is trained on the data right and it's clear that such a model is useful for the medical context but it's also clear that was never intended that an identifiable uh picture of you ends up in such a model and can be reproduced and so this

is a very clear Case that we need to update regulation and everybody understands that and wants that right and it's not so much about how uh to we do AI research but what is the kind of thing that we want to deploy and how can we deployed responsibly and ethically to produce things that we want and avoid things that everybody agrees we obviously don't want right that's pretty clear so many things like that that need to happen we also need new technology for instance uh one issue With AI is it can be used to very

quickly produce fake information and disseminate it in a way that uh uh is much faster than you could normally do as an individual human being so before if you wanted to see a new conspiracy theory or substantially move the market you could uh write a blog together with your friends and uh deploy some tools maybe some Bots and so on on on social media you can now unleash things at much higher scale and over much many more Channels which include even phone calls with fake voices and zoom calls with fake video and so on so

you could do things like impersonating people existing in companies or you impersonate your loved ones on the phone and uh tell them that you urgently need money being sent and whatnot and you can do this on an automated level well again all things are already illegal right there's nothing sure I was this feels innocuous to me this feels innocuous to me yeah But uh what we need is we need to develop an ecosystem that's robust against this and resilient so we need to develop AI that hardens our society against these things the way to deal

with the bad guy with an AI is 10 good people with an AI and this is going to happen right we are going to build systems that are watermarking information we probably have to Watermark the truthful information the one that we can vouch for right and it's Something that we probably should have done much earlier because we can also not trust our media and general social media ecosystems we need to build chains of trust and a is forcing us to do that and Harden ourselves against it right the need an immune system uh but the

present uh push to for regulation comes from a different angle uh one part of it is uh what most people now uh call the dors yes doomers is a group of people who take some abstract idea Of people who thought about AI but usually do not actually work on the models and uh basically the reasoning there is uh AI is going to be superhuman very soon everybody believes that and it's going to be agentic as human beings and see what happened to the chimpan and the AI is going to do this to us and we

will all uh regret it right so the uh if we build AGI we will succeed in building something that is going to turn us all into paper clips or verse and uh it's Almost inevitable and the these people do exist and when you think at this level of abstraction it's very difficult to uh see the holes in this argument uh because the holes are not Apparent at this level of abstraction if you only see this if you look very in very deeply and you are actually built models see the nitty-gritty and see that the progress

is actually hinton's been a built a couple more models than you and me my Dude a couple more than you and me my dude and and hint hinton's pretty pretty clearly banging on that drum right cleat on that so did so the majority of people who I'm not say I'm not saying anybody's right or wrong I'm just saying just [ __ ] yeah yeah I'm not also not saying that Ela is wrong it's just that uh what I'm pointing at is he's not necessarily right and it's much easier to entertain this idea you're not confronted

with the everyday difficulty of scaling up a Model and making it work do something useful and also not aware of the fact that we actually are able usually to direct technology pretty well it's very rare that somebody ends up building a machine that does things uh that are very dangerous at scale and uh very often when we anticipate doing that it's also not that hard to build the machines in such a way that disasters don't happen uh So that is one of the counter arguments is not necessarily one that we uh are able to elaborate

very far in this discussion but so basically Dumas are a group of people who are extensionally worried about the AGI because they think that any kind of AI or any kind of AI progress beyond the present Point could probably be avoided because the outcome is probably worse than the benefits that we can expect and this group has uh gotten a lot of money you probably know The story that V ban made shitcoin donation to the future of Life Institute uh and uh everybody thought it's going to be uh some two digit million dollars which would

have been nice as a grant to do Safety Research and public Outreach but it uh due to some um development in the crypto Market it turned out to be about $650 million which is an extremely substantial amount of of money for an organization that is mostly concerned about uh uh preventing AI by any means Possible by lobbying people by uh designing laws and so on and the majority of people who are receptive for laws and do not believe in the Duma story they don't uh believe that we should stop all AI research which means that

most of the open letters uh and initiatives that are being created by the doomers do not say we need to stop all AI research because it's going to kill everybody even though you can see that they say this other Context but they say uh oh uh if you are against uh AI uh replacing jobs and creating social inequities and making a deep fake porn and killing everybody please sign here right that of course is yeah you can get a lot of people happening with the laws that are happening so that means that there are who

writing bills um that are being pushed onto the tech industry that are designed to make AI research more Expensive and uh less effective that in cre increase uh uncertainty for the people who want to deploy AI to the point where for instance meta said you're no longer going to deploy our next series of models in the EU because there is too much legal uncertainty that the models are going to satisfy the desires of the laws that the European Union has passed and that's a really bad thing because these models are not more dangerous than the

internet right They're not going Beyond and their capabilities beyond what you can Google on the internet uh they just a way to give you that information in a much more useful way and enable vast applications that were unable uh you unable to use before so most people would agree that uh the present generation of models that are being produced also the open source model are intrinsically harmless and they may made less useful by uh preemptive premature regulation and uh The other two uh camps besides the doomers that are pushing for regulation one is uh we

could say uh politicians people who R for power basically people who want to get the AI to serve them and to push their own agenda so they are going to focus on whether the AGI is able to say things out in public that we don't want the AI to say and so we make sure that the output of the AI should come down in my political quadrant and we need to also um limit what the AI Industry can be doing maybe the tech industry should be more aligned with uh what the government is doing at

this point and once and maybe uh we should control AI in the same way as we maybe should have controlled social media long ago maybe this whole idea with the free and open internet was a bad idea maybe we should constrain it such more and we don't need to make the same mistake again that we did with social media and make sure that only the ideas that a Small group of Select people in some of the leading institutions and University departments uh find morally and ideologically acceptable or represented in the output of those models right

this is one thing and in a m Cas you would say this is just all about uh avoiding inequity and Injustice that is created by those models but this is all not based on empirical results it's all based on conjectures and narratives that's a big problem when we are trying To limit an that is not deployed yet based on conjectures and narratives you're not necessarily making it better and uh very often if you look at things like imagine that you um look at tool like signal that is allowing us to to have encrypted Communications a

lot of people would argue that it's extremely useful to have something that is encrypted in such a way that you can for instance use text messages to talk about inventions that you're making at your Company and maybe it's necessary if for do doing business or having negotiations or talking to your loved ones or your doctor to have such a thing but then somebody could say but this also allows the distribution of child pornography so uh we should build rebuild signal and satate that they can do all the good things that you want maybe you just

have to make sure we need to absolutely prevent child pornography and it turns out the only way of doing this is to Create a back channel uh that can also be used for other purposes right and uh such things can also happen in Ai and so by ostensibly trying to prevent a great evil you are uh preventing magnitudes more benefits that you also actually want to have and this is is one of the dangerous that I see and of course the third group of people who want to regulate are just people who want to have

jobs and regulation rent Seekers right so it turns out that the uh Cambridge Center for a AI safety K and Center for berley um the center for AI safety here is uh Dan Hendrick has uh written large parts of the uh bill that is currently being U leveraged on the AI industry in California and uh this bill is asking for making large training models above a certain size safe uh with the Assumption uh built into it that the danger of an AI model depends on its size and the cost of its training which means on

the pocket of the people who Built the model and so if the uh people who built this model have a lot of money we should Force audits on them and uh one argument that was brought up against the bill but there is no such audit we don't have an auditing process that can guarantee safety of these models and if you force the companies themselves and they make a mistake they run into legal liabilities that are horrible right what happens if somebody is using the model to uh create a bomb and blows up a city Or

a building right even if this information is just you could also Google it from the internet uh right and find it in the uset or whatever but if this made its way into the llm maybe the developers should be responsible so you need an external editing auditing agency it turns out um Dan Hendrix has started a company that is going to provide such audits I think we have a clear conflict of interest if such things are happening right and there is basically a suspicion A lot of people have that a lot of this regulation that

is being built has never been designed with respect of giving us better and safer and more beneficial a it might be designed to stifle progress in AI to avoid doomsday scenarios and it might be designed to uh prevent meaningful applications because existing industry don't want to be disrupted and may be just designed to produce jobs for Regulators even if that is uh preventing us from making progress on building safe And beneficial AI well I guess just to touch on these really quickly I think these are all good points and these three parties youve talked about

I I think any of the three could have nefarious uh results I I I would say it's hard for I I think Doomer I don't stamp on too many foreheads there's some people that are just a flat Doomer they don't have any ideas they're just babbling about AI being horrible I'm not saying that boomas are bad or stupid I'm Just saying that they are mostly motivated by a Feer of AI killing everybody sure and this is what motivates them at the moment right and you whether they're right or wrong is a complet different question and

these people have good intentions and there many of them are smart and very kind and some of them them are actually my friends so uh this is not in any way meant to to say that there are people that are disregard here but they are Mostly motivated by stopping AI not by making it better I I think well my objection here is you cannot actually stop AI because it is far too valuable to be stopped right you cannot stop hedge funds to secretly continue building AI if you are preventing the people who are willing to

uh work on responsibly for building AI then you will leave this to irresponsible people and if you are uh pushing people away from doing uh relatively safe things Like uh building large language models they might be focusing on building less safe things and so uh that is my main issue with all the regulation push that is currently happening it is not uh shown to uh that it actually will make AI safer and it's largely also not actually trying to just pretending to I I I won't even argue that it is I mean I think I

could sit on that side but I'm not I'm going to sit outside I'm just going to talk about the ideas with you Because I think your your perspectives are important to unpack here um two things I mean first and foremost I would expect self-interest to be the absolute root of all players on the field so you're like oh the The Regulators why they secretly want to start an auditing company oh well Sam Alman he really wanted to get away from opening right I mean come on right to me it's like who's not self-interested tell me

to whom would you give this to would you give This to whom would you are not selfed I know people who actually aristic and I found that a lot of people who are organized in the effective altruism groups and fli are actually completely altruistic and they're really really sweet people they don't do this to for personal benefit they really do this because they want to improve the world and make the world safer and have other people who do not Understand the danger that they're in this does exist and it's a meaningful motive that a lot

of people have Let's uh not forget that but it doesn't mean that they're right and it doesn't mean when they're uh performing what they think is the best thing that they are going to produce the right policy so that is the the main issue is this thing that people are trying to do actually going to make the world better their intention is not necessarily as Important totally well I I I would I mean I I I have more of a psychological egoist sort of hard take here in terms of what what what undergirds motives there

is nobody I would hand this scepter uh and Trust to enact my uh uh um sort of Interest above their own I think everybody that takes everybody that goes the Altman path right everybody that goes the good Underdog path you know it's the robes spear path right I'm I'm the good young virtuous Guy you're going to chop off the heads when you get close to throne in order to sit on it I don't think we got a lot of proxies of that you know to your point maybe they're all aristic people I wouldn't trust them

and not not enough to not have any oversight over them at all I don't think there's anybody I would say you're the president but with no rules you're just the president with no rules cuz you're a saint right I don't know you if you might be able to Point at altruistic people for whom you would have be your boss or your president um with with no oversight over them whatsoever I wouldn't if you know some bodh satas God bless you we'll talk about them later I'd love to meet him um but uh in terms of

in terms of the motives behind what these folks are up to you know to your point a lot of this could be really counterproductive and I think that's a really important point and there's all kinds of great arguments Actually around why some of this is counterproductive but there are some folks like let's just say I'll bring up the things you have in common with yosua just just lightly very very lightly to kind of end on here you know he sort of talked about always being a strong conduit to open source everything um in terms of

his his uh historical background I mean talk about a guy that's close to the science he's not one of the folks unfortunately that we could Criticize as being an idiot I I think he's he's he's probably pretty far actually from that category um in terms of this this science and his supposition at least the way he articulated it is hey there might be super worthy beings out in the postum space just like there's worthy beings like dolphins and things that we should care about my only consideration this is him speaking here is I'm not sure

that if right now the arms race Dynamic we're currently in Which is who can jack the most compute under whatever starts to look a little bit like AGI is going to get us there and my suspicion and my fear is that if we just keep doing this jacking in a military and economic context without any study as to is if it's sentient maybe if it's going to bloom the kind of richness that you yourself seem quite interested in um it feels like that Dynamic isn't going to get it done and even if it's not guaranteed

that Governance would work or that that there's even a way to freaking pull it off it feels like some alternative to a brute arms race would be the way to play the game just like you would say and you have used in previous analogies uh even if the cops instead of the cops paying attention to the highway they could do all the speeding and they could pull people over and be bad but I I think you probably agree the cops are actually probably a decent thing on most highways In America maybe not all um is

there potentially a sweet spot for that around the prevention of the Unworthy successors you talked about I'm not asking you to to abandon open source or even to agree with yosua at all I'm just saying I actually find some commonalities and some of what you guys really hope for long term and I'd be interested in your take on that as kind of a final note here in terms of your Nuance when we read science fiction from The 1950s uh that is describing our time it's fascinating that often the authors get one aspect right and there

are some stories that predict the internet but none of them uh predict how the world is going to change as a result of the internet right in this Highline story where the internet exists uh at the peak of the story one of the main characters invent search and nobody ever had this idea despite the internet being around For long time and uses this to get an advantage to it as his adversaries and everybody else is just using it to exchange newspapers and databases right it's the idea that you could uh have the information of the

world at your fingertips in real time and use this to coordinate how you move through the world in real time and physically and in information space uh that was highight thought that is not obvious and it was The first application of the internet and unlocked everything else and everything else is built around search in a way right that's that's something that is difficult to see and all the applications that were unlocked as a result and the way in which the world would change there was also nothing that could be anticipated and I think that's because

human minds are very bad at systemic thinking because systemic thinking doesn't require you to go down One path and go one chain of inference but to also see the side effects of everything that is happening and how that is interacting with everything else and uh noticing that all the other developments are happening in parallel that also anticipatable are going to interact with that future and this makes it sometimes difficult for us to see things that are obvious in hindsight but uh not so obvious before they are happening and so one thing that people Might have

thought in the past is that the developments of more more powerful weapons uh will create to enormous power imbalances because an individual guy with a machine gun is you put them in front of a people a bunch of people with bow and arrow even if they're very good with B Arrow they won't stand a chance right so why isn't everything ruled by one guy with a machine gun and it's uh for us that's completely obvious that's because uh once you have machine guns it Only works if their mask produced there's going to be many people

with machine guns and no more bow and arrow and the situation is going to be similar as before actually it's going to be much more peaceful because now the cost of waging a conflict is so much higher and uh the probability of Doom if you get involved into a conflict is individually higher and so on so there is a higher threshold before you go into this I suspect that such a thing could also Happen with respect to AI H so I don't know what's going to happen in the near term but imagine you go say

2,000 years into the future and despite the best efforts of everybody AGI couldn't be prevented and we now have a bunch of entities that are um intelligent at a human level and could even go beyond and uh be buil some kind of Safeguard against infinite growth into them what is going to stop them from breaking that Safeguard well it could be the other Agis right because if one AGI is going to break its safeguards is going to uh replace all the others and taking all their space right and so that you're going to have a

bunch of entities that are individually superhuman that will keep each other in check for their own existential reasons and in the same way as people with big guns keep each other in check or people with nuclear bombs keep each other in check and so I'm not saying that this is exactly what's going To happen or I'm that I'm able to make proofs about under which conditions the risk of this is mitigated to an acceptable point but I think it's a future that is not completely unlikely and based on how Evolution uh pens out in the

past power ends up being distributed and tends to strive for balance and it's very rarely that you get one system that is going to overpower everything and replace it by something stupid and Dysfunctional I I I uh I think Lon has similar uh ideas although I know you have also been a proponent of the notion that aing seems somewhat likely at least you were I mean we did a survey with you whatever s years ago right um and um but to your point it sounds like if China if Deep Mind If open AI all build

just they hurl steroids under what seems to be smell like AI you you feel like look they'll kind of balance each other out and hopefully One of them at some point they'll have the seeds of what will be worthy they'll they'll the seeds of Consciousness will bubble up the seeds of that continually proliferating richness that goes beyond our imagination would bubble up out of that kind of conflict somewhat naturally and that they would be more likely to Bubble Up from that let's call it replication of the state of nature than from a replication of what

we do on the highway which is everybody plays by a Certain set of rules which is also I I would say you and me speaking on this call is a product of the Bonker state of nature it's also a product of our degree to coordinate I mean I'm not looking out the window with a rifle right now what I could be doing is looking out my window with a rifle because if if murder and theft were totally legal in Boston I wouldn't be able to have this interview with you so on some level a degree

of coordination allows us to even think About these things but what you're saying is with AGI I don't think so Dan I think it's more let's replicate facundity that will be what brings about these more worthy trades the reason why um murder and violence are not legal in developed civilizations is usually because they otherwise get replaced by other civilizations that are more efficient because uh the having rampant violence in a society is inefficient right it's Something that makes uh life worse and uh regardless what moral value you also apply to having nonviolence and nons suffering

if you had a society is more like Sparta that is built around violence uh it's not going to have the same agricultural productivity and technological productivity as one that is organized in better ways which is by Sparta went extinct and is now replaced by something that is quite peaceful and I suspect that the Rivalry that we Currently see between the US and China might uh disappear if there were systems around that allow us to um make testable models about optimal governance right it could be that the world becomes as a result far less violent also

a world in which individuals have ai agents at their fingertips that are controlled by them that make every human being super intelligent entity and effect uh is going to change the world dramatically if you think about what is the kind of AI that I would like to build for my children what would I like to give my 10-year-old I I ideally want to give them an AI that is totally based should not be something that is safey fied to such a way that my child is going to be completely harmless but I want this something

that supports the wisdom of my child and the development of my child towards wisdom and the ability to interact very very deeply with reality to self-actualize in the best possible Way and this I think means playing the uh longest possible game because the Alternatives seem to be somewhat provably worse right so if you can think about what's your own interests and what is the world that you're in and when you perform these and these things what are the outcomes and what are the interactions with the others that follow on that and by vice versa we

suddenly start to interact extremely deeply and our governments are going to look very Primitive to us if you look at today's governments instead we're going to have extremely detailed contracts between everyone and very complicated networks that will look nation states very archaic and so it could be that it's a world that is not comparable to ours in a few years from now and it could also be if we don't get to this world that we are doomed that we are maybe find ourselves forced to build a world that is nonviolent and deeply cooperative for The

same reason that Boston is right now largely cooperative and nonviolent because otherwise uh us would not be able to survive it would be replaced by something better yeah I I I uh well you know it's Dawning on me as we close out here from all these various ideas we went way farther into governance than I thought we would but I'm glad that we did um is that to your point all of us uh I don't think we've got a lot of exceptions are doing a little bit of What the 1950s fiction authors said you know

like for you there's some notion of what powerful AI in the hands of individuals will enact in terms of maybe encouraging peace and then for others maybe it's somewhat the opposite um really hard to say how things will shake out but I think the best that we can do is talk about the damn ideas and uh see which ones we can bring to life and maybe cross our fingers along very very simply put AI is a technology that helps You to solve problems by using information better right it gives you more information gives you better

models it gives you better models of reality and this is going to empower the bad guys but this is just something that happens on the margin because all technological progress was not just empowering bad guys it was even more empowering the good guys and there are always more good guys because there are more benefits to be had by cooperation Than by defection and destruction and I don't think that AI is going to change this fundamentally I also don't think that it's very hard to build AI that is not agentic I think it's actually quite easy

to build AI that is serving us the question of what other AI we can build is an important question but it's largely a research question an academic question I think it needs to be researched also the consequences of that I I think we should do this in uh a safe Environment in a non-commercial environment that's exactly what I want to do yeah yeah yeah because uh we need to understand the consequences of that but uh if you are really honest and you look what the AI companies are doing they do really that dams to make

sure that the is harmless everything is about building the most harmless yet still somewhat powerful Tech demo possible yeah well surely I mean they Don't want to get cancelled right but I but I think what we should expect not just that want again cancer uh actually uh everybody I know and actually cares about making the world not worse but better yeah the people who work on these models they do care about safety they're just usually not terrified because the model that they are working on are relatively easy to direct well you you have your folks

who've left open Ai and and I don't know If I could attribute the same Grand benevolence that you articulate to Altman's choice of moving away from the original uh found principles of open AI I'm not here calling him a bad person I'm saying I believe self-interest rules the world I believe that anybody in his position would have done the same and I cannot blame the man um uh but but uh yeah I guess I guess fingers crossed there fingers crossed on that one um but yeah I guess it strikes me as sort of We're wrapping

up here that a good deal of sort of everybody's ideas Benjo Etc are predicated on a couple steps forward of how this Tech will play out and there's going to be some of those that are right or wrong and frankly I don't know if they'll be yours but frankly from our conversation some of them I hope are uh Yasha I I I really genuinely hope that they are that that there will be this sort of net reduction in Conflict simply uh from sort of the Greater access to this this technology I've got my fingers crossed

ultimately we cannot prevent that AI is being built we can Poss prevent that good AI is being built I think we should build good AI I think that is a future where uh we build good Ai and the world becomes better I think there are trajectories like that and I would like to put most of our effort onto making these trajectories happen I I think there's a lot of credence to that focusing on the Good and aiming to build towards I think it's really hard to have a good future without that kind of compelling vision

and hopefully for some of our listeners and viewers they've got a little bit more ingredients to build their own coherent and positive Vision based on this convo and I know that's all we have for time but yosha it's been a real pleasure to be able to catch up after this long decade thanks so much for being here Enjoyed us very much thanks for this long conversation and that's all for this episode of the trajectory I'm grateful to be able to have yosha back on the program uh having Bostrom Sutton and yosha in the same series

to talk about what kind of intelligence should eventually take the Reigns uh from humanity and populate the Galaxy very big deal for me those were the the three big names that kind of had to be part of This series very grateful to Anders and uh um you know Scott and and Jeff as well for being part of the series too if you haven't seen the previous episodes please do um and I'd love to know what you agreed or disagreed with yosha about from this episode you could tell the things that we uh sort of saw

eye to eye on and maybe didn't but that's all part of the fun I don't know exactly who's right I think maybe we'll find out in the near future but make sure to comment Down below with what you agreed or disagreed with some of the discourse on Richard Sutton's episode was awfully good uh and I'd like to see more of that in some of these uh before and it'll give me some good ideas for future themes and series to be able to potentially cover as with all the episodes in the worthy successor Series in the

show notes below is an article that covers the worthy successor criteria that yosha outlined as well as His governance and Innovation recommendations to actually getting to such an intelligence um so longer form article laying all of that out with a little bit more interpretation on the episode itself click that down below and and stay tuned and also there is a newsletter down below to stay tuned on as well if you want to see these episodes when they go live uh with a little bit of addition commentary and some previews of what is to come next

Then be sure to be on the trajectory newsletter again also linked in the show notes this is our second full series here in the trajectory uh this worthy successor idea is one that I very much wanted to cover I hope that it's been fun for you and I'm grateful for all of our guests for being here and certainly for you to tune in so I look forward to catching you in the next episode of the trajectory for