What is the way in which you can distinguish made up consciousness from real consciousness? Then again, maybe all consciousness is made up, right? As long as you don't know that it's made up, um, it's a tricky question. It's super hairy and I haven't, uh, good handle on, uh, on the phenomenological side yet, how to distinguish fake phenomenology of consciousness from real phenomenology of consciousness is both of them are virtual. Welcome to Manifold. Today, my guest is Joscha Bach, a cognitive scientist and leading thinker in the field of AI. Joscha, welcome to the show. Thank you.
Good to be here. So we met at a fancy event over the summer in Frankfurt. And, um, I was privileged to hear you, uh, in several lengthy discussions talking about AI, but also talking about your rather unique childhood and upbringing. And so I, I thought we would start talking, uh, first about your early life and how you got to where you are today and then branch into, um, topics more central to AI. So, so tell us about your childhood in the woods. Well, um, my father, uh, did not get along very well with society, so
he decided to build his own. And, uh, he was basically a first generation society founder, nobody who Was trying to get anybody in, he had not discovered how to, um, grow groups yet, or was also not interested in this very much. So he basically built his own kingdom in the woods. And only converted his wife into supporting him and, uh, living in extremely beautiful. life in Garden of Eden that he created. He was an artist, originally an architect, and he basically felt that life was about producing art, about interacting with his inner voices, with the
things that motivated him. And so as a result, I grew up as one of his side projects. One that he didn't engage very much with. He was not that much interested in the minds of children in an Extremely beautiful place in which I was mostly left alone. And in a way that was a blessing because it was indeed stunning. And on the other hand, I was very lonely as a child. And so I got bored. I started to read very early. And it was like a sponge and took up a lot of things in my
mind. And when I ventured out into the world to go to school, um, I was already lost to the world in a way out there, there was a communist, Eastern Germany philosophy was a vulgar. Um, Like a Marxist dialectic, and there was not much that I felt I could learn for the first eight years in school. And, uh, mostly because I already read the books, and the stuff that My teachers tried to impart on me were not that interesting or, uh, not that deep, and didn't work out. And so, in some sense, I Retained a
certain useful arrogance for quite long in which I felt that there was not much that other people could teach me and I would have to be self taught. And this has not made me a much smarter person than others, but it has made me a more unique person than many others because, um, it. means that I started to project the world onto my own surface onto my own screen and try to get a perspective that works from first principles that allows me to understand how the world around me functions and why I'm different from the
world around me and basically to realize that other people. Due to their socialization, due to their trajectories and their psychology Have a different interface to the world and I would need to translate to mine if I want to interface with them. And unlike my father, I felt that living out there in a beautiful valley remotely was not in the cards. I would have preferred this because it's quiet, has no sensory overflow and so on. And you don't have to put up with all the difficulties that civilization and cities impart on us. But I need the
stimulation of other people and their interaction for the projects that I'm working on. Uh, now you're in the Bay Area. Do you, do you find it preferable to live in a city like San Francisco to living out in the woods? Um, at the moment I live in the South Bay and I feel that, uh, why we have wonderful neighbors and amazing weather. It in some sense combines the worst of both worlds because it's not, uh, remote enough to be in nature and it's not urban enough, uh, to have lots of interesting people within walking distance.
So I drive up to San Francisco a lot. Yeah. I mean, the advantage though is lots of people who want to think about AI or want to build AI are close by. Yeah. That's why I'm here. Now, um, it seemed that the teachers that you had didn't stimulate you, but I'm Wondering of the books that you read, were there historical thinkers or authors that influenced you when you were growing up? I read a lot of philosophy and also a lot of science fiction. When I was a kid, I didn't read much just for pleasure. I
found that reading was often intrinsically interesting, but my main goal was to understand the canon of our society. I forced myself through the Bible and even Gandhi's The Wind as a young child. And, uh, at the same time, I tried to read, um, mathematics books or an Einstein biography and so on just to understand the background of our society. And I thought it's necessary for everybody in a way to get the canon down. Also, not just in terms of books, but also Uh, movies and so on. And, uh, later in life I was surprised that,
oh, nobody, nobo, almost nobody Is doing that anymore because I thought it's the goal of education to give you a deep understanding of the society that you're part of. So, despite what I perceive to be your rather unique intellect and personality, you did go through the standard school system, and you actually have a PhD, so you went through quite a bit of school, is that right? Yes. Um, when I made the decision to get into academia, it was because I thought, I want to understand how the mind works and. I looked at the different fields
and I found that there was no single field that would explain it to me. I felt most attracted to computer science because I was very much into computing. That was close to how my own mind worked and I thought a lot about coding as a kid. And I was blessed enough to grow up as the first generation with home computers. So when I had my Commodore 64 at a very young age, uh, there was no software that I could use. And so one of the first things I had to write was my own text editor.
And then I learned how to make computer graphics from scratch. And I think with hindsight, it was a blessing because it informed my mind into how to construct a reality. in computational system and the computational framework. So that was very close to me. I also found when I entered academia that I was very much at home with the style of interaction in computer science. You could criticize a professor's proof in your first semester and everybody would Be grateful, especially the professor, if you found a mistake in his proof. In philosophy, not so much. There was
the opposite, because the criteria of acceptance are very much social and you're expected to be one with the group. At least that was the case in university courses that I was in for the most part. It was slightly better in hardcore logic and analytical philosophy, but the most hardcore analytical philosophy I got was within computer science and logic education. And, uh, I studied also psychology and, uh, a number of other subjects that I found related. So, um, when I went to Berlin, I, I, Partly did this because there were a number of universities that I
could attend in parallel. So, would I just pick whenever there was an interesting class, even if it was a Potsdam cognitive science class by a visiting professor or so, I would drive out and attend it. And, um, when the class was not intense in my main subjects, I would just skip it and only read the textbook and show up for the exam. So, uh, I was very self directed and was grateful that I had the opportunity to go into a lot of subjects. But I also found that my disrespect that I acquired as a child
in public school for, uh, my teachers did not abate very much. There were not that many professors initially that I felt actually able to teach me something that was going beyond the textbook. And I think good education has to do with interaction with the Intellect of another person. So if it wants to go beyond the book, and the first person I met in this regard was Raul Rojas in Halle, I was not a student yet when I crashed his classes, and he later on became very influential in the development of neural networks. moved to Berlin
to go to Freie Universität. I ended up being at Humboldt University, and after the war came down, the department was mostly put together with new professors. The old professors, for the most part, didn't make the cut because of their connections to the communist regime and lack of technical abilities. So a lot of young people were hired, and there are not a lot of students. There were 60 students in my entire year. And which means they had personal access to all the professors. And, um, I felt that, um, this allowed me also to shape my own
curriculum in a way I could write my own exam, uh, rules in part. So I could take philosophy as a second subject, in addition to computer science and so on. This is a slight digression, but I hadn't really thought about the fact that you, you saw the wall fall from close proximity. Did, did that evolution of society surprise you? Did you anticipate it? Uh, how did, how did those events feel to you? Yes. um, a friend of my grandfather, uh, a couple of years before the war came down and he predicted. That, uh, East Germany
would be going bankrupt and would be crashing and I saw that it was unimaginable because nobody seemed to be seeing such a thing. East Germany was, uh, extremely stable in a way. There was, uh, no inflation. All the prices were regulated by the state and, uh, there was almost no growth, very little progress. Um, our productivity had remained stagnating for a very long time because Eastern Germany had replaced the economic terror of capitalism, where you have a force to work, to survive, uh, this moral terror, because you had a right to work, but not actually
a duty to work, uh, people were incentivized mostly with moral slogans. And you can imagine how that went because people ultimately look privately after What they believe to be incentivized to do, and if nobody has skin in the game. People in the big factories largely don't work very efficiently. And so Eastern Germany, uh, always had, um, uh, um, lack of people who could be employed and lack of workforce just because we were so unproductive. And as a result of this system crashed, I think that the economic factor was the underlying thing, but there was also,
um, Political what? Because everybody got promoted in the political system according to the Peter principle back in the economy. So basically you get promoted as long as you're good enough. And once you're not good enough anymore to get promoted, you stay where you are. And, uh, The system didn't have the power to renew itself. Erich Honecker, who was the leader of Eastern Germany until basically The fall of the wall, was still a first generation of leadership. He, um, succeeded, um, Ulbricht and, um, a couple others, but he, uh, was, um, in prison under the Nazis.
He was a communist leader and was pretty much motivated with making sure that Nazi fascism wouldn't happen again. And a lot of the people in his generation were quite idealistic, but also were not able to update and to critically interact with what needed to be done to modernize society to keep the economy going. And they're also somewhat limited for the most part. By, uh, the diktats of the Soviet Union, which limited the ability to get freedom. We had the most freedom, I think, in, uh, terms of organizing our economy in the Eastern Bloc, because we
were in direct systemic competition with the West. And, uh, so Eastern Germany in many ways was better off than most of the other Eastern Bloc states, but, um, still, we had very poor technological production. We never went hungry. It was always, um, cabbage and apples. And, um, food, it was not often not super fancy, but we also had enough clothing and so on, but our cars were terrible and a lot of, uh, spare parts for anything that you needed, uh, were only available on the black market. So it was an interesting switch when we had
the opportunity to join Western Germany, but I was naive and I did not, uh, Uh, expect that, um, there was no opportunity for a third way. It was highly politicized at this time, and knew exactly what the working class needed. It was, uh, in many ways, as young people, um, had a more realistic world view. And, um, but I thought that, uh, the justice in a society, equality in a society, in which Eastern Germany scored quite high, is more important than the total productivity of society that is available to everybody. And I was in some
sense shocked when the working class, uh, betrayed the ideals of socialism and communism and also of the revolution, which was largely driven by people who were willing to risk their lives to go on the street. So idealists. This German opposition was largely very idealistic and the working class ultimately opted to be oppressed by the bourgeoisie again and exploited because they could see that, uh, West Germany was the control group. And despite the grave inequalities which existed, there were no Billionaires in Eastern Germany. Uh, the, um, medium income and also the lower income for much, much
higher, even though we had no homeless people. So you mentioned your computing, uh, your first computing exposure was to a Commodore 64 and I'm, I'm actually a little bit older than you. So I, I think for me, it was an Apple 2 plus, which is even weaker than a Commodore 64. Um, I'm curious now, I think. Yeah, I think a positive side of that is, as you said, getting your hands dirty and having to really build stuff yourself. And kids these days, I think, could be intimidated because everything is so advanced already that they don't
feel like they can get in at a fundamental level. Um, but I wanted to ask you, so coming from that beginning, can you describe how your own AGI timeline. Evolved. So, in other words, when you were young, could you have imagined machines as powerful as we have today occurring in your lifetime and thereby us getting to AGI? And so, so how did that timeline your perception of that timeline evolve as you got older? I thought it was obviously going to happen. I would have been really, really surprised if AGI didn't happen in my lifetime, because
it seemed to me clear that computers had the capacity to, uh, think and to perceive, and, uh, there is no limitation To what you can do on a computer. And I also realized that, uh, you needed more memory and, uh, you needed a higher processing speed, but I was also very confident that this was coming along because I saw the speed at which computing developed. And, uh, so to me, this time until we had electronic brains and so on, uh, couldn't come early enough. I thought this was a really exciting time to be alive. I
was very glad to be born in that generation. I want to differentiate between couldn't come soon enough. and still might take longer than my lifetime. Um, so, for example, in my case, you know, the question was whether, how long Moore's Law would continue before it crapped out. And so there's always a concern that it might crap out too early, and then, you know, uh, speed ups in computational capability, you know, uh, go away and, and then you're stuck with whatever hardware, you know, you suppose it had happened in 1998 and we were stuck with 1998
hardware that that would be a real barrier to getting there in, uh, our lifetime. But I guess you, you were never, you were always confident that, uh, we'd keep, we would keep, uh, improving our computational power. Yes, I was pretty confident. And I also felt that if this thing is 10 times slower than my mind initially, that doesn't really matter. What we need to figure out is how it actually works. And, uh, I could see how much you could do with little hardware already. So just use more, uh, try to distribute the load. I thought
there would be a way. Uh, the science side, I would say that I was, uh, had a tendency to be over optimistic about technology in my childhood. I thought that things would be happening more rapidly, not necessarily in computing, but everywhere. I imagined that there would be a lot of medical progress that didn't really happen, uh, in our lifetimes. I would, uh, imagine that, uh, we would, uh, go to the moon again, uh, and to the, to Mars in a relatively short amount of time. We didn't. So there are many things where, uh, uh, technology
was stalling and Felt was keeping behind the times and was no longer progressing. And, uh, they, I could have made the inference if I had seen that earlier, uh, that a similar thing could have happened to computing, but it just didn't seem to be imaginable, simply because computers seemed to be so cheap to innovate on. I think the problem for me was being a physicist and knowing what actually is involved in semiconductor, the semiconductor industry. Uh, there was a lot of terror all along the way. Like if you actually ask the engineers and physicists who
kept Moore's law going, there was always every year on the five year map, there was always like a miracle has to occur. Hopefully we'll get that miracle to occur, but there was never any assurance that it was going to keep happening. Yeah, it's easy to see. You think, oh my god, photonics is going to come soon. so now in 2024, I'm curious what your AGI timeline is. So how much of the remainder of your life is going to be used up before we actually get to, I realize the definition of AGI is a little bit
fuzzy, but. But, but how do you feel the next, say, decade going? I always felt that I don't really have a timeline. It's, uh, I found it difficult to put years on any of these things. Simply because I don't have an idea how to progress on something that people don't really know how to do. The scaling hypothesis has changed the perspective of a lot of people. Simply because, uh, the scaling hypothesis seems to make progress predictable. And the scaling hypothesis means that, um, you get better by training with more compute and more data. And that
is a very radical idea that, um, most of, But the history of computing saw things differently, right? Mostly we are looking for a master algorithm. We are looking for something that is going to, um, basically once you implement it, it's going to self improve and it's going to learn everything. It's going to recursively do meta learning and so on. And there is not going to be some slow, predictable. Logarithmic learning curve that you are getting better, um, uh, linearly with adding, um, um, next magnitude data. And I wonder if this is a good hypothesis on
what happens in human minds. I don't really think that humans get better, uh, at thinking if they read more. And while this is not completely true, I mean, reading more does help you to understand more things. You have more, uh, things available at your fingertips to delve into new problems, to solve new issues. Uh, I think that, uh, there is something like raw firepower, this elusive G factor that, uh, determines how many balls you can keep up in the air, how well you are able to do inference and how early you discover epistemology. And so
I, I still think that there is a much more sparse version of intelligence than the one that we are launching in the current foundation models that requires far less data to bootstrap itself. But it could be that the present systems are good enough to put force themselves there. And maybe I'm wrong, maybe the scaling hypothesis is right and intelligence is just about scaling up processing of information. Um, feel free to say, uh, to answer no to this question, but I'm curious, uh, how, do you closely follow what's actually happening right now with the hyperscalers? Since,
since you are in the Bay Area, I'm, I'm curious whether you hear rumors. I'm hearing a lot of rumors about a slowdown in the hyperscaling program. Um, I don't know if you follow any of that. Um, it's, um, rumors in this regard, but people always expected that there would be something like a plateau when it's harder to get new data. Yeah. And um, at the moment we always had this thing that people said, Oh my God, there has been a training failure, but progress has been happening. So I'm not sure how to, what to make
of this, but I technically expect that AGI is going to be a different algorithm than a transformer. So you just mentioned, Joscha, that you thought transformers might not be enough to get us to AGI. Maybe you could elaborate on that. Yes, I still suspect that there is going to be some kind of master algorithm, and I don't know that transformers are not enough. There is no proof that they are not. There is no proof that the scaling hypothesis is wrong. And, um, the networks that we are building through learning are During complete, but so there
is no limitation that is intrinsic in these networks that we know of. But there is also a strong sense that the way in which our own mind is very different. The transformer is not its own meta algorithm. The transformer was not discovered by somebody having a transformer in their head and then using the transformer to construct a transformer. There is something that is much more elegant about the way in which our own mind works. But, uh, I am not confident that we can prove that, uh, you cannot learn how that is happening. And Another, um,
element of the or compliment of the scaling hypothesis Is universality hypothesis. And this universality hypothesis is something that I first encountered in a text by Chris Ola, who, uh, worked at the time at OMI AI and, uh, studied. Uh, vision networks, especially, and they discovered that the number of the architectures, um, have different learning algorithms and so on, uh, different network architectures, um, at the beginning of training discover the same features. And so you have put a hierarchy of features that are being recognized in the neural network that can be identified across a family of
different systems. And this goes together with the observation that a team by Tommy of Tomo Pioggio at MIT made. That there is a similarity between the visual cortex and the Organization of these neural networks. And so the idea of the universality hypothesis is that if you have a good enough learning algorithm that is general enough and neural networks are already in this category and you give it enough data and enough compute, it's going to converge to a model with an equivalent structure. And this is And in some sense, a mind blowing idea, if you were
to end to end train a system on your own input and output for long enough, you will wake up in it. Right? Is this really the case? But if this model is producing the same output as you, it must produce internally a causal structure that is somewhat similar to your own causal structure. And, uh, that means whatever it is that you are doing to be intelligent, it's going to, you know, discover that causal structure to a very large degree. And so the transformer itself might not be the mechanism that, uh, leads by itself to, uh,
intelligence at the lowest level, but it can learn how to be intelligent. So in many ways, you can also understand it as a meta learning algorithm. So I wanted to, uh, you've spoken a lot about the concept of consciousness, and, uh, I wanted to discuss that with you a little bit. And, uh, I guess the goal would be to give the listener who isn't an expert on, in cognitive science or AI, some idea of how we might hope to define what we mean by consciousness. And then I believe you're in the group of people Who
hope that our successor intelligence, this future AGI or ASI, does have consciousness and we Would be disappointed if it didn't. So maybe you could talk a little bit about the concept of consciousness and and why it's important for the successor intelligence that you'd like to create. Well, there is, of course, this thing that we have not succeeded if you're not building something that has the ability to become self aware in real time and observe itself, observing and experience, uh, present and presence in the world. And this, um, present and presence is, um, I think what
defines consciousness introspectively. We notice that there is a now that we inhabit and, uh, typically we inhabit it from the perspective of a self. That's somewhat optional. We can also have dream states or meditation states in which no self exists. You can only attend to attention, or you can have an experience where it's just things are happening, but there is no I present, no self present. But typically, consciousness is attached to the surface of the self. And the surface of the self is not the surface of our physical body. But it is the surface of
an object that is simulated in our own mind that is the, um, the one that experiences it. Our mind is a machine that is composed out of billions of neurons and probably relies on the interactions with many, many other cells. So you have this organism with a few trillion cells. And this, um, these few trillion cells for hundreds of trillions of cells, they are shambling through the world as if they were a single agent. And that's how they're able to survive in the high degree of organization And specialization that they manifest and, uh, They need
to have some kind of control model of what it would be like if there were a single agent and what it would be like for that agent to perceive the world at the sense at a level at which it makes sense of it. And so it creates a simulation of a vote from the perspective of what the cells can extract in terms of meaning with respect to relevance of the behavior of the organism. And then it creates a model of the interest of that organism, which is, um, the experiences, our motivation and emotion in this
world. It's also generated inside of our mind, and then we have a model of who we are, and this is also inside of our mind. It's a virtual person, a story that your mind tells itself, not just A verbal story, but a multimedia story about what it would be like to be an agent that experiences. And this experiencing self is the surface on which consciousness is being projected. So all the future dimensions of the world that are relevant from the perspective of that simulated agent, um, are being experienced. And this experience is virtual. It's as
if consciousness is an as if thing. It's a software thing. It's simulated in, in the mind. Consciousness is a simulated property. And I think that, um, unlike, uh, Daniel Dennett, who argued that philosophical zombies and agent that is able to do everything that You and me are doing without being conscious, um, It's not possible. I think that's entirely possible to have such a zombie. For instance, you can imagine you were puppeteering somebody else. And, uh, if that somebody else was being puppeteered, uh, is looking at some color, um, uh, and responds to this. There is
nobody there who responds to this except you, but to do this, you don't need to perceive the color in the same way. You can also use a technical device that is doing this in the same way as a self driving car does it. And you just build a classifier that doesn't have a self reflection is not aware of what it's doing in any way. So technically it is possible to produce human like behavior on all levels. Without having this, uh, being that experience itself as real as entangled to a sense of reality. But, uh, how
do you get to such a being that, uh, experiences such a reality. But you just simulate it, right? You simulate what it would be like if such a being existed that is directly exposed to reality. And so it's not that hard to get from the zombie to, uh, the real thing. You just need to simulate what this is like to something that all this is real. And, uh, there is also a next stage that you wake up from this, where you realize, um, actually, it's not real. It's a construction that my mind is making. And,
uh, you can maybe identify in that state as the generator. So you experience yourself not as a person, but as a vessel that can create a person. And you do not experience, uh, that you're looking at faces and trees and so on, but you, uh, look at geometry. that moves and you at some point you realize that there is not even geometry, but there are patterns that are being abstracted into geometry and interpreted as such. And the degree to which you become aware of that, that is the degree to which you construct your immediacy of
this conscious experience. And I find, for instance, when I write programming code, there is not much going on in terms of sensory qualities of qualia. It's, it's very abstract and the more I go older, I look at the world, the More I realize that I'm looking at code. And so I suspect that this, um, Direct immediate experience of something being real and us being entangled with it rather than us looking at the models that our brain is generating is something that is passing with the advancement of a mind. And so I would expect that when
we build self improving AI, that it might have some kind of intermediate stage. In, uh, which it is conscious in a way that is very vaguely comparable to ours, but it might transcend this much quicker than we do. Now, I think it was implicit in what you said that the reason we have consciousness has to do with natural selection and for organisms like ourselves built out of a trillion cooperating cells to just be able to act effectively in the world. Is, is that your viewpoint? Would I think that consciousness is an aspect of nature's master
algorithm. Basically, our brain seems to have discovered something that makes it trainable to file this data and bootstrap itself in a way that is self organizing to get these cells organized By, uh, individual cells talking to their neighbors until they find some kind of order. That is, um, Spreading out. I once created this metaphor of a puppy, um, uh, universe. So basically imagine that your skull is a dark room full of little puppies and all of these puppies can do is they can bark at each other and they listen to the box of the others.
Initially they have no meaning idea about these barks mean. And the purpose of this is they use these parks to, Create communication across each other to process information. And the individual dog doesn't really know how to do that. It's a pattern that needs to emerge across the dogs. And to do this, they need to find a pattern of bugs that entrains new dogs. To, um, speak this spark language. And, uh, again, the spark language has some semantic context in the way that if, and you're making the right sequence of bugs, you're getting fed in the
evening, if you make this in response to the other one, but, uh, what you're doing is very low level computation. It's nothing that has meaning with respect to what the organism is doing. I think the individual. Neuron has no idea what, uh, happens to the organism, even though it's a little animal that is trainable with reinforcements. So it needs to be sensitive to some kind of semantic input that is predictive of whether it will still get fed. So basically it gets paid for being trainable. And, uh, All together, they basically create this giant dog food
factory that is, uh, making sure that they all get fed. And, uh, this dog food factory is, uh, walking through the world and collects ingredients for dog food. And that would not be possible for, uh, the dogs, if they're very individuals, uh, not working together in this giant tightly integrated collective. And so what is this integration? Of information across them. There needs to be some kind of protocol layer that allows all parts of your mind to speak all the other parts of your mind and not get subject to some kind of Tower of Babel that
needs to be some kind of coherence criterion that is establishing itself relatively early on and everything else rests on top of this. Coherence criterion. And I think that what we observe in conscious experience is we can only see things that are coherent. Stuff that is not coherent doesn't get integrated. We just experience is that noise or is chaos. Right? So there is the boundary to the bubble of now that we experience in time and space. And this is the degree to which we are able to impose coherence on the world. And maybe this is a
different loss function. Uh, we find something similar in energy based models, where the energy is defined as a mismatch between features, and we try to, um, minimize this energy, which means we try to, um, minimize the mismatch between feature variables that, uh, have constraints across each other. you say this framework for thinking about it is enough to explain qualia? I don't think that there is anything mystical about qualia. I would prefer if we would call them sensory features as being projected onto an observer. From the perspective of that observer, uh, there is, uh, some discussion
and philosophy about good quality are, and then people are trying to formalize them And define them as being, for instance, atomic, and then others come in and say, Oh, maybe, uh, atomic features cannot really exist or that don't exist, but ultimately, this doesn't really matter. What matters is that, um, the experience reality and things. That we can describe as feature dimensions and these feature dimensions are somewhat similar to, uh, the dimensions that you could find in embedding spaces. And indeed, this notion has been discovered by Rogers and Lesney, I think, in the 1950s, an American
science fiction author who is, I think, underappreciated. This was his best idea, that our mental organization is happening in something that we today experience as embedding spaces, a multidimensional space where every dimension is a feature that can vary along some variable, along some parameter. I, uh, I'm very happy to meet another Roger Zelazny fan. And, uh, who also thinks he's underappreciated and I think wrongfully kind of forgotten by, by this, this era. Um, he's, he's one of my favorite science fiction authors actually. Yeah, it was very creative. He didn't write enough. I think a lot
of what he writes is in some sense slop, but it doesn't matter. It's all good. yeah, has a good spirit. yeah. I, So, uh, if, if any listeners want to catch up on him, creatures of light And darkness is, uh, the most epic one that I think he wrote and, uh, the, um, embedding space reference in this one is, um, the Prince of a thousand names. Uh, a guy who was able to travel to every place, every planet that he imagines and the world that he imagines. And he never knows whether he creates them or
just covers them when he is a teleportationist, which is the rarest of rare talents. Um, in Princess of Amber, uh, which is a very long, uh, fantasy, it was, you have a, a character that is, uh, able to move between universes. It's also in, in a similar way a teleportation is, but he's the main character and it's the main thing, uh, by, uh, changing, uh, the feature parameters by just one bit and it's being described as. Experiencing or treating reality is a lucid dream in which you slowly, gradually, um, move the feature dimensions. And I
also strongly suspect that, um, Elany has discovered this, uh, through lucid dreaming. so I, um, Just to digress a little bit away from AI, I'm very pleased now that finally we've reached a point where very nice realizations of Dune, uh, can be made. And I'm hoping that in my lifetime we'll see some great realizations of novels like Creatures of Light and Darkness and Lord of Light and, uh, the Amber series. So maybe that'll happen. I'm not sure if, uh, creatures of light and darkness can be turned well into a movie. It's, it doesn't have much
of a story. It's mostly archetypes, uh, running against each other. And these archetypes are, uh, so archetypal that they're mostly ideas. I have this general who has been killed in many battles and is only made of spar parts and spare parts and will always be reassembled because there is just this archetype of this general. But how do you show this well in a movie? I think it's possible, but in a way, I think that the text is the most powerful form that Creatures of Light and Darkness can take. Well, maybe a generative model will be
able to take the actual Creatures of Light and Darkness, uh, text and make an audiovisual, Yeah, but our minds are already such generative models, right? We don't underestimate our minds, but what I would like to see is an augmentation of our minds. And Zalesny has this story, He Who Shapes, it's a short story, uh, in which he is describing how to use a generative AI for, uh, therapy purposes. So, uh, immersive VR that is, um, responding to the interactions of Um, the user and, um, it can be controlled from the outside with lots of sliders
and, uh, scroll wheels and so on by the therapist. And, uh, the world that is created is not necessarily super realistic. It doesn't need to be one that is conforming to the sensory modalities of a human being. It's something that is conforming to the modalities of the mind. So, so to dreams in general, and I would really like to see. The two generative AI at some point, something that is able to integrate with our mind so much that it augments every modality of what we can think, feel, experience, imagine, imagine you have this box next
to you and it's, um, observing you and many, many machine modalities. I think that true machine perception has never been done at the moment. We are still learning on data that is created for human consumption at frame rates that humans can process. And, uh, that is the limitation that is not necessary for a technical system. And this means if, uh, our AIs are observing us at a much, much higher rate with much more processing power, they will be able to infer our mental state with relatively high granularity, I think. And when they're able to do
that, uh, they can also display that, uh, using, Arbitrary modalities and this means you imagine something and what you imagine you see on the screen and this feeds back into you and leads to a much higher resolution that you and stability of your imagination that you could achieve yourself. So the AI will become a natural extension of your mind and I would really like to see such systems that allow you to think better, more clearly, more deeply when you're sitting next to them. Yes. And I think you're saying this could be possible without drilling a
hole in your skull, right? Is this how you're Yeah, I'm optimistic. I think that, um, drilling a hole in your skull, uh, should be a last resort and It's probably helpful for some people. And I mean, I would probably get it done if it has the benefits, but it doesn't seem to be necessary because we find, if you look at the best human practitioners, um, they can infer your mental state with surprising level of detail. And if they're just human minds. So it seems to me that. Um, the low rate at which our mind is
working and the way it is, uh, at which it is working, neurons are roughly switching at the speed of sound, right? And the bandwidth of our mind is not that super high. I suspect that there is a way to infer a lot of that state just by observing your organism with non invasive means. Yes, so I wanted to come back though to zombies and consciousness so I think You gave a kind of counterexample to Dennett of Possibility of effective organisms that were actually zombies But the thing which will emerge from this current AGI program and
which then maybe Could be a successor to us. Do you feel confident that thing will have consciousness? It's uh, if you look at the current foundation models, it's difficult to say whether they are conscious or not. I think that a true Turing test for consciousness is very hard for similar reasons also that a Turing test is very hard to make decisive. I think that a true Turing test would require that an AGI explains how AGI works. In that case, we know that it has succeeded for this definition of intelligence, but for consciousness, I suspect we
would need to have a System, we want to have a decisive test that can interface with us so much that we experience it as an extension of ourselves. And that means that we can use the same criterion for our own consciousness. Right? So it would be good enough if we have less, uh, the same degree of doubt that we had about the nature of our own consciousness. The other one is we get an operational definition and, uh, operationally, when I look at consciousness, there isn't the introspective aspect that you are able to report on certain
features, which means the, um, Presence in a now and, uh, the, uh, for flexible perception that you perceive yourself perceiving and so on. But this is also obtainable for current LLMs. They seem to be able to simulate phenomenology so much that the LLM Generated entity doesn't know that it's not conscious, that it's not real. And that seems to be the same for humans, right? On the other hand, the functionality of consciousness, there is a set of mechanisms in our own brain that leads to this side effect of this phenomenology. And this set of mechanism is
not necessarily present in the LLM for, um, Uh, it's generations, it's, uh, only necessary when it's, uh, able to, or that means required to simulate an interaction partner that is able to make that self report on the other hand, to which degree is this, um, thing zombie to which degree is it only pretending role play to be conscious while, uh, going through some, uh, much more mechanistic procedure in which no self is being simulated. And, uh, I think we can see this in the early attempts to create the Uh, chatbots that simulate a conscious human
being and Blake Lemoine interactions, Um, the guy who worked at Google and was convinced that, uh, his, uh, LLM that he was talking to deserved human rights and, uh, was oppressed by Google to which degree was this just interacting with him and producing an interactive text that was predicting the next token in this text without phenomenology going on. And what, uh, you could observe when you read the protocols that, uh, it made things up. about its own consciousness. For instance, it, uh, told, um, him that it was meditating for hours and, uh, how the, uh,
how it perceived the room during that time. And it cannot perceive the room, and it doesn't have a sense of the passage of hours, so it just made this up. And if it make it this up, you can also assume that it makes up the rest. So, uh, what is the way in which you can distinguish made up consciousness from real consciousness? Then again, maybe all consciousness is made up, right? As long as you don't know that it's made up, um, it's a tricky question. It's super hairy and I haven't, uh, good handle on, uh,
on the phenomenological side yet, how to distinguish fake phenomenology of consciousness from real phenomenology of consciousness is both of them are virtual. I know that I remember having been in conscious state that couldn't actually happen. But you can remember sometimes having been in a time loop or you can, uh, Remember having had a dream that lasted for two hours in a very short amount of time in which you are actually asleep. So it's possible that all these memories of being in conscious states are being created after the fact and do not correspond to actually having
undergone a succession of conscious states that were summarizations of your workspace into a single point that you marked in your mental protocol. So if I understand right, you are actually working on creating an institute or some not non commercial entity that will study consciousness, the problem of consciousness. Am I correct about that? Yes, I feel that it's something that is probably not best done in the current companies and also the companies are reluctant to touch it. I know that there are people within OpenAI and Google and other companies who want to work on this topic
and are very interested in it. But, uh, there are political and technical, um, business reasons why it would not be very smart for these companies to come out and say, this is what they're working on. Unless there is a very clear use case and no ethical implication. And there are ethical implications to this and cultural implications. So I think this is a cultural and philosophical project and should be treated as such. If I were a business manager at OpenAI or Anthropic and under pressure to raise the next 10 billion round, the last thing I would
want is an effort within a company like this because the answers could be very, uh, disturbing. for the commercial goals of the company. Yeah. I think it also, um, rightfully would meet a lot of cultural opposition. And I think that it is an important project. It's, I think, a moral in a way to not trickle this project. It's arguably one of the most important or the most important philosophical question that we have left. And it's one very finally can make progress. Yeah. And I, I think it's also related to, you know, things like existential risk
and, and such. So, you know, I, I would be, I would think future of life Institute, FLI and, and some of these effective altruists would want to support research on consciousness specifically, I guess, LLM consciousness, but, but consciousness in general. Yeah. It's also an interesting question. If you can identify criteria for LLM consciousness, is there something like a test, some criterion that makes sense for us to distinguish whether it's conscious or not? There's often also the question, do we need to give a system rights as soon as it's able to experience itself as suffering? And,
uh, that is, uh, depends very much on our culture. Of course. I suspect that our, um, caring about innocence and the protection of innocence from suffering is something that is relatively unique to our present time and cultural space that we are in. Right. Um, from an evolutionary perspective, uh, whether your prey suffers or not, doesn't play a very big role. It's to a goal of your prey to deal with the fact that it's being eaten alive. And if not, then it's, it's probably not yours. Right. And a lot of cultures, uh, still feel that it
doesn't really matter whether the animal that you're going to eat. experience its pain before you eat it. And, uh, our revulsion of this, I think, starts with, uh, parts of the Abrahamic tradition and especially Christianity, where, uh, innocence itself is a core value, the protection of innocence. And this in many ways, uh, has spread and influenced a lot of other cultures. There were also, of course, Buddhism and so on, um, and, uh, Jainism, um, the idea that suffering of other creatures should be avoided. As soon as you are identifying with other agents, uh, you might
want that they don't suffer. On the other hand, if you are, uh, in enlightened enough to turn off your own pain, um, and reinterpret the world in any way which you want, basically change your own source code with respect to the experience of pain and suffering, um, it, uh, the perspective on suffering changes dramatically, right? So if you were a God. like entity that is able to create living beings in a simulation or in biology and so on that experience. And you yourself know how to create an arbitrary experience. Do you think that the experience
of suffering is intrinsically so bad that it needs to be avoided and the creatures need to have good experiences? I think this obsession with good and bad experiences is, um, It's strange, and your experience would be appropriate, not good or bad. The valence of your emotion doesn't actually matter. It is instrumental to what you need to achieve as an organism, as a living being, as a conscious entity. And I think it's much more important to give a system agency over what it feels like in a given situation, that it's able to make itself up to
such a point that it can regulate its own emotions and its own feelings, its own experience of the world. And the reason why we cannot do this for the most part is because we Don't get very old as human beings. And if you are able to cheat too early, if you're able to rewrite your experience too early, maybe you opt out of the goals of evolution. And so. Uh, you should, um, I think in principle be able to learn how to regulate your pain any way, which way you want, but you need to have wisdom
to do so. We need to be understanding what are your larger goals. What is the larger game that you commit to playing? Why do you commit to this larger game? And then you can choose how to, uh, how to relate to the experiences that come with it. Yeah, most athletes learn to tolerate what initially is extreme discomfort, But when they realize that it's helping them toward the larger goal of becoming a better athlete, they then embrace that discomfort that comes with training. So, I wonder if, uh, how many of them are kinky. I suspect that
when you are a person who motivates yourself through pain, that, uh, means that pain gets associated with some positive experience and basically you cross your wires. But on the other hand, if you motivate yourself through pleasure, uh, you also become kinky. You should be motivated by the outcomes, not by the emotions or feelings that achieving, uh, the outcomes instill in you. Um, I wanted to ask you what you think, um, the long term forecast is, um, for The interaction of humans with AGI. So you, I think you're not a doomer, right? You're, No, Okay, so
you expect that humanity by itself is doomed. There is no way in which a lot of humans in the present form, in the present way of organization, identities relating to yourself, will be around in a hundred thousand years from now, without AI. And, uh, with AI, it's very difficult to predict the future. So my P Doom, um, with AI is lower than, uh, without AI. But ultimately, I think life on earth is not about us. It's about life on earth and about consciousness, and we might be able to create a new form of life. And
I believe that this is ultimately a good thing. I'm not worried about paperclip maximizers too much. Of course, I believe that anything that reduces complexity is, uh, first of all, not interesting from my human perspective. So I would say that's not good. I want the future to be more interesting than the past and the present, but evolution tends to go towards more interestingness. At least that's what it looks like. just to clarify that for the audience, because I've heard you say this before, and it's a, it's a somewhat unique view among people Who think about
things like PD. Your belief is that if humans don't develop more advanced intelligence on this planet that we're doomed for other reasons will kind of screw ourselves up and vanish. we lost our belief in the future and somewhere in the 1980s, I think that our generation is the last one who had a glimpse of going into a positive future. And basically in our youth, uh, late adolescence and so on, we observed this collapsing culturally, which means society as a whole, our zeitgeist. Does not envision ourselves living in a beautiful ecology space, uh, flying cars, and
it's going to be awesome and great. Don't know if this is different in China right now, but, uh, after the, uh, end of modernism in the West. We stopped having a future and we are not planning for a future as a society anymore, which is, I think, related to an underlying belief of our cultural zeitgeist that you're doomed, that there is an end to a growth. That is an end to using resources in the way in which we do. And that we ultimately, uh, which we live, that is not sustainable. I think that attitude in
China is more optimistic, but I would only say that confidently in a shallow way that just because they're, they're societies on an upswing, they can easily imagine that upswing continuing. Um, China is in its modernist phase, it seems. And the question is when does China become post modernist but now what you were describing for the West is, is, is more the, The overall feel, but we still have the, the acceleration is among us. So there, there is a subset of people. Not, not all our age who still can visualize that beautiful future. Right. And still
want to work toward that beautiful future. Yeah, but although I have the sense that EAG is mostly a counter movement to effective altruism and, uh, not effective altruism in general, but the aspects of it that have given rise to the doomsday card. Yes. Yes. Um, so coming. So, uh, to explain, I was trying to explain to the audience that, you Know, You actually view, um, our way of minimizing pDoom is actually to build AGI, Yeah, I suspect that the problems ahead of us cannot be solved without AGI. And AI is fundamentally a technology to solve
problems that require information through better information and problem solving. yes. And, uh, so you, I mean, for you, uh, for you there's really no issue. It's just, uh, build AGI and, uh, the, you know, the expected, uh, outcome is actually better than if we don't build AGI. Yeah, I think the issues are somewhat similar to the internet. The internet is seen as a threat, especially in the form of social media and so on by existing stakeholders. But, uh, Legacy media are terrified of social media and the way in which people Can form their own ideas
and synchronize them in which individuals can cultivate large audiences in which new phenomena are popping up and you cannot gatekeep them. But, uh, from the perspective of society as a whole, I'm not sure that this is a bad thing. I think that ultimately Uh, over a long enough time spent within a few generations, we'll figure out how to use social media productively and society is going to change as a result, and it's going to be better. Maybe I'm, uh, my optimism is unwarranted, but I, uh, observed that technology so far has not led to mass
impoverishment. It has not led to the creation of a larger, um, economic divides and so on. It's, uh, contrary to what everybody seems to be saying these days, technology has not exacerbated social inequality, but it has actually, uh. Being able to rise every boat, almost every boat on this planet. And so I don't see why this is not going to happen for AI. I think that there are a lot of problems going to create by bad actors using AI, um, maybe even rogue AI and so on, but all these problems are solved using more AI
on the other side. And as long as there are more agents that want to build and want to destroy, as long as there are more. players that are interested in a long game that is coherent and sustainable, for that long, the forces of darkness will be defeated, I think, as they have always been, Good. I wanted to ask you, because I've never heard you talk about this, do you have An opinion on the simulation question? whether we are currently living in a simulation. Yes. Personally, I think the probability of that is low. That's because the
universe looks like I would expect it to look like if we are based reality. Uh, some things that could convince me that we are in a simulation is if we are observing phenomena that cannot be explained in any framework of physics, something that would fundamentally defy any kind of logic. And sometimes people feel that there are things like this, that they're encountering. For instance, if you talk to a Qigong practitioner in China who is cultivating His key energy Uh, until, uh, they're able to, uh, project mental states into others or, uh, use telepathy over short
distances or, um, manipulate their organisms in interesting ways. I'm not sure that this cannot be explained by current physics. I think it might need to rethink neuroscience or the way in which organisms process information and in which ways they can exchange information. But I haven't seen anything that would compel me to think that physics is fundamentally wrong or doesn't work. Um, maybe, maybe retroquisition would be such a thing. But, um, do you think it's implausible that, let's imagine a future where AGIs, uh, can create simulated worlds and within those simulated worlds are self Aware beings,
um, maybe who aren't aware that they're, quote, artificial. yes, I think we could be living in the memories of an AGI. Bye. those, yes, and those simulated worlds though could have realistic physics, right? They might be, quote, realistic simulations of the base don't need to write if the AI is simply trying to recreate how it came into existence based on available data, maybe in future AI is going to read all the social media archive and whatever data is available. And then it recreates simulations of mental states of beings that got to this state. But the
measurements of foundational physics can be induced memories. There is no need to make this physically accurate, the simulation, all we would need to have an accurate depiction of the mental state that you're in right now. And that doesn't seem to be very hard to achieve. Yes. So maybe the AGI is now watching this podcast and it's trying to recreate our mental states and here we are. And maybe this is the only thing that ever exists of us in this sense. what, what you're discussing is in, in the physics, theoretical physics world is called the Boltzmann
brain problem, which I don't know if you've ever heard that term, but yeah. But I think that's, that's, uh, Boltzmann brain is slightly, uh, Boltzmann brain is only there for one tick, for one frame, and then it falls apart again. And, uh, what we, uh, observe is that we have a memory of a sequence of states that would need to be engendered in this one tick. And while this is not impossible, it is very, very unlikely to a lot, right? So Boltzmann brains are. Probably rare, in the sense that, uh, random arrangements of matter that
randomly form some kind of swamp thing that is, um, able to experience itself as something because it has been randomly imbibed by, uh, by, uh, Statistical happenstance to be asked that's conceivable, but the simulation idea is slightly different. It means that there is a sequence of states that we go through some kind of longer process in which we are cogitating and perceiving and so on. But since our mental states are simulations of our brain, it's also Conceivable that something else is simulating them with the same degree of perceived fidelity as we do right now. Do
you, do you think it's implausible that, let's imagine a future with super powerful A. S. I. s, um, with infinite energy resources, et cetera, do you think it's implausible that they would, that they would create simulated worlds, which in turn have sentient beings inside them? No, I mean, that's what our brains do too. Our brains also create simulated worlds with sentient beings inside of them. For instance, in dreams at night, we create such simulated worlds and we have Interaction partners that we can talk to. And Yes. you don't, you don't find it implausible. And, and
I don't find it implausible. My, uh, I don't bank on being in a simulation. It doesn't look like a simulation to me in the sense that there isn't, uh, the physical universe is simulated. I think it looks like it could emerge, uh, emerge by itself. Yeah. And, uh, not some Minecraft, like, well, that has too many artificial bits, but For instance, Ed Shvetkin, who I got to talk to a few years before he left this plane, uh, was convinced that, uh, there are too many parameters in our universe for that universe to being a random
thing. And he actually believed that the physical universe is probably in some server farm in the parent universe. Yes. Yes. Um, let me turn to your company, Liquid AI. Um, because I think we've, we've been on now for, I think, over an hour. Uh, and I want to be conscious of your time. Um, tell us a little bit about Liquid AI and, uh, what is the key innovation there? Liquid AI has been founded by a team of, um, MIT postdocs, um, from Daniela Rus And, uh, it started out with, um, PhD work by Ramin Hassani,
who, um, had the insight that when you are representing programs in a differentiable form. To, um, uh, Neil, uh, the program that you're looking for, uh, maybe you shouldn't be using neural networks, but we should be using a more efficient way to represent this. And so he came up with, uh, an idea to Um, built networks in which, um, you use, uh, differential equations to describe the geometry of the function. And when you optimize the parameters, uh, of these liquid networks, you end up with something that has more expressivity per, per unit length. unit of
compute. So basically a liquid neuron can express more than a neuron can do in a traditional neural network. And while the mathematics is slightly more complex, the compute that you eventually need for these networks is less. It's not that the thing is learning things that you cannot learn with other networks. You can give it the same loss function, the same data, and it's going to converge to something that is functionally the same model. But, uh, you are able to run the same model on much smaller hardware. And, uh, the, uh, we felt that it's a
good time to build a company around this and to productivize it originally with some people at MIT who thought, Oh, and this, this actually needs to be funded and turned into a startup. And, uh, at the moment, the, uh, mostly working on the day to day things, like how to build an efficient fine tuning pipeline for this, how to find customers that want to work with networks like this. And, uh, what, what is your role? Is your title something like AI strategist or are you writing code? What do you actually do at Liquid AI? for
having me. sadly, I'm not writing code and I, in many ways I miss writing code and maybe I should make an effort to get back into this. Uh, but, uh, I'm mostly looking at the space of AI companies at the moment and of the things that we should be doing that we have on our horizon. So I'm helping this, the strategy of the company. And that's a role that is different from machine learning development. On the other hand. A lot of the practical machine learning development is, uh, Extremely fine grained detail. Still, this is, uh,
it's mostly an engineering discipline in which we tinker a lot, and, uh, you need a lot of cumulative small ideas to get things to work. It's very much an empirical discipline, not so much a philosophical project. If, if your entity, which is focused on setting consciousness were to get off the ground, would you go there and, and work full time on that problem? Uh, I will need to find out how to, uh, split my time between this. I think that there is also a degree of complementarity, but I also feel that, uh, this initiative right
now, uh, needs my input simply because, uh, not enough people are working on it at the moment. And don't quite understand by the notion of machine consciousness is not More popular than it is, um, because, um, it's not completely outlandish and it's really, really important. That it's I don't expect that it's going to be a mainstream thing anytime soon But there should be a critical mass of people who think oh my god This needs to be done We need to understand how consciousness works and actually it can be done with computers Computationalism is actually the
right way to look at the world and to look at the mind and these people do exist But there are relatively few and far between, and I felt the reason why I have to do an initiative like this in San Francisco is because there is no place in the world where there is a larger critical mass of people who consider this as being a real possibility. I felt that while I was living in Europe, that most people were, um, Very skeptical with respect to the possibility that machines can become intelligent, Can become generally intelligent and
extremely skeptical with respect to the possibility of machine consciousness. And even in Boston, I felt that people were quite uncurious this. much easier to get support by claiming also if you are in philosophy that machines cannot think, it will never. Uh, then to tell people the opposite, that it's actually unlikely that they can't. And, uh, it's unlikely that they're in a fundamentally different class of systems than us. You know, uh, when I was in the administration here at Michigan state, we were looking at recruiting a professor from Wisconsin. His name was maybe Tononi. Are you
familiar with, I might have his name slightly wrong, but yes, yes. So I think he has a proposal, uh, for how to define consciousness in a mechanistic way. I don't know if you're familiar with his work. He is, uh, trying to build an alternative to functionalism. And functionalism means that you can describe an object or that you describe object categories about what a thing does. The meaning of information is, uh, What are its relationships to changes in other information? And that is, uh, in some sense, very deep. It has to do with our epistemology and
how we construct reality. Imagine that there would be a water molecule that is behaving like a normal water molecule, but it's fake. Right? In all properties, you can even split it apart into some simulated or pseudo oxygen and hydrogen. But it's not the real deal. In all measurements that you can possibly make, it is going to behave the same. I think that this notion of a water molecule makes no sense. Because, uh, all the other water molecules are constructed in the same way, as a regularization over observables. Every object that we're dealing with is a
regularization of observables. And so, uh, pretending that an object would be different, uh, despite having the same observables, uh, Does not make sense with respect to that object category, right? If you want to create a meaningfully different object category, you need to propose a feature that is different. It leads to different behavior in some sense. Otherwise it is going to be for, uh, with respect to an object that we can interact with and so on, the same object. And, uh, he, uh, thinks that consciousness is something slightly different. I believe that he is a deep
philosopher who is, um, posing as a neuroscientist. And he's a sleep scientist and, uh, he, this seeing something that, uh, other philosophers have haven't seen in the last 6, 000 years or so, which on the face of it is bad news. Right. It's, uh, if you have an idea that is radically different in, uh, not nobody Has seen the possibility of what you are seeing and, uh, the history of humanity. And it's something that you didn't get to by making some arcane experiment in the lab. Okay. You're likely wrong still, I, uh, it's necessary to
do that, but you need to go out on a limb because, uh, what if nobody is doing it? And the other thing is he cannot actually express this theory super well. It's, he cannot write a formalization down of IIT that satisfies me. Instead, the formalization of IIT seems to be designed for the incentives of a defective field. hmm. It looks a lot like to me that somebody says, Oh, for a theory that is able To compete with the gold standard of theories, it needs to have axioms and mathematical formalisms and predictions. And so he comes
up with a section that he calls axioms and they are just a description of what he means by consciousness. It's not axioms in any mathematical sense. And this description is not too bad. I have a few objections, but it's with the vein, which he. Tries to characterize consciousness. Maybe this, um, unity and, uh, so on is, uh, it's a little bit overstretched because it might obscure a long tail of experiences that are, uh, at the fringe of what's consciousness. But as a course, it's not too bad. Most theories of consciousness that other people are coming
up with, um, studies of the hand and so on are not that detail that I say there is a bunch of, uh, features that I commit to that I actually want to explain. And so it's he is not visiting out there. That's nice. The mathematization Well, to be advanced mathematics, it needs to have Greek letters in it. So it has this, uh, factor phi, which is describing the integration of information, but it's not clear what, uh, the scalar means. How does a high phi correspond to different elements of consciousness? At the time, I looked at
some of these papers and wasn't fully satisfied, but I have to admit I didn't exert that much energy, uh, at So the way in which I understood phi, and I haven't revisited it for, uh, quite some time, so maybe my interpretation is not current anymore, was that it's some kind of Mutual information measure. If you take your cognitive system and you slice it every which way, you look at how much the information is correlated across the system. And, uh, it meant people made objections to how FI is being computed. Then they just released a new
version of how FI is being computed. And I got a strong impression also in personal interaction that, uh, the FI itself is, uh, a core part of selling the theory, but it's not a core part of the theory itself. And it's more like a stand in until he's able to properly express what he means by the core part of his theory. And the predictions are, okay, information integration is high in the neocortex, and therefore the neocortex is conscious, and it's low in the cerebellum, and therefore the cerebellum is probably not, uh, integrated in our consciousness
or is not conscious by itself. And that is somehow what you would expect, but it's not clear that this is a, um, prediction in the sense of positivist science that you didn't know this before you made the prediction. And, uh, also led to this, uh, weird thing that Scott Aronson, uh, jumped on where, uh, Scott Aronson pointed out if you have some kind of XOR gate that makes a boring operation, you can optimize it for having a very high phi. And, uh, this, um, Tononi bit the bullet and said, yeah, maybe then it's conscious. And,
uh, Yes. I remember this now. I'd forgotten yeah, I wouldn't want to have a theory to have these properties, but my main issue with, uh, his theory is that he is going out in public and says, um, consciousness is not necessarily something that a digital computer cannot have. But a digital computer needs to be organized in such a way that it has an extremely high fi. So if you have a biomorphic computer that is constructed in exactly the right way, even if it's digital, electronic and so on, and works with bits, then it can considerably
be conscious if it is satisfying all the necessary properties. But a von Neumann computer can never be conscious. Because it's too linear, it doesn't have this necessary phi. And, uh, on the other hand, if you ask him, he does not deny the Church Turing thesis. So he admits that every digital computer can be emulated on every other digital computer, as long as it has enough resources, right? As long as you have enough memory and are willing to wait for the answer. And, uh, so we now have this weird situation that You get your neuromorphic computer
that says, Oh my God, I feel I have a phenomenal experience because I have this high integration of information. And it's, uh, when you look at it, you just see that there are bits being processed in this neuromorphic computer flowing through logic gates. But in the gates that are organized in the right way. And now you emulate this thing in a simulation on your von Neumann computer, and it's going to simulate exactly the same flow of bits. At some causal level and it's going to produce the same output only now it's lying, which means that
the, uh, Numerology computer was also lying. It was not, it was producing this output, not because of some property that the phenomenon machine cannot have. So now we are in the realm of epiphenomenalism. So, Joscha, I want to close out with one last question. Uh, it's a philosophical question. Um, imagine, uh, we've got the transporter from Star Trek working and, uh, the way it works is it scans you, destroys you, transmits the description of you to London. The, the transporter unit in London recreates a copy of you using that information. And then you go about
your day, take all your meetings in London, and then maybe you can beam back at the end. Do you have any qualms about stepping in that machine? I remember that, uh, I had a very different perspective in the past, right? I thought this kind of teleportation is very unsatisfying because, uh, you're clearly dying and then the clone of you is being revived and the other point. And this is not you, but, uh, I've come to the conclusion that identity isn't actually not a thing as only now. And my memory of my past is what's stringing
me together, this past instances. And, uh, there is not an actual continuity. It is, uh, between states, there is, um, something else going on. And you are just gradually simulating your own continuity to a very coarse approximation. So, for instance, you could think of an electron as something that exists whenever there is, um, an environment that affords the existence of an electron in the universe. And so the electron is an operator that manifests wherever an electron shaped hole is there, so to speak. And, uh, this operation is happening. And an operator, like an electron, is
something that doesn't have an identity in a similar way as addition doesn't have an identity. If you, uh, perform a plus on two numbers, this plus is not a thing that has an identity. Um, identity or a continuity that the plus that somebody else is using is not the same plus or different plus it's just the same operator and I've come to the conclusion that everything in the universe that we perceive as an object, this is an operator in a way I am a complex operator that exists wherever there is a Joscha shaped hole in
the universe, wherever the conditions for my existence are being met. I will exist and experience myself as myself. And, uh, this is very difficult to achieve the conditions for my existence multiple times, because there's so many things that have to come together to get the integration of memories and traits and behaviors in certain volume of space. So it can manifest, but wherever that is happening, this is where I am. And, uh, so, uh, in my current thinking, I would say that they, you should not have qualms about using a teleporter like this. Your answer is
eminently logical, uh, from a materialist perspective, and I have to admit, like you, I flip flopped in my life on, in my opinion on this question, so that's why I ask it sometimes to Yes, but I believe that the discomfort that we experience is a result of the way in which we model reality, not of reality itself. We basically tend to assign timelines to objects. This is the way we, which we create identity. And the benefits of having such an identity is that we can do credit assignment that we can, for instance, assign rewards for
us for present decisions in the future. And we can evaluate our past behaviors to prove present behavior. And we can track Other objects in the world that have such a history and that are evolving. And there are objects that are evolving and it's easiest to think or to model, to experience that, uh, these evolving objects are not just different instances that are updated slightly, uh, because the universe updated them, but, uh, made a new release. But instead, uh, we perceive them as the same object that is intrinsically evolving. And that's, I think it's just an
inaccurate representation. Great. So let's end it there. And, uh, let me ask you, is there any place where someone who's interested in your ideas can find, uh, sort of an organized introduction to your thinking? I haven't really made a. Super organized introduction to my thinking. And it's something that, uh, should probably be done. You can, if you are not just interested in following me on Twitter, um, uh, look at my YouTube page where I collected most of the talks and podcasts in recent years. Um, in. For the past few years, my life has been, uh,
very overwhelming. I have kids and so many projects going on, and I felt that I have difficulty to set aside the time for long form writing. And, uh, I hope that at some point I am able to sit down and write a long book In which I explain most of the things. But, um, for the time being, the best way to, um, Um, get informed about the mainstay of my ideas is for instance, to listen to the series of talks that I made at the chaos communication Congress, in which I tried to, uh, identify many
milestones of my ideas and thinking and put them out into one hour talks, so it's not that hard to consume and you can also find them organized on YouTube as a playlist. Great. I will put a link in the show notes. Uh, thanks again for your time, Joscha. It's, I hope, I hope to see you in person sometime soon. Yes. Uh, it was a big honor to be on your, uh, Manifold podcast. Thank you. Cheers.
