Le mouvement Pause AI avec son fondateur Joep Meindertsma (Anglais)

[Music] I'm here with yep minder my hope I'm not butchering your name too badly and with DD Kel with which is this vice president of the transhumanist association the French trans humanist Association yep can you introduce yourself before we start asking you question about pause AI right thank you Gan yeah I'm a founder of pause AI but before that and in also still large part of my life I'm a Software entrepreneur so I make databases and basically two years ago I decided to spend more and more of my time trying to prevent super intelligence from

being created and suddenly I find myself in a life where I'm doing activism and lobbying and all these things that I didn't expect myself to end up doing but yeah here we are so that's basically me okay so yeah we're going to dive deeper into to your your journey and your worries but yeah what What specific events or breakthrough LED you to believe that the development of AI need to be posed which motivated you to start POS AI so I think I think most people before GPT and gpt3 and gp4 in the AI space were

were convinced that it would take at least like a couple of decades before AI would reach human level and then gpt3 GPT 4 happened and the the thing that worried me most was seeing that these things are really good at programming because if you are good Enough at programming at some point you can probably modify the algorithms that you're running on which could lead to even better learning algorithms which could lead to more powerful AI models and we enter this you know this this this short Loop where timelines could be way shorter than I would

have hoped to to have and and one of the specific capabilities that that convinced me personally was that auto auto GPT turned GPT 4 into an autonomous agent so it had Like this this this reason I think like in in heinstein it was more of a coping mechanism where I thought well at least these chatbots won't be agentic right they won't be able to do things because they don't pursue an objective they just answer in chat and auto GPD basically showed okay if you put it in a simple Loop and you feed its output back

on itself and let it do some actions in your terminal it can actually become an agent so for me that was a personal Point where I was okay like my all my reasons for not being too concerned about these models are now proven wrong so I need to get into action thank you you quite interesting how do you respond to critics who argue that posing AI development could hinder beneficial technological progress you know that's the main let's say argument of people in favor of technological progress like probably you and me so how could how could

you concile both aspects so I Think it's interesting to see that you know people like you and me right we're all we all like technology we like Innovation we're all interested in technology advancing Humanity I studied Innovation sciences and doing something that prevents an innovation from happening goes against pretty much all of my intuitions and my gut uh but to to be specific I don't think we should Hal all AI development or we should pause using AI I think we should continue Using Ai and we should still continue to develop some forms of AR or

most forms of AI but specifically AI models that are generally capable and for example that are better and better at being able to potentially improve themselves these are the ones that we shouldn't be making and yes we are missing some innovation in that regard I can imagine if we at some point build a super intelligence that will innovate everything to an extent where it's existentially Dangerous but I still believe we can have a lot of beneficial Innovations just with narrow AI models with with tool AI models we have like classifiers in medical domains we can

use the existing generative AI models to completely transform many of the existing sectors in the world we don't need to have a super intelligence to have a lot of benefits from from AI how do you feel about this would you also say well yeah sure I I mean personally I'm I'm very close from your opinions but yeah here it's also for people listening of course and concerning not anymore let's say existential risks on yeah Global existential risk but other do you also think that AI could erode diminish the democracy and destabilize the economy and maybe

for maybe also you can Define in a few words what is for you really the biggest existential risk of AI it's two different questions but I think it's always good to to make the Distinction between yeah really existential risk and risk who could be very bad but not existential yeah I agree there's a lot of different risks from AI maybe I can start with the things that we're already seeing to some extent so I I think our first encounter with AI has been in social media platforms where we have these recommendation engines that basically predict

what we humans like or what we engage with what we click on and That is already having quite a dramatic impact on our democracy because they you know we're all familiar with with Filter Bubbles and how social media feeds have an impact on things that we vote in elections so this is I feel like this is our first encounter with AI this is only going to get more and more dramatic the more and more these capabilities advance so so right now we have capabilities that creat that that can allow anyone to create huge amounts of

of fabricated Content like completely fake news there are now it's now pretty easy to have an AI model just spin up completely fabricated fake news websites on on a large scale and it's now also possible to Target individuals using llms to convince them using you know more information about them so there there was a study last year which showed that gp4 was more convincing than a convincing human expert at convincing other humans because it could like make A psychological profile of you and use that basically to weaponize its persuasion abilities uh this is also something

that can destabilize democracy because these AI models are in the hands of a few and these few have maybe different things that they want in a society than other people have and that's just you know with current existing AI models the more powerful we let them become and the more they can do the larger these risks become and the More other things they also would unlock so so so for example if we assume that at some point there is Agi right which could for example replace our work right it would replace the work of people

trying to influence decision making it would automate lobbyists what would that what what what does it mean if you have automated lobbyists that are better at lobbying than individuals that would probably mean that the one who has most access to compute or these AI models Would be able to completely control democracy in a way that's I I find it very difficult to imagine a society where democracy is still like a functioning meaningful thing where we also have an AI that is more powerful than humans I think it's very hard to combine these two I don't

see how it can be done maybe it can be done I don't see how yes that's definitely true and I think even for when we look at destabilizing the economy if we have fgi Then something that can automate many jobs if not every jobs because you lobbying is quite an advanced High skill job and then what we are we going to do what it could be I'm not saying you know working is great and we should all work and that's the meaning of life but we need to do the transition correctly and if the transition

happen in in in in in a year it's going to be a mess so yeah yeah it could be great I I do agree that like if we if we manage to pass through This this this this this Narrow Path correctly and we end up in a situation where we don't need jobs because we all live in some sort of you know completely automated yet still Democratic Society that would be amazing but I don't think we get that by default yeah no exactly and and so if you had to pick maybe one capability that you

will Fe you really fear will arise you mentioned coding was maybe the one that you think oh that's the the one I didn't expect with the Current level of of of Frontier Model is there something you anticipate in the future if we continue to push forward and you say like this is really the the threshold we shouldn't pass or something like that yeah for me that's also going to be coding related but it is the capability to find zero day exploits in code bases so so zero day exploit is basically a a problem in a

codebase that no one knows of which can be used to to hack that code so these zero days are Extremely valuable you know security agencies pay up to a million dollars for for finding these zero days because they are so valuable because they allow you to you know hack into systems and control them remotely you get all these you know completely sci-fi sounding tools that actually exist like Pegasus which can you know hack random smartphones you can just they you know security agencies can literally control your entire phone Remotely through these zero days and and

what currently protects I think our our internet in our society that is so dependent on digital technology is that it is really hard to find these so because this threshold is really high you need like a million dollars to find one zero day and you probably need like a couple to effectively hack a Target we're basically protected but when we have an AI model that can find these zero day exploits then suddenly this Cost drops from like let's say million dollars to maybe a couple of dollars you can just run it you can find and

suddenly anyone can find thousands of zero days in all code bases that gives us like some sort of window in which all developers in the world will need to patch their software and close all these security bugs or else one [ __ ] sorry excuse my language but one person can just comp completely destroy a lot of the the the the internet infrastructure That we have and that that to me is is one of the most scary dangerous capabilities that we're getting really really close to okay interesting but maybe so there are many AI laps

and many experts that agree that there is a real risk that AI could even end Humanity there are even people saying that it's almost sure that it will end Humanity but uh can you maybe what is for you the most compelling argument or arguments for the existential threats so now not Only for the society like you describe until no but really let's say this kind of yeah apocalyptic scenario what's your opinion about about those if if I recall correctly you had Lon shapira as a guest not that long ago and and I'm going to say

pretty much the exact same argument as he made so when we're talking about intelligence many people mean different things but what I mean by intelligence is the ability to to take actions that lead to a certain outcome to achieve Goals so something that is super intelligent is better than humans at achieving arbitrary goals given some finite set of of resources and I feel like if we assume that there is an A system that is better than humans at achieving goals that by definition is also able to beat humans at achieving goals and that would logically

mean that whatever goal that system has is the thing that will be achieved now if we don't know exactly what kind of goal we Should give to that system it could very well be that that system achieves some sort of goal that in doesn't include humans actually that should is what we should expect so for example the paperclip maximizer story is like hey we have an AI model that tries to create as much as TS with creating as much paperclips as possible at some point it will realize that humans are made of iron which you

can use for paper clips and that's a completely rational logical Idea so so I feel like because there is this thing called instrumental convergence where where whatever goal you put into a system you can expect some sub goals to arise that is the most powerful argument in my opinion that it could be an existential risk right because preventing from being turned off is one of those key things that an AI would want to do even if it doesn't really care about some some some specific other things Because not being able to not being turned off

is extremely useful for achieving pretty much any other objective in the world Gathering as much resources as possible is also very useful getting rid of competition is also very useful preventing that humans can interfere with your objective is also very useful so what be rest is how powerful do you imagine a system to become and if we don't see any like hard limits to how intelligent system can can Become then I think the logical conclusion is at some point it will want something different than what humans want and that means we are like in an

evolutionary competition with that thing and we're going to lose yeah yeah that's the I think a good summary of the of the big issue and well you mentioned Le sh I know you asked this question a lot do you have a p Doom what's my P Doom well I I used to say it was about 40% but I feel like it has grown a little bit Because I feel we're getting closer to these extremely dangerous capabilities and we as a society we're not acting as rational adults we're just letting these AI companies Advance further and

further and further so I would say now it's more close to like 60% I do still think that there are ways we we can make it through but I don't think it's the most likely outcome at this at this point unfortunately and I guess what's your time frame like when you say maybe 60% Is in is it before the end of the century before 2050 something like that yeah this is I I would expect super intelligent AI capabilities to arrive in somewhere in the next couple of years with large Arrow bars like it could happen

next month I wouldn't be surprised if that happens but it could also happen you know in a couple of decades also wouldn't be very surprised if it takes somewhat longer right but but in in in 100 years yeah I think then We'll we will have like reached that point where either we're all dead or you know we we we have some achieved some sort of stable state where humans and AI coexist yeah maybe one comment about you know if there is no technological progress in 100 years that's also sure that we will be de the

three of us that's yeah so there is there are existential risk and there is something existentially sure at the moment it's aging but it's well partly outside of The the scope of this of this discussion now I have a a question of almost a question of vocabulary so the word the words existential risk and doom they can be perceived as too negative and also many people don't know about well certainly pedom and probably also most of people don't know about existential risk so somebody that maybe somebody maybe you know David wood a kind of techn

progressist active in the UK said to me A few months ago it would be better to use the the bir AI safety activist so to put the important aspect on safety and not on the risk you know what do you think about this this question and maybe why did you choose as an official name pose Ai and not like safety a Sai yeah yeah good questions and to be honest I'm still like pretty un sure about what is the best way to approach all this what is like the strongest marketing you know way way to

to to best best way to Describe these words I I really dislike that for example the word Doomer is so prevalent now in the media because I feel like it's it's at least for me like just a wrong description I'm not a pessimistic type of person I'm pretty optimistic which is why I still think we can fix this whole mess right but but when it comes to to to to the name pause AI I wanted to have some something that is extremely clear in its ask like what what we want is to Halt development we

Want to pause it and if we say we want safe AI well safe is kind of hard to Define right what what what what does safe AI mean like in the end it's true I would want provably safe AI systems which I think we'll we'll get there in a moment I we'll discuss it in a moment but safe safety and AI can also be interpreted as things that are like implementing guard rails before you deploy AI models or preventing AI models from saying bad words to people or Making certain jokes and I kind of dislike

this like how the AI safety terminology has been captured by companies now so companies like open Ai and anthropic getting to Define what safe AI means even what AI alignment means where I think it's it's quite different from what has was what was the original meaning in the AI safety sphere which is like how do you make a system that you can make sure it doesn't end up killing you and right now it's like how Do you make a system that you know in 99% of the cases asks what the user wants which is a

very different kind of problem so so I I think language is important but I would like to stick closely to words that describe halting progress because I think that's the most important policy ask right now and and regarding NE negative language about for example Doom I do I do kind of like the the P Doom as in that it kind of forces people to think about probability Right you want people to to think not in absolutes or or like will AI kill us all or won't it right you got to ask people why do you

think it's likely it either may or may not so that's I think it's it's a powerful meme P Doom in that sense yeah yeah I think so as well um and again trying to to be a little bit more precise as well than saying I think it's likely or it's unlikely or or you know because if you say it's unlikely what does it mean exactly is it one Chance like for I don't know 10% 20% it's it's still a lot I mean when we talk about the end of humanity how much percent would people actually

accept I think for most people if you do surgery right in your back or something and there's like a 1% chance you turn out to be paraplegic or you die most people would just not take the surgery unless there's like a really really good reason to take it like if you're in constant pain or whatever but but for For AI risk the average estimate from AI researchers at this point is like 14% how how is that acceptable how are people not completely freaking out about this I I just I'm so confused by the lack of

emotional response that people have to this problem like I I feel like an alien sometimes when I discuss with PE people this with people and they're like yeah yeah maybe there's a risk that we're all going to die and then like they just go along with their day as if Nothing has happened right I think we yeah it's easy to deny these things I guess yeah maybe if I if I may say on one side I totally agree but also we live in a dangerous world now you know for example the the risk of global

nuclear war is still high and is even probably higher than before and the AI like it exist now is probably making the risk higher because there is a big risk of starting it very fast yeah so you have to kind of choose between having Better AI with risk and having the situation that we have now without better AI but with the other risk so yeah not not easy at all what's what's your point I suppose that your point is yes we can accelerate AI but not AGI something like that can you say that in your

opinion about this in a few words so how to how to mitigate other risk and existential risk related to AI at the same time o that's difficult I I must Admit that I'm mostly focused on AI risks also because like there are so many other things you could dive into we could also talk a little bit about bio bios safety risks but I I I think that most of these risks can be averted through Global Co coordination and cooperation I think most of the existential risks for Humanity arise from technical development from from technologies that

we either haven't built like for example supercharged Viruses or artificial super intelligence and preventing that these are you know the these these these Innovations are being created I think is one of the best things we could strive for and we don't need we don't need AI in itself to achieve this we just need to work together as a world and get you know get our incentives aligned so to speak and I'm pretty hopeful that we can as in we did before right so for example in the end of the 80s there was the Montreal Protocol

which internationally banned the use of chlorocarbons which was creating like an ozone layer hole and I think we I think we can do stuff like that again we can ban certain Technologies we've also banned cloning in a way like nobody is doing human cloning anywhere on Earth uh nowhere nowhere on on on on on at least like not in large scales that are extremely dangerous I think if we can get AGI to the same level as we do with human Cloning where there's maybe one Rogue scientist somewhere doing it then we're probably way in a

way safer situation than now where like companies are explicitly doing this and burning billions of dollars trying to achieve this dangerous objective yeah that's a good point and speaking of your Global coordinations how do you how do you propose implementing an international pose on a AGI or super AI yeah do you have to is it I guess it's it's it has To be International we can't just say oh in Europe we won't do super AI but I mean we're not really far in the in the race anyway but like if if because there's a lot

of people who will kind of use the adversarial competition if if the US doesn't do it China will do super so we have to compete and this is kind of what we've always done and but I think that's a that's not really a good way to see the problem yeah like it's it's true that we that we're we're Definitely in a race right like where the idual incentives are to you know Advance Innovations and create more more and more powerful AI models whereas collectively it would make more sense to take things slowly so so the

problem becomes verification like if if there is if you basically say okay we're all going to pause how do you make sure that the others are actually pass passing and not creating AI in secret and there was this paper from Akash basil which was Published a couple of months back showing I think like 11 mechanisms of of Chip Fair like like verifying that AI models aren't being trained so because these training runs are still mostly pretty big you can Target data centers you can put people in data centers at different countries to make sure they're

not training their next a very big AI run you can use thermal cameras on satellites to make to to find like energy signatures but more most Importantly uh in order to do a large scale training run you need a lot of compute and compute is really difficult to make at this point only a handful of companies are able to create AI training chips basically 90% of them are coming from the tsmc factories in Taiwan 100% of them are being made with asml lithography machines so the fact that we have near monopolies and even true monopolies

in this supply chain to me makes it seem like okay we can do Something about this we can just point to where these machines are in the real world and by the way these lithography machines they have actual remote off switches right because asml and the Dutch government are afraid that Taiwan will be invaded in China and that way they want to prevent that China gets access to all of these lithography machines and basically win in in the in the chip Wars but but this this to me makes it hopeful because we we can there

There's like a centralized supply chain and we can do verification externally we can do verification with on Shi governments which is also kind of interesting you could add like little chips next to the AI chips which do things like make sure that the AI chip doesn't run dangerous code or it needs some sort of license or it needs to report back to a different server so there's also technical approaches to this which you can enforce for example At the lithography stage so I'm pretty hopeful of the the feasibility of preventing the all the largest training

runs but we need to act fast because these things are getting faster and faster and faster at a ridiculous bace and modern AI models are often even smaller than the ones from like a year ago yet they have the same or better capabilities so I'm not sure how much time we have but the optimistic side of me says well we should try anyway and Maybe we do prevent the super intelligence from being created in some sort of Garage on you know a bunch of MacBooks right yeah yes because that's the question I was also thinking

is we are probably going to see better algorithm better ways to train AI on a smaller infrastructure so and and and we see some big compan is releasing open source model or open weight and obviously when you have an open weight model you can Jailbreak it very easily And remove the the yeah the safeguards so how do you see the open source discussion into posei do you do you think it will be possible to have a way to enforce a pose even if we have a lot of Open Source models around doing stuff yeah yeah

it's it's becoming more and more difficult to implement a pause on smaller models so for example this new Nvidia digits computer which is like a $33,000 supercomputer you can put on your desk and it has a p flops of Compute capabilities that means it can train 200 billion parameter models or like fine-tune 200 billion parameter models you can do it at home in the near future that's absolutely wild and I think that if we want to Halt AI like POS POS AI development we you should probably prevent these things from be becoming better and better

but as of now like okay cats kind of out of the bag with this capability level right and with with open source models it's a very Difficult subject for me because I I'm I'm a software engineer who makes exclusively open source code like I I love open source I'm like I'm I'm like married to the whole community and the idea of open sourcing things and with AI it does have a lot of Advantage because you can do Safety Research you basically prevents the the the monopolies from getting all the income and and of course you're

also accelerating Innovation and this to me is is the dangerous thing We're accelerating AI Innovation because these models are open source and indeed these can be jailbroken and even a model from you know Facebook that is prevented from doing cyber security offense capabilities can be jailbroken and you have like a new open weights model that suddenly can do all of these offensive cybercity capabilities and right now they're not smart enough to be extremely d dangerous they are somewhat dangerous but in the near future they could become Extremely dangerous and I'm I'm if I'm honest I'm

not sure if we can pause the development of these these smaller models becoming dangerous so we kind of have to you know hope in a way that the most dangerous AI models will also be the largest ones and we can Fair least Target these yes yeah I think so because I also heard some arguments saying oh we'll have basically some sort of good AI versus bad Ai and they will you know and if we have like maybe if we can Regulate the big ones so no super artificial intelligence but we have like capable models and

some people some Bad actors they use them to do harm but we also have um you know obviously white hats doing you know using AI to fight and so that that's kind of a war that we already seeing in the hacker Community because you know there is always attack and defense uh and when you have a new virus being developed quickly you have antiv virus and then it's it's always The yeah it's very much the question like how would the offense defense play out with with with AI I personally believe that offensive capabilities are often

like at an advantage so for example you mentioned virus and immunization right like getting an vaccination for example like probably creating a new virus in a lab is going to be way cheaper than developing and spreading an a vaccine because the vaccine you need to like actually travel It across the world and you need infrastru structure whereas the virus can autonomously spread right it's it's kind of like it spreads itself yeah so so that is like there's one offense defense problem there with cyber security you have the concept of the window of vulnerability whereas there's

like the time that developers need to patch their systems in which the systems are still vulnerable which is very fundamental and that also creates an Attack factor and with a super intelligence I would expect it to be able to spread across all devices like it could it's smart enough to hack into everything it could hack into your laptop your smartphone maybe even your thermostat I'm not sure like how well connected that is but you should expect it to you know collapse onto a single T that spreads across the internet I think that's a very likely

outcome yeah that's what a very smart Agent would probably do yeah so okay do you also think sometimes about the second best option to avoid the end of humanity if the pose doesn't work because as you said we have a limited amount of time to do it the windows of opportunity is closing soon is there maybe some other options we can hope for yeah so what the pause does it buys us time to spend more time on on safety and spend us more give us more time to think about what kind of future do we

actually Want because once we have a super intelligence and it starts to do the thing then it's too late to change it I think like the alternative is making sure that we get it right on the first try so that's like doing AI Safety Research working on alignment maybe even just dreaming about Utopias I think if more people would think about what an UT to Opia would actually look like how the power structures would work and we we just just write it down that could be Useful maybe you can like give it to the AI

model and at open Ai and anthropic and say like look if you ever have a super intelligence just give them this document and at least it will have some sort of you know goal to work towards that isn't like maximize paper clips or just create more a more powerful version of yourself right but to be honest I don't think we have great policy options that are second best we can do like guard rails we can add some guard rails To AI models where we say well if the AI model is extremely capable at these specific

dangerous domains then you you cannot deploy it but then still the capability exists within a lab which is still dangerous because at some point someone will create the similarly capable model somewhere else or it will be dangerous inside the lab itself because it probably has access to the internet right yeah so so I don't have unfortunately I don't have a very really Really good second option yeah about this these about what I would say I'm calling that sometimes existential hopes so you can imagine that we explain to AI what would be the best world and

also we use AI to have a better world with healthy resilient and happy immortality it could be a solution by by the way you very probably know Nick bom and yes he so he wrote this book super intelligence already 11 years ago but recently he he wrote a book called Deep Utopia and that Was about yeah what would be the perfect world but it's kind of a philosophical work and he doesn't speak so much about AGI as yeah as this aspect I would say existential hope and artificial and existential risk uh maybe no one more

question about so we all agree here at least but also probably theorically most people that we have to have a safe AI what Matrix of Benchmark would you use to determine when it safe to resume uh to start to to to go further for AI Development let's say dangerous AI develop development and maybe even more important to Define by who you said that International agreement would be the the best solution but you know it's very difficult there is an AI International Summit in a few weeks in in Paris maybe we will speak about that later

but we can also some people imagine that one agreement between let's say the the biggest players USA China and a bit Europe would be enough what do you think About which benchmarks and created by who it's a very difficult question I think for for for which AI models shouldn't be regulated we should probably just look at things like parameter count uh AI size and do some evaluations after it's being trained but we shouldn't train those that are like bigger than than the Threshold at all I think this should be defined by some sort of international

agency of AI safety right we got the the the the UK Aisi and the US aisi I feel like maybe a collaboration between these where they you know put some draw some lines in the sand and say like okay this is this is too much we shouldn't do this thing that would be a way better Universe than we are in right now and yeah but but but at this point it one of the core issues is that these agencies aren't trying to push for a slow down down or a pause which to me is is

kind of wild right like for example Ian Hogarth he wrote This article like two years ago it was we need to slow down the race to Godlike AI it's one of the best AI safety articles I've ever read and then a couple of months later he started leading the AI safety Institute in the UK which I think is great and a the UK AI safety Institute is doing a lot of important work with regards to evaluations and benchmarks and guard reils for AI which which is good but there's no more talk of trying to slow

Down it's it's kind of like outside the Overton window in the political Elite spheres I think because they're scared of slowing down their own country too much right which is the the race condition or being accused of being anti- Innovation or anti- economy so it's a it's a tough political situation I think which is also why I'm a little less optimistic than I was two years ago maybe yeah I I agree with you but you the race aspect the the biggest let's Say potential enemy actor at the moment is China and in China they seem

to have the two Trends one yeah to accelerate but one also big to slow down so don't you have in your answer you were speaking about us UK and so on but not not at all about China can you why not so China is is I think surprising yeah maybe of course it's China it's it's already not easy to know what's happening in in US Europe and so on so China it's even more complicated Of course but but still yeah CH China is doing a a lot of interesting things in the AI space because they

they have had more strict regulations than the UK and the us at some point they just banned all Ai chatbots and in its entirety in in in their country and they have said like like on on International meetings they are the only National actor that has explicitly mentioned the possibility of a pause so for example in the in the UN Security Council meeting if I'm not Mistaken they were the only ones who brought up the option to pause which I think is is a good sign so I'm I'm more worried about the US and I

am about China at this point but I do think that China is is China is definitely racing I think yesterday they announced a plan to invest $41 billion into the Chinese lithography market so I talked about asml which is like this Dutch lithography company which right now has a complete Monopoly on euv machines Which is used for all these AI chips and China is basically saying hey we want we want that too we don't want to be dependent on a technology that we can no longer access because the US and the Netherlands have prevented China

from getting access to these lithography machines which is essential for chip fabrication and right now they're investing a lot in their own company smic which is you know catching up to some maybe more rapidly than a lot of People had had anticipated and Chinese AI labs are also getting more results than many people have anticipated so for example the Deep seek V3 model which was released a couple of weeks back was trained on just 40 gpus which in AI land is almost nothing yet it matched the capabilities of most of the frontier models which is

completely wild right it's basically showing that the Chinese they start getting more and more of the hardware they also are pretty good at The training and software perhaps even better than the the US to some extent so yeah it's it's I I it's like they're in a race and they're they're they're worthy opponents worthy adversaries but we need to think of them as being part on the same team right we're all team human we all want to prevent the super intelligence from taking over so let's just grow the [ __ ] up get together in

a room and prevent this thing from killing everyone it's pretty obvious when you Think about it right like just just don't do the stupid thing yeah because when you look at the AI act from the US I mean at the end of the day it kind of lead China to find other solution think outside the box and maybe they're going to make progress where we shouldn't make progress maybe they're going to make like amazing new algorithm as you said that that can be trained on 40 gpus maybe that so they they're accelerating another branch of

dangerous capabilities And if we had just said oh we're going to collaborate you can have you know as many chips as you want maybe they would be like okay the the race will be slower in a way that's a shame good for well be yeah yeah and I was wondering also if because you said that the it seems to that the PO AI is is almost too late for some leading intellectual or the the government they don't even talk about it like do you know if very prominent a researcher who are very worried about The

risk like inter or Benjo what do you do they think about POI and have they said anything positive about it I suppose but I don't know exactly I know that that Benjo thinks were doing we're doing important work I think Hinton Hinton is pretty pessimistic about the feasibility of slowing down he did call he did explicitly say like I think you know two years ago or almost two years ago something like oh it would make sense not to build these things in the First place um so I think he's ideologically aligned but he just feels

like it's it's kind of like it's not going to happen right like we're not going to be able to pause but yeah it's it's kind of striking to me that if you look at who are the number two no number number one number two and number three most cited AI researchers that each one of them is publicly saying look this thing can kill everyone right like Elia Sater Joffrey Hinton Joshua Benjo They're all saying the same thing publicly so it's maybe we should maybe we should listen to them right yeah yeah that's a very it's

bit like Cassandra effect I guess yeah it's like I really feel like we're living in this this this Looney Tunes Universe it's more ridiculous than the movie don't look up in many ways like H how how is this not how is this not completely obvious to people that we're doing something that is extremely dangerous and we're just Letting it happen it's it's completely wild to me I just I just feel so alienated yeah yeah and and uh so you mentioned a little bit about alignment that we have to get it right if we don't pause

and let's say we just go ahead and we finger cross we succeed that a alignment it means we really I guess how hard do you think the alignment problem is do you really expect that it's extremely hard maybe some people think it might be impossible Roman Yosi said It's like building the perpetual motion machine or something like that do you I don't know if you've studied the alignment you know literature there's there's some Avenues maybe like mechanistic interpretability or something like how hard do you think it is I think it's I think it will be

way harder than people tend to think it is but not necessarily impossible like I I I feel I don't feel extremely confident about how hard it's going to be but I Feel like there is a lot of naivity about easy ways to get there so for example you mentioned mechanistic interpretability like I don't really see how mechanistic interpretability will lead to fixing the alignment problem like even if you can look into the brain of an AI model which is like something that is absolutely you know good to have from a safety perspective although it could

also accelerate capabilities I don't necessarily see how that means That you will be able to control it because the fundamental problem is it only has to go wrong once only one time it has to decide well I'm going to spread myself and take over and if it's capable enough you're [ __ ] and and you need to have like a system that is 100% foolproof like it never has to go wrong otherwise you still collapse onto AI takeover I think that's also the point that Roman yski is making like it it has to be completely

bug free you can There's no room for error because one error and it's game over that's what that's why I think like alignment is going to be really difficult but maybe there is some sort of pivotal act that can be done like if I would have a super intelligent AI one of the first things I would try to do with it is prevent that a different super super intelligent AI will take over the world I would probably try to get it to implement a pause or something Like that right just to prevent AI capabilities from

growing further in the world but even that could backfire maybe I want to say two two things and and that you we about that maybe about the risk that that AI is that AGI is better better or more powerful than us in a short term situation I remember already 12 years ago there was IBM Watson winning for JA party you know and at this time I thought yeah no it will be very fast something like IBM Watson will Be online soon and yeah it's better than humans for the most complicated question that's questions that you

can imagine so it will be better than us for everything soon and it was not at all the case I I mean IBM tried to develop IBM Watson for health and it was a failure so yeah can you imagine that it's going to kind of slow down for 10 years that was my my first remark and your reaction and the second remark is you said at one moment nobody wants AI to be better than us and To that Humanity disappears there are a few people who think that it's kind of yeah after the humans we

will have consenus artificial intelligence and we will fuse with this intelligence and it is better let's say for yeah the whole universe can you say a few words about those two aspects yeah I I think first let me take the SEC the second one because like it's it's a bit more emot to me so I think like in an abstract sense you could say something Like yeah maybe there will be like a successor species and that you know that takes over and it's more intelligent so we would like kind of have to make room for

that new species to you know enter the world but then you have to bring it back to your to to reality around you like would you be okay with that model like killing your friends right like that successor species like killing your friends and your children and like destroying the entire planet like would You be okay with that I I think that it's like this this to me is like a Monumental way of of of coping in a way it's like we have this very dangerous thing we don't want to think about how dangerous it

is and what we actually have to do in order to prevent this from happening so we like kind of give up and accept our our horrible premise in a way so I I feel like saying okay to human extinction is is basically you're betraying your own kind you're betraying All all the people who you love it's it's such a horrible thing to do I I I find it difficult to to like understand it but I think it's I think it's a coping mechanism basically and I'm sorry I I completely missed the first question sorry to

interrupt you again but yeah that's not not anymore an existential risk but also some people think we will fuse with you know but okay that's not anymore than really an existential risk sorry well that's interesting like I I Think the fusing thing like earlier on I had some intuitions about this like yeah maybe we can like combine and we have like some sort of transhuman thing but I then I thought it doesn't really make sense because you have if you have this thing which is more capable than us like we wouldn't we wouldn't bring anything

to the table like it would just do its thing without us so I I don't think we like I don't think the fusion paths are very likely maybe it would be good if Something like that happens so so I'm not sure I'm not sure if most people would agree with this yeah because it's it's a bit it depends the different the the difference between the two level of intelligence is because it's like are we going like in my daily life I don't you know I don't really have an ant with me and I don't you

know I don't help ants and birds so much I mean I don't mind them you know they're fine but as soon as I see maybe Something that I don't really like it's a different story if it's an insect so you know maybe the insect when they saw human are arrive on the planet this oh maybe this species can be good for them or you know they can give us more stuff but um and and it's the same with chess for example since the AI are more capable than us as at playing chess if we have

a human in the loop it's basically noise the a is much faster without the human toit us so why why why Oh that's a perfect argument yeah yeah yeah I think the question didd asked first was also I think he said something about Watson if I can maybe re ask quickly basically do you expect maybe another AI winter that that maybe a possibility we might realize oh I don't know like the Deep learning Revolution is plateauing so we don't have enough data or the compute doesn't lead to anywhere we need to find another breakthrough so

may maybe this going to Take 10 20 years before we see a new AI summer and this is enough to make progress on on alignment it it could be like I I think it's within the realm of possibilities it's so hard to predict like what what will happen but I feel like what has happened in the past is that pretty much everyone underestimated the rate of progress pretty consistently and that includes myself right I have also like I've been basically too pessimistic about how long it takes so But but but I feel like it's it's

we don't know and we should air on the side of caution like even if there is just a 10% chance that this current Paradigm will scale up to super intelligent AI we should not you know we shouldn't we shouldn't take that gamble it's so incredibly dangerous like even if it was just maybe just 1% I would argue we shouldn't do it the risks are too high if but if I had like if I had some sort of certainty and proof about that it Will take at least 20 years and I would probably not have started

POS AI but you know just the uncertainty itself and the the amount of Arrow is pointing towards shit's going down really really soon that's that's very for me and like enough of reason to to say shut it all down yeah yeah and maybe you didn't say that explicitly but I want to say it well I suppose you will approve explicitly if when let's say the people in the street or major the large Majority of people when you ask them and they understand that there is a serious risk of destroying Humanity in the process they well

almost 100% of them agree that we have to POS so since we are in relatively Democratic countries we should take this in consideration also and we don't at the moment yeah the next question is about the The Summit in Paris on AI in February so it's the 10 and 11th of February what is your expectation for this event so I don't Have a high expectations for this specific Summit unfortunately so basically when we started B and we started protesting we were asking very very explicitly for a summit we said like please organize an international Summit

work towards a treaty like this is the only place where it can happen and it needs to happen there and then Rishi sunak organized the first AI safety Summit in Bley Park we were like really ecstatic so happy that was Happening and then you know that Summit came and went there was a blly decoration but no serious regulations being created then there was a mini Summit in SE which didn't really do a lot of things and now the next one is is is going to be in Paris this is going to be the big Summit

but as as soon as it was announced the name was changed from AI safety Summit to AI action Summit and it was very clear that the people who are organizing this Summit they have no Idea what they're talking about like I honestly think that they have no idea what kind of risks are involved and they're not taking this seriously I think it's it's it's absolutely a shame to see how little Focus there is on AI safety during this this next Summit like the program I think there were like five categories and none of them were

basically diving into the safety issues the existential risk from AI so I I I Feel like our leaders are screwing us over like they're they're they're they're not they're not being the adults in the room they should be focusing on the small things that things go really really wrong in very short amount of time they need to like consider the worst possible outcomes and prevent these from happening and right now they're just looking away they're just ignoring the problems they're ignoring all the warning calls they're ignoring The AI safety researchers they are ignoring the AI

experts like benj Benjo and Hinton it's a it's a disgrace I yeah so my expectation isn't great this is also why we're organizing an international protest we're organizing a protest in Paris but also in 14 other cities across the globe to get more attention on on AI safety basically to to ask the people demand that the people at this Summit will spend their time in a useful way to prevent the worst from Happening because like that's the only place in the world where it can happen individual countries cannot stop this it needs to happen at

a at a treaty level at at a summit level and this is like a huge opportunity and I feel like it's one of the most important things we could focus on and get people to to to to spend their time persuading attendees and politicians that they have a moral obligation to prevent catastrophe from happening yeah go concerning the Organization of the next Summit so it's organized by France and the situation is even worse than what you say it's not that they don't speak about existential risk that's uh in some documents they explicitly say what is

not true that there is a kind of a consensus that there is no that that the existential risk are not so big or even don't exist it's very very strange and yeah you you know mising also for the people listening the the the situation at the Moment is in France it's kind of paradoxal because you you know French people are very often kind of contrarian and afraid of technical technological progress but here it's the other direction the president maon and the most wellknown AI scientist on The French speaking World Yan L and the people making

the AI mistal they are all in favor of let's say accelerating without many Nuance now they they were officially how can you Say that contacts with the Civil Society and contact with citizens and one of the proposals of the citizens was explicitly stop all AI so not only AGI you know but stop all AI so this was a conclusion of this study of what people think but I'm Almost sure that they will be they will not speak about that when they will speak about consultation of civil society and concerning Civil Society also there was there

was one point explicitly yeah speak more about Existential risk but there also I'm afraid they will not be they will the the leaders will not speak about that so like you say it's important to have actions but I I will ask you what what are the actions that we could have also not only POS people in POS a but other people who are interested in this problem and I will give you my point of view I'm afraid yeah I of course actions are important but those actions has to be before the summit not after that's

Well of course also after but they are more important before the summit and I'm afraid if you don't have contacts directly with let's say High people in charge it's not going to have a big impact but about this last point I think it's really important like I think anyone can have a far larger impact than they think if they just send an email to these high people in charge like it it is absolutely wild how few people are trying to reach out to high politicians Or ministers directly almost no one does this right so they

don't get that many emails and like I'm I'm constantly trying to do this right I'm constantly speaking with posi volunteers just getting them to send emails to politicians it's very hard to convince people to do this because people find it very awkward to reach out to someone who is that high in power right there like this this power asymmetry and we're basically taught as children like you You know you kind of leave the really powerful people alone you're not worthy enough to speak up to them but please do it please send an email to a

politic it is really one of the most overpowered things you can do with your time I've had I've personally had big successes just reaching out to politicians and saying Hey listen I'm concerned about AI here's some information about this just link to a bunch of studies link to quotes from from high status AI Researchers and that can get you a conversation and in that conversation you can talk about the policy measures that are needed but we need people to speak up and reach out to these politicians because otherwise they just don't know and it's like

completely outside of their their their their their sphere so I think sending emails to politicians is the number one thing I would recommend to people number two thing is join the protest because Protests are a way to show the world that you care enough you care so much about this topic you're willing to spend your free time go on the streets hold up a sign right like that that is something that is I think is really powerful and it it gives us a strong signal and po politicians do listen to that but just and but

like also what what you two are doing right now like getting more attention to this problem just speaking about it discovering like like other Opinions about it that is also something that I think anyone can do find a stage small or big get people to talk about this but just don't just don't keep it to yourself be loud be active inform the world we all have a moral obligation to do something because right now we know about this thing and we make we we have like an obligation to protect the rest of our species and

and inform other people of these risks because who else is going to do it Right yeah yeah exactly I think so I I guess to conclude if someone wants to know more after this conversation they can go to posi the website I suppose and you have a like I guess what what's the list of things people can do online and look for more resources and send emails to their politician of course yes yes and and join PA like the French branch of Pauli led by F he's doing amazing work in France like and he's got

a very very cool community of people around him All like spending their volunteer time trying to save the world it's it's really inspiring yeah and I just want maybe for people who listen they can hear my conversation with Maxim on this podcast as well if they want to have the POS POS French version but that was that was very very good thank you again y for your time it was great speaking with you and keep up the good work informing people about these important topics thank you [Music]