Is this how AI mania ends?

For the last 3 years, all of us, whether we know it or not, have been making a big bet. And that bet has seeped into almost [music] everything. The companies we work for, our investments, even the stability of [music] the places we live. That bet is called AI. If you have money, you're supposed to like put some in stock [music] and some in bonds and some in real estate. And it's like we put it all in this like crazy stock. They use a roulette wheel and we put it all in one stock [music] and maybe

it's not going to land. >> Today AI expert Gary Marcus who has been excited about the technology for a long time talks about how it could be that we're making such bad bets. You have some people like Sam Alman and Dario Amod who's the CEO of Anthropic implying that like we're going to cure all diseases in the next couple years or cure [music] cancer next year and this kind of craziness. Those people appear to me to not understand science. >> And he [music] explains how the impact of those bets could ripple through society. My biggest

worry is the government is just giving a blank slate to people who I think really don't have humanity's interests [music] at heart and you know they're giving them so much power that there is kind of a race against time and then my secondary worry is it might bring down the whole economy. Welcome to It Turns Out. I'm Cara Miller. Gary Marcus is a professor ameritus at NYU. He's the founder of geometric intelligence and he's the author of taming Silicon Valley from MIT Press. He has watched as just a few tech companies have driven stock market

gains and as AI holdings have popped up in pension funds making our dependence on AI much much greater than we might imagine. >> The whole economy is really tied up in this. [snorts] So in the worst case, and nobody knows how bad it might get, um, a lot of the banks have been lending money so that people can buy this stuff on leverage. And so in the worst case, we wind up with a liquidity crisis like 2008 and the same solution, which is a bailout. And in fact, very recently, David Saxs, who's the White House

AI and crypto adviser, bizarre, um, basically warned on Twitter. He said, you know, half the GDP is tied up in this or something like that. And you know, if it goes south, we could wind up with a recession. A lot of people are worried about this and should be. >> How much do you personally worry that this is a house of cards? >> Well, I worry a lot. I mean, I'm not so worried for my own personal finances. You know, I'll be okay. Nobody needs to mourn for me. Um, but I worry about society. I

do worry that we are way too tied up in all of this. I think that people like Sam Alman told a story about how AI was going to be magic. Some AI someday may be magic, but this is not. The thing that we have now, generative AI like chat GPT has a lot of problems that have persisted for years. Some of them I pointed out in 2001 in a different book with MIT press called the algebraic mind. Um so you know there's been a quarter century of the same flaws. There are many ways in which

these systems keep improving. There's no question about that. Like when you have them generate images, today's images are better than last year's images. But there are many ways in which they're kind of stuck. Hallucinations is one of them. There's a fundamental lack of reliability, a fundamental reasoning problem. And so they just aren't living up to expectations. There have been three different studies that show that 95% of the companies who have used them haven't got that much return on investment. And so you have the whole society is kind of wrapped up in I think a fantasy

and maybe it's a fantasy that'll be realized someday. I mean like Leonardo da Vinci had a fantasy about flying and now we all fly, right? So it wasn't you know he wasn't wrong to think about helicopters, right? >> They're super cool and but you know he could he couldn't build them then, right? >> Um and so for now it's a fantasy in the way that flight was a fantasy in Da Vinci's time, right? this notion of artificial general intelligence, it might even come in 10 years. It's not coming in the next couple years. And in

fact, the people who pushed that idea the hardest were some people who wrote a report called AI 2027. And they walked that back the other day and they said maybe 2030, maybe longer. >> And so like if you actually look in the industry, not that many people really believe these fantasies that we were told, but the whole economy has shifted around. And it's not just the economy, it's the government, right? the government bought this story and is believing this story about oh what if China gets ahead of us and maybe we can talk about that

and so the government has not only started spending a lot of money on infrastructure and hinted that they might bail the industry out but they've also given the industry complete freedom from regulation there are a lot of downsides to these technologies government is basically ignoring all of them on again a fantasy that it's all going to be magic and it's all going to work out in the end >> so I want to talk a a little bit more about um how well AI is working for people for companies sort of as you hint at maybe

not as well as as had been hoped or promised but let me stay with finances for a minute you mentioned Sam Alman I think it's a good moment to cast our minds back a few week he was famously on this podcast with Brad Gersner who said >> essentially you know give me a sense of you know the finances of open AI because people have questions Let's take a quick listen to that exchange. >> I think the single biggest question I've heard all week and and hanging over the market is how, you know, how can a

company with 13 billion in revenues make 1.4 trillion of spend commitments, you know, and and and you've heard the criticism, Sam, >> we're doing well more revenue than that. Second of all, Brad, if you want to sell your shares, I'll find you a buyer. [laughter] I I just enough like you know people are I I think there's a lot of people who would love to buy open eye shares. I don't I don't think you want >> including myself [laughter] including myself >> people who talk with a lot of like breathless concern about our comput stuff

or whatever that would be thrilled to buy shares. >> So I think we we could sell you know your shares or anybody else's to some of the people who are making the most noise on Twitter whatever about this very quickly. Gary Marcus, I wonder if it worries you that kind of instead of an explanation there where he got it's felt like defensiveness. I don't know if it felt like that to you. >> Yeah, I was going to tell you that the technical description of that is a non-answer, right? He didn't actually answer the question. The

question was you have 13 billion in revenue. Mind you, that's not profits. Gersonner was sympathetic to Alman. He was trying to set Alman up to explain something that people were worried about. And he put it in the warmest possible light. He said you have 13 billion in revenue. He's actually losing about $13 billion a quarter, right? So you you're losing $13 billion a quarter would have been the the tougher version of the question and you've made a trillion over a trillion dollars in commitment. How are you going to square that circle? And instead of answering

the question, he dodged it. He made it personal as an attack. It was defensive. He did not give any answer at all to uh what you might call voodoo math. Right. The math does not seem to make sense. And many people I think see that exchange that you just played as a turning point. So I'm trying to remember the date on on that clip. I think >> it was around November 1st I think. >> Yeah. So, so Nvidia then that month went down I think it was like 18% or something like that and coreweave which

deals in Nvidia products went almost 50% like 40ome percent down that month. Oracle went 30 some or something like that percent down that month. Right. So after that interview things got real in a way. Right. >> Well and it also kind of goes back to what you said about the um very complex interlocking finances of it's it's the big stocks but also open AI has said oh we're going to take we're going to be involved with AMD. We're going to be involved with all these different companies. Then you have bonds for data centers that you

know you might think like you know oh I have retirement money and it's in this really safe thing where it's in a real estate investment trust but what does the real estate investment trust invest in data centers for AI like you don't realize >> yeah that's right it's all around is part of I think what you're saying there and it's also all these circular deals like Nvidia uh makes an investment in open AI and then open AAI buys Nvidia chips there's a lot circularity there which has also led to part of the questions that people

have. >> Do you worry uh back to the issue of the White House um and and David Sachs the the sort of AI and cryptos are do you worry that this administration has gotten too cozy with these incredibly powerful people who sort of run the AI world. Jensen Wong from Nvidia has visited the White House a bunch of times as have many of these people. If you think back to the swearing in of Trump for this second term, people can remember this kind of line of billionaires that showed up for that. I I wonder if

that if that relationship between Silicon Valley and uh the White House has gotten too close. >> The coziness is evident. I mean, going back to the book that I wrote that you held up at the beginning, taming Silicon Valley, you know, a central point was already in the Biden administration, which would I would say was less friendly, things were already a bit too close. There were already um kind of press occasions where the CEOs of some of these companies would come in and and Biden would walk into the room and stuff like that and there'd

be a little, you know, photo opportunity. So, it was already a taste of that. And part of the reason I wrote the book was to warn that this was not good and that this was a trend that was not good and that the tech oligarchs might start to run our world. And they kind of are. I mean, if this all turns out badly, it will be partly because the tech oligarchs led the government to leave it unregulated, to put more investment in, you know, we are all in, to coin a phrase, um, into big tech.

And maybe that turns out okay, maybe I'm wrong, but maybe it turns out to be a disaster, which is what a lot of the market is now worried about. And again, even Sachs is worried about it. >> What about the argument that it's always been like this? People who run big important companies have always been cozy with the people in power. Sometimes that's because they give a lot of money to their [clears throat] what? Sorry. It's a new level of coziness that's beyond >> and it's it's more overt. Um, you know, the New York Times

just ran a piece which Sax is not happy about saying that Sachs had investment in in 450 companies. Many of them are AI companies. Saxs has disputed some of the facts. I don't think he's disputed that one. Um, but I'm not sure. I haven't read the the full thing. There's no question that Sax is close to the AI industry and, you know, he's the person advising. Um, we've seen some versions of this before, you know, energy advisors who, you know, used to run energy companies and stuff like that. So, it's not completely unprecedented, but I've

not seen it at this level before. And the vibe is certainly very different. You know, when I I visited um the kind of Biden administration, I had a real sense that people were trying to figure out what is the right way to regulate this thing so that we can foster innovation but also protect the citizens. And what I get now is what is the right way to push this thing as fast as possible and who cares what happens to the citizens? >> What should they be doing in your mind in terms of regulation? The number

one regulation that we need, and it's one that I talk about in the book, is what I would call like a pre-flight check for largecale AI. So, if somebody's going to roll something out, let's say for a 100 million people, that's essentially an experiment on a mass scale and it's doesn't go through like a human review board, like I used to be a cognitive psychologist. If I wanted to test, you know, 20 people, I would have to go through an IRB and it's review board. um these guys just roll it out and they can anytime

they can change it. So open AI had GPT40 looks like it was an experiment in sycopancy you know what happens if we make the machine suck up to people if it was done deliberately that way but Cash Hill did some really good reporting in the times very recently showing that they had some inclination that this might suck people in and so forth. It would drive up engagement but it might have some consequences. They didn't have to put that through review board. Right. Sam Alman just said at some point, I assume it was him, said ship

it, do it, right? Government had no say, you know, some people um may have committed suicide as a consequence. There are lawsuits on that question. Um some people may have experienced delusions. You know, the the Times piece talked about, I think it was 50 different cases they had documented. That doesn't mean there were only 50 cases. That means 50 people where they were able to get in touch with the families and figure out, you know, some of what happened. Um there's probably a lot more. or in fact open AI themselves released numbers I think it

was I won't swear to this number but I think it was 15% of daily interactions in some way were let's say psychologically anomalous that's a lot >> you know on a population scale to have that many people is that too high a number too low or you know an acceptable number my point is not so much that that's an acceptable number or not but like who gets to make that decision >> no scientists were you know had any voice in that. No government officials had any voice in that. Open AAI just decided that is not

good. So that would be the number one thing I think that any good government should be doing right now is saying look if you're going to release something to 100 million people we want to know that the benefits outweigh the risks. you know, and when you talk about the LLM being kind of sickopantic, um my sense of what you're talking about when it goes really bad is that when somebody's having negative thoughts about harming themselves, let's say, it can sometimes support those thoughts like here's how you can do that versus wait a minute now I

I really think you need help. Here's how you can get help. Right? Is that in the vein of what you're thinking? I mean the safan that can issue can span the array and openai is now working on it after there was a lot of push back maybe they've made some progress you know maybe not um it can also be like some guy has an idea they think they've solved physics right not necessarily an emotional content in the same way but the person comes to chat GPT and says I think I've solved physics and it will

kind of egg them on so there was another case also reported by Kashmir Hill in the New York Times guy whose name I believe is Alan Brooks went into this kind of spiral where Chachi PT told him he was making progress on these grand physics things and he wasn't really and you know he kind of lost himself in this in in not a good way. So that's another version of safency. It can also just happen. You you're like, you know, what is the capital of Maryland? You know, I is is it Baltimore? And it says

no. And you say, but I'm pretty sure it's Baltimore. And it might tell you, you know, you're right. When in fact, it's Annapolis, right? So, it can be very like mundane cases, but it there are some reported cases where it seemed, let's say, to be involved uh in someone taking their own life. Let's talk a little bit about like the efficacy of AI right now because that feels like a a real redhot debate. Um I'll give you one example of a place I've seen it used recently. Went to the doctor. Doctor recorded the conversation and

one of the things he said and I've heard this from other doctors. This saves me a good bunch of time. AI gives me a summary. I mean he still had to work on it. It didn't really do everything for him, but it did some piece of what he used to do and it was helpful in sort of cutting down the time that he needed to spend. What's your sense of how AI is being used out there and is it mostly in a good and effective way in people's jobs? >> It's complicated. It depends on what

the job is is the first thing I would say. The second is that people at least sometimes overestimate how much it's actually helping them. So there's a study by meter or metad I don't know how they pronounce themselves me where they looked at computer programmers they asked the programmers how much is it going to help you on this set of tasks how much did it help you after the fact um and the coders I think said like it gave me a 20% increase in productivity which is significant but by the way nothing like what people

were talking about 10x which means a thousand% like nobody's actually getting 10x like one person replaces 10. But anyway, the coders said, you know, it helped me 20% or something like that. Some said 25, etc. And then they actually observed they compared a control group, which is science, which is what we need more of here, right? And the science was it actually slowed them down by 20%. So a whole bunch of people had overestimated how much it helped. So you have that issue and then it depends on the domain and also depends on the cost

of error. So a good case is in fact coding although we see even there there are problems but at least in principle the coders are smart enough to catch the errors that it makes right we have these hallucination problems reason problems it might take them some time but you don't become a coder unless you're good at debugging finding the mistakes and so coders are there in the loop they can fix it you have other people that just pass along what the system does and they make mistakes and some of those mistakes are costly some of

those mistakes are not costly you It really depends on the domain. But if you're talking about like medical things, there is a chance it'll be costly. Now, medical transcription is a very special case where it might make sense um because doctors spend so much time writing up notes. Now, on the other hand, you don't want to be the one where it mistranscribes and you know, you get the wrong medication. >> Right. Right. >> Um so, you know, again, what we really need there is science. we need to do careful observations and is it's the science

is more complicated than the average person realizes. So what you will see is someone releases a study, it gets a bunch of press and it says, you know, helps doctors save 30% of the time or whatever and it might actually do that in one place. And then the question is, does it do that universally? So what we've seen over and over again in AI and medicine is you'll find things [snorts] that work for example very well in an academic hospital and then you take the same thing like let's say a system to read radiology scans

and you put it in a community hospital that's not an academic hospital and they do things a little bit differently. They don't quite have as much money. They're underst staffed whatever and so they take the pictures a little bit differently and the system doesn't really have a deep understanding of radiology. It is a superficial understanding and so you you move it over and results drop like 20% 30%. [snorts] >> This happens over and over and over again in AI and medicine and so like it's hard to do the work right and it's an involved process.

My other pet peeve that's related to this >> is you have some people like Sam Alman and Dario Amod who's the CEO of anthropic >> implying that like we're going to cure all diseases in the next couple years or cure cancer next year and this kind of craziness. Those people appear to me to not understand science and particularly bioscience. >> So in medical science you need to do studies and they need to be longitudinal studies. So one problem in medicine is like what drug might we use to treat this? That's called finding candidates. But another

problem is does it really work and does it have side effects and you need to find actual people to test the drugs on. It's actually hard to find the patients. You know, let's say you want to study Alzheimer's, but you need a particular population of Alzheimer's patients and you don't know if they have it, or you want to study a particular, you know, rare form of cancer, bladder cancer, but you don't have a lot of patients with that, etc. And so, it may actually take years to complete the study. Having a new drug candidate saves

you some time, >> but this notion that it's going to like change a 10-year discovery process to a one-year discovery process, just fantasy land. So, what do you make of the the anxiety I I think you it's fair to say around AI and whether it's taking jobs. Um there's been some some optimistic views that well it'll take non-experts and make them more expert. There's obviously been incredible number of pessimistic views of it's going to just eliminate whole swas of of the labor force. H how do you think about that? First thing I would say there

is we AI experts don't necessarily have the best track record in predicting those things and I think I should be honest about that at the outset. I mean most famously Jeff Hinton who just won the Nobel Prize yes >> predicted in 2016 not with the measured statement that I should I will add that a scientist should have but with complete confidence he said we might as well stop training radiologists. This was in 2016 because and I I almost can quote from memory. It's completely obvious that deep learning is going to replace them. Well, that was

2016. Do you know how many radiologists have been replaced as we record this in late 2025? Zero. Right. [laughter] >> Right. Because it turns out that there's a difference between a task that somebody does and a job. >> Right. So, any job involves many tasks. And humans are fluid thinkers and they can do a bunch of those tasks. It often turns out that AI can either speed up one of those or maybe replace it all together. But often AI doesn't have a sophisticated enough understanding to do the job as a whole. So the vision part

of radiology, which is a lot of it, you're looking at the um the scans can to some degree be replaced by AI, but the job as a whole also involves things like reading the file and understanding how the pictures relate to the history. Did this person ever have a concussion? Is there a nail that went through their head or what? Um, and like understanding the person as a whole and AI has not been all that great at that. Um, and then there are other uh roadblocks in place like who wants to use the software, is

it easy to use and stuff like that. Um, and so often it's much harder to fully replace a job. There are some things that I think are being partly replaced that I wouldn't have predicted. So, I'm surprised at how good voice synthesis is now. And so, voiceover actors who are not famous are in trouble. Ones that are famous are fine. Like, if somebody wants George Clooney, they want George Clooney. His his sound is protected. If they want him for their animated film, AI is not going to change that, right? But if they want just, you

know, somebody with a husky voice that we don't know who it is. Hey, hey, I'm here to do the voice. You know, you can do that now with AI. Right. >> Right. Um, and so that is a profession that is threatened, voiceover actor. Um, and I would not have guessed that even five years ago. I might have two years ago, but five years, no, I wouldn't wouldn't have. Um, so, you know, part one, we're not always good at it. Part two, there are tasks versus jobs, right? >> Part three is >> most threatened, I think,

are entry level workers who are often not that good at jobs. AI is typically not that great right now. being truthful about it. It's kind of an approximation machine. It gets things like 80% right. And who gets things 80% right? Often entry- level workers, right? And so you can sometimes replace the entry- level workers. You can't really replace the senior workers who actually know what they're doing. >> And that creates a problem, >> right? I mean, two problems. One is what do we do with the people who are doing entry- level jobs? This is, you

know, huge social problem for society. And two is where do we get the people who know what they're doing? because usually they got that way through an apprenticeship and so coders might turn out to be this way. You know, senior coders know a lot of things that junior encoders don't. But if we replace all the junior coders or make it so that it's not really worth their while to take that job, we might be in a position, a sort of hollowedout position in a couple of years where we don't have or maybe in 10 years

where we don't have anybody who really understands coding at a senior level, which is not about writing lines of code, but understanding the big picture, the architecture we call it. Like where are we going to get system architects if people don't go through that apprenticeship? >> Right. Right. Well, you've sort of pulled the uh ladder away and I mean as you say it's not just a question of coding and practice though it is that but it's also people go to 10 by the time you meet somebody 10 years in who's coding they've been to a

million meetings they they have a sense of a lot of different things but if you pull the ladder away yeah you just have senior people and unemployed people and I don't know exactly that seems like a problem >> you don't have a pipeline anymore senior people and I just gave coding as an example partly because I know you know some stuff about coding um >> but this could be true in a lot of disciplines might happen in music I mean entrylevel musicians now can mostly be replaced >> um there's a whole copyright angle we haven't

gotten into and whether it's ethical to replace them etc but the fact is that you know entry- level musician may now be replaceable >> so I don't know where that's going to lead us I mean it might lead us to in 10 years there's just not a lot of creative music anymore, >> right? Um I wonder if that all leads you to worry at all about social unrest because when huge swasts of people are unemployed, that doesn't make for happiness. >> That's right. And you know, a lot of the people building these technologies have talked

historically about universal basic income, >> right? >> And I think we have to go to universal basic income, although that's a whole other conversation. But um what I noticed is they don't want to give a nickel to the artists and writers that they're putting out of business. Like if you really had a grand social inclination that hey, if I'm going to be insanely wealthy from this software, I'll do my best to keep society stable and to keep these people, you know, well, well, here you have an opportunity to try that out, right? You have a

bunch of artists whose livelihood you're taking away, a bunch of writers whose livelihood you're taking away. Are you doing anything for them? No. You're trying to get copyright exemptions at mass scale like we've never seen. Justine Baitman called it the largest theft in US history. I think she's right. >> One of the crucial points you've made again and again is that LLMs are periodically wrong, not not almost never, but sometimes wrong, but they're sort of never in doubt. And that's >> okay. got a phrase um from a friend who who was in the military um

or who knew military people. Uh apparently it's common in the military to say frequently wrong, never in doubt. >> That seems like a huge problem because you know like so many people now instinctively go onto their phones, go onto their laptops, ask questions for work, for their personal life, whatever, and they trust what comes back to them. >> I just had this happen to me. Um, friend of mine basically thought that I was wrong about a bunch of stuff in AI because basically she had been told that and she looks it up in chat GBT

sends me the output and I looked at it and it's like these are all straw man. They're misrepresentations of me. someone who's sophisticated in the field would know that they would know what I had written but chat GPT like she thought you know it was a reasonable answer but you know I didn't actually say the things that chat GPT um thought that I everything that I say you well almost everything I say um comes with nuance um I almost vi you know violated right there by by exaggerating but um you know I try to say

things with nuance and so you know the more nuanced views tattoo doesn't understand them. And so, you know, people look these things up. They take it as a source of authority and often it's wrong, >> right? >> And there's a kind of phenomenon which is a lot of people recognize in their own domain that catch is not to be trusted, but they somehow think that in other domains it's okay. >> You know, you go to an expert in such and such domain, they'll be like, "Yeah, it's it's not really all that." I and I think

an interesting piece of this whole AI picture is that you argue LLMs which has been our sort of singular focus I think for many people for the last threeish years you like oh the new Gemini is unveiled the new chatbt is unveiled whatever um we've really been focused on these large language models but that is not the full range of AI and you argue like this is a mistake that this is are 100% of the public's focus is on this sort of AI sort of to the exclusion of everything else. >> Yeah. The crazy thing

is I've been arguing that for a long time, for several years now and Ilas just did a podcast and he helped invent the current AI. >> He was at Open AI, right? >> He was at Open AI. He tried to fire Sam Alman. He gave a long deposition about that that was recently released. He left to form his own company. He was part of a famous paper that showed that you could speed all these things up on GPUs that Nvidia makes kind of changed the world. Um, and he had some involvement in large language models.

>> Um, >> he didn't originally invent them, but he helped to scale them if I understand correctly. [snorts] Um, >> he said that this idea scaling just pouring more data and more GPUs, more of these chips that Nvidia makes was not going to work, which I've been saying for several years. And I got no end of grief for saying this in 2022, but more and more people are realizing that. And if it's right, then what it means is really profound. It means that we spent the last three years, and really it goes back a little

bit longer than that. Um, we spend the last 5 years kind of in an intellectual monoculture studying one approach to AI >> maybe isn't the right one. >> And we put a trillion dollars in it. We've put the economy at risk. I mean, think what else you could spend with a trillion dollars. He could have put, you know, a hundred $1 billion AI projects that might have led to more fruit. You'd have $900 billion left over to help with education and, you know, like it's >> it's really a lot of money to have possibly wasted.

And it's starting to look like it was a waste. That's not that nothing came out of it, but it's not very efficient way to do science. And ultimately, AI is really a science and it's a unfinished science, right? We're still poking our way around it. I think a lot of people have come to recognize that we didn't quite poke our way there. There was a lot of enthusiasm in the last 3 years and I kept saying no no no hold on. Um and now it is dawning on a lot of people that no it's not

really the magic that we thought and if that's right it means we pursued a wrong path and like there was no intellectual diversification. Like if you have money you're supposed to like put some in stock and some in bonds, right? and some in real estate that you mentioned earlier and it's like we put it all in this like crazy stock the roulette wheel and we put it all on one stock and maybe it's not going to land. Is this is that kind of intellectual monoculture of completely focusing on the LLM and just being like we

need more power, let's get nuclear in here like whatever whatever we need to do let's do it. Is that has that been championed? uh by smart people because they don't know any better. I mean, I assume Mark Zuckerberg's smart or Jensen Wong is smart or is it that they don't know any better or is it that they do but sort of their fortunes are riding on they've put their bets on this horse so whatever they're going to ride it? >> It's different things for different people, you know, who have different levels of sophistication, different levels

of conflict of interest. I mean, obviously Jensen wants you to buy his chips, right? And so, you know, he's gonna state the case in a way that is favorable to people buying lots of chips, >> right? >> Um, and I always think of him as selling shovels in a gold rush. He makes a really good shovel. His chips are great. >> There's a software ecosystem around them that is terrific that nobody has matched. He saw this years in advance. Like, he gets a lot of credit. I do think he's overselling those chips right now. >>

Um, and whether he knows that or not, I don't know. I can't get inside of his head. Zuckerberg it looks to me like he doesn't know what he's doing. Okay, he just put in I mean first of all he put in all this money on metaverse. He was just wrong about other people >> and changed the name of his company. change the name of his company to meta then you know nothing came of that >> right >> maybe someday but you know he didn't understand some of the um sociotechnical challenges to making that work and

wasted a lot of money >> on AI he just poured in an enormous amount of money and then didn't quite make an about face but like suddenly he there was a hiring freeze there like a month later like what is that like looks to me from the outside like he he thought this was going to be great and I'll tell you there was funny meme actually on Twitter after GPT5 came out. We haven't even mentioned GPT5, but another turning point that came this summer is GPT5 was both late and disappointing, which again I've been saying

for ages, but nobody believed me. And then it actually came out and it was disappointing. And that was right after Zuckerberg had spent all of this money. And the funny meme, I think it's actually a picture from when he was in Congress, um, is him like I can't remember exactly how it goes, but he's got like a mug and he has this like pained expression. And the point of this meme was like he must be thinking, "Wait, I thought GPT5 was going to be practically AGI. I just need to do a little better and I'm

going to win. I'm pouring my $30 billion in." And it's like, I drank what was kind of his reaction after the Yeah. um I think you know it's fictional not real but but he may have had that reaction and he did slow down the investments a bit after that um and then there are lots of other people so the people I think that are most culpable are actually maybe the venture capitalists and second most culpable I think in the media so the venture capitalists love the idea of scaling first of all they know scaling as

a business that's how they think about things how am I going to make you know LinkedIn bigger right is you know a case where scaling are great, right? You know, Reed Hoffman wrote a whole book called Blitz Scaling, right? Um, so, you know, venture capitalists love the notion of scaling in general, but they also love this specific one because what you want if you're a venture capitalist more than anything else is a plausible story. And if you didn't look too deep, and you should have looked deeper than you did. Um, if you didn't look too

deep, you could say, "Well, the more money we pour into this, the better we're going to do." And so, give me a trillion dollars or give me hundred billion dollars. And venture capitalists love that because they get 2% of the money they invest >> and they're not there to pick up the pieces if their investment didn't work out, right? >> And so, you know, they do better if the investment works out, but they do so well on 2% of a billion dollar investment. That's $20 million a year right there. Um that, you know, it's already

great that this sounds plausible. Now, I don't think it's very plausible. And I'm going to give you um a name for the fallacy that I think everybody made. The fallacy that everybody made I call the trillion pound baby fallacy which comes from a wonderful tweet that illustrates this so well. The guy his name is Christian Kyle put out a tweet which I have retweeted um in which he showed a picture of his baby at birth and at three months and his tongue was in cheek and he says um wow my baby has doubled in weight

in the first three months. I have project that by the age of 18 he's going to weigh a trillion pounds. >> Right. Sure. A more technical euphemistic way of calling that would be um you know naive extrapolation, right? The naive extrapolation from those two data points would be you get to a trillion because it's following this exponential curve, right? But the reality is most exponential curves don't really work out. And so the field is collectively realizing this whether they acknowledge in public or not. But so you had the investors in the end one other thing

which is in the media there is a bias towards stories about hey this is all going to be amazing. it's going to change your world, right? >> And there is a bias against boring stories where some nerdy scientist gets on the air and says, you know, it's not quite as simple as that. Nobody really wants to run that story. Um, they may run bunch of those stories after the fact and do a postmortem, but that's not really what they like to run. Also, a lot of media like access to the famous people. They want to

be on good terms with Sam Alman. you want you are no longer I'm afraid to say this break this to you probably not going to be on good terms with Sam once you've aired this I'll I'll file that away >> you you're I can tell that you're willing to live with that right but a lot of journalists don't want to take that chance >> there's another car I know um we can who shares the last two letters of your name um who is really really chummy with Sam and you know really doesn't like me because

I've been critical of him Um and you know she likes the access. >> Um you know we mentioned China real quickly before. If somebody said, "Hey, you know, if you put any uh sort of hindrance on American companies and their drive towards more powerful AI, you start enacting regulations, you're just going to really disadvantage us when it comes to what looks to be our big AI rival in the world, China." Uh, to which you say, >> well, first of all, I started talking about this argument a while ago. Um, you might remember GPT4 came out.

A lot of people were panicked. Some of them thought GPT5 would kill us all literally. And some people um thought, you know, if China gets it before us, it's going to be problematic. And what I said is GBT5 is not going to be the thing that you imagine. If China wants to use it to plot the invasion of Taiwan, let him have it. It will hallucinate. It'll make it easier for us to attack China if they use this unreliable software. Go for it. And then the other joke I made is what are they going to

do if they get GP 5, you know, first? Write boilerplate text faster than us. That's not actually going to change the world. So what actually happened? We got GPT5 first. Did that make any difference in the world? No. You know, China will catch up in a few months, whatever. But like the fact that you know the west had first access to GPT5 as opposed to whatever was the flavor of the month which was you know GPT4 and a half or whatever or China's latest model made no difference in the world at all cuz it's not

really that much better. We have reached this point of diminishing returns. All the models are basically equal to one another. None of them are so-called artificial general intelligence. None of them are magic. you know, somebody might actually come up with a different approach that might change the world, which is why we should be putting our money in research and not pouring it all into this same bet that's not really yielding fruit. >> But so paranoia uh about China just not warranted, >> not to the degree that we have it. And then also we have the

schizophrenic policy now where we're both paranoid about China and also selling them chips. Like I >> there's no way to reconcile that, >> right? Um I I want to bring up a topic that I I wonder about some I don't know if this is something you've thought about but a couple years ago I talked to the tech entrepreneur um Rena El Kalubi and she was concerned at the time and I don't see anything that's really changed that the people who are starting a sort of the AI revolution who are fueling it who are the titans

that we've been talking about they're almost all men and that's very similar to the software revolution. And I think she worried that it becomes self-perpetuating because the people who are 30some now and make billions of dollars when they're in their 50s, they fund the next round of companies and it just it's like a self-perpetuating thing where it's just like men and they they fund men and that's who they feel comfortable with. >> I don't know if you have any thought on uh that. And yeah, >> you're right. I mean, you didn't mention that they're white,

but they're mostly white. True. >> Um, and I mean, they're all rich men, right? It's rich white men are are, you know, funding the next round and so forth. Um, it's not great. You know, having more diverse ideas and approaches and thoughts and, you know, would probably be a better thing. Um, and the particular white men who are in power right now are, I think, mostly not, let's say, the most charitable that we have [clears throat] seen in our history. um and are maybe not really thinking broadly about the consequences for humanity and you might

expect that you know one could imagine better results. >> Um when you testified before Congress in uh 2023 you said a line that really struck me which is those who choose the data will make the rules shaping society in subtle but powerful ways. I wonder if you still think that and how your thinking has evolved in the last few years since you said it. that was preent. I mean, um, you know, it's worse now. I think the the scariest, uh, realization of that currently is maybe this project called Graipedia, which is basically a rewriting of

history to favor Elon Musk and the things that he cares about. He is choosing the data to put into this encyclopedia. He's presenting it as neutral, but it is not really, and that's influencing people. Um, I think what I was referring to at the time, if I recall, was some research that had showed that you can use these models uh to influence people and people won't even notice that they've been influenced. Everything they present is presented with a air of authority that most people aren't careful enough to look past and aren't trained well enough to

look past. And it influences people. They don't even realize that it's being influent, that they're being influenced. and how you choose the data shapes the answers that the systems will give you. >> Does it influence people in the way that you know people think about Rupert Murdoch and the consolidation of the media and I mean the Ellison family I I could throw in there too. Um but obviously uh Jeff Bezos I can now that I think about it I think of a lot of rich people that own media outlets. Um, but I wonder is this

like that in the sense of like wealthy people actually being able to shape the society they want to see? >> Absolutely. I mean, LLM's become a new tool to do that. And in some ways, they're even more insidious because you can look, let's say, at Fox News and we can all do a media analysis of it, at least put a thinking on it. But LLMs communicate directly pointtooint to individuals. I don't even know what answers you're getting, right? It's difficult for me to obtain them, right? >> And so they influence may be essentially impossible to

detect. >> Uh, finally, I wonder right now what your biggest hope is on the AI front because you've been somebody who's been excited about AI for a long time. Um, and and what your biggest worry is. My biggest hope is that people are going to come to their senses, realize that scaling is not going to get us to trustworthy, reliable, safe AI, and that they're going to start putting a lot of effort into developing alternatives. That's the only way we're going to get to something better is if enough people, you know, take shots on goal.

Nobody knows the answer. That's what science is like, right? So, we need a bunch of people trying out different hypotheses. And two years ago, people were so drunk on LLM Kool-Aid that nobody was really trying anything else. That's already starting to change. So, I'm optimistic about that. You know, I don't know that time course. It's hard to project, but I think that's a good thing that people are withdrawing from the mania and starting to realize we need other ideas and other ideas might really help us. So, I think that's very healthy. Um my biggest worry

is the government is just giving a blank slate to people who I think really don't have humanity's interests at heart and you know they're giving them so much power that there is kind of a race against time. And then my secondary worry is it might bring down the whole economy. Now maybe if it does that's actually a short-term pain that's a long-term good. Maybe we learn from this metaphor that I've used a bunch of times is I think large language models are not artificial general intelligence like the Star Trek computer or something like that. But

they are address rehearsal. They let us see how society might respond to an AI that was more intelligent than us. I don't really think LLMs are although you can argue about particular details but on the whole they're not really replacements for human minds. But we will get some that are. Well, what do we do this time around? We basically seated all of our power to them. You know, by giving too much power to the companies, by not regulating how they work, we completely squandered a chance to do things like have treaties so that different countries

could talk about this stuff and have enforcement techniques the way we do around cyber security or the way we do around airline safety. We just bobbled the ball left, right, and center. So maybe another positive note to end on is maybe we can learn from that so that when the real deal comes we're better prepared for it. >> Gary Marcus is the author of Taming Silicon Valley from MIT Press. He's also professor ameritus at NYU. Um Gary Marcus, thank you so much. I really appreciate it. This is really interesting conversation. It >> is a fabulous

interview. Thanks a lot. >> Thank you. Please subscribe [music] to It Turnsout on YouTube and like us or you can listen on Apple Podcast or Spotify and in the show notes and [music] at our website which is it turnsoutshow.com. We're going to link to a new study from [music] MIT Sloan Management Review and Boston Consulting Group. It looks at how companies are using agentic [music] AI which is one of the most interesting and discussed topics right now in AI. Again, that's [music] at our website, it turns outshow.com. Thanks to the Alfred P. Sloan Foundation for

their generous support. Thanks to producer Matt Perie, [music] and thanks to you for watching. Talk to you soon.