How Does FAKE NEWS Spread? | Renee DiResta Full Interview | The Jordan Harbinger Show Ep. 420

Welcome to the show today a really interesting friend of mine Renee daresta she's a researcher who studies influence operations and propaganda in the context of pseudoscience conspiracies terrorist activity and state-sponsored information Warfare today we're going to focus on Russian influence operations online so election hacking via social media there's a lot to this it's a really big show so we're going to Jump Right In all Right enjoy [Music] thank you how did this get on your radar I know this sort of started with you doing uh trying to do the impossible which is find a preschool

for your kids yes um so I had my My First Son in December 2013 and in California in San Francisco you have to get on these waiting lists like a year ahead of when you want it Um so in December 2014 was when I had to get them on these lists anyway I started Googling around um looking for schools but also I I wanted to look at the vaccination rates because that had come up um I used to live in New York and uh it was kind of required but in California you could just opt

out because of personal beliefs you could just say we're not going to vaccinate and that's it philosophical exemption Um so I didn't want to go to a school like that and I downloaded a bunch of data um and was really disturbed by how the rates had been really trending down over time and I wrote a blog post about it late November um looking at like how private schools were a million times worse how Waldorf schools were an absolute disaster just trying to say like California parents like we you know we should be demanding Better here

um I called my local representative I'd never done that before but I felt like that's such a caring move I was something about that why do we have these exemptions and um yeah and they were like yeah no nothing to do um nothing to do about that you know good luck and then the Disneyland measles outbreak happened a month later and I called back again um and I was like hey you know how about Now and they were like actually yeah you know we have a center in Sacramento who's going to be um institutional you

know it's kind of introducing a bill to deal with that situation um remove the exemptions and I said you know I'd love to help maybe I could you know do some data analysis for you guys I'd love to um just be useful in some way and so they put me in touch with two other moms and we started this group called Vaccinate California and we I started a group I mean we made a Facebook page just to be clear there was no legal filings there was nothing that you know that that this was going to

be like a big deal we didn't really know what we were getting into so we just made a Facebook page and I started running Facebook ads because we needed people to call and to say uh to support the bill and so I started running Facebook ads and I got you know I learned how to do Ad targeting and look-alike audiences and a bunch of other kind of um more sophisticated ad targeting tools and in the course of doing that you know we also started um kind of putting out content about the bill on Twitter and

um you know I I was like part of SF Tech you know I uh been on Twitter for years and was kind of completely blown away by like activism Twitter um particularly by how well organized The anti-vaccine movement already was and the way in which you know these fake accounts would kind of come out of nowhere they would say that they were from California they'd been created yesterday uh they were communicating they were being given instructions on YouTube videos that were being put out every night saying these are the hashtags you're going after here's the

memes you're going to use here's the folder with the memes in them and I was Kind of like wow this is really amazingly organized operation um on Facebook what I started noticing was if you search for vaccine for running any of your ad targeting what it started giving you was anti-vaccine suggestions so if you typed in vaccine it would it would push you a bunch of anti-vaccine topics and you could Target anti-vaxxers really easily but I was like well how do I find like parents who want High immunization levels in their schools like and then

I realized that there was like this asymmetry right like most people vaccinate but they don't stop to think about it they definitely don't become activists about it they vaccinate their kids nothing happens they move on with their lives and you know and they don't get on social media to talk about how nothing happened um so I was kind of struck by this asymmetry and by saying like Okay how do We Rectify this because it seems like all of the algorithmic boosts are driving things in the other direction um and so I I started looking at

mapping the network so I did some network analysis with a data scientist named glaslow 10. and we started looking at um how the anti-vaccine parents on Twitter who had been anti-vaccine for years and years and years were beginning to reach out to the tea party for example to some of the militia movements to some of the Second Amendment activists all of which are you know um again like kind of communities that had been on Twitter for a really long time I haven't heard about the Tea Party in like I know it was the two-party in

like 2015 20 you know like late 20s early it was early 2015. yeah it really was um so this was before the you know the kind of um but this was way before the 2016 Campaign this was all happening in January to June 2015 or so so pretty early on um and I started writing about how social networks were enabling different kind of persistent factions of people different communities to find each other uh and to find uh you know to kind of like marry hashtags was what the anti-vaccine kind of activists called it so that

if you wanted to push your anti-vaccine content into tea party uh Communities you would use the Tea Party hashtags alongside yours and then they were kind of using these Basic Marketing tactics actually for uh for policy activism and I thought that was really interesting because it made me feel like everything was going to be a marketing campaign for an idea any policy that went out the door from now on was effectively going it was going to use these tactics and so the people who were best most adapted deploying them were Going to be the groups

that got their message heard because What mattered was share of voice right on social media what matters is how much of your audience can you reach Can Your Meme become the sticky thing that everybody says is your content does that have virality potential where it's going to be forwarded along how are regular people going to see your content and engage with it how our influencer is going to pick up your content and engage With it and so I started looking a lot at the Dynamics of just this really one Niche bill in California that was

highly politically polarizing there was a ton of media um you know content going on about it I got docs I got harassed you know I was not expecting that either you know that was a surprise for you at that time and now it's that time of course it was a surprise now it's like obviously right yeah Um and I you know I was sort of like wow so also the the cost of being a public Advocate has really changed now too you know I I hear I think I'm just like a mom and right and

and that's not by the way the docs by the way for people that don't know is like they put all of your personal information online essentially to say call Renee diresta at her place of work and at home and like bother her and her family because she's doing something bad so I don't think everyone Knows what doxxed means but it's like we're it's it's worse for people who aren't already public figures because of course someone can just find your office and call you you work wherever you work but it's it's bad when you're an activist

and you have a bunch of crazy people calling your husband's workplace and telling him nasty things about you which is kind of the point of doing that right right and so I used to have like I used to have a little Tumblr blog where I put up pictures of my kids and you know like like well I had one kid at the time but um you know wedding pictures like just your life the way that normal people engage on the internet locked all that [ __ ] down yeah took it all down you know it

was sort of like you know Twitter's response to me when I what happened was like um effectively it was like you're a limited purpose public figure now like good luck to you when they started making YouTube Videos about me also and I was like I didn't even I'd never heard that term before I didn't realize what does that mean limited purpose public figure is a legal designation that says that you have chosen to inject yourself into a particular narrowly tailored conversation which for me was immunization policy in California which meant that I was a public

figure in the context of that topic that I had chosen To engage on um now legally that would mean if I were to file a defamation suit I would have to you know prove an intent to harm as opposed to just uh you know intend to harass which is uh protections are different for these different tiers of publicness I guess and I thought that was bizarre too because I thought well in the age of social media anybody who again like throws up a Facebook page puts up a Twitter account has you know And weighs in

on a conversation if the internet gets mad at them and turns them into a public figure like boom there you go you know you're right you're gonna have a choice like before it was you had to run for office that's one of the sort of traditional public figure uh occupations that's listed in the law um and it's been a long time since law school but I think the other one would be like if you are an actual celebrity right and like regular people would Recognize you right and so you have all of the downsides of

being a public figure but none of the real upside of like getting money thrown at you to wear Chanel perfume and like getting tables first bodyguard you know okay no downsides only right so that was like it was a really hard slog I was very very proud of being able to get that done we were the first state to do it and something like I think God 30 or 40 years actually in the law to be clear Was was what again the last that you have to have a medical reason to opt your children out

of vaccines that was it you had to have some sort of like medical justification as opposed to I don't want to um yeah and so that was uh you know so I mean the reasonable people can disagree on whether you know how they feel about that law and where they think the line should be and what the state's obligation or responsibility should be Um I feel like reasonable people shouldn't turn that disagreement into doxing harassing threatening and you know right that's where we kind of get into the realm of like what the internet delivered as

far as policy conversations versus the olden days where you'd uh you know kind of you know fight it out through some op-eds in a newspaper right right um so I learned a lot about that I I really Felt like I got this introduction into Network activism um and I think it was the fact that we took a very quantitative approach to understanding how these groups were operating how they were networking how social media algorithms how curation and amplification were really amplifying this content um and I had been a venture capitalist so I had done a

lot of investing in tech companies and looking at Tech business Models looking at implications of Technology on society and I I really felt like um what I was seeing was this um this this shift this thing that was going to fundamentally change uh political campaigning policy advocacy um activism online in a very very fundamental way and then what wound up happening for me was the White House reached out and through uh through Todd Parks a former Chief technology officer And he said uh you asked me to um to meet up and he said you know

some of the work that you're doing we're trying to do some similar work looking at how terrorist organizations are using social media for recruitment advocacy and propaganda um we think that the way that you've been writing about this and talking about this and you know seeing this develop is something that we'd like to uh we'd like to have you kind of come Down to to DC and participate in some of our conversations about this in the context of this other group um so that was not you know really where I expected my career to go

but uh that's where it went so yeah um so I took a I went to DC for about three weeks and um worked with a group of people who were who had done a lot you know I was not a counter-terrorism expert but this was Isis of course was what was going on At the time so what was happening at the same time in um mid-2015 was Isis was beginning to become a really big problem on social networks particularly Twitter um if you were to follow One account it would suggest more accounts for you uh

there were what they called Isis sort of Fanboys so they were the core accounts that would occasionally post the beheading videos and things those would come down quite quickly Twitter would take those down uh but then there were The people who would try to serve as like the amplifiers or the people who were like hey Twitter is censoring this content but you can go find it over here on this other platform directing people to things like LiveLeak and other places um so really working to amplify the content amplify the message and the thing that we

started talking about was um what happens when social platforms became um kind of vast tools for Uh for for propaganda what happened when they became vast amplification tools that really in a very different way than propaganda had been done in the past it wasn't being carried out only through the media it wasn't being carried not only through broadcast or print it was a participatory process it was ways in which you could produce uh content a meme uh you know a YouTube video you could go make your own content and then you could disseminate your own

content And what was happening was these very small organizations um were able to follow that process put the content out and then real people would come and serve as amplifiers uh leaving this this question of if you were to take down or delete the accounts of the people who were amplifying it were you kind of stifling free expression so this was where that that that kind of like laundry the manufacturing process of one group Producing the content but then regular peer-to-peer dissemination was something that we'd not really seen before it was something that the internet

really enabled in a way that prior broadcast mechanisms and print just hadn't you know it wasn't really possible and this took Washington and Everyone by surprise right they were thinking like oh well if we find a channel that has Isis propaganda we'll just close the channel we'll just close the website we'll do Something to that now when it's like thousands of users or tens of thousands of users on social media you have to invent software to track it and then if you shut that there's all these Free Speech issues right I assume that they then

said I I know there's there's always like the slippery slope fallacy where it's like well if we shut down anti-vaxx then what else are we going to shut down and I I would imagine they use that with Isis although what's the Argument if we shut down beheading videos then what's next no it was um one man's terrorist is another man's freedom fighter and if the U.S government requests this be taken down what if a more authoritarian government requests takedowns in the future that argument did come up actually this the slippery slope was very much a

part of it um there were a couple interesting you know questions right one was um first it was not a surprise to them Actually there's been a DARPA program in 2012 called social media and strategic communication smisc and because the DARPA program is open anyone can actually go and read the archives so anyone who's interested in go read the research that came out of Smith and DARPA is our defense research kind of like invented the internet and looks at technology in terms of how we can it can either be weaponized against us or how we

can weaponize it is that kind of a Yeah and I think the motto is um to uh gosh what is it it's two prevent and create strategic surprise I think is the obviously yeah how do you both prevent it and uh and you know put it to use it's a defense department um defense Advanced research projects uh it is yeah I don't see a motto anywhere but I haven't looked that that hard defense Advanced research projects agency agency that's what that is okay So DARPA had this had this program from 2012 to 2015 called uh

social media and strategic communication that actually asked this question what happens if hostile governments terrorist organizations you know unsavory characters begin to use the internet uh for you know for these kind of Nefarious purposes particularly propaganda and um spreading false information so it was not a surprise um I think the question really became What could the government do about it though and there were certain limitations in place so at the time the state department could counter message but because of U.S law that says that the government is not allowed to propagandize to its own citizens

the state department messages that went out over Twitter if they might be seen by an American had to be declared and attributable to the state department which meant that the entity that was Kind of tweeting back at Isis sympathizers on Twitter was this U.S state Department group um that so it's sort of like well it's not like the U.S state Department is really going to defer somebody who's a you know on their way to down this radicalization path becoming a nicer sympathizer that's not a realistic kind of authentic you know that's that's not going to

be the most well received counter messaging uh of course you're Not going to de-radicalize somebody with the state department tweet it's like the opposite of influencer marketing it's like we want influencers to wear our cool shoes and show how useful they are and then it's like please buy our shoes yeah like what no turn away the hashtags that they were using was like thank turn away it was not it was not the most uh it's not the most well executed um no early kind of uh counter counter operation but they recognized at least That they

had to do something right so um so there was a widespread acceptance of the fact that this was going to be the sort of like the new normal that information were through terrorists you know through and at the time as we were having these conversations in October 2015 we also already knew that Russia was operating on U.S social platforms that also was not a surprise um the extent was a surprise the Specifics were a surprise but at the time as we were thinking about what would the U.S government's response to this kind of activity be

what should the tech platform response to this kind of activity be even in the very narrowly scoped Realm of counter-terrorism the question really became um well if regular domestic like moms in California can do it and if Isis can do it you better believe that like Russia and China can do it right And so it turned into a little bit of like uh well maybe the kind of counter messaging entity and the whole of government response to the stuff shouldn't be some random State Department Twitter account maybe we should be forward-looking and thinking you know

think about um the fact that this is you know I felt like I I felt I couldn't tell if it was like the Chicken Little or Cassandra you know Right something is changing here like I really feel like we should be giving this um some some like pretty big gun you know big guns in terms of um get a lot of like the best brands in the country thinking about what's going to happen here uh because it seems like it's gonna be a disaster and uh and yeah I mean like I said so Adrian Chen's

article about the internet research agency had come out six months prior The article called the agency in the New York Times uh so again the fact that Russia was running fake Facebook pages and doing these things was was not news um but the real question became like what happens when it's all happening on a private platform so the government doesn't really have that kind of jurisdiction you know what are the regulatory mechanisms how do you compel a platform to take action if you if you choose to do so so let me pause for a Second

because I think a lot of people they look at Facebook groups and there's an algorithm that's doing some of the work right it's not necessarily just information Warfare actors from Russia and China it can be if I join a group that's anti-vaxx and I've done that before to see what's going on in there especially when I do shows about that kind of thing it's kind of a quick jump to like Q Anon California right and like pizza gate you know truthers 911 Truthers like those are the things that were in the sidebar at the time

it wasn't like biking in the Bay Area right it wasn't normal stuff it was or you search for covid-19 or coronavirus it's not like coronavirus Information Network it's like coronavirus truth Coronavirus Warriors coronavirus uh whatever like it's it's misinformation rabbit hole that you can't you almost can't even find the real stuff you just kind of the algorithm is just like oh you believe Dumb crap here's a lot more dumb crap for you to deal with so there's a reason for that so um when you build a social network particularly if somebody's onboarding you want to

show them things that are relevant to them which means that you build a recommendation system and before it has data from the person before it's instrumented the user and follows their clicks and what they engage with and what they do uh you're kind of a blank Slate and so what it does is it does one of two things it shows you accounts that are geographically relevant it keys off the information it has to suggest interest based on certain demographic characteristics certain things that maybe you have selected um but the other thing that it does so

when you engage with content it can show you more content similar to that type so that's called content-based filtering so it says you like gardening here's you Know 50 more gardening Pages um then there's the other thing which is called collaborative filtering which is you in its Aggregate and its understanding of who you are based on your clicks and your behavior and your demographics and all of the data that it amasses on you if you engage with this type of content people who are statistically similar to you who engage with that content also engage with

this other content so if you engage with Gardening maybe you're also a cook so even if you've never searched for cooking we can push cooking stuff at you because 75 of people who like gardening also like to cook right if it's an oversimplification but that's basically how collaborative filtering works so it says that some percentage of people who like this thing like this other thing so what happens with conspiracy theories there are some people who are into the anti-vaccine conspiracy for example Because they have health you know they're concerned about toxins or they're concerned about

um you know kind of uh that sort of Purity people the you know um my baby is born perfect why would I inject the thing into them but then there are other people who are involved in the anti-vaccine movement because they believe that they're a vast government cover-ups and the New World Order and the Illuminati and Pharma and You know these sorts of um the people who are more on the conspiratorial someone somewhere is harming uh you know is harming people and I'm not going to be a victim of that those people are more likely

to be receptive to other anti-government conspiracies or conspiracies that are alleging that evil nefarious horses are at work and that's where you start to see even if you originally go to the anti-vaccine group because you are a natural health like Juice you know person um it still says well of the of the 10 000 people in this group 7 500 of them are also members of this pizzagate group Ergo we're going to serve you the pizzagate stuff too even if you've never gotten and seen it so I also started getting you know I had joined

a bunch of anti-vaccine groups and pages when we were doing the vaccine law and so I had this account that was regularly you know I was getting Recommendations for pizzagate and for Q Anon and so that that tie that kind of conspiracy correlation Matrix kind of uh became pretty clear really early on and that was another thing where we were trying to flag that and say hey maybe maybe this is one reason why you know some radicalization is happening maybe pushing ordinary people who have never gone looking for pizzagator Q and I'm like pushing it

to them and saying hey You should see this you should see this you should see this maybe that's not the most kind of ethical design um for a recommendation system maybe that's not how we want our Foundation Systems to work yeah remember when we thought cyber warfare was going to be like attacks on power plants and infrastructure and now it's like your gardening group suddenly links you three groups later and you're talking about how essential oils can cure covid-19 or Drinking bleach can do it it's like it's just it unfortunately that algorithm has sort of

mastered finding people who lack critical thinking skills and then like pushing their boundaries a little bit little by little and even people who are otherwise sensible can have their boundaries pushed like this especially if it's done over time and in a calculated way and it's like this algorithm that's designed to serve you a bunch of ads and keep you on the Platform for as long as possible just happens to also be really good at that right good at pushing people's belief systems yep so it's not just the platforms it's not just the algorithm you mentioned

earlier the the internet research agency I don't think IRA has ever been a good acronym for anything not a great track record for that one what is the internet research agency sounds so innocuous sounds like an academic Think Tank So it's a nurse say uh it's an entity in Russia um attributed to an individual namely Evgeni pagosian uh pregosian is uh an oligarch a Russian businessman with close ties to Vladimir Putin President of Russia and the internet research agency um began operations in around 2014. I think late 2013 established 2014 operational and it originally was

to uh it originally operated largely like a Comment Farm kind of leaving uh leaving comments and then as Twitter became a bigger thing tweets on content related to uh to topics of interest to the Russian government so that originally started locally so Ukraine the annexation of Crimea mh17 the Downing of the jet the Malaysia Airlines jet over over Ukraine and so it began to uh its goal was really to kind of nudge the conversation in particular directions to distract to do the things that you would Do you know China had actually set up an operation

called the 50 cent party so sometimes called the 50 Cent Army back in 2004 so again there are certain entities there are certain governments that that believe that kind of controlling the narrative first domestically first within your local kind of sphere of influence the internet research agency rapidly expanded outward and by mid-2014 early 2015 was actively targeting Americans as well American Citizens so it didn't start as an entity to disrupt the presidential election of 2016 that of course is the thing that it's perhaps best known for because of you know the investigation and the you

know the that's been very much a topic of conversation Russian interference but it did start several years prior to that and it was a multi-year operation uh it really one thing that's kind of remarkable about it is how it took very old active measures tactics things that Had been carried out in the cold war the Soviet era and I think the real um Insight was that propaganda was going to shift with the shift in technology and this again this was something DARPA saw too Russia executed on it right and and prioritized it actively prioritize it

actively made it a strategy actively made it something that it put to use for for geopolitical influence operations and so that was again that that recognition that propaganda evolves to Fit the information ecosystem ecosystem or architecture of the day the new information architecture was social networks the new information architecture the means of transmission was peer-to-peer so rather than executing simply on Old Cold War propaganda tactics of placing an article in your newspaper and kind of laundering a narrative up the chain of newspapers they realized that you could make a meme push the meme to people

who Would be receptive to it and then have those people act as your disseminators and it would become this process by which um content designed for the internet designed for people very much like a extremely modern social media marketing approach to propaganda was was what this entity really kind of took and and ran with and and they're trying to create tribes right these aren't just like meme posters I know some of the earliest Pages were that they created these like Facebook pages were for the religious community for the the right and then they kind of

like switch morph whatever to to Black activism and then it's like Pride Texas Pride LGBT Pride Texas secessionists and they're targeting these in-group communities why are they doing this what's up with the tribes and it seems it seems random to somebody like me looking from the outside or someone that's new to this looking at LGBT Kansas Pride how is that possibly related to somebody trying to change an election or trying to change public opinion it seems so Niche how are the 500 people who are LGBT in Kansas in Kansas going to do anything like what's

the what's the strategy here so the internet is really made to help you find people who are like you right there's so much of that is is built into The design it's prioritized it's actively prioritized this is why Facebook developed groups you used to go to Facebook with your real friends and you would see what your real friends were putting out there but if you're real friends you know either weren't that interesting weren't that active on the platform for the platform to keep you there coming up with the structure of groups is a fantastic way

we help you find new friends we help you find new Interests and particularly in a you know kind of increasingly polarized U.S a lot of that was people who would come to Facebook to kind of like fight the culture wars right you know they find their political you know group and get entrenched and uh become you know really active and you started to see this on Twitter as well again these kind of like persistent groups of people who were um really aligned around a particular shared interest you had your internet Friends all of a sudden

right and and sometimes you know for people who are more into like fighting and trolling you remember the 2015-2016 campaign right Real Americans are just on there with say like they put their um their Pepe the frog or their Blue Wave or you know whatever like you signal now it's like the DSA Rose the nail liberal like Globe you know whoever you are is like boom it's like right out there and you know your affiliation you're just Wearing it on your uh your your little like username line there it's the Maga hat of your social

media account right like there's even lobsters those I don't know they're like a Jordan Peterson thing the rose of socialism I don't I don't understand why those are that way I've asked about the rose and I got an explanation for God but yeah like then there's the blue hat emoji so anything that you can possibly imagine yeah the avocados for California you know kind of Making fun of the idea that if you just eat less avocado toast you can buy a house you know yeah that might there's something to that though actually you can at

least pay your mortgage with that so so we're wearing this on Our Sleeve we're fighting with each other in like you can even just find people to mess with by searching for that particular hashtag or that particular Emoji right you can just pick yeah and so you so you think of the internet as like a Networked series of factions right just this kind of War ball against all you know everybody's out there uh trolling what the internet research agency does their big kind of innovation is they realize that they can create Facebook pages uh and

they can pretend to be members of these communities and so the content that's put out is 90 of it is really about um solidifying that in-group Dynamic right It's about saying I as a Texas secessionist believe this I as a um as a black lives matter activist believe this right and so the point of the pages what they're doing is they're really instilling pride in that group we are descendants of Confederates you know this is not about racism this is about our pride and our Confederate Heritage right so most of the most of the narratives

even for the groups that many Americans would see is like highly Polarizing was reinforcing that um that kind of in-group Pride and then when you do that there's a kind of a presentation of other groups as the other right so the question then becomes if I a real American have this identity and this other guy's identity feels kind of counter to mine um you know the question that that kind of is sort of underpinning a lot of it is like who is America for so if you believe in a you know kind of like a

Finite set of resources why are we giving money to refugees when we should be supporting our veterans and that was a theme that was like hammered home constantly um why should black Americans vote if this country has never done anything for them right um and so for us is how it's phrased it's always crazy it's like I am a member of this community speaking to people who are just like me and so it's Not it doesn't read as like media from on high or you know uh um kind of like Ivory Tower intellectuals or media

pundits talking to you it reads like you're having a conversation with people who are just like you and that's the so there's that element of um you're receptive to the message because it's coming from somebody who is theoretically just like you and then the other thing but unless you're like a Russian 20-something 30-something that lives outside of Moscow they're not really just like you they're not religious like you right why did why the focus on the black community specifically or was that just an example you picked no no that was that was actually a huge

percentage of the content was focused on the black community uh we were just you know so so the Senate requested from the social media platforms from Facebook Twitter And uh and an alphabet Google um which included YouTube that they provide all of the data related to this operation and then I led one of the teams that analyzed that data set and there was another team that did the same analysis with the same data we were told not to communicate because they wanted to make sure that uh that they could check you know make sure that

different teams on the same things because this was such a political live wire at the Time this was 2018 that we were doing this work it was before the Mueller report came out and so what we got when we were given this um this data set was this look about you know 400 gigs worth of stuff um several hundred thousand memes 10 and a half million tweets so just this this kind of Corpus spanning multiple years of their activity and then what we were tasked with was saying what are the kind of tactic techniques and

procedures that This particular adversary used uh what what were they doing what was the goal what were the messages how did it work and you couldn't look at it and not see uh the extraordinary effort put into expanding racial tension and particularly by targeting the black community when you say memes I think a lot of people know what those are but when I think meme I think Kermit the Frog sipping tea I think uh somebody Rick Rolling me when I'm clicking on a Link right those are what I'm thinking of are we talking about the

same thing yeah so meme um like the kind of geeky academic definition goes back to Richard Dawkins right the idea of a cultural Gene a unit of culture and much the way uh your genes kind of in aggregate form you as a person um memes and this kind of building block you know in aggregate kind of comprises culture right a particular type of Culture so a meme is something that is intended to be spread it's intended to propagate it's not only the kind of square cat picture with the white words on it which is I

think what what has come to be thought of it's also um catchphrases it's things that sort of signal a participation in a in a particular Community if I were to say this is fine probably maybe 50 of your audience would immediately see the dog on fire room right you know if I were to Say winter is coming maybe you get more people right yeah this is your brain on drugs is that one that's right from the 80s Yeah so basically anything I remember from the 80s or 90s is potentially a meme because everything else is

gone that's how effective these things are sticky yeah well they're intended to be sticky they're intended to um you know people will use them in a certain way if you were to go and look at like the K-pop community on Twitter I am not a K-pop fan I'm not a not fan I just don't pay attention to it but there are all these you know they're these like hashtags there's like a whole vocabulary right there's like a little like uh kind of like in-group language of of ways in which people who are part of the

community talk to each other and so there's one of the things that's really unique about the internet is ways in which searching for certain hashtags Certain phrases certain words kind of like is the gateway to finding that entire faction the entire Community um right right now BTS is done Rabbit Hole yeah but what the uh what the internet research agency did if you look at their Instagram posts every single Instagram post has like 40 hashtags down at the bottom because they know that that's how people are going to find them so if You're searching for

you know in the most basic form hashtag Maga right you know was the you know one thing that the Trump campaign did very well was they their slogan became you know a really kind of it could be reduced down to an acronym and that acronym became kind of a meme um people and it became a thing that that people would use to Signal their support by putting the hashtag in their Twitter bio by putting the hashtag on Instagram so you make your community discoverable in that way and the other thing that's really interesting about memes

they don't require very much uh thought usually right it's it's something that kind of immediately hits It's got some emotional resonance it's funny it's pithy it's uh tailored for kind of Internet style communication the olden days of propaganda um a lot of times it was like kind of long form narratives you would read a Persuasive article or an article that made an insinuation you would feel a certain way maybe but it required a lot more time with the memetic propaganda with just the um I you know I believe that veterans um veterans before refugees like

and Sheriff you agree that's all you have to say right uh and you've communicated information about yourself information about what you believe a political point of view like and share that takes two Seconds that there's no real heavy lift you're not asking someone to read a thousand word article they click a button and it's moved on to their Network as well and so that's the kind of propagation that happens so it's like virtue signal plus propaganda and of easily snackable shareable piece of content yeah and to be part of the community to be part of

the all you do is click the share button are these things are these things being Created by Bots or real people or both because I'm I'm always on the fence right I I do a lot of stuff about the Chinese Communist party and like organ trafficking or whatever and you see the wool mouth the 50 Cent Army they'll post and some of it is clearly just like an automated thing because they've done zero looking into anything that I've done um or I'll get DMS when I do stuff about Russia like I had Clint Watts on

the Show and he was like watch out you're going to get a ton of Russian Bots and Russian uh hate mail and I was like ah no problem and my DMs were just alive with people who are like you should have been an abortion like stuff like that like horrible things and I was like oh this is just automated I bet this person tweets it to everybody who they is on their like hate list but then other things look like real people other things like someone will engage and some Of them seem to just be

really sort of dumb folks from wherever but other times it's clearly a foreigner because they're like what is it called like subject verb agreement is off and everything yeah the articles are wrong pronouns are wrong stuff like that yeah um well so you raise an interesting thing there so so I'm at Stanford internet Observatory now and I um I look at both China and Russia and we've actually done quite a lot of work on China recently because one of the really interesting things you know well the internet research agency um they are still I believe

Russia is still kind of um most effective most sophisticated at understanding how to Target Americans and that's in part because the you know you asked earlier why racism that was a very common theme in the Cold War also Right even the you know um how can you say you're a free Society if black people are treated so badly and don't have rights right and the sort of Civil Rights era um so they they have a very kind of deep bench to work with on understanding how to how to what messaging works with Americans as we've

seen China come into the game um as we've seen China expand from the woman which focus on the domestic Chinese internet into how do you execute those operations targeting people internationally and that's a really interesting question so what we've seen from China uh first of all propaganda is a core part of Chinese government public diplomacy right so there is a propaganda Bureau this is not a thing that is done surreptitiously or secretly it's quite you know quite quite out there um so they have a large State media apparatus quite well developed that's Attributable so that's

what it's sometimes called White propaganda it's the attribution is quite clear as opposed to what Russia was doing which is sometimes called black propaganda because the attribution is nebulous it's kind of opaque it's hidden in the shadows it's actively misattributed as opposed to knowing you know where the provenance comes from so what we've seen from China is first of all they've taken their broadcast Apparatus their television stations their radio stations their print news and they've begun to establish presences for all of those other Publications on social platforms as well so they have hundreds of millions

not an exaggeration I think cgtn China Global TV network has over 100 million Facebook fans on its on its page it's I think one of the largest yeah I mean to put that in perspective I think CNN has about 33 million right so this is uh you know sort of like 3x uh CNN Russia Today has something like uh between 7 and 15 independent they all have regionalization for their channels um so what you see is the overt apparatus ported to Facebook even though Chinese citizens are banned from plot from Facebook they can't go on

Facebook the purpose is not to reach the domestics the way you have with some you know uh some of the Chinese State media and the mamal it's to create ways of reaching audiences globally and so they Run ads and so you'll see cgtn boosting their content for example about coronavirus they began to run ads uh kind of in Earnest pushing out the Chinese State position on coronavirus where it had originated how the country was handling it all of the kind of glowing stories about well China sent PPE to Italy you know here's a China built

this Hospital little you know so so these sorts of positive stories so they're using the Facebook apparatus as Yet one more channel in an overt propaganda strategy then the other thing that they do is they begin to run Bots and that's where you get into again that actively misattributed these are just ordinary people on the street talking but again they are they are controlled by the attribution is made to the Chinese Communist party it's a little bit difficult to make an attribution to like where in the Chinese government this is attributed to so they So

Facebook and Twitter say kind of China or CCP um but what we see from those accounts is as you're describing it's very much like the Wu Mao kind of ported to this other platform it's not very well targeted the personas are not very sophisticated it's not like Russia where they had these personas that were so um convincing that real people real big influencers Donald Trump Jr janashta Sousa Jack Dorsey were retweeting these accounts because they really seem to be a black woman activist a you know marvel romance style like you know um kind of trump

supporter right and so where Russia invested years in developing these personas and making them convincing what you see from China is this kind of throw spaghetti at the wall right the accounts don't even have plausible names they rarely have a profile picture they don't have a great Bio it's just crap yeah it's like a it's like an anime or like a stock photo and it's cropped poorly yeah it's almost like that even the creation of the photo was automated using like basic Ai and it just didn't work very well it's like a screenshot from a

photo of something else like it just you're right and then and the name is like Huggy Bear Rose and you're like this is not that's not a real name it's a bunch of random words that sounded cute put Together somebody who didn't speak English fluently thought sounded like a name that um you're absolutely right you see this they're not very persuasive but but one thing that we're looking at right now is trying to understand the motivation for that like why would you run an operation that's just such garbage like what is the what is the

point because when you when you go and you try to take over a Hashtag or put out a hashtag or do something like this Twitter is watching now right there are Integrity teams that are designed to find state sponsored operations on the platform this is the kind of Legacy you know the improvement post uh post 2016. um so the question is like why do you do it and one of the things that we're thinking about at the observatory is not all propaganda is designed to persuade not all activity is designed to persuade Sometimes it's designed

to distract sometimes it's designed to just make it too hard to find the good stuff so if in the case of like the Hong Kong protest there was this one moment and remember this I was sitting in an airport so I didn't go somewhere and I was on Twitter um and this was right when the Hong Kong police had shot a woman in the eye um I believe she was a medic or a nurse or something that shot her in the eye And uh and the there was these extraordinary photos and of the woman on

the ground you know the tear gasoline is very very compelling visuals um and Western Twitter was paying attention all of a sudden right it was really paying attention to this moment my Twitter feed which is not primarily China Watchers was was all talking about this moment and it reminded me of um of Iran actually when during the Arab you know kind of right after the Arab Spring There was a woman I think her name was Niha who was uh murdered and her face became an icon all right and similarly with China this woman who was

shot became an icon and the protesters began coming out in solidarity with eye patches and things covering their eyes sort of you know for their um the sort of woman who became an icon and when you have that moment where the protest has a human face has an icon right that's where you see the government oftentimes Will come in and we'll realize that this is now a thing that it has to respond to this has just gotten quite big right now there's a face there's a humanization it's not abstract protests it's quite personalized and so

what we started to see almost immediately after that incident happened was all of these accounts coming out of the woodwork to talk about how the West had it wrong her own side had shot her that you know she was really a plant this was a false flag The Hong Kong police hadn't done it so ways in which when that happens you see this barrage of conflicting narratives they don't even have to make sense they don't have to be cohesive they just have to be in the hashtag and they have to be so that when people

are searching for information about this protester or about this moment what they see is this content which is designed to cast doubt on what actually happened and by flooding The Zone with alternative Explanations you saw Russia do this when Malaysia Airlines when the flight came down also you know six or seven different explanations for what had happened but how it wasn't them when Jamal khashoggi was killed the Saudi Bots turned on Boom here are all the different reasons why Jamal khashoggi was not murdered was just missing was ins you know was still in the embassy

had fled so this is a journalist that was murdered in the Saudi consulate or Embassy and just they cut him into little pieces and they got rid it's horrifying so what they're doing is they're competing for attention right so and and they realize I guess that they don't need to get a critical mass of all Americans to do anything they can just focus on these Pockets diseases can take root literally or figuratively swing States can be swung right and you focus on these little bubbles just like in real life that make Fringe perspective Seem like

a normal or prevailing one so you have like anti-vaxx 911 truth there's birthers pizzagate Flat Earth and in this case like you say you flood the Zone with something that people just go well we can never be sure what the real answer is because these guys over here are saying that mh7 to this Malaysian Airlines flight crashed they're saying it was shot down by Ukraine they're saying it was shot down by Russia these people are saying it was Shot down by the U.S these other people say it was a mechanical thing we just don't know

even though it's like 99 of all the evidence points to one thing if enough people scream that it's not that and that it's something else and that then it casts doubt on what the real explanation might be and it doesn't even matter so what's interesting is a lot of people in Irish I originally thought this too they're just going to argue their perspective hey this was Shot down by a Russian uh militants in Ukraine no it wasn't it was shot down by this or it was mechanical failure but if they say like 10 different things

it's it's even more effective than just arguing persuasively arguing their one counter argument if you just cast doubt you don't even have to persuade people right it's easier to just create fud fear uncertainty and doubt yep and so I think one of the things when we see the Chinese activity It may not be designed to be persuasive right and they may not care if they lose those accounts because it's really easy to spin up another cluster of accounts when you need them or to go buy a cluster of accounts off the black market if you

need to and so there's a a difference in strategy you know Russia ran this multi-year long game and and it takes a lot of time and investment in those personas in that operation and then of course when they Were discovered and they lost those you know they lost about 3 800 Twitter accounts and a little over kind of 200 in aggregate Facebook and Instagram pages when you lose that audience you have to decide whether you're going to go reinvest in growing it again over again a multi-year period one of the things that we're seeing is

the idea that maybe particularly for American politics you have enough people who already have that Point of view you can just go in and amplify it why create your own accounts so one of the things that we're looking for for the 2020 election is not so much that same you know um replicating the tactics of 2016 or replicating this multi-year long-game propaganda operation by this influence operation but perhaps instead taking those highly polarized existing factions that exist from you know Real Americans and just Amplifying the contents that you create that um you know what could

possibly have happened in this moment on this night well here's 10 different Real Americans who have 10 different points of view about that we're just going to amplify those you know those different perspectives and that's something that we've been seeing you know the um we've been seeing the state media take That route right RT speaks very differently or is it just RT now they got did they get ready well if it was Russia today but now the now the proper name is RT um RT speaks very differently about uh the Floyd protests for example then

it's subsidiary uh redfish does right redfish really leans into the American left amplifying content from the the kind of like the left-leaning communities RT takes the more Law and Order approach You know articulating the conservative point of view and so here even in a completely attributed way if you're looking at Russian State media narratives on one side they're talking about um how America is burning on the other side they're talking about how cops are destroying communities and so these are again these are both um points of view held by Real Americans so they're just kind

of putting it out There and then they're using their their broadcast tools and their ability to reach people to kind of put out these kind of conflicting um you know the sort of opposite sides of the story and then retweeting and finding prominent accounts that hold these particular points of view and amplifying them and trying to get those ideas more exposure have there been leaks from the IRA like have we heard from people that have Worked there that are like Yep this is totally real because other people are like oh it's not even a thing

that's just an excuse that we're being told of course it's just more fun right fear uncertainty and doubt but have we heard from anybody that's worked out it's like no that my job was to cause crap on Twitter yes um actually repressed did some of the early uh high quality kind of exercises that um didn't See that coming yeah I know so there was actually a woman who went undercover her name is uh Ludmila savichok and I probably just butchered the pronunciation of that but she um she went and worked as an IRA Troll and

then uh kind of came out and and told stories about it and a few others have also told stories about their time there they had these people who've come out who've given you know who worked in the troll Factory and then told their Stories and they described something that sounds very much like a social media marketing agency right um we have quotas we had stand up during stand-up these are the kinds of you know guidance that we were given for what kind of posts we should put out the Twitter people operated a little bit differently

than the Instagram people um you know they had to understand the nuances of the platform and React to what was kind of dominant and trending On the platform that day uh they talk about sitting next to the person you know I'm running the black lives matter page and they're running the Confederacy page uh and sort of you know how are they going to do things that would kind of start fights and increase tension so you're you know the troll next to you is tasked with targeting the other community um one thing that I think is

very different About the IRA that gets lost sometimes in the conversation about the memes though is that the IRA actively engaged with real activists also so one of the things that that we talk about when we think about like cold war active measures is the idea of Agents of influence right people who work for another government that are not telling you who they work for but trying to influence you to take a particular action right like that show The Americans right yes yes you ever watch that I love that shows yeah I loved it yeah

of course you do yeah that makes sense and they had like the South African people who are like anti-apartheid or something like that but they were just being used by the Russian government to cut stir stuff and they they were they were just like college students and one of them ended up I think spoiler alert if you haven't seen the whole series they end up like Killing him for some reason or he ends up like getting left behind in some operation so it's like yeah they're using these Americans and people who mean well just to

just as tools to further their own ends it has nothing to do with the actual mission right and so there's um they find people who are ideologically aligned with what they're pretending to be and and social media makes this easy right you don't have to have a sophisticated you know Handsome spy and Alias the way you know um Philip Jennings is the character in that show you can just be a you're you're just a you're an avatar on a Facebook profile right and but at the same time you're the admin of this page that I

follow and I'm a black lives matter activist and you look like you're the person running a black lives matter page and you DM me and we start having a conversation you say hey can I give you some resources for a protest do you guys Need some money we can't be there in person because we live too far away but would some posters help right and so you start to see this process by which um there's extensive amounts of direct messaging that's happening right they're really communicating with the activists they're working to uh you know to

to a useful idiot is the kind of like you know unflattering term but um to create people who are unwittingly doing what they want them to do because of this Perception of shared camaraderie and so and they do that on you know they do that targeting a range of communities they hire Hillary Clinton impersonator they pay for Hillary Clinton impersonator to show up to a trump rally sitting on a flatbed truck like in jail you know um they hire a self-defense uh teacher to teach the black community Austin so we had to defend themselves at

protests right so again these are these things That were there and they're actually paying the guy and they're paying them via Paypal they're communicating with them via VIA telephone and text and he doesn't know that this is all just an influence operation he's just like great a gig that I can yeah okay yeah amazing most people don't think that you know the person who reaches out to them on you know Facebook is a Russian troll that's not generally people's minds go right no ironically most of the people That probably believe in a lot of these

conspiracy theories would be the people that would be suspicious enough to believe that except in this case since the crazies agree with the trolls they really won't know because they think they're all on the same side and that's that's probably how I'm sure you heard about this there's like a Texas secession movement and like a pro-muslim protest can you take us through that I don't remember the exact details but This was like a classic ridiculous example of what happens yeah so they had a page um called United Muslims and they had another page called Heart

of Texas Heart of Texas was the Texas secessionist page United Muslims was the the pro-muslim page which they used in some very interesting ways um they used it when they wanted to they ran like a Muslims for Hillary protest where they wanted people to show up and be proud Muslims for Hillary which of course was designed to antagonize the other side so sort of a very kind of um like kind of dirty way to to use that that group but what they did in this particular case is they had there was an Islamic Center and

I think Houston Texas um I think dawa Islamic Center if I recall the name and what they did was they had um the Heart of Texas page come out to protest the islamicization of Texas was what the Facebook event was Called and then um the the United Muslims were called to rally on the same day to defend um I'm trying to remember what they said the the specific wording it was a kind of um to you know kind of put out a you know kind of pro-muslim presence to counter this anti-muslim rally and so people

sitting in St Petersburg created two Facebook events and you know solicited the members of this group which were I Think both were over a hundred thousand members these were not as well these are not small Pages no um so they had these uh these different pages and they put out these events and then people rsvp'd and then they showed up in person and so so you had two different groups on opposite sides of the street at the same time and of course the organizers aren't there you know yeah yeah they're right they live in Moscow

or St Petersburg Right the organizer I wonder have they ever done anything where there's a protest and like seven people go like Hey where's the guy that set this whole thing up like no one's here or like hey the other side didn't show up what they kept doing and what you see constantly over and over and over again in the memes is they're trying to recruit people to be their local on the ground representative so like hey will you photograph Um get involved in the cause come photograph our protests come be a reporter for our

fake newspaper so they're constantly trying to find Real Americans so that they do have someone to send in person to these events so that's of course that is part of the operation what winds up happening with this Islamic Center rally is the police have to come and kind of like keep the peace between the two sides um and then if you go on YouTube I Actually went to YouTube because I was curious if there was first person video footage Houston Chronicle covered this you know major Texas press is out there reporting on this this protest

and you can go back and actually read the read the coverage from the day which I think now has a disclaimer up at the top linking to their more recent coverage talking about how this was executed by Russia um but you have this YouTube footage you Actually can go and see man on the street you know a woman with a camera phone recording this stuff so it actually happened and there's uh and it was uh and people uploaded the the footage of it to YouTube which then of course provides the trolls with more fodder because

now they have video right and and of course this has other Real Worlds this has other real world impacts in the UK I don't know if this has happened here but people are burning Down 5G Towers because they're like this causes coronavirus the 5G so they're destroying and burning these things down and then of course the probably the most common easy to point to example of disinformation in the USA costing people's lives is a lot of the anti-mask stuff is normal Americans but a lot of it is definitely foreign interference I mean you can just

sort of tell has all the Hallmarks of it have you looked at that at all it seems like that's an easy One for them to get involved with we have a hard time so one thing I'll say we have a really hard time with after and the way the so the internet research agency um when the platforms provided the data to the Senate the platforms did the attribution so I used to get like [ __ ] from people on the internet like well she said it was a Russian troll but how does she know no

the platforms did Digital forensics looked at things like metadata logins shared email addresses cookies you know certain types of behavior did this attribution kind of trace the network and one thing that we found when we looked at the data related to how that the attribution methodology um Twitter provided some of the metadata they were registered you know the Baltimore City News fake Twitter account was registered to a Russian B-Line phone Right you had American pages that were being run with you know people logging in from Jakarta and so there was not very much of an

effort to conceal the operation because nobody was looking they paid for the ads in Rubles you know so this was not a really yeah yes oh wow wow that wasn't they're like I'm not even gonna try and cover this it was just a right out there you know feline B-Line is a Russian carrier sort of look at all device data and were like Jesus Christ my vpn's not working don't even turn it on no one's even looking don't even worry about it basically um but that's changed now right so anytime you uh any any in

any kind of um cyber security or you know um you know kind of conflict situation uh when you expose the tactic of the adversary the adversary should evolve to no longer use that tactic right and so what we have now is is an interesting Challenge where there are Integrity teams looking there are researchers looking regular people are cognizant of the fact that Russian Bots or Chinese Bots or Saudi Bots you know all any nation's Bots exist on Twitter and so what you have instead is um but it's it's much more difficult to say this is

being Amplified by the Russians and one of the reasons why we try to hold off from doing that is one you don't want to delegitimize existing Movements and that's partly not to delegitimize them but it's also partly not to um absolve them of their responsibility if it is kind of domestic actors that are that are you know um pushing out lies or misleading video footage or whatever else um so with the coronavirus one of the things that that we look at at uh you know some of the work we've done on at Stanford is we've

looked at how different countries all around the world We looked at Russia China Saudi Arabia Iran Venezuela Brazil um uh how each of these governments has responded to coronavirus both domestically and internationally ways that they've used their state media apparatus and then the ones that have gone one step further and use this kind of covert you know Twitter Bots fake accounts um you know fake news that that sort of more uh Darker side of the propaganda Spectrum and what we see is um the there's really there's messaging that's put out for a domestic audience then

there's messaging that's put out for an international audience and the international content is often designed to you know it's just kind of like a public diplomacy effort we want to look good so here we are talking about what we've done but what really matters is um particularly for more authoritarian Regimes it didn't handle it so well ensuring that their population believes that they handled it well to preserve their um their ability to govern right and so what you see is conspiracy theories that the state puts out to ensure that you know in large part for

the domestic audience because they want the domestic audience to believe this was a matter outside of our control dark Forces beyond our borders brought this virus here and this is what we have done To control it but really the coronavirus was created by the Americans in a lab before Dietrich was kind of where you know some of these sorts of things went um and and then you have uh entities like Russia Today which say like the Iranians are saying that the Americans kind of Daisy daisy chain of just asking questions and um it's been really

fascinating to watch you know the thing with coronavirus is the entire world is paying attention to it every government Has had to you know handle the crisis in its own way the entirety of the population of the planet pretty much has all been searching for information about the same thing as various countries have gone through phases of high death rates or lockdowns um so it's really been a remarkable opportunity to study how narratives move and how this interplay between the kind of traditional like sort of fever swamps of the internet being legitimized Through State media

and state influencers in this um this kind of free-for-all so in some ways it's been less of a our Russian trolls in there and more of a in this sort of free-for-all in which every faction is participating simultaneously what are the narratives that are spreading what are the mechanisms by which they move through communities and then what is the impact that they have do we start to see people Change their behavior in response to allegations of a drug working or not working allegations of masks working or not working uh how does that become you know

incorporated into people's identities their partisan identity in particular in the U.S and that's the kind of stuff we've been looking at do we know if the United States is doing this as well to other countries like are we doing this in gosh I don't even know like Central Asia Um South America it seems like we would have done but I don't know are we doing mostly like Hey we're trying to form a democracy and we're promoting democracy or is it kind of like you know what screw it we're doing the exact same thing I we

have not found an operation attributable to the United States so as I as when we were chatting earlier and I said there are certain laws that prevent the United States from putting out Particular types of propaganda on so or Communications even on social platforms because of the uh risk of it being seen by American persons and then you know violating certain laws um similarly doing kind of mass ingestion of social media data to do analysis and find influencers or messaging that's also uh not you know that's not legal um like there's a certain privacy laws

that prevent the US government from Doing that so in some ways these sort of laws that were put in place for kind of older um communication ecosystems have limited is the sense that I've gotten U.S government's ability to do this so even though everybody says you know you're just not looking for you're not talking about what the US is doing we don't have any uh any activity attributed to the us and that is not because people are afraid to find it or under willing to Look for it it's just that's that's the you know that's

the kind of current state right now so the platform Integrity teams have been pretty clear that you know they will take down uh any content that they see as inauthentic and that's uh it's kind of where we are right now the Assumption being that you know domestic groups do come down periodically um activist groups occasionally come down you know there's a variety of Reasons sometimes um the platforms will pull things down because of coordinated and authentic behavior is the term that really usually means that there's like a state actor involved um and so the platforms

will then Twitter will make the data set public Facebook will write a blog post and outside researchers will do an analysis and publicize their findings we do that sometimes you know we're one of the teams that does those analyzes Um but the yeah I mean the other way that things the other way that things come down is like for spam just kind of coordinated distribution like um no allegations at any foreign actor or domestic actor or government sorry there's a no foreign actor or government is involved um but just like I have a Blog and

then I have 10 other Blogs whose job it is to promote that one blog and so it becomes more like a Spam type takedown yeah that makes that makes sense the coordinated inauthentic Behavior has got to be tough to catch that's a whole technological discussion I wonder what other Solutions are available to us for this because especially in this like I think Sam Harris calls it an epistemo I'm going to try and say it I had it the other day uh epistemological Free-for-all where facts do not exist and I've always wanted to use the word

epistemological because it sounds hella sophisticated but what Solutions are available to us for this if people kind of don't believe any facts now or if that's like the idea that they're trying to generate or the mindset they're trying to generate I think that we have to fundamentally rethink curation and recommendations online and I think that that is a bigger Issue than State actors or not State actors I think that's a question of information integrity and I think that's a question of um you know it's it this is where you know people will yell on the

internet and say like who decides you know well somebody's already deciding the algorithm is already structured in a certain way so I don't understand why we've decided that these algorithms that are like 10 years old or sacristanct and We can't rethink it in any way so I do believe and as you started to see with anti-vaccine content in particular um with q and on now as well the platforms are saying we're going to remove it from the recommendation engine it stays up on the platform meaning if you want to go find it you can go

you can go find it but we're not going to return it in search results we're not going to return it we're not going to promote it in the recommendation engine And if somebody is looking for an answer to a topic we're not going to serve up garbage in response to that inquiry and this is it's weird to me that in the you know span of like five short years or so we've gotten this idea that um you know we you know who can possibly decide how to set the rankings and weightings we've you know there

is a system that's in place that's only that old I I don't think it's unreasonable to think that That we could change it um Google in particular recognized pretty early on that serving garbage in response to health and financial inquiries for search results had potentially very serious negative Downstream impacts for people's lives and so it came up with a framework called your money or your life this was back in I think 2013 that the framework was introduced 2015 it was improved it didn't apply to YouTube at the time Interestingly because for a long time the

search function was seen as kind of different than the social function in the sense that when you're going to search you have a question if you have cancer you want to know what's going to happen to you you want reliable medical information not juice fast [ __ ] but if you're on YouTube you maybe want to be entertained right and so the idea of you know none of these platforms were not the social platforms they were not Developed to be information libraries and repositories of human knowledge where people go find answers to their financial and

health and polit critical questions on those platforms they were to help you like find your friends right and find your knitting club not your your scientific and medical Authority for how you're going to treat a disease or you know operate in the world of of a global pandemic right and as they've been put into that position as this is What they have inadvertently evolved into um I think it's I think it's eminently reasonable to say we have to be rethinking what we're curating and what we're recommending uh in the context of looking at the potential

Downstream harms on the subject of the foreign actors though that's something where um you know for the 2020 election uh We've really done a whole lot of work in outside researchers like you know our Team at Stanford or um governments tech companies the Integrity teams at tech companies policy teams civil society which you know particularly for like the African-American Community that's been very targeted they have a real strong voice now saying like hey come on you know these um fake trolls pretending to be black people are really detrimental to our community what are you doing

about them Um the ways in which academic researchers all of the different kind of stakeholders that pay attention to and study this topic uh now are working together in concert saying okay the 2020 election is coming out uh how do we have a narrowly tailored you know nobody wants to be the fact-checking police but how do we have a narrowly tailored uh kind of multi-stakeholder ability to say um here's voter suppression narratives here's voting misinformation related to Voting itself and how do we try to ensure Integrity in the election ensure Integrity um in the outcome

of the election make sure that people's degree of trust in the outcome of the election remains High the election remains legitimate how do we all take the the tools and capabilities that we have and work together so if we see something that looks like anomalous activity we're not going to be the ones to make the Determination about whether it's Russian Chinese or domestics but if we were to give it to the social platform and say hey you guys should take a look at this hashtag you should take a look at this page take a look

at this account they're the ones who then have the capability to make that judgment behind the scenes or similarly when someone like the FBI says this site appears to have Financial ties to Russian activities Russian intelligence Services that they're buying ads and rubles yeah right so this is where you start to see that all of the different you know we all have different capabilities how do we ensure that that we're sharing information in an effective way uh particularly as the 2020 election approaches if we can sow doubt and be negative with these narratives can we

do positive things with it as well like it I'm kind of thinking of explosives right they can Be used for construction but only like a small part of that would be construction you can blast the rocks and then you can build the tunnel and that part takes a really long time but then when it comes to destruction they're really fast you don't worry about cleaning up the mess if you blow up like a shopping mall or something right you just worry about the Destruction part can we also do positive things with these techniques and these

narratives or these narrative building Techniques and Technologies it seems like there should be a way for us to go hey remember when everybody started to believe that covid-19 was caused by 5G why don't we use that to show people that uh oh gosh I don't know good hygiene or like staying in school is a good thing you know can we is anybody thinking about that or is it just like hey man one problem at a time seeing a number of medical journals um You know running convenings uh you know kind of immunization research teams working

alongside marketing teams not because they're interested in the marketing piece but again this idea of everything is a marketing campaign for an idea now so how do you ensure that you are communicating in a in a way that's conducive to the modern environment this is one of the things that's been very frustrating to me as the uh as the kind of covid stuff began Happening um in January we started seeing the anti-vaccine narrative so we were paying attention to coronavirus pretty early on because we you know we pay attention to internet conversations globally and this

was a big topic of conversation in Southeast Asia even in January um and what we started to see among Americans was the anti-vaccine activist saying um oh my God they're going to use this As a way to Institute programs of mandatory vaccinations all right and so that was the sort of thing where we're like okay this is a this is a thing that if there there is a vaccine like the people who are in the government and the manufacturers and the scientists and the public health authorities have to do better than a press conference here

and there and like a PDF and uh you know the because the problem that that you're getting at is that There are low levels of trust in authority and there's low levels of trust between communities and that is not a social media problem that is a societal problem and social media can exacerbate it social media can reflect it but ultimately the question becomes how do Americans respond to their uh to their authority figures and do they trust them and if you know this this the president that we have interestingly funny enough he was kind of

a darling of The anti-vaccine movement when he was running in 2016 he you know was talking about how he's tweeted in the past you know kind of tacit support for the idea that vaccines cause autism in a range of these other kind of theories so they were big supporters of the president then now as this operation warp speed and this idea that the you know US government is looking for and hoping to find a coronavirus vaccine they're not quite sure what to make of that actually So but interestingly the people who opposed the president who

do generally trust science don't trust the president and so you have this interesting Dynamic of you know would you take a vaccine that Donald Trump's FDA says is safe and that's one of the narratives that you see even from people who are inclined to trust scientists and Science in general believe in vaccination vaccinate their children are hesitant because they don't trust this particular Administration and This particular administration's commitment to science so the thing that's interesting about social media is it really kind of tears down the veneer of infallibility right you see your leaders having right

you know there might not be so just go for it but it's the idea that like you know you come away with the sense that like nobody's steering the ship right nobody's in charge and The other thing is you know you see the media get things wrong a lot there are a lot of stories early on with covid-19 about oh it's just the bad flu right nobody yeah um there are a lot of people who were very angry about that there then there was the Mass guidance issue CDC and World Health Organization didn't update their

guidance until people on social media who were not even epidemiologists had been screaming for weeks that we Needed to be using masks then there was a sense of well random person on Twitter seems to be right and the authorities are lying to us or wrong so how can we trust the authorities when they come to us later so you just have this the sense that um I think we have to rethink how authorities communicate in the era of social media I think people have to and that goes for kind of media as well in The

sense of saying hey look we got this wrong we didn't have the you know or not even we got it wrong with the information at the time this seemed accurate new information has since come out that renders that obsolete or incorrect and here is where we are today right and or communicating to people we are 25 sure of this now and here's our best guess 25 you guys are all on here looking for information we're going to level with you this is what we know this Is what we don't know here's how we should be

thinking about this right and it requires a very different style of of communication than I think these institutional authorities are used to um and I yeah I think that that ultimately is going to have to change in order to restore trust um in yeah you know in authority and media almost like thinking in bets right have you ever read that book or heard of Your book yeah yeah oh yeah so yeah thinking about right like right we're not 100 sure and we can admit that because I I get why an agency says you have to

do it this way because this is what we we know right now because if they say well we're really not sure a lot of people are going to go well I'm gonna go with the guy who says that he's sure even though he just pulled the information completely out of the air and has no facts to back it up Or research it just sounds more persuasive but the problem is once you have to change your mind then people go well how do I know this isn't one of your things where you're just going to flip-flop

on me but if you say look we're 60 sure it's this way and if we do it this way it's at least safer than doing it the other way until we know for sure a lot of people are just going to be a lot of folks like us are going to go okay good enough that sounds like a Plan but I think there's just going to be a lot of people that are used to hearing certainty and only will settle for that and that's a problem social media manipulation is right now what happens now that

deep fakes are coming they're going to be super easy to make um deep fakes for people that don't know is when like I mean I can only think of porn examples which might be a little incriminating but whatever it's like they put someone's face on a body of Another person and they might even fake the voice and make it look real I mean how do we counteract that if I see a video of Renee the rest of being like hey all this stuff I just said on the Jordan Harbinger show and Joe Rogan about the

IRA was just complete garbage I just made it up as being paid by Canadian intelligence I'm not likely well I am now after this but normally I'm not likely to go let me just double check that I saw in a video it has to be Real right has to be real yeah it's I've been thinking a lot about that lately um you know as a society like we adapted to photoshop right there we adapted to see right there there were these sorts of like so I've been curious both about what is the impact of the

technology what is the societal acceptance and internalization of the technology um deep fakes are interesting because for those who don't know uh with with Photoshop there's a thing that you can Go back to right there is a um a uh you know original material and actually funny enough Adobe has actually been really like leading the Vanguard of this um desire to ensure that there is some sort of like crypto lot you know um cryptographically verified kind of um certification that says that this is the image as it was when it came out of the camera

and here it still is or so you can see the manipulations that have Taken place on an image or something when you have something that's generated by AI there is nothing to go back to so the interesting question is um you know for audio or video do people believe it right now we're still in The Uncanny Valley it's not quite right it's not quite there particularly video there are certain detections like ways that you can uh observe um something that a computer has Generated uh there's there's little things like um you can see when somebody's

uh heartbeat is pulsing you know the color of their skin changes ever so slightly this is how your iPhone can see um if you know like red to normal red to normal red to normal wow I did not know that that's incredible yeah there's ways that you can tell um if a you know if you're looking at a video of something that is alive I'm not an expert in this Particular type of detection but um what happens now though is we have the ability to generate text and still images and video and audio and I

think though we focus a lot on the video because it's very very Sensational as you mentioned it also did kind of really take off and Porn first um yeah that's where most people get their news these days is right out of PornHub like new tech oh let me go look it up Right now porn started taking it down out of non-consent actually out of the the idea that the people who were in it had not consented to be in the video imagine looking to porn sites for like guidelines on how we're gonna yeah how to

develop policy for it um so that was kind of an interesting interesting thing Facebook and Twitter have since come down on um generative media they've recognized that it's different than something that's just selectively edited Selectively edited is allowed to stay up oftentimes with an annotation generated uh will come down if it's seeing is being manipulative question is what happens when you have generative that is labeled as satire uh does it fall under artistic expression you know so there's a lot of these kind of policy gray areas but the thing with the video and audio is

that usually um you know they they are used to create Sensational Moments right so this is a hot mic politician or a sex tape politician has been you know of course because everything goes back to that um so those are very Sensational moments that then inspire people to go and try to authenticate the video find out who created it find out how it spread so there's a lot of attention that kind of immediately goes into understanding that particular video the thing that's been more interesting to me has actually been Like gpt3 which is the open

ai's text generation AI so this is like an AI algorithm or program that writes like a human is that kind of what that is text yeah you give it a prompt um you give it a prompt you can give it kind of constraints around how creative it can be um the AI has been trained on content on the internet so it's Read Wikipedia right and so it has a body of knowledge which is very interesting and so as I've Been using it um just to see what comes out of it um depending on what prompt

you give it and how whether you constrain it um to uh kind of stick to what it knows or allow it to be more creative you will get content back very different content you can submit the same prompt over and over again and it'll return very different types of content to you you can have it right long form essays you can give it tweets and it'll return back Kind of tweet link things I've been working on an essay actually and I thought him having a hard time with the closing well it's an article on AI

let me give it a gpt3 and see what the AI generates for me for my closing um and I did a couple really interesting things like once or twice it pulled in characters I had not mentioned yeah I was writing about Ai and all of a sudden it gave me a paragraph on Edward Snowden And I thought like okay you know I wasn't expecting that um once or twice it returned links to like you know returned regurgitations of academic papers that it had probably kind of you know read at some point consumer right um been

trained on and then a couple times it gave me back like um it once it suggested a title for the prom I gave it a couple paragraphs that suggested a title and then it gave me You know when you go to like you read an article and it gives you the list of like related articles on the bottom oh yeah you know it like generated a whole bunch of titles for things that it thought would be like related to this content that it hit that I had given it um which I thought was sort of

an interesting an interesting response to get back but what the takeaway for me was like it's not perfect you know it is sloppy And at times it goes off the rails into these like you know the kind of like freshman philosophy essays yeah I mean that's as far as I ever got with it so it probably just looks like I wrote the essay but yeah [Laughter] or just like garbage just like word soup where like you write enough words on a page and you hope that that something has come out of it but really it

hasn't right yeah you hope your teacher's Grading it like while watching Netflix so it's not perfect but it's remarkably good and it with like some mild human curation you could see how if you wanted to run a Twitter bot um you could with some you know give the you know give it a prompt run it a bunch of times pull the stuff that comes out of it evaluate it you know check it off and push it into a queue and then go right and it's original content and That's the thing that's so remarkable about this

stuff you can't trace it back to something somebody else said it's not a copy pasta it's not an engram that is like you know tied to some Community or something in the past it's uniquely generated content and so it really is interesting it kind of in a way you know democratizes the ability to run a commenter army because all of a sudden you have this machine that does the work of producing all the content that you're You know kind of you know 50 Cent Army had previously been paid to do right so you have this

ability to um to create something that's virtually undetectable it's cheap it can be put into an environment where it's not Sensational people aren't going to go look at it it's just read as like this is the voice of the man on the street this is the opinion of this commenter uh and so what I think it ultimately does a Lot of these things is they they it means that we have to have a kind of a different relationship to our idea of identity online right when you can no longer trust with the content you're reading

is being produced by a real person when you can no longer distinguish then the question becomes like one is you know do we care do we evaluate the comment based on the marriage but as we've talked about in this conversation if you create the Perception that a whole lot of people have this opinion and you've just done that you don't even have to use copy pasta now you have an AI generating that then you really do have the ability to influence through this repetition and this persuasion and this idea that everybody's talking about or thinking

the same thing so then the question becomes how do we know that these are real people real identities that are tied to this content And so I think that the next thing that we'll be thinking about is actually is actually that question really it's um to what extent is you know how are we going to think about identity in an era when faces are generated you know when I was uh when I spoke with Joe Rogan this was in like March 2018 I said there's this website this person does not exist.com and it generates faces

the AI is generating faces and each face is different so if you reverse image search It nobody's going to find it because it's not a stock photo and it's not a real person and as that technology improves again you can create so you can create personas with these computer generated faces that are untraceable and we've already seen three or four influence operations I you know I was talking about it with Rogan in March of 2018 uh November of um 20 sorry I was talking about I spoke with him about it in March of 2019. let

me say that again This year's been five years long it's it's right so it's really confusing being that 20 20 yeah has been at least three to five years in my in my perception so I was having this conversation about this technology in um March of 2019 and in no November 2019 we had the first takedown based on this you know a network of pages that were attributable to an entity affiliated with The Epic Times called the BL I think I think it's For the beautiful life or something one of these kind of fake you

know fake is not the word one of these like kind of spammy Facebook pages um was creating personas to share their articles and these were fake accounts with AI generated faces to just produ you know going into groups and sharing content affiliated with this publication so we've already seen ways in which people who want to manipulate are using all of these different Technologies as They become available um and so the question becomes okay how do we think about in advance what kinds of Frameworks we're going to want to see to ensure that people can trust

the information and content that's in front of them trust that that person who's saying it is real yeah I mean that's the challenge and then as we move into VR it's like and add another layer of where like voice And video like face and everything is fake it's just it seems like a a pretty uphill battle Renee thank you so much this has been fascinating it's enlightening it's also a little disturbing because it almost seems how optimistic are you that we're going to be able to solve this like what happens in other countries like Estonia

that have been hit with Russian influence operations for decades like are they doing okay or is it just nobody trusts Anything they see ever I think there are certain countries that have a higher degree of trust in government than we do right now right so like Sweden has been a target for operations but people trust the government when the government says you're you know the society's operations look uh people internalize that and then they feel better educated and the uh impact is less that doesn't happen here unfortunately it turns into well this Political party said

it's a thing and this political party you know just is saying it because they hate that other political party so it's become a partisan debacle here I think the I mean I'm optimistic look people have adapted to emergent Technologies before propaganda has been around for a very long time I think the question becomes do we have the um are we sufficiently prioritizing making changes to improve our Information environment right to recognize that this is you know this is the new normal and how do we want to adapt to that either by way of Regulation or

by way of changes you know making algorithms more um understanding of the downstream impacts that they have so I think that that's kind of where we are today I wouldn't say solve I don't think that you solved this information I think it's more like a chronic condition right you Manage it you adapt to it you help the population be resilient to it and that's where we have to be going Renee daresta thank you so much thank you hope you all enjoyed that you can find a lot more just like this in the Jordan Harbinger Show

podcast feed which you can find in apple podcasts Spotify or wherever you listen to your podcasts now click here for a little change of pace with actor Danny Trejo or right here for More election hacking with actual computer hacker Harry hirsty he talks about the vulnerabilities of election machines and of course click here to subscribe to the channel