[Applause] you have almost all been subject to a computerized predictive algorithm of the following form if you like X you may like Y music movie streaming services online shopping services routinely look at your pattern of behavior compares that to previous customers and tries to make predictions for you sometimes it works and sometimes it doesn't work now you may be less aware that you've also may have been subjected to a predictive algorithm of the form if you are like X we may not give you a loan banks financial institutions routinely use your personal attributes compares them
to previous customers to assess the risk to the institution in terms of giving you a loan or mortgage you may have also recently been subjected to a predictive algorithm of the form if you are like ax we may not give you a job employers and in some cases university admissions are using predictive algorithms to make hiring and admission decisions and if you've recently had a run-in with the criminal justice system well I'm sorry first of all and second of all you may have been subjected to a predictive algorithm of the following form if you are
like X then you may go to jail courts now are routinely using predictive algorithms to make decisions in the courtroom one particularly popular and now widespread algorithm is used at the point of arrest to make bail decisions and conceptually it's a pretty simple idea a defendant who's been charged with a crime is information is extracted from them and is fed into a computer algorithm and that computer algorithm outputs a risk factor and that risk factor is meant to quantify the likelihood that person will commit a crime in the future think of a really simple version
of the Minority Report without the creepy people in the pool and if you are assessed to be high-risk well then the judge may deny bail and you will be held in prison awaiting trial and if you are assessed to be low-risk then you may be released pending trial now in 2016 investigative journalists from ProPublica published a troubling report on one particularly popular and widespread predictive algorithm that is used in the courts around this country and what they found is that this particular algorithm was significantly disadvantageous to black defendants what they found is that if you
were a black defendant you are nearly twice as likely to be predicted as reoffending when you did not that happened 44 percent of the time if you were black and only 23 percent of the time if you are white that is a staggering difference and at the same time you were almost twice as likely to be predicted as not offending when in fact you did if you were white as compared to black that happened 47 percent of the time versus 28 percent of time so in other words the algorithm is biased to black defendants against
black defendants in other words if you are black and you are a subject to this algorithm you are more likely to be assessed as high-risk and incarcerated than if you are white so shortly after this article was published a student of mine Julia Dressel came to me and she said how is this possible we're computer scientists how are we allowing this to happen how are we allowing software to make life-altering decisions on people when they have these types of inequalities and it was the right question to ask and it got us spending about a year
and a half journey trying to understand what was happening here and how and if we can do better and I want to tell you a little bit about that journey and a little bit about what we learned along the way so before I dive into the algorithmic details and sort of understand where this bias is coming from we started with a really simple question we started with the question of are these algorithms actually better than you than the average person because after all presumably we use these algorithms because they're more accurate more objective and more
fair and they remove any biases that may exist in the human mind and that's a good idea but we don't know if it actually is true or not so we set out to determine that so here's what we did we recruited four hundred participants from an online crowdsourcing platform and we asked him a very simple question we gave them a short paragraph about an actual defendant this was a with 8,000 people for whom we had demographic information we had information about their past criminal history if any um and then we knew the risk score according
to the software that is being used in the court and then they were tracked for two years so we actually know if they reoffended or not and so our participants read a short paragraph was seven pieces of information the age and the sex of the person and then five pieces of data that summarized their juvenile and adult criminal history they did not know the race of the person importantly okay and then we simply asked them please read this short paragraph it's an online survey and just tell us if you think this person will reoffended in
a two-year window following their last arrest yes no yes no yes no here's what we learned the computerized software had an accuracy of 65% our humans had an accuracy of 67% these are random people on the internet responding to a survey for a buck and they spent about 10 seconds reading that little short paragraph and making a determination they are as good as the commercial software being used in the courts today oddly they made the same errors as a software they were more likely to say that a black defendant at a rate of 37 percent
would reoffended --nt compared to 27 percent for white defendants and they're more likely to say that a white defendant would not reoffended at a rate of 40 versus 29 percent how is this possible they don't know the race of the person and by the way neither does the software so both the humans and the software have a racial bias and they no the race that's weird okay so first of all it's a little troubling that random people on the internet are as accurate as a software that's been deployed in our courts and we're gonna talk
some more about that in a minute but there's two questions that came out of this one is how is it that humans and the software have a racial bias when they don't know the race of the person and number two is how is it that random people on the internet being paid a buck to answer our survey question are as good as commercial software that is being deployed in our course and we needed to answer those two questions now at this stage what we should have done is reached into this commercial software this predictive algorithm
and asked how does it work what's going on in here what is it what information is it using how is it doing that mathematically and really understand how the algorithm is making the mistakes and why it's not more accurate unfortunately the software is proprietary unfortunately the software is a very tightly held corporate secret that has been protected by the company and we do not have access to it and neither do the courts by the way so here's what we did we reversed engineered it we said okay fine you don't want to tell us the answer
we'll figure it out ourselves and so let me tell you a little bit about how we did that so here's what we're gonna do I'm going to assume that I'm going to only try to make a prediction of a person reoffending from two factors these could be any factors you want it to be how old you are how many juvenile convictions you have how many years you were unemployed whatever whatever numbers you can quantify and imagine I just have two of those factors for now just to keep things simple and when we do is I'm
gonna reach into that database uh that I use for my human participants where I know the demographic information I know whether they reoffended or not and I'm gonna plot in this space factor a factor B all the people who did reoffending data point here every point here corresponds to an individual and their corresponding value of factor a and factor B and now I reach into my database and I do the same thing for the people who did not reoffending factor B and so now I populate this space of these two factors and I know who
did and who did not refine and my job is to build a classifier and the simplest possible thing you can do in this space the kind of thing we teach our undergraduates at Dartmouth College is to draw a line through the space that's it that's all it takes and all we now say is well anybody that is to the left of that line is at risk of reoffending because that's where the majority of the offenders are and anybody who's to the right of that space is not at risk now I have a classifier and notice
that I've trained the classifier on data that I know the answer to but now any individual that comes in anybody in the audience I can compute your factor a compute your factor B plot in the space are you to the left or to the right and I have a prediction for you okay so now the question is what are the factors what are the factors that allow me to predict whether you will commit a crime in the future if any now remember I said that our human participants had seven factors age sex and then five
factors that quantified their previous criminal history and importantly not race so here's what we did we did hundreds and hundreds of classifiers on all possible subsets of those seven factors we didn't know which ones were the most important ones but we wanted to start there and we build hundreds and hundreds of classifiers with all possible subsets whether it was five or two or one or all seven and then we just asked which classifier is the most accurate which one gives us the best predictive accuracy and we learned something really surprising from just two factors how
old you are and the total number of crimes whether that was a juvenile a dolt is enough to get the same accuracy as the commercial software so commercial software is an inaccuracy of sixty-five percent our simplest possible classifier with two factors and only two factors is it 66 percent and our classifier by the way still had the same racial bias misclassifying blacks at a higher rates than whites so how does this help us answer this question as to what's going on here right how does this help us determine why is there racial bias and how
is it that this commercial software is as good as random people on the internet getting paid a buck to do a survey one of the two most important things the classifier is number of prior crimes right I think we can all agree that age is probably race blind but let's talk about number of prior crimes here's what we know we know that in this country if you are black you are much more likely to be arrested charged and convicted of a crime the New York Times just last week for example had a report that said
in New York City black people are 15 times more likely to be charged on low-level marijuana charges than their white counterparts number of crimes is a proxy for race because of this social inequity because of this inequity in the criminal justice system and so race was slipping into the classifier unknown to us and had we not reverse engineered the classifier had we not figured out what are the two most important factors it would not have all been clear how race was slipping into here and so the race bias is coming in because we are feeding
data into this thing that tells you the race of the person inadvertently number two how is it the commercial software isn't that is making life-altering decisions on people is the same as random people on the internet with little to no criminal justice experience well as it turns out what we learned is it's actually a really simple classifier here's what you do you take two pieces of information how old somebody is in the total number of crimes they've committed and here's basically what it comes down to what the classifier is doing and what we think humans
are unconsciously doing is the following if you are young and you've committed a lot of crimes you're at high risk and if you're old and you've committed very few crimes you're at low risk that's what it comes down to right it's pretty simple now you could argue that simplicity's not bad it's not bad to be have a simple classifier but I argue that hiding behind big data data analytics artificial intelligence machine learning you give the appearance that your predictive algorithms can make more sophisticated inferences than they actually can when the company says we use the
latest and data analytics we use the late and artificial intelligence it appears as if they can make very sophisticated inferences about the future for criminal defendants when in reality they are actually doing something incredibly simple and why does that matter it matters for the following reason imagine the following scenario you were a judge in a courtroom and there is a defendant before you and you are told that a proprietary software built on the latest advances in data analytics and artificial intelligence and machine learning predicts that this person is at high risk of offending you would
imagine that that judge would take that recommendation relatively seriously it sounds compelling now imagine that the judge is told that a random person on the internet tweeted that they think that person is at high risk you'd imagine the judge wouldn't give a damn and rightfully so but here's the problem that seemingly sophisticated software and that random person tweeting on the Internet had exactly the same accuracy and so it does matter it does matter how simple and it does matter that it is no better than humans because it tells us how much credibility it tells us
how much weight to put into these predictions and I argue that weights should be very very little given what we have learned so in this study we learned two important things and they go beyond these predictions in the criminal justice system one big data data analytics AI ml are not inherently more accurate more fair less biased than humans simply hiding behind technology does not make you better and this is particularly true when the data that is being fed into these algorithms mirror social existing social inequalities and the danger here is that you hide behind that
right you forget that you're feeding data into the system that is what is happening in the world around us these predictive algorithms are meant to overcome the limitations and they are simply mirroring them and that is very dangerous number two it is dangerous and it is reckless to unleash predictive algorithms like this without thoroughly understanding how they work without thoroughly understanding their accuracy and without thoroughly understanding their fairness and that is true in the criminal justice system it is true for employers and universities making admissions and hiring decisions it is true in financial institutions making
loan and mortgage decisions those three things are some of the biggest things that we do in our lives our civil liberties our financial well-being and our employment and we are turning over decisions to computerized algorithms and that is very worrisome to people like me as technologists I will mention by the way that the much-talked-about GDP our that went into effect in Europe recently has given European citizens the right to audit the data and the algorithms that are used against them or on them or to them if they believe they are incorrect that is a very
thoughtful and very good idea and it's something that we in this country should start thinking about there was a larger lesson here in the 1980s we were doing with highly complex issues having to do with medicine and biology we had amazing advances and we were really struggling with how to reconcile that with our social norms with ethics with morality and with religion and we were really struggling in society to come to grips of that in the u.s. propped up a bioethics panel and that bioethics panel was populated with incredibly smart people from all walks of
life from the religious part from the scientific from the ethics from the legal and from the policy and from the economic side and they were they were their task was to help us manage these highly complex issues that we were dealing with and we didn't have a roadmap for how to go forward I advocate for the creation of a national or an international cyber ethics panel a panel to help us contend with what is turning into an incredibly complex cyber landscape whether it is the types of predictive algorithms that I'm talking about here that are
being on sign-u is the public whether is how the internet has been weaponized by hate groups and extremists and terrorists whether it is the privacy issues that we've been struggling with over the last few years whether it is security issues whether it is how the internet is being used to traffic children in the sex trade all the problems that we are seeing online we have to start getting a grip on technology can be a force for tremendous progress and tremendous good but as we have seen over the last few years left unchecked it can just
as well plunge us into a digital dystopia thank you [Applause]
