Alignment problem. It was released just last week. He offers theoretical contributions to the industry documenting in his bestselling book the most human human outer rhythms to live by. In his new book, question such as if we continue to rely on Artificial Intelligence to solve our problems, whathappens when ai itself is the problem. Greenbergs last seen Computer Scientists tackling the obstacles of technological implication of an advanced new society that has become more like on technology. His work on Technology Leaders such as eli must once put the question in his 2000 book the most human human for his night table reading. In his new book the alignment problem is dependence on ai questions whether or not the system can help us or the systems simply are a replication of an inevitable human desire, how can we handle ontechnology. Here he argues that all these things make decisions for us, essentially we will be discussing a lot of the next hour and i want to ask your questions to watching live , read our questions in the chat on youtube. So i can work them into our conversation. Thank you brian and welcome. Thank you forjoining us. You for having me. This is not your first book. I want to ask kind of the ultimate question that is the obvious question we ask an author but were just setting the plate for our station here and that is why did you try to tackle this kind of thing now in this book. Its a great question. The initial piece for this book came after my first book had come out as you had mentioned, vanity fair reported that on must, it was his best bedsidereading and i found myself in 2014 attending a Silicon Valley book group. That was investors and entrepreneurs and they had seen this thing about elon must reading a book and decided to joiand to my surprise he did and there was this really fascinating moment at the end of the dier when the organizers me and everyone stood up to go home for t night. And elon must urged everyone to sit back down. He said no, but seriously, what are we goingto do about ai . Im not letting anyone leave this ro until either you give me a convincing counterargumentwhthere should be this or you ca give me an idea for something else. And it was quite a memorable. And i found myself drawing a blank. I didnthave a convincg argument. And i was aware of the conversation around ai, some people its like a human expansion level risk. Other people are focused more on presentday ethical problems. I didnt have a reason why we shouldnbe worried about it but i didnt have a concrete suggestion and that was the general consensus in the room at that time. So this queson of so seriously, whats the plan . I was finishing my previous book and i really began to see around 2016 a Dramatic Movement wiin the field to actually put some kind of plan together. Both on the ethical questions and the sort of further into thefuture safety questions. And both of those movements have grown i woulday explosively to 2016 and now. And these questions of ethics ansafety and whats in the book i described as the alignment proble how do we make sure the objective that the system is carrying out is in fact what were intending for it to be. These things have gone from card kind of marginal and somewhat philosophical questions to really today making up i would say the central question. And so i wanted to tell the story of that movement to figure out in a way the question, whats the plan. What are we doing . One of the interesting thoughts about getting into this was theres a lot of opportunity, complex Technology Going into this but i dont think a lot of people are aware of that the systems already being applied into lifeanddeath situations. We when we were talking to the program, you picked out a number of examples in the helplessness they hope to perform one of those examples could not be more true and that his algorithms are being invited judges notin california but what im getting into in california , where instead of ch bail, a judge will use an album rhythm to determine wheer or not a suspect is released or is allowedo only try and be small californians are boarding that proposition. That would reaffm along that would replace it with this algorithm system and its a very complicated issue but what reay surprised me is the naacp and Human Rights Law Firm this proposition because of built in and equities of the alrithms. So whether we get into that kind of build from there. Wh happens with those problems that algorithms encounter d how do we get there . Is a nearly 100 yearlong series of statistical orhat in the 1920s were called actuarial methods and parole. This attempt to create a kind of science of probation and parole. And that as i say started in the 20s and 30s but really tookff with the rise of persal computers may in the 90s. And today its implemented almost every jusdiction in the us, and there has been an increasing scrutiny thcome along with that. And itseen interesting watching the public discour on some of these tools. So f example, the New York Times was writing, the New York TimesEditorial Board was writing through out 2015 the letters saying its te for new york state to join the 21 century. We need objectives that are evidencebased, anwe cant just be relying on the whims of folks in robes behd the bench. We nd to bring science. Shortly, that position changes. And i just months later the end of 2015, the new yo times writes articles saying algorithms are putting people in jail. Algorithms have succded racial bias, we need to throw the brakes and calling out my name this particular tool which is called compass which is one of the most widely used tools. Throughout the united states. And the question has this really ignit an entire subfie within statistics around the qution of what does it mean to say that the tool is fair. This system is designed to make predictions about whethersoone will reoffend. Its if theyre released on probation or if their pending trial in a pretrial. What does it mean to get ese concepts into law. Things like disparate treatment for equal opportunity etc. Working keep a minimum protections etc. What does it mean to turn ose. And how do we look at tool like ts and say whether we feel comfortable actually to the point. It was interesting when you get a handle on how a black suspect in the white suspect for similar crimes, similar backgrounds and much more likely. [inaudible] including one what suspect. [inaudible] what were, so what goes into the baking of that cake that built these biases because biases were notintentionally into it. But there still hard baked in. Yes. This is a very big conversation. I think one place to startis to look at the data that goes into the systems. So one of ththings that these systems are trying to do is predict one of three things. Lets just think about the pretrial case for now. A tool like compass is producing three different things. One is your likelihood to not make your court appointment. The second is to commit a nonViolent Crime while your pending trial athe third is is there a Violent Crime pending trial . The question is where does the data me from . And if you look at Something Like care in court, the court knows about it by defition. So thats ing to be fairly unbiased, regardless of who you are. If you look at somhing like nonViolent Crime , it the case that foexample if you pull yog white men and young black men about their purported marijuanusage they self report that they use marijuana at the same rate andet if you look at the arrest data a black person is 15 times or kely to be arrested for using marijuana that a white person is. In manhattan, and in other jurisdictis it might be eight times aslikely in iowa. Varies from place to place. Thats the case where its really important to remember that the model to be able to predict crime but whats actually predicting is rearrests so rearrests this nd of in person and systematically so proxy for what we really care about witches crime. Its ironic to me because part of the project of reseching the systems i went back into the historical literature, when i first startegetting views which is inillinois in 1930 at the time, a l of the objections were coming from the conservatives, from the political right. And ironically mang the same argument being made now from the other side so conservatives in the late 30s were saying it a minute, if a bad guy is able to evade arrest and the system doesnt knowthat he committed a crime , hes innocent and though recommend his release and recommend other people like m. Now we the argument is ing made from the left whh is to say someone is wrongfuy arrested and convicted , they dont have any concted indexing data and will recommend that attention of otherpeop like them. Its really the same argument , just amed in different ways but thats a vy real problem and we are starting to see groups like for example the partnership for ai which is a Industry Coalition with aumber of groups100 different stakeholders recommending that we dontake these protecons of nonviolent rearrests as seriously as we take predictions of nature. The second component i want to highlight here, its the second thing is worth covering is thisquestion o what do you do with the prediction once you have . So lets say youve got a higher than average that youre going to fail to make your court appointment. Thats a prediction. Thesecond question is what we do with that information. One thing we do that information is put a person in jail. While they wait for their trial. Now thats one answer and it turns out theres an emerging File Research that shows you send them a text message reminder, they are much more likelyto show up for their court appointment. And there are people proposing Solutions Like providing Daycare Services for their kids for providing this subsidizedtransportation to the court. So theres a whole separate question which is as much as is going on, as ch scrutiny is being directed at the actual algorithmic prediction, thers a much more systemic question is which iswhat do we dwith those predictions . And if youre a judge and the person is gog to failto reappear , ideally youd want to recommend some kind of text message for them as opposed to jail but that may not be available to you in that jurisdiction so you have to kind of go with what you have and thats a systemic problem, not necessarily algorithms per se and the algorithm sort of caught in the middle. Lets take it out on the crime and punishment area into the business area and you talk later in the book about hiring and amazon coming uwith this ai system to help it cold job applicants and what they were finding was that a lot them, the reason for this also were make into the way thsystem was being trained in the way the system was being used d also when you get to this, it also has the question, why were you trying to find people but tell us about that and how did they get into it . This is the story that involves amazon at the yea 2017 but by no means are they unique examples here. The example of amazon bu like Many Companies they were trying to design a little bit of the work load off of t human employers and if you have an open position and you start paying x number of residents coming in, ideally youd like some kind of algorithmic system to do some triage and say ese are the resumes that have rked or theyre lookinat more closely these were lower priorities. And somewhat skewed or ironic twists, amazon deced they wanted to weigh applicants on a scale of 1o 5 stars so they rate prospective employees the same way. But to do that, they were using a type of computational language model called word vectors and without getting too technical, for people who arfamiliar with the rise of Neural Networks, these Neural Network dels were very successful. Around 2012 all of them started to move to computational linguistics. And in particular there was this very remarkable faly of models that were able to sort of imagine words as these points in space. So you have documentwords based on the other words that were unified and 300 dimensional space. But ese models had a bunch of really cool other operties. You could actually do nd of arithmetic with words. King midas man plus one. D search for the point in spacewas arer to that. And get clean. You could do tokyo minus japan plus england and get london. And these were the same so these numecal representations of words that fe out of this Neural Network and it being useful for this drivingly vast array of tasks. And one of these was tell us try to figure out the quote unquote relevance of a given job. And so one way you can do it is just say here are all the resumes of people weve hired over theyears. Throw those into thiwork model and find all this point in space and thenor any new resume, see which of the words have the kind of positive attribus in which have negative attributes. Sods good enough but this team started looking at this and they found all sorts of bias. For example the word women was assigned a penalty so if you are going to do a Womens College or Womens Society of something, then the word women on a tv would get a negative deduction. So yes, getting a negative rating or whatever because its further away from the more susceptible words that its been trained to watch for. Doesnt appear on the typical resumes collected in the past. And its similar to other words also here. Of course the team, the red flag goes off and they say releases attribute from our model. They start noticing its also applying deductions like womens field hockey for example. Or Womens College. So they get rid of that. And then they start noticing that its picking up on all these very subtle choices that were more typical of resumes and females so for example the use of words executed and capture. I executed a strategy to capture market value, phrases that were more difficult. And at that point they basically gave up, they scrap the project entirely. And in the book i compare it to something that happened with the boston Symphony Orchestra in the 1950s where they were trying to make the orchestra which had been a little bit more equitable so this they decided to hold auditions behind a wooden screen. But whats someone found out only later was that as the audition or walked out onto the wooden floor, the horse they could identify whether it was a flat shoe or a highheeled shoe so it wasnt until the 70s when they officially requested people remove their shoes for entering the room at finally started to seegender balance. So the problem with these language models is that they basically always adhere to shoes, their detecting the word executed, captured and the team in this case gave up and said we dont feel comfortable using this technology. Whatever its going to do, its going to have some very subtle patterns and the engineering detects before, the gender balance is off there so its just going to sort of replicate that into the future which is not what we want soin this case , it just walked away but a very active area of research, how do you the bias language model. Youve got a point in space and you try to identify spaces within this giant thing which represents gender disparity and you leave those dimensions while serving the rest. It is area and its kind of ongoing tothis day. How much did azon spend developing that its a great question. Thre pretty tightlipped about it most of what we know comes from an article where, i wasnt able to da lot of followup but as i understand they actually not only gave the product but they dianded the team that made and distributed their engineers to other things. Im asking because i assume millions were put into that that could have hired an extra hr person. Another example i wanted to get to and might be coming from a different angle is you talk in the book about the fatalities that ppened because of the wathe car was recognizing this person. Again, explain that. This was the death of Elaine Herzberg in tempe arona in 2015, the first pedestrian killed by a self driving car. It was an r and d uber vehicle and the transptation safety review or came out and i go into some of th in the book and it was very illuminating. To read the kind official breakdown of everythinthat went wrong cause it was one ofhese things where probably six or seven separate things wentrong. Had only been but for that entire slate of things going wrong it might have been prevented. One of the things ppening was it was using sort of the neur network to apply object detection but it had never been given as an exame of a jaywalker. So in all of the Training Data that it had been trained on, peoplealking across the street were perfectly correlated with viewers and they were perfectly correlated with intersections. So, they dont how to classify stuff thatoes into more tha one category or classify how things are not in any category so this is again one of these active Research Problems that the field and making headway on only recently but i this particular case the woman was lking a bicycle and so this sets the object rognition system kind of into this uttering state where at first they thought she was a cyclist but she wasnt moving like a cyclist and then and thought it was too pedestrn but recognizes she had the bicycle and thought just its something object thats been blowing or rolling into the rd but no, i thinkts a person, no its a biker but due to a quirk in the way the system was built every time it changed its mind about what type of entity it was seen it would reset the motion prediction so its constantly protecting this is how a typical pedestrian would move or a cyclist would move et cetera and extrapolating as a result where i think they will be in a couple seconds from now. If that intersects a car that will do something but every time it changed its mind started re computing that prediction and so its never stabilized on a prediction. So, there were additional things here with overrides that the hoover team had made because in 2018 most cars already had this very rudimentary form of self driving like automatically break or swerve and to override that and at their own system in and those in weird ways but i think the object recognition thing itself is, for me, ver