Transcripts For CSPAN2 Weapons Of Math Destruction 20161016

Of fraction of what were told we should be spending. More than 30 years after the creation of the u. S. Department of education, students taxpayers and the country are not better off. But we can be. After decades of waving the constitutional barrier to a federal role in education, under the guides of partnering with state government, it is time to dissolve that partnership. And abolish u. S. Department of education once and for all. You can watch this and other programs onlines at booktv. Org. [inaudible] hello everyone. Thank you so much for coming out tonight my name is davis. Part of the policy honored to welcome you here hills out for a new book weapons of mass destruction. Really excited to get into this conversation, but i do have a few housekeeping notes. I want to let you know before we get startedded, if you will attack a moment of to silence your cell phone and no interruptions or otherwise. And also i want to mention that format of this event and youll have the opportunity to ask questions. And we have cspan recording the event tonight i was really appreciated if you raise your hand if you have a question and i can bring this to you so we can have all of the questions recorded and everyone can hear and get in in on the conversation thank you very much for that. Well be doing a book signing after the event is over and right here with where the white table cloth is there shell be there to sign a book you cant pay for it here but you can pay for it at the front in the bookstore so either head out to kick out and come back to the copy or get it signed here before you go out. I also want to make mention that this is a publish event great and developing over the past year and a half, were located here as well as if you at the calendar you can see where they are throughout the city. So you know, i first will learn what it was when i solve a rue rubiks cube but we live in an age of the all gore rhode island increasingly the decision that affect our lives, where we go to school get a car loan and pay for our Health Insurance are dismaid by humans but by mathematical moldses everybody does courting the same rule and bias is eliminated but kathy oneal reveals in urgent and necessary book, the opposite is true. The model used today are unregulated whefn theyre young. Most trouble aring they reinfluence the discrimination. So bank of america and land reviews oneal is io deem person to write this book. An academic who turned wall street and scientist, who has been involved in occupy wall street and recently started auditing company. He was one of the strongest book was speaking out for limiting ways to allow algorithm and against notion that because it is implemented by unemotional gene cannot have justice. Review said it is an unusually reasonable book adopting governing aspects of our life. So m. I. T. Department and the college switch over to private sector working for the hedge fund, and metrics of a Software Company he left in 2011 and started working as a native science in the new york startup and reviewing in 2013 and lost lead program and Data Journalism in 2014. Hes a weekly guest on slate money podcast and may know her for her blog oneal was with the conversation by doing pullback Computer Science and university of maryland so join me in welcoming kathy joe. [applause] so yeah Computer Science professor so algorithm leak out here for yall. [laughter] so maybe we can actually jump right in and get some like definition and ideas out there and also talk about one of the examples in your book because we were chatting in the back aside you always were coming in and saying one of the propghts ive done in my lab is to take data from twitter and facebook to use that to predict your personality trait. How many of you have done a i. N. T. Personality test . Businesses want you to take those and you dont need to because we have algorithms to look at what you leave behind and find them out automatically and use ways that you may or may not like. I wouldnt like that. [laughter] yeah so were going to talk about what a weapon of mass destruction is and then we thought might start with an example. Just then we can sort of like draw out the characteristics that make it a weapon of mass destruction and by that i mean like theres some all gore risms that i care about and most i dont care about. Ones that i care about i think are potentially really destructive i call them weapons of mass destruction but start with an example to make it real. So the example if you dont mind me starting there so theres this guy named kyle beam he was a College Student in atlanta and he wanted to get a parttime job at clovers Grocery Store and he had a friend who worked there and his friend it was leaving so he was like if i talk to my manager. Youve got any job. You have to fill out the paperwork online. And so he started to fill out paperwork and 50 of the job applicants in this country required to take a Personality Test. Online before he got the interview. So if he failed this Personality Test which he did then he wouldnt get to the interview process and this was a standard procedure for lots of mnl minimum wage jobs especially and luckily people dont find out that they failed it but they never get called out but his friend told him that he got red lighted thats what it is called and then unusual in a second way which was that his father is a lawyer. And you know most applying to minimum wage doesnt have a father who is a lawyer and father asked him what kind of questions were there on this test because youre very, very qualified to get a job at the grocery bagger at a Grocery Store straight as in high school. Whats the problem . And he said welled dad the questions were a lot like the ones i got at the hospital when i was being treated for bipolar a Mental Health assessment. Do you know about it . It is actually what we could predict. So it looks at are you an introvert and that measures how emotionally stable are you. So if you get anxious or anger easily a high score. I do. Or if youre supermellow and nothing both rs you then you score low on that. And easy to find that out through a test but those signals also show up in all kind of other data we leave behind too. Oh, my god thats so frightening. So what happened was he talked to his father so glad to meet you and glad that you dont have to take this test to get this about someone. But what happens is his father was like thats illegal. You can you cant make them take a health exam including a Mental Health exam. And thats the american disability act thats trying to prevent this sort of creation of an underclass of people with disabilities and Mental Health problems. So his father is suing on behalf a lawsuit on anyone who ever took this test and add by the way that kite kyle went ahead to get a job at six other Large Companies in the atlanta area and got the same Personality Test and red lighted for all of them and prevented from getting a job in his in his area. So thats the first example i want to talk about. Do you want to add anything . See if that takes us towards this definition question because this you were talking about a test. But, in fact, a lot of these kinds of insights about people come from these moldses that are built on algorithm and bunch of data and they work opaquely and the fact that they can hammer people or have bias even though on the surface it looks like they dont have bias that is from the nature of your weapon of mass destruction concept. And apply. Go ahead and define weapon of mass disruption with three Properties First is that it is widespread and impactful. So it matters like algorithms the there like i built in any basement because im a Data Scientist i build them all of the time. Nobody cares about ones that dont listen. They start caring they should start caring when it aivetses a lot of people in important ways because it was widely used. People who built this all gore rhode island located in bos son and sold it to people, companies to use instead of their hr people so widespread and impactful. Second is that it is secret. So kite kyle did not understand how he was being scored and people who took this Personality Test is is not understanding that they were being scored. They didnt know this was a hoop they have to jump through and finally it is destructive. So it destroyed kyles chances of getting a job. But it is actually descriskt in a largeer sense characteristic of weapon of mass destruction is and terribling gore risms to create and reenforce a feedback loop which is destructive to the society as a whole. Here it is destructive larger feedback loop is like systemically refusing to influence people with certain disorders. I think you one of the examples of like both like most in the book is when you open with which was a local one. And it highlight it is the really secretive black box nature of the algorithms including if by the way, the Computer Scientist really crappy ways of validating them a and that was their use in the Dc School System for getting teachers fire sod talk a little about nap second example so michelle was school is chancellor for some time and she instituted this policy where by some people would get fired if they had bad teacher assessments and some people would get bonuses they got really good teacher assessments now assessments are complicated but short version is that most of the ways that a teacher are assessed is like not very they dont have a lot of spread so most people get acceptable or very good. And those people who really want to discriminate between bad and good are frustrated with that. They want more spread in their scoring system. They want like more people who are terrible and then eminent, goods, fair, better and very good. So theyre frustrated by the fact that have few of the ways that we could we now aseis teachers have any kind of spread which thng of information. So they instituted this this assessment this new kind of assessment called growth score or the value added score of a teacher web and i wont go into it too technically but the very broadway of just thinking about this is the idea is that the teacher the teacher is on the hook for the difference between what their students should have gotten versus what they actually got. So theres an underline model that estimates what each student in a class should get so if youre all in my class like im teaching you fifth grades youre all fifth graders right now. Each of you got at the end of fourth grade you each got a score for your standardized test right,let say you got a 75 out of 100. Then you should be and expected to got a 75 for a fifth grade test. Now, lets say you actually got an 80 then i would be given credit for five points that you got more than you were expected. Okay, does that make sense now as to not declare how expected score is actually determined per student. And it is really complicated and its a source of uncertainty. And its also another uncertainty what you actually got at the end of fifth grade or what youll get at the end of fifth grade because afterall like tests vary every year, and kids get different scores depending on morning or afternoon or whether they have breakfast before the test so that kind of thing. So theres leak two uncertainty and the teacher again, remind you teacher is on the book with a difference between two things. Okay. Now, actually if you think about this this original expected score that each of you got, the difference between the expected score and the actual score is called error term, specific, also sometimes called noise. So the teachers are being assessed based on these error terms which is if youre not a dont be confused we in statistics call this stuff essentially pretty meaningless and, in fact, this way, the scores actually came out with almost meaningless. Pnt so anyway going back to washington, d. C. , i interviewed this woman who got fired because her assessments were too low. Again it wasnt entirely due to the valued at its corporate it was 50 of the but the other 50 with these things that dont have any spread. Most of the information her score was this terrible value added score. Okay, one more thing to tell the about this, her score is that a lot of her, she was a fifth grade teacher. A lot of of fourth graders have gotten good scores at the end of fourth grade. A lot of the kids come into a class had gotten actual fourth grade scores but they couldnt read and they couldnt write. And so she was suspicious of there really very good scores. And, in fact, she is every reason to believe that the teachers of those kids cheated on the test at the end of the year in order to get better value added model scores for themselves. Because then their kids got better than expected so they can good scores. So they set her up to get worse than expected. Does that make sense of . She has good reason to believe this. All sorts of unusual erasures. I dont know if you remove that the scandal that industry but unusual erasures which never lick a systematically investigated. They got hushed basically. In any case she got fired. Lets go back over the three characteristics of the weapons of mass destruction to its widespread. The scores are being used in more than half the states mostly in urban school districts. Its secret. Sarah didnt understand her scores and she was suspicious of it but when she appealed that they said you know, no, its there because its mathematical. And finally destructive. Not only did it destroy sarahs life because she got fired. Like, i should say she got rehired like the following week and sort of affluent suburb where they dont use the scoring system. Thats going to the final think which was destructive. I think youll be surprised if you know about Michelle Rhee and why she was brought income which is theres a nationwide kind of are on teachers going on which is fine but that teachers, get rid of them and then we will fix education. Its been like a bunch of president s who want to be the president that fixed education. We can talk about whether that even make sense but in any case the idea was to get rid of the bad teachers. In a Larger Destructive feedback loop engendered by the valued added model is it hasnt gotten rid of bad teachers. Its been getting rid of good teachers because good teachers have been quitting, retiring early and most of all they been moving to affluent suburbs that dont have this scoring system. Right now we have a nationwide shortage of teachers. I would argue this regime of this ill take value added model has been very, very bad. At i think something that is especially in cities about this, we talk about algorithms and they can sound tacky and four, things we dont interact with, but another the algorithms that were talking about that we both work on have tons of some letters, sometimes are identical under date the algorithms like the netflix that recommend shows for you or amazon that recommend books to buy. We totally know how to handle those. Amazon or most of the time it superb or and not insightful. You bought a stephen king book and how about these other stephen king books . Youre right but not helpful right . Sometimes its totally wrong and you just go like where did this come from . Sometimes you get these beautiful glimmers of insight. So bad bought this Survival Guide and recommend i buy this 18inch knife that folds out at the handle. I never wouldve thought of that. That was a great insight. You dont just by having amazon recommends. You dont watch every show netflix tells you to watch but the algorithms you are talking about, people dont treat him as one area in a human decisionmaking process. They treat them like this oracle of truth even when theyre poorly validated, even when the people who use them to understand. That makes them more destructive because you override all of our great human intuition with this little bit of math. Gradepoint. I would argue this worse than what you just said. You know how, let me give you two examples. You know how the facebook drinking stories has been messing up lately, weird fake stories . Guess what. We noticed that ever give feedback to facebook site thats not a real story. Right . That something that doesnt happen to the teacher value added model. There was no ground truth for those teachers. There was no secondary assessment of those teachers to compare their scores to. They were told this is your score. The guy who got six out of 100 was shamed. An excuse to get 96 out of 100 he was like what, that means i went from being a terrible teacher to a great teacher . Theres no feedback mechanism. Teachers cant say actually im a really good teacher so please update your model. That would be the equivalent of like saying netflix, i do want to see that movie because its awful. Please dont show that to me. They need a rock button. Please dont recommend that. Teachers do not have access to that way. So parent or administration knew they were good, they could go in and override. Exactly. The thing i want to agree with is that people really did just trust the scores because they were mathematical. The way i came about, across this algorithm is my friend who is a principal in brooklyn under high school in new york city, and she started complaining to me about her teachers getting scores. I just said can you tell me how theyre being escorted . What is the algorithm . She said i asked a contract at the apartment of education and they told me, you wouldnt understand. Its math. And i was like, thats not good enough. As a mathematician let me just say asthmatics as both to clarify, not confuse. Thats not a thing to do with math. Thats recognizing math, right . Its not okay. I asked her to keep pushing and she got through three different layers, each later sang to her you wouldnt understand it, its math. Three times. She finally got this white paper which was unreadable to me. I cannot understand what this white paper sing. What i did was i filed for a freedom of information act request to get the source code for the algorithm. It was denied. And by the way, i should say that the reason i thought of that is because the New York Post had filed a freedom of information act and request and gotten, successfully gotten all the teachers at scores and teachers names and scores and they published it as an act of public shaming for the teachers. Really awful. I was like if they can get the scores than i should be able to get the way the scores are made. Makes sense. I was denied. That i contacted some at the institute, the Data Institute that build the score three friend of a friend come and explain to me i would never get that source code because they had a contract with the city of new york was too good now in new york city would get that source code, whatever that this stuff was made. It was secret from the. Which is to say secret from the officials in the department of education, so they cant explain to the teachers how they are being scored. Part of this book really, the book as a whole as a manifesto about how math is being weaponize. I thought maybe you could give us an overview of how you came to the place. You were building these for finance for a while, and then you kind of became the enemy of finance. My words, not yours. But join the occupy wall street is probably not something youre hedge funds wouldve smiled upon, right . Definitely. So quick, many years of your life. So i joined occupied in 2000. Gov. I joined my hedge fund in the early 2007 is, and like right headed straight into the crisis. Walked in and then boom, crisis. I was like really the solution very quickly because the people that i thought were experts really didnt seem to know much about what was going on, didnt really understand the market which is to not say i understood a perfect but they didnt seem to understand that much better than i did. At the heart of it, and by the way, the person who introduced us talked about rubics cubes. Thats why i fell in love with math is rubics cube. How beautiful are rubics cubes . This idea, mathematics thinkers pure, clean, beautiful, almost artistic endeavor. And then back to the financial crisis, at the heart of the financial crisis was this mathematical life, the opposite of the beauty of a rubiks true. The aaa rating for mortgagebacked securities, which given he when i think of them as mathematics but they are. They were thomas is that people who were really good at math which ph. Ds, or at the back crunching the numbers and promising that these mortgages were not going to go back. They were not doing that the it which is basically creating alive and selling it for money the trip theyll the aaa ratings, the scale of the market, mortgagebacked seekers was very, very large, one of the reasons it was such a big deal. Thats one of the things i realized was that it was a weaponization of Peoples Trust in mathematics. People trusted math. They trusted mathematicians, and like math itself wasnt the problem. It was the people were basically corrupt and they were shooting themselves, shielding their corruption from peoples prying eyes by calling them out and sang dont look here, you wouldnt understand this, its math. And its right. You have to trust us. Thats kind of the big thing that i realized, this isnt the right way to deal with math. Then i left finance it became a data site is and then i was like, doing almost the same thing as there was a finance. Instead predicting market im predicting people. Thats what they decide to do. They use Historical Data to build algorithms to predict protect people. You are predicting personnel does. That didnt seem so bad, but until i came across sort of the teacher value added model antigen, which i was like thats not good. And then i think the moment that i decided to quit my job and write this book was caused by this one interaction i had in my startupstart up with his Venture Capitalist came to visit. I was thinking, he was thinking of investing in our company. We all sat and listened to them talk about the future of tailored advertising which is what i was working on. I was working on Advertising Industries et al. , like on expedia. I that those relatively benign. You offer some people like a hotel room, some people you dont offer the motel rooms. You are giving opportunities to some people and not giving opportunities to others. A very kind of mild way of segregating society. Does that make sense . You guys deserve these algorithms, you guys dont deserve these algorithms. I dont think it was particularly evil but then this guy gave us this idea of what is future dream for the world of the internet was. He was like i had this dream that like someday it didnt sound like Martin Luther king. Let me take that big. He said i had this idea, like this is what i hope to see in tailored advertising. He said hope to see something that i get offered trips to aruba and jet skis, and to never again have to see another university of phoenix add. Because those are not for people like me. And everyone like around the laughter. And i was like, what . Want . I thought that would ever happen to the democratizing force of the internet . Like this is the goal of the people constructing the modern internet, which is to segregate people and silos in my class so that we, the technologists, the people who are lucky who are getting scored well, score high, with opportunities. We have like and things would like to play with competent people on the other side of the spectrum can be preyed upon. I make money off you from your vacations and i make money off them from exploiting universities that give worthless degrees, right . Rick and i make money off of them . If any cercla i can make money off them by exploiting the. Right. If you think about the way tailored advertising works theres an auction for the eyes of whomever youre talking about. Its a very efficient because its all source of demographics. And again, if you talk to someone like me, they are very good at finding and joining yarn adds because unlike reportable deal alpaca. They know my number, right . Thats okay because like i love jarred and im like i want that yarn. But like for some people, the most profitable thing is to get the person to take out a federal loan which goes straight to a forprofit college, saddles them with the debt and does that give them an actual education. We have seen some forprofit colleges been posted recently, which is great news but it was an enormous, still is a relatively large industry, but that type of the time Venture Capital dr. Mcnutt but i had never seen a universe of phoenix add. I did note what it was also. It was in 2012. I looked up and i saw that apollo group, the Parent Company of universe of phoenix, was the single biggest google admired that quarter. Thats when i realized like im not going to see the failure of this kind of algorithm. Everyone here saw the failure of the aaa ratings on mortgages. Everyone in the world saw that because it was so loud when it exploded. But who is going to see the failure of this kind of algorithm . Because we segregate ourselves online to such an extent that with so many passionate when someone falls prey to that false trap, we do not see. In fact, we blame them for it. How are we doing on time to . I can keep going. Great. Where do i jump in . Ive got like 10 questions im going to squish them all into one. One of the things you talk about in your book is like potentially we have people are building these algorithms, take something equivalent to a hippocratic oath. Certainly does to ethics training for people like us. It makes people accountable in the Computer Science department to talk about ethics i think. Theres training is a good part, but i think the question i came away with is, if theres money to be made by exploiting people, it seems like thats going to happen result with the mortgagebacked securities we saw with enron. Like back in 2001 pc folk also enron traders, if you remember back then, shut down the power grid in california just to make money on energy futures. Theyre talking about the old people who are getting screwed basically because they have to pay these enormous rates for the electricity. It seems like the market is set up where if we can use these algorithms, if we can use anything that is vaguely within the rules to exploit people to make money, we will do it. And i wonder what your thoughts are on that would change that part of the system. So im not expecting all corruption to end, right . Period, right . But if you want one particular form of corruption to be challenged, and that is the corruption of where people claim that because of an algorithm it is beyond morals. Every algorithm has invented values in it. Embedded. The people who create the algorithms, who own them are the one to get to decide what those morals are. We need to start challenging that. I have a very specific reason to define the algorithms like i do, the weapons of math destruction. I perform triage. This is the service i done for the world as i perform triage on all those algorithms. I told him where to focus our attention, and for those algorithms, many of them are accidents, accidentally terrible algorithms. Some of them are just terrible because people who are greedy will always want to do that. So i do think that for the algorithms that rise to the level of may be good down to the level of something that is potential weapon of math destruction we need a more scrutiny. In the to be laws about the. A lot of the stuff as you mentioned, there will be no free Market Solution to this. The free market will not solve this problem. It actually makes money. Its profitable to be discriminatory. You cant ask someone please stop being discriminatory because of the Great Society even though you make more money doing it. Thats just not going to happen. Thats why we have laws about things like fair lending. We have the fair credit reporting act. We have equal credit opportunity act. And Discrimination Laws that were written in the 1970s that essentially regulated things like fico scores come and theyre not perfect but there are a hell of a lot better than we have for whats going on in big data. I am basically arguing not for one, the public, stop trusting algorithms. Pushback, demand accountability. We give secrecy you know accountability. So especially when you are being assessed after job you should be able to know how that is happening. My calendar the ss exactly . You should be able to ask that question. Other also asked the rules and regulations around things that rise to the level of widespread and impactful. I think just to close this out. If what you declared in the book which is one ive been thinking a lot about in another domain is if you have an algorithm that seems unfair was implications work differently, and thats because of the human basically right . If you have an algorithm thats discriminating and it doesnt consider race but it does consider zip code, thats because whoever programmed it didnt think using zip code, didnt consider the fact discriminate based on race. If you have a chat bot that started doing holocaust denials and nazi stuff commits because whoever programmed it didnt think that maybe we need to think about minorities and women because they get harassed online all the time. If you have an industry thats made up of people who are greedy or who are ignorant of the problems of people not in my class, they can be a good other sorts of problems are algorithms create anything this accountability should you grace also gets to us as developers that we do think that these problems, we need to know about it and not just about optimizing for profit or the output. Thats well said. I want to add one last thing. Theres been recent publicized examples of spectacular failures of algorithms coming from like this twitter bot and the facebook algorithm stuff. Thats what wiki to see, like most of the bad algorithms that are ruining peoples lives are never seen by the public. They are businesstobusiness dealings, like the one that was the personal the test for kroger supermarket. If we think things can do things that are known to be under public scrutiny are that bad just imagine whats was going on when there is the public that can scrutinize these things. I think youre taking my mic now, right . Thank you. [applause] we will start in the back. Thank you very much. This is incredibly interesting. Talking about the algorithm as a device for controlling sort of, for establishing or maintaining a control system, and algorithms, this device that none of us would understands presented is valid. It reminds me of other things we have in society that accomplished the same function like perhaps constitutional law or things that no one really understands that they are legitimizing a structure that probably ought to be getting to the bottom of. These other things how usually like a priestly class thats where the only people who understand it. You guys come from this in algorithms. What is this class and what are they thinking about this and are you a renegade from this class or is everybody thinking about this . Whats the deal . Because i dont know of any of these people, really. Thank you. Yes, it is a priestly class and much to my shame, a lot of the people who are in this class with me refer it that way. Its like actually pretty flattered to be told you are magical because you are a mathematician, right . Not enough of us are worried about her impact on the larger society, but i do want to say when i started this project for years ago, i quit my job to write this book, i was really panicking because i didnt see Anyone Around me who was worried about this. But now four years later i have an entire community of people, including Computer Scientists sociologists, anthropologists, technologists, that are all super concerned with the stuff. In particular ive been hanging out with some people who, like me, are interested in developing tools and auditing tools specifically to, like audit the black box, look in the black box in some sense. I do want to make that mysterious. What i mean by that is, in a social logical experiment we are borrowing from sociology. Light, if sociologists want to see whether hiring practice was racist they would send, like a bunch of applications with similar qualifications and have like black names and white names and see if like the white names get more callback for interview. You can do that with an algorithm, too. You can just do similar things with the algorithm. Its crude and its the first generation of funding tool for algorithms is the type of thing we need to start doing. Long story long, we do have work to do, and our priestly class of technologists who work with the data, but we do have a growing group of people who want to think about this. And by the way, also like a field that should be developed right . Like i feel in 20 years there will be conferences about this kind of thing. Right now theres nothing. As far as i know theres not even a journal thats interested in polishing things like this, but there will be. And by the way, one last thing i would say that the beginning of this field has already taken root in Data Journalism. If you look at like propublica, the recent work with auditing, we talk about recidivism models but these models that judges use to send people to jail, you know, viewed as racist as one of the reporting teams at propublica after that of data using foia request and audited and found it to be racist. Theres work being done by this in the early stages. Thank you. This is wonderful. Im a physician, a little bit late, but i wanted to get to areas in which its might have something to do with medicine. Theres tremendous confidence in the medical community could you speak up . Im sorry. Im a physician. Big day comes up in two ways. The first is that the medical community and the Business Community are convinced that all the data going into Electronic Medical records is going to give us brandnew insight into taking care of people and health. I wonder, its not your field but i wonder what you think of this . Im skeptical. The second thing is, another big push in medicine is to force individual positions to take risks for the patients that they take care of. That is to say, if a patient doesnt require so much care, you do well. If the patient requires more care, than you anticipated, then you are screwed. We are not insurance companies. I wonder if there are algorithms that will be developed so that physicians and insurance companies, even though everything is supposed to be on the up and up and we dont do preexisting conditions, they will use information that is publicly available of recently publicly available to figure out who is at risk and who is not . Great question. Im not an expert in your field, as you said, but i do want to say that i worry as well as you did about all the hype around how much data can be in the medical field. So i would focus as a nondata person, you know, and i would focused even though i am a data person, if i were you, on accountability. And explicable testing procedures for this type of thing. That may explain by analogy. The problem with the teacher stuff, i said it before, i will say it again. Theres no ground troops. But compared to what . How do we know this is any good . Like, how correlated visit to the thing we know is good . It so bad, right . Really good it agrees with you in the wake of what you know is true and you Start Talking in a way you are not sure of yourself. That example testing there is an example i wanted to throw out there. And algorithm, imagine we had good algorithms for that we could predict future illnesses, that is not necessarily a good thing and that is what people get wrong when they talk about Precision Medicine or all this stuff. In the hands of your doctor trying to help you stay well it was done well. But in the hands of an Insurance Company who can charge you more for future illnesses that is a terrible thing. Something not addressed by obamacare, it is not future conditions. Imagine in the hands of walmart, not saying it is but a Large Employer who gets to choose who to hire or not hire based on future insurance, that would be really bad. There is no guarantee even if the algorithm is accurate that it is used for good. I am kind of in this space building those algorithms and they terrify me, as we were talking before we published a paper where we are looking at people who are starting treatment in aa, the first Alcoholics Anonymous meeting, we analyze twitter profiles from before they start going to aa and 85 or 90 accuracy they stay sober, they are working on that. The twitter accounts over the course of the pregnancy and the day they gave birth predictions about 85 accuracy or postpartum depression great for your doctor this you go to your ob gyn and she can push a button that monitors you for this and all the cases that is really scary and there is work in this community looking at things like diabetes obesity, heart disease, we are getting very good, the science is really interesting that we can do this. On the other hand the implications, the fact that there is no regulation to these algorithms, no one can even know if you are doing it is terrifying. I spend a lot of time talking to Big Companies saying you need to be careful about this or bad things will happen. It is a scary space at this point being down there building and looking at these algorithms. I have a very dystopian view. We are all terrified, right . I didnt even know you could do that with twitter, that is awful. Talking about the medical profession reminds me of hit the, so in relation to vocational testing why isnt that as enhanced as worthy of being protected by this law as medical information that is protected. Thank you for saying that, that is one of the things i called for in my book. But even better if you had read the book. I thought the perfect algorithm in the hands of the audience gathering around. I am a reporter with higher education, and the potential benefits in the cases you cited. For instance with the teacher example, it seems it was just the use of the information or wrong analysis but it seemed it helps identify an instance of cheating. I wonder what you are it is the way they are used. What you say to businesses in the second example using Personality Tests, for them it is a matter a matter of saving time at the end of the hiring process. I am sure you identified what the response is to that. The only reason it was cheating in the first place was the structure of the model itself had too many incentives. When you have enough incentives in place and highstakes, it causes cheating. I dont it is circular reasoning on this regime detecting cheating. I am not an education expert, i dont have a solution for education. I want to point out the lowest bar of all my examples because it is a bad model. I will tell you my evidence for this besides some people didnt even teach. And i told you guys at the New York Post, they were Just High School math teachers in new york city did something really clever. I couldnt look at this model. He found teachers that maybe taught seventh grade math and eighth grade math, supposed to be an overall score for a teacher that would be consistent, and eighth grade math and things like that. When he found uniform distribution, practically as likely to get 90 as 4040 all over the map. Almost a random number generator. So there is no reason to think this is informational. 24 correlation, to say you couldnt possibly take a given teacher and say this is your fault, you cant do that, not Strong Enough evidence but a 0 correlation at the district level, useful information but i want to throw in, useful information about test scores, test scores are probably not enough information. Not and education expert, i dont think so. In terms of the hiring practices, the Personality Test it was illegal and secret, we give it evidence eventually that it is a Mental Health assessment which we think it is. We have evidence for that. Assuming it is true you cant figure out directly but assuming it is true the problem is it is against the law, you cant discriminate when you hire. I dont have a problem with using big data to help you filter a resume but they have to be legal. At the very least. They should be transparent as well. A very challenging picture and frightening picture of these algorithms, the average Facebook User has no idea what an algorithm is. Those people most perfected by the failure of the housing crisis dont know any algorithm had anything to do with it at all. How do you get this information out . That is important because people have no idea of the impact of algorithms on their life. And most people when the survey was done didnt realize facebook had an algorithm. They are not aware of the algorithm. The discussion around the trending news stories, they are not seeing everything. And some populations have tricky things where they figure out how to get their friends to see their posts. If you put something your friends might not see it. The teenage population, its branding into the post like coke and pushed up into the ranking. That is a longterm public all so make people aware of the algorithm. One interesting point i spend a lot of time looking at is you look at things like this pew study, a great representative study, awareness of the algorithm but also concern about the impact clearly correlated to socioeconomic status and race so people who are poor are less worried about these algorithms and less aware of them. Talking about being exploited by these algorithms people who are most likely to be exploited, i wont be exploited that much but people likely to be exploited or less aware, there is nothing bad going to happen. And interesting things for the data. Your book is a great way to get it out there. And the massive education, so powerful, and a literacy we need to start developing to at least know they are there even if we dont know the details of how they work. It is not enough to say everyone should be aware and everyone should protect themselves because what jen just said, the reason i wrote this book was i saw this time and time again as a class issue. These are tools of social control, people at the top are controlling people at the bottom you should have been aware we need rules about this, it is not good enough. Close to that. I am a big fan of your blog. I work in healthcare, work in healthcare payment, using this algorithm to determine what you were talking about. It is okay to be using these algorithms for hiring practices. And the solution in terms of algorithms themselves, dealing with what rises. And the Data Scientists, and inequitable welfare and affect. The thing about promise of data, the promise of big data is we can make things more fair. I am not holding my breath but i think we can do that. Part of it is the realization that we are not done, part is developing tools to audit and see if it is currently fair and how do we fix the algorithm, how do we make it fair . The good news, lets compare a company that has discriminatory practices with an algorithm with discriminatory practices for hiring was the problem with the companys if you interview people and ask them how they choose to hire, or maybe they dont realize they are being discriminatory. Whereas an algorithm will not lie. And that is good. But you have to have sufficient trust in the out rhythm to be able to do a good job. Can you hear me . My partner and i did a fair amount of work earlier this year we experienced exasperation and frustration at the fact that Bernies Campaign and other democratic campaigns use data from a Company Called them and what we are finding was the data was sending us in terms of canvassing to a lot of neighborhoods that dont match up with the demographics so on our end it was frustrating the data didnt seem to be working that hire as didnt see just because there was data in a certain demographic didnt mean that was the best demographic target but beyond that on a broader scale i am a bit concerned about the fact that it seemed to be contacting a certain type of motor and not a different type of motor going to wealthy neighborhoods and not a lot of less affluent neighborhood so people are reminded to vote and others arent. Have you looked at all the advantages . To look at them. In terms the algorithm, the existence of it is not the best thing for society, dont think there should be a private company that owns the data on voters and links it out to campaigns, what can we do about that . Where is the power struggle . I have a chapter on politics the subtitle of my book is how big data threatens democracy. This is the second part. Just having exacerbated any quality, a lot of people have confusing and badly scheduled lives they dont have time to be engaged but the threat to democracy comes from the data, targeting stuff in the way it is uneven. You are absolutely right the targeting homosexual people and people who are either people who donate a lot of money so they have bigger voices or people in swing states who are swing voters, exacerbates feedback loops where they spend money on people who definitely spend money on people who may vote. Having said that, if somebody is supposed to vote doesnt vote that is corrected and they spend money on that person next time around but somebody who doesnt vote doesnt vote they are continually ignored. A socioeconomic in terms of voting. The larger issue is this, what is efficient for campaigns is an efficient for democracy. What is efficient for campaigns is everybody here profiling you perfectly and says the campaign once you to care. That is not what is good for us as a group. What is good for us as a group is public discussion on various issues but that is not what is happening. If rand paul targeted me he would find things i agree with him on which is breaking up the big banks. If i went to his webpage to see what he things of other stuff, he would follow me with a cookie and say lets show her that. They are controlling the information i have about them. That is not democratic. We want information about candidates to be utterly open and have more information about them and what is happening. Understanding who people vote for, we are pretty good at it in the last election we had 97 accuracy but here is the scary part. We remember in the last election something you saw, here are your friends who have voted. And Something Like 1 2 person more people to vote if they showed that then if they didnt. They want to get find other people they think will vote for donald trump and show them the things that get people to vote but dont show the people who will vote for somebody else. Suddenly they are having a Significant Impact on the election itself. That seems relatively apolitical. You are giving me a skeptical look but calling facebook out specifically, the fact is the companies that control what you see with these algorithms have the option to have real impacts on elections by understanding, who is going to make you go vote and hiding that from people evidence of that happening, is a real possibility with the technology we have today and one of the most conservative things in this political space. Only reason i look skeptical is the population of people on facebook is already not bipartisan. If they showed everyone it would be fair. It shows everyone would have more impact on the democratic election. Time for one more question. I looked for an Online Advertisement on political websites, on the course of the last year a lot of major advertisings running ads as the american political life increasingly radicalize is. Have noble intentions and half being able naacp, we dont sell tshirts to people who what is happening is whenever we pull out of the market, in this particular case, online advertisers sell weapons and videos about how Hillary Clinton has a video, and ideological segregation, advertisers pulling out of rightwing websites the feedback loop steadily increasing and are getting really biased, google cars go to really big things. This is something that is done the great remarks you know of, hundreds of thousands discovered through algorithms and account managers, advertising rates. I wonder if you can comment on that. A larger event in my book. I dont know how to address it, the growing partisan divide in our country is not due to algorithms. I agree the way we live in on facebook and talk to each other is part of it but not the only part. I dont know how to address the problem. Thank you so much for coming out. And thank everyone else for coming out, a great talk and great conversation. We will have books available for purchases in the bookstore and finding afterwards. [applause] [inaudible conversations] be change here is a look at the most borrowed books this week from nashville public library. Topping the list the lifechanging magic of tidying up. Our look at this weeks most borrowed books at the National Public library continues with Anderson Cooper and his mothers dual memoir the rainbow comes and goes followed by dave ramseys the total money makeover where he provides tips how to manage money and alleviate debt. And Elizabeth Gilberts latest, big magic. The list concludes with john lewiss remembrance of the civil rights movement, march, the first installment of a 3part graphic novel series. That is a look at the most recently borrowed book from the National Public library. Tune in for coverage of the southern festival of books this weekend. This is an unprecedented level all those moments, in 1999 speaking specifically to the idea of the explosion of hiphop and the global export of that and what does it mean for those particular images, not necessarily these images that are being projected to the world, how are we wrestling with the truth of the artist expression and also recognizing how it is born of a form of oppression and stereotype and flattening of identities how can we reconcile, and respect one abstruse and to know that it is influenced by systems you may not be a part of. And the necessity of blackness is to the american identity, in order for the United States to operate the way it does, it needs a bottom, system of hierarchies and capitalism is that, the needs and explicate of class and blackness provides that bottom class and consistent, always have that bottom class that blackness is always there and we create demonize we have latinos, muslims, and it different points, ethnic groups, we are part of that italians, irish, german all exploited classes of people, but there escape was the fact that whiteness needed to reproduce itself to form strongholds so that they could be embraced in that fold to assure that blackness is always at the bottom. There is a need to flatten those identities and demonize, to make invisible blackness the humanity of black people to the minstrel shows, postcards that depict black men being lynched, watermelon and exaggerated features, calling us lazy and all these Different Things are the work that the humanity of black people, need to be seen in terms of housin