Analysis, Artificial Intelligence and threat forecasting. This is part of an intelligence and National Security conference, cohosted by the cia and George Washington university. Thank you. If i could ask everyone to please find their seats. So i get the privilege of actually moderating this next panel. And i can tell you based on the conversations weve had already, youre in for a real treat. These are some really big brains tackling some of the really big issues the Intelligence Community faces. While were looking at, our focus is on forecasting conflict and instability. I think youll see this will expand in many other ways to look at the role that technology plays for the red. The bad guys and for the blue, the good guys, and how we can try to get out in front of some of these issues. Awesome panel. Starting with dr. Stay ecy dixo. Thank you, stacy. Andrew holman, who is Deputy Director for Digital Innovation at cia. So all Things Digital and a very cool reorg within the agency. Rob high, who is Vice President and chief Technology Officer at ibm watson, thank you, rob. And last but certainly not least, fran moore, who is president of fpm consulting and a former cia director of analysis. So weve got an Amazing Group hire. And just to very briefly set the stage. Today is the 60th anniversary of the launch of sputnik, which clearly triggered a space race. Recently, the head of the, head of state of the same country opined that whoever owns Artificial Intelligence owns the world. In essence. This is also on the heels of china launching a Quantum Entanglement satellite capability. So it seems like, to paraphrase samuel clemons, where his, the future may not repeat itself, it does tend to rhyme. I think were seeing some of the rhyming now. But im really optimistic that the good guys are getting their arms around this. So lets sort of peel back the hype from the reality. What should we be thinking now, where is it going, and im not going to ask you to answer whether Vladimir Putin is correct, but clearly, you have a h head of state highlighting this. Stacy, you want to kick us off . Ill start off with a basic definition, Artificial Intelligence is when machines mimic people. We see it every day in our lives in something simple as mapping programs, getting us from point a to point b, personal assistants on our phones. Were thinking of analytic tools that can help the analyst make decisions better, learn lessons from them more clearly and better. I would agree that there is i wont speak specifically on what putin said, yes. So whoever rules a. I. Rules the world . Potentially. It open up the possibility to automate a lot of things that we havent done in the past. Certainly, theres a lot of investment around the world in Artificial Intelligence, and we have to make sure were monitoring that and keeping up with that. We have great researchers and scientists and practitioners in this country that i think will be competing well against those in other parts of the world. So im optimistic that ais really going to have more benefits to society than threats, but we have to make sure that we are watching out for the threats as well. Andrew, i minean, ai is more than simply processing algorithms and technology. What is the agency doing right now to advance the field of antis pa tor eye intelligence and what role does ai play in all that . So as an intelligence organization, the first kind of order of value that we provide is revealing the secrets denied to us by our add vir sarees. So exquisite intelligence collision, signal intelligence, the first sort of business for us. Increasingly, what our policy makers expect and the value that we provide is help us understand how major world events unfold. What are the major muscle movements that move to periods of instability. So we also have an expectation and can provide value in taking what we know from many decades of social sciences and using advances in the yew pick wit and power of computing and sensor data to learn from that, that social science experience and apply that computational ability to enable the higher human cognition of our analysts and our officers, and buy time and space for policy makers by helping them understand where those emerging conditions changing that require a noix,foa shift, a pivot. So the demands on us increasingly are on the more diffuse muscle movements where its a higher volume and fragmentation of data. Oftentimes public sentiment, many types of data sources, diverse sources that we can apply them, computational methods to and Computing Power to enable again that higher human cognition. And andrew, and i wanted you to touch on this and then bring fran in and go to rob. But, you know, isnt expectationsetting a big part of this discussion . I mean, the Intelligence Community, after they miss something, its sort of a witchhunt and what have you, and a lot of fixnger pointing. But the reality is youre estimative, youre not claire voints. Absolutely. People cant call what they see because theyre afraid they might get it wrong. We try to manage expectations by nightinoting, were talking improving our forecasting to reduce the uncertainty, manage the complexity and yes, we can. And we have made advances in improving our ability to forecast unstable events, and that will be a consistent investment we make in a field of forecasting and antis pa tor y e intelligence. We have to invest in the field and develop tool suites for our analysts and officers to manage that complexity, were always going to be in that constant state of investment in the field as it advances. So important to understand the difference between prediction and forecasting in that context. And fran, youve lived, and youve been the leader of so many in this community at the Central Intelligence agency and elsewhere. I mean, looking at it through the eyes of the analyst as well. Where do you see this kind of playing out and how do you enhance, how do you empower their voices, even if theyre occasionally wrong . I think you got to get back to the heart of the analytic mission. The analytic mission is to provide insight to policymakers so that they can make sense of events on the ground and either advance u. S. Interests or protect u. S. Interests. So might a policymaker like it if we were in a position to predict, to give them an accurate prediction of an event . Absolutely. If were in a position with confidence to provide that is correct that, ultimately no surprises is what were looking for. We recognize if you could predict that will, you would be in las vegas. You wouldnt be here work being no the Intelligence Community. But what we need instead is to understand what is happening in ways that give us space to make decisions. Either to do something that advances interest or protects it. So i think for analysts, its actually the understanding that sometimes were not going to be in a position to make that prediction, and the best we can do is to provide understanding of whats taking place on the ground. And i think Technology Offers us an opportunity to extract from information what is latent knowledge now, l latent informan now but could become actionable in that context. When integrated with what experts know. So whether were talking about Artificial Intelligence or forecasting, its the marriage of the power of that computing capability with the subject matter expertise. [ no audio ] ultimately everything we discuss today has to be useful to that end user which is [ no audio ] and ultimately policymakers to make good decisions. Rob, so it, at ibm, youve got some massive processing capabilities. Youve got some Amazing Technology and thinking around these issues. Where do you siee coming dow on some of these matters and how do you make it work . How do you make it work effectively . Because there is a lot of buzz around the technology but theres also [ no audio ] what are your thoughts on all this . And not to put you on the spot, but you see the real animation coming out of uncle sam or out of the private sector . Ill come back to that question. But perhaps to answer your question by [ inaudible ] first of all, i think not withstanding everything else, ai, and certainly, you might hear from some of the celebrities who like to weigh in on this topic. The purpose of ai is not, our current focus on ai is not about replicating the human mind, not in an entirety, not in a general sense. Its purpose primarily is around what i call augment ng the human mind and amplifying the [ no audio ] that allows the information coming in. Were seeing in the field to be distilled. And yes, theres some reasonable similarity for the reasonable processes and nonetheless, we can activate the human mind, to see perspectives that might otherwise be overlooked. To help people see through their buy as. Biases, the trigsers that cause us as humans to have a new idea and to bring that to whatever problem were dealing with, whether thats in intelligence setting. In private sector, of cork turs the similarities are to financial investors, recognizing that most of the decisions are going to be based on predicting and anticipating and forecasting what the markets going to do or in the case of health care doing the same thing for identifying the right treatments for a patient, recognizing that in this case youre predicting and anticipating and forecasting how this patients going to respond to that treatment and likewise, what kinds of side effects are going to experience and so forth and how you deal with that as well. So i think in all these cases what matters is that were concentrating on the parts of the human cognitive process that today, for most of us represents a limitation in our cognition, limitation in our awareness, limb tag limitation in the way that we reason through a problem and bringing ai in at that moment where its ability to distill information will make a difference in the process. Rob touched on a critical point there, and ultimately most tech breakthroughs are at the edge. Looking at it from iarpas perspective, are you starting to sigh more see more in the convergence . Yes, youve got different d dimensions. The adversary can express more pain more cheaply. But id be interested from iarpas perspective how you think about applying some of this, and in particular some of the convergence between human tlerngs technical means and how that all comes together. We started out some of our research in forecasting looking at data itself. What was available from all the publicly available information out there that you can derive intelligence from. We coupled that with human judgment. How do you lit people make decisions about events in the future and grade them in their success success or failure. Machine learning that can parse through all of it and that comes faster than humans can process it oftentimes, how do you put that together, the best of machines together with the best of the human forecasters to figure out if you can make more powerful forecasts. To give Decision Makers more time and more space. We see a lot of promise in bringing it together and allowing machines to help analysts think through the processes that theyre using both in coming up with assessments and looking at the Lessons Learned. Walk them through the processes in a way that sometimes is difficult. In theory, the machines can be developed to be unbiassed in their approach. But they are, they are created by humans. So youd have to watch the human bias that gets put into there. But in theory, you can help bring an analyst through making designatures decisions. For us, the power is in the combination. Seeing it in the workforce at the agency, any early Lessons Learned and observations that i think are important . Because the, a good collector may not even, got is a big part of what they do every day and they cannot necessarily convey that. They just know it when they see it. How does that sort of play out . Your world . Couple points. One is benefits of this field and of this technology and the marriage of the technology to the forecasters. Theres utilities simply in including for our operators in enabling their Situational Awareness. And, as you look to the way that the sensory environment globally is evolving and the digital dust that our operateeaors leave beh, theres greater opportunity to have a greater Situational Awareness of the environment, the targets, the tracking. Were trying to find who we dont yet know, but know from analogous behavior, maybe theres a pattern in there or something to detect to find evolution of a plot. But the other point i would make is, and to the point that you made to rob was wheres the greatest innovation happening, in the federal sphere, the government sphere or commercial sphere. I noticed rob didnt answer. You about i want to hit this, because for us, its really an important point that one, its not an either or but deemands a close partnership. We simply cant we do this internally behind our firewalls, were just not going to be the advancers in this field. Weve got to have that very close partnership. And you are leaning forward, not only with the traditional Big Industries that are doing phenomenal work here and the Defense Industrial space and the integrators but also silicon valley, right . Yeah, theyre partnered very closely with silicon valley. We just have to have that close partnership. Awesome. And fran, i mean, sort of having now been out of the community for a little bit, anything youre seeing that you would have, im not suggesting done differently, but just sort of having that private sector, because i think that really is a critical point. Its become a little trite, Public Private partnerships, but its also genuinely important, and i think that whats important is not the partnership, but its the people that are actually partnering, and they understand, they step in one anothers shoes and know what they can, what piece of the puzzle they can bring to the fight. So any things from your end . One of the most incredible insights once you leave the organization is the isolation, as much as we try to reach out in the organization. The isolation of working in a secure environment like cia creates and the intensity of the mission and the need to on a daytoday basis be completely on juyour mettle to not miss Th Development or event that you are responsible for warning for. I think it tells scopes your ability to focus in an i have in insular way. Its very difficult, no matter how much the agency tries to help officers reach out. So, when i listen to this conversation, whether were talking specifically about working with the technology, working with data scientists to be explicit about the process through which subject Matter Experts arrive at their conclusions or even in some of the more traditional, what i call lowtech ways that weve developed over the last 30 years for anless alysts. There is always an ahha in those exercises. The debate always has been do you provide the output of those processes to the customer . Or is it use the simply to make sure that the analysts and tour has widened and that its open to the possibility of change. So when i look at what im seeing in the private sector is very often those processes are incontinueded to improve the interm quality of the work being done. Its not considered an enduser product. Its a qualitycontrol product. So one of the things i would say is that debate in our own organization on the analytics side, if we make the investment, does it have to produce a prediction thats going to lantd on the president s plate . Or is the process itself and the ability to challenge othur assumptions the actual end goal that ultimately our more it traditional means are informed as possible. And i think thats such an important, its more than just warning. Its context, setting the stage. So you can make sense. Right. Of environment. And its Situational Awareness. The freedom has to come first inside the organization to think expansively without being constrained by the need to produce for the enduser, whos the customer. One thing i might note that the agency did that i felt was quite innovative and i understand quite effective is getting their analysts in the field and getting those who are operators a better sense of what the analyst can really provide, and that sort of, in a tech environment, how many times have we invented tools and the operators werent part of it . And you sort of have the operators coming up with macgyver Like Solutions to everything. So we really do have to move if that direction. Rob, id also be curious, director pompeo in his opening remarks referenced the fact that collection continues to outstrip our ability to process data. And h and this is an ageold signal to noise. We got a lot of data, but what data really matters . Data is the new oil. It is gold, but not necessarily if you dont know how to mine it. So id be curious what youre seeing there and howatson can be brought to this fight. And actually, i need to bring out that this is the exact same thing occurring in the private sector as well. That it is motivating. Not only ibm, buff realt really entire industry to invest as much as we can in ai. It is about how do we get insight from all the information produced out there in every aspect of our lives. Just think about the changes that have occurred in our lives over the last ten years with the introduction of smartphones on a daily, its not even just a daily interaction. Its an every moment of our life is spent, away from it is really spent somehow engaged with these devices. And that is both an opportunity meaning thats a place in which we can really get advantage by having a system thats able to ingest and make sense of information, allow us to make daily decisions, small and large, but its also the source of a lot of information that were producing. So it creates a cyclical effect. But i will say that i think there is, and just to tie this back to your last question, this s synergy that occurs between private and public or between the agency and work that were doing outside in the world, both in terms of the differences that occur inside the agency, point is, the point here is that, you know, these cognitive systems, at least the ones i work on are very driven by ling weuistics. They derive their understanding not by doing a knowledge search or trying to compare against a whole set of structured knowledge meta data but rather looking at the linguistic patterns that we as humans use to convey and communicate our intention and meaning. Theres so much more information in that than there is through the traditional approaches that weve taken to understanding. And what was interesting was, you know, the work that we did early on with watson for jeopardy, playing the game of gener jeopardy did drive us to do levels of linguistics to perform that particular task, the task of taking a question, a clue that was intentionally nuanced with puns and subtleties and understand that drives that question and finding the answers, bring it into a highly specialized domain where the linguistics are in fact a little bit different and sometimes substantially different really gave us the opportunity to open up the system to another level of understanding around human language. That in turn then, you know, obviously, we werent taking this information. But the same link which g kwis particular properties that were used there are used. So i think theres an opportunity to share across those two spaces just if nothing else because of the way the different disciplines and the domains bring you a different perspective. Im actually glad you brought that, because i always come back to the issue. Your outcomes only going to be as good as a, the enterers, b, the questions youre asking or querying. That is where machines can help process unstructured data. And more for our government, for fran as well, one of the things i always found interesting is in the hot wash process, after something had occurred that maybe you didnt recognize or pick up or provide and indicate a warning. Do you go back and look at that data to get a sense of what you could have missed and then previous incidents to then drive new indicators for future events . Does that make sense, andrew . It does, yeah. And do you do that . You want to be in a constant state of reevaluation, what are your assumptions, and do you interpret the data differently. You have to do that. And to the extent we can use computational power and Artificial Intelligence to do that dynamically with this, thats really where you want to be. You dont want to be in an epic fail, epic fail. That points to really the future of how we provide that Situational Awareness and warning and anticipation in a world where were not living from, as fran brought up, where youre looking at enduser product delivered on a periodic basis but when were in a much more dynamic push data dynamic forecasting realm where you can provide greater accuracy in your forecasting as your models improve and as your data converges. So constantly learning. Yes. And presentationly in a more dynamic night to our consumers, and i think thats a really exciting field for us in the future. Its really present now. And from all sources. All source ths. It could be words, weddings, bad guys. With our exquisite intelligence and coordination with data layer. Just to add on to that, because i think its not even just a matter of retroactively going back and looking at past events, fundamental system, the system fundamentally relies on being taught, which means that it had to have been exposed to experiences in the past that are now relevant to making predictions about the future or, you know, when i say prediction here, i mean around the basic things like can we predict what ling westicly that particular statement might mean when considered in its context. So we began by looking at Historical Data and use that Historical Data then to teach the system how to recognize those things, how to look for the patterns of meaning in the signals that theyre representing and as you said, continuing that process by taking even Current Events and just near past events to further refine that. And what youre really looking for there isnt its ability to have this ahha moment, because frankly, thats not whats going to make it a difference for the system, whats, what its, what you want is for the system to draw people, to create their own ahhas, right . So you want the system to be really good at finding the little subtleties in information and meaning that when brought to the person causes them to have that moment and then come out with a conclusion thats meaningful, so i agree with your point that it really should not be using the system to present the final conclusions. They should be used to facilitate the end process. Fran, nip yanything you want unpack further on that . Yeah. I may have more experience on this panel with encountering intelligence failures and being accountable for them. I was the head of analysis at the start of the arab spring. What might we have taken away from tunisia . The subject Matter Experts, the 1. 5 fte, recognized that something was unusual in the persistence of demonstrations in tunisia. And the analysts working that area understood that a whole series of factors that underlie instability were inching upwards on the heat scale, if you will, in the region in the five years before. In the two to three weeks before the actual departure of the hid head of that country, the analyst recognized something was different, but in the heat of the moment when things began to move toward the capital, made one of those predictive analytic calls that analysts are eager to make in the moment, which is mr. Policymaker now that its moving toward the capital, that the governments going to do what theyve done for the past 23 years. Theyre going to take control of the situation. And ultimately, thats not what happened. So when you look at that, what Lesson Learned can you take away from it and where can a forecasting capability really help . Where do we need to understand instability and areas where we dont have huge resource investments . So think the inverse, right . Do we want to use this against our top ten intelligence priorities . Yes. But if one of our low area intelligence priorities is going to kick off a crisis of a decade, where can we be warned in a way, and, again, not when we were writing for the policy make irbut whe maker but where the analyst recognized something wasnt right three weeks before hand, could we have gotten them to a different place had we had a model in place, a complex behavioral model right where Artificial Intelligence was helping to parse those weak signals in advance. Would we have gotten it absolutely right . I dont know. But i do know that even the human beings working that area recognized that when ben ali got on the airplane that we were headed for tech ton eck shift in the middle east. The magnitude and scope they didnt know. But the implications of the Ripple Effect were instantaneously obvious through the subject matter expert. But could we have gained three or four weeks and not been flatfooted on tunisia if wed had behavioral models on stability in that region up and running in ways that tweaked the imagination of the analyst. I think the answer is yes. And thats, i think thats where we really could have added more value. And we have better capability now to do this. But the understanding, whats the fragility beyond the first precipitated ve precipitated event that could have positioned us, this is likely to have a cascading effect. We have greater ability to do that now so we had some research that was going on at the time. We werent focussed on that part of the world. We had a very limited resources. We were looking more at latin america, but we able to forecast some of the protests happening in latin america. I think our sweet spot is being able to focus on we only had one and a half Million People looking at a particular area. Technology can help you, and have a situation where you are looking at the data for that particular part of the country. It would have been interesting to see if we would have seen that. But that is where the potential is, to be able to bring in sources for places where you dont have the ability to put more Human Resources on them, let the automation help to you triage and bring in the Human Resources. Thats the great point. Because at this end of the day, youre responsible for the entire world. From a decisionmaker, policymakers perspective, think are expecting consistent coverage of all things all the time. And the reality is we have predators who are going to take advantage of crises and use that time to start acting themselves. So i think that is a great place where i think technology clearly help with that. And the one thing i might note is, i mean, with all technologies, its really in the application. So we can come up with all the tech, but its ultimately how its applied. How is it being used. How with relearning where it becomes steal a becomes stale and static and its no longer growing. I do want to open it up to the audience but i want to get that one word. Will ai and some of these technological advances help red or blue . When you look at cyber, the reality really remains with the attacker. But i think technology can help the defenders as well. Im curious. And im putting you on the spot. Red or blue, ai . Im going, its going to help both. Theres no question. Thats the right answer. But we are the bulls eye for many, many, many countries, so cumulatively speaking, the stakes for the adversaries are ultimately going to be very, very high. And thats a challenge for us because we have many targets. We are the bulls eye for most. So vote for purple. And a recognition an attack surface keeps growing exponentially too. Rob . I think the technology is going to play a minor role in deciding that. Its the quality of the data, the availability of the data and the integration of the results of the processing into our processes. Theyre fundamentally determined who gets that advantage. So to be honest with you, i think we have complete say in this, right . We get to decide for ourselves how well we want to leverage this. So, as good as the user is, basically, what youre saying. Andrew . Yes. Its not either or. Its both. And the key thing, and ill key off i think where director pompeo started its how we apply our offensive capabilities. Our red capabilities to inform our blue. And to the sense to which you can gain or sustain that competitive advantage is how we use all of our capabilities in that way. And im glad you brought that up. Because at the end of the day we play by rules. A lot of our adversaries dont. So that might tilt it to red. Stacy . . I im going to go with the blue and the red. I think there is a space race on, and im not sure we fully grasp what this means. Lets open it up to questions and please take a moment. We have a question here, ron marks. And we have a question here and a question there. So. Hi, ron marks on the board here. Hello, fran. Let me ask you a question because these people are all very interesting to me in terms of what theyre producing. Youre on the firing line. Youre on the coal face, and youve goat t youve got the people sitting on the coal face. What do you want them to know about these guys . What do you want them to understand about this process . There are a number of people who still believe that the best analysis is in between your ears and all this is very interesting. But how do you want them to balance it off in terms of the new analysts that youre bringing in. Im glad you asked that. At the university we also want to