Test test test test test test test. This is a test. This is a captioning test. Test test test test test test test test test test test test test test test test test test test test test test test test test test they do break down a little bit in cyber in terms of going out and labeling objects for full motion video, there are no objects on the ground. We have it and we have people, buildings, we work our way down from there. Cyber a little bit difficult problem to deal with. What is the baseline . If we take the baseline behavior, i have to know what baseline is, its a appreciative maintenance or humanitarian assistance astro relief case. So if i go back to starting with a data problem on cyber, its the most basic problems that everybody begins with, data access, data quality, data content, data classification and data standards. What we have to do is reset a little bit. Our challenge without getting into the Technical Details is we have 24 Cyber Security providers all of whom are collecting data in different ways. The starting spoipt come you po come up with a cyber frame to come one a starting point for data curation and content, sharing and storage, just on that agreement, i think well have much more success down the road as we bring in commercial vendors to do product evaluation. Were challenged right now they didnt quite know what data they are going to be seeing. There is not an image net ekwip quiff lent. We quiff lent. Our first of three lines of effort is what we are calling event detection, currently basic, something bad has happened and monitoring and the third is network mapping. All of those have the same basis of a data problem. So by going back to the beginning and working on a cyber data framework, which is nothing more than could we agree on a common set of procedures from now on of data coming in. If thats not the starting point, we dont have the decades worth of really nice clean kurated data thswami would say. Its not true for the department of defense as the intelligence community. You made the point earlier that every Cyber Security company is now a kieber ai cyber ai company and i would mike i make the point in the last decade, Many Companies started branding themselves as cyber and Cyber Security companies and that gets into the definition of what problem are we trying to solve, right . So a decade ago, we talked about Cyber Security. We were probably talking about antivirus definitions. Right . Now were talking about a living breathing ecosystem of the world and as general shanahan said is define normal. How do i even know the difference between whats normal and abnormal so i can detect anomalies. And we simply dont know. We dont know the answer to those questions right now. That makes it very challenging to develop solutions. So this community here this Cyber SecurityCommunity Needs to be thinking deeply about how do we know its normal . How do we detect variants in the system, how do we know our systems are appropriately secured against Cyber Attacks we cant yet define, that fundament amly is a challenge. Ai can help with it. Ai is not a magic bullet. Its not jacks magic beans. Right. It can solve some problems really really well and other problems its you know particularly the kind of ai we are talking about now, the machine classifiers and so on. You request solve those problems really well. Not everybody problem boils down to that thing very well. One of the pitfalls i see many commerce fall into the high expectation trap. Azy not a Silver Bullet by any means and when you set out actually saying like the best way to go about it is you smart small and actually it rate and check to see how well, it solves the problem. Its almost like the journey that you will be on for actually fought just months, even years to come and earic is completely right, i will fund a project, its going to be big, its going to be massive. I will see how you will be doing in six months or aer 82. If not, by definition, your chance of success will be low and you are absolutely spot on, that this is something, its almost like a journey of discipline, how you have to progress. And if i can add as well on the data piece, you know the challenge today is not just trying to wrangle it into a good form, its also determining whether or not you can trust it. And that gets into some of the challenges with the that poisoning attacks for instance, where you may have perfectly good looking data, but,llen on quall of the data we have from a formatting or occur it aing perspective is has someone actually tampered with it. So that gets into some r d challenges how do you make sure the data is pristine, its the way you intended for it to be. Its not included within that perhaps some examples of where are you learning maybe unwillingly that a particular data set is not, is either is or is not indicative of an attack of some sort. So thats an extra challenge on top of either not having the data or having good quality data as you might have that, can you trust that you have good data . And this idea that trustworthiness, the data is critical. You can imagine our job is to see over the horizon enough time to impact the difference. Well, in an era of adversarial networks, producing deep fake videos and fake images of people and fake audio and you know being able to substitute anybodys face in anybodys video, yet there are parlor tricks right now, but they have you know if you look enough down the road, it has the implication of it being very difficult for us to separate truth from fiction and that makes the job of intelligent really really hard, right . If you dont know the difference between truth and fiction, you got a big problem on your hands. So the finkind of things we are focused on is what is real and what is not valley really really huge. Its applicable to the cyber domain as every other domain in which we look at these problems. So based upon the previous conversation, you know, were starting to address some fairly bake use cases and were starting to move towards adoption. You have a captive audience here. In terms of research and development and where you are looking, id like to kind of work, hone in on where you are looking for new ideas, where should this community be investing for the future . Do you want to start us off there . Sure when you think about ai and Cyber Security together, there is ai for Cyber Security and there is also the Cyber Security of ai and both of those have important r d challenges to them. You could imagine using ai for Cyber Security and doing things like being able to understand your adversary and how theyre attacking and maybe look at past behavior, past history and use that to perhaps predict what future attacks might look like, for instance, and thats certainly an interesting r d challenge from the ai for Cyber Security. In the other direction, Cyber Security for ai, looking at challenges like how do you make sure that a model that an ai system learns is not reverse engineered to somehow detect Sensitive Data or information that you dont want your adversary to learn about, i already mentioned data poisoning attacks and there are a number of other of these kind of challenges that you want to have your a. S. System trustworthy to ensure when you use it, it will do exactly what you planned for it to do. That in and of itself has a lot of r d challenges as well. The National ScienceTechnology Council every three years puts out a national or a federal Cyber Security r d strategic plan. So theyre preparing that Digital Foundation and compute. It says as government we need to be fast followers. We are in the interesting position probably for the first time we are not the leading investor in a technology area. We are not a minority investor. The World Economy is the investor. In 2016, mckenzie estimated that there were 50 billion in Global Investment in ai and Machine Learning and they estimated that there was about a billion dollars in u. S. Investment at that time. So 50 on the end is billions. Yes, were spending more since 2016, dod, darpa has announced their strategies. We dont publish our investment. But you can imagine the private Sector Investment has accelerated and investment has far exceeded governments expectations. We have to be fast followers of the world. Next we have to invest in the gaps. As a government we have to invest in things the private sector has not invested in, or is not as invested as we are. Think about a bell curve. Where is most of the private sector, the middle of the bell curve, where your shoppers are, dollars, collision, add, eyeballs. Darragh shannon has ab pro. Whats my problem . Into the low bell curve. High probability, things happen out there. Thats not where the Machine Learning has been invested in the last decade. Thats where we need invest. Were the intelligence communities our job is to see over the outcome. We need to be investing in long range semantics and meaning and only in. Because ultimately its good but its fought good if you have. I need to know why there were airplanes on the runway yesterday and theyre not today. I want to below they are, what their intentions are. Ultimately the job of intelligence is to understand that. Yeah. How quickly things from the private spectre. In amazon we tend to use the phrase, its day one in the age of internet, even tow now we are more than 20yearsold so that shows like how we tend to think. But in the Machine Learning world, its so early that i joke around saying like, yes, its day one but we have just woken up and havent had a cup of coffee yet. Its that early in terms of how much early we are in this game. So there is so much r d that is still yet to be done in terms of kind of feels like develop the internet like in early 90s and so forth. So in terms of what we need to see in r d, its not just about making machine models, development accessible and getting data aknowtation done. Annotation done. Like when it produces a result, what we see when we worked with the healthcare customers, that the consumers of this Machine Learning model, hey, are you scheduled for surgery, you might want to pay, which is more optimized. They were not willing to trust that result unless you explain saying like historically if you had done this, you will be 40 efficient and so forth. There is even these elements of explaining these results, so that people will trust it more. Its going to be a lot more important and these are some of these areas that are still under research to me. And we are to invest a lot more and fought just in the private sector, but also in academia and assets. We partner with nsf on these topics as well and Fund Programs and we look to do more. An idea just to cut to the chase, without trying to bridge a gap a little bit from the r d side, a more practical fielding element comes down to trust. If we look to future of war fighting or defense of which we no longer are measuring action, counteraction in minutes or seconds even but millie seconds and microseconds, trust becomes t what we are trying to achieve, how do i get there . If are you doing research or developing a pro you cant that is a 98 performance in a pristine Lab Environment and doesnt work in the cases that dean mentioned in a dirty dod environment, its not help. For me, proving it can work under those conditions, its a partnership, us two a vendor giving them the data they need to show or perform in those instances, i would go along in saying we also need to be thinking much more about ai and counter ai or adversarial ai, a red teaming approach which has a program of automateing some of the red teaminging as to give the humans, the people more time to think about the contextual things that are going on behind the scenes, counter ai is a future we will be dealing with, its analogous to warfare, counter and counter action on down the road. That is something upon us now. We need more thought put into that across a commercial enterprise. I think thats a really interesting observation. The sector last year identified two new war demands, soober and space. If you imagine the future of combat, you know its adversarial ai against our ai and you know how are we going to adapt in that new war fighting demand . So certainly exciting times. We have about a minute left. Lets go around, each person has about 20 second for any parting thoughts. Dr. Parker. Well, certainly, if you look at the president s American Ai Initiative that was assigned in the executive order that was, happened in february. There are a lot of these issues that priority in the initiative in the national strong. You look hat work force issues and trying to make sure we have the people we need in the ai spice, which includes the ai applied security space so that we can be in these areas. You look at data, there are actions in the executive order about making data more available for ai r d that can help in Cyber Security. There are a number of key areas that we have touched on that the federal government is taking a number of actions and to try to help the nation move forward to ensure and maintain American Leadership in ai going forward. Well, actually, i think we are just about out of time, so to the panelists, thank you, and a good discussion today. I appreciate everyones time, so, thank you. [ applause ] meers of the panel for a great discussion. The next panel is on preventing a cyber 9 11 and is moderated by bill loomis program hacker one. Joining bill on stage is jeff brown, chief Security Officer for an Intercontinental Exchange in New York Stock Exchange. The hon rablg karen evans assistant secretary for Cyber SecurityEnergy Security and Emergency Response under the department of energy and kerry wright, Vice President of Product Management at end ace. Bill over to you. Thanks, everybody, for joining us. To start off id like to let our panelists we got a brief introduction there talk about their current relate and what they are doing in the area of critic infrastructure. Jeff, if you want to start us off. Thank you for having me. Just a quick correction in the intro, jeff brown. Im the head of something called new york city Cyber Command and the chief information Security Officer for the City Government of fork. We have the mission to defend all of those technologies that deliver Via Technology services to new yorkers each and every day and we also have admission to bring Cyber Security to new yorkers through solutions and awareness in ways that helps them navigate away from the threats they might encounter on the internet. Your question. Okay. To your question about how we think about Critical Infrastructure, we certainly as a large City Government have parts of the portfolio agencies like the department of Environmental Protection that has you know ics, ot, Water Service that new yorkers rely on then we also think about the criticality of things that are deemed Critical Services that new yorkers have to you know rely on the time of need, Something Like our 911 environment, Public Safety arms, et cetera, thats how we think about it. Karen. I am karen evans and i am the assistant secretary for Cyber SecurityEnergy Security and Emergency Response, otherwise known as csesacseser. We deal with Sector Agency requirements as it relates to all hazards both natural and man made, so the Emergency Response function is really high right now on our efforts of our team due to the hurricane so i have hurricane responses, i have cyber responses, i have the Energy Security piece. I have dnd, dmd. We are responsible if you are familiar with the National Response framework, we are the esf12 coordinators under that with our sectors specific roles and we also have specific authorities that are designated to department of energy under the fast act of 2015. So i think ill stop there and pass it on. Sure. Thank you, karen. So, kerry roth, im from endice. Endice is one of the sponsors here. Vice president Product Management. I am fortunate enough to get around the world and talk to a lot of subsecurity teams and help them with their analytical tools to analyze things deeply and roll out different tools to defend the network very interesting insights i hope i can share with the subpractices in the subsecurity teams as of today. Cool. Im jerry perlow. Im the real chief Information Officer