Studies and got my masters at Carnegie Mellon. Then i left c. M. U. Joined i. B. M. Research for three years or so and i came back to c. M. U. Because i liked c. M. U. And pittsburgh that much. Peter what kind of things do you work on . Raj i work on what are called embedded systems, the technical term. These are things that embed computers inside them but the user basically uses different devices. For example, your smart phone is a sophisticated device but you dont worry about the components inside there. Televisions, for example, it has a computer inside it but it is meant to be a television. Think about a projector. It. As a computer inside of that toaster that we use on a daily basis that cost 10, it has a tiny computer embedded inside it. These are embedded systems that act smart but its a dedicated device of some other kind. Ive been working on embedded systems since my doctorate and it turns out vehicles that drive themselves are a classical example of embedded systems. So a car is a car that transports people and goods from point a to point b with components embedded inside them. Peter how did you get into the business of Autonomous Vehicles . Raj great question. Ive been working with General Motors, the carmaker since 2004. General motors has been working with the researchers in our Department Since the year 2000. I started implying my expertise into embedded systems in the automobile. In 2006, the Research Advanced by the agency, the research arm of the u. S. Military announced a competition. The competition was the urban challenge. For vehicles that drive themselves, because anybody in the car, they need to drive for about 60 miles in fewer than six hours. In an urbanlike setting, with other selfdriving vehicles as well as human driven vehicles and following the same rules of the road that you and i have to follow on a deal he basis. With that competition g. M. Became our biggest sponsor. We had about 20 other sponsors but g. M. Was the biggest of them. Because i already had a strong working relationship with g. M. , i became an integral part of the team that worked on the vehicle, which ended up winning the competition and the 2 million prize. When our team from Carnegie Mellon won the competition, g. M. , who was the biggest sponsor of the team, said hey, our team actually sponsored the winning team and they said because it is about urban driving, a clearly has implications to the Consumer Market segment and they started a second lab on campus focused exclusively on automated driving and ive been running that plan that lab as well since its launch. So our relationship with g. M. Continues to be extremely strong and very loyal. Peter so, does g. M. Own the technology that you develop . Raj for the technology that they sponsor were grateful for their support but it is actually owned by caron darn owned by Carnegie Mellon university and we have some Licensing Agreements with g. M. Peter and we learned more about the experimental cadillac as we got ready to take it for a drive. What is this monitor thats up here in the top . Raj that is one of six laser sensors. Theres one in the front of the car. Theres one in the bumper here. Right there. Thats the second radar, thats the third on the other side of the bumper as well. Theres one behind the side back window of the car and there is one on the other side exactly opposite that. Peter what are they reading . Raj what they are doing is theyre sending out multiple laser beams and when the beams basically hit the on the, they bounce back and come back to the transmitter and because we know the speed of light, we can actually calculate how far that vehicle is. Because there are multiple beams scanning as well, you can get a profile of the object. And because we have radar all over the car, it shows whats happening around the vehicle in realtime. Peter is this car communicating with anybody but itself . Anybody but itself . Raj its capable of actually communicating with properly andpped traffic lights similar radios. Its a automated vehicle. T. A. V. In short. Peter i also see some cameras inside the car up here. What are these . Raj we had six laser sensors. Three cameras and six radars as well. Two camera, one is inside the cabin. One is actually pointed downwards looking for lane markers on the road. The other looks at traffic lights so you know the status of the traffic light. The third camera at the back of the vehicle is for backing purposes, you can see whats going on. And there are also six radars. Theres actually one behind this cadillac emblem. We replaced the metal logo with a plastic logo so the radar can see through the plastic. Theres one a radar thats behind the bumper. The bumper is made of plastic so the radar can see through the plastic as well. Theres another radar on the other side of the bumper and there are two on the side but you cannot see from the outside. Internally theyre enclosed. You dont see from the inside either. Theres radar at the back in a plastic bumper also. Peter is this car seeing 360 degrees . Raj it is seeing 360 degrees all the time, as opposed to humans, who have to turn our heads around. Peter so, professor, when you get in this car, whats different in the look than a regular cadillac . Raj we tried to make this car look normal on the outside and the inside. On the inside it pretty much operates and looks like a normal car. Just like you would rent a car at an airport and pick up the keys, and the layout looks slightly different but were still able to drive. Its basically the same thing. You bring in your keys, get the the echo going manually vehicle running manually and start driving. Looking at the dark board there are two things that have changed. A button on the dashboard, an emergency stop and theres a button behind the stick shift. Think of this as the autonomy equivalent of Cruise Control engage button. So you would actually engage this button to go into autonomous driving mode. So what you need to do is rotate this switch and then pull it upwards to get into autonomous driving mode and that is a very conscious action on the drivers part. Peter is this car licensed to drive the streets of pennsylvania . Raj in pennsylvania, the laws allow a vehicle to drive itself as long as two conditions are satisfied. Number one, a licensed human driver in the drivers seat, and number two, that human can take over control at any point in time. On those two conditions, vehicles can drive themselves on pennsylvania public roads. Peter but this could drive as a normal car as well, correct . Raj of course. This autonomous board, you reflexively push this down. But you would not have the time to do that, you could grab the Steering Wheel, turn it and press the brake pedals or the gas pedal and the vehicle would still respond. Something could be happening that allows the human to take over. And then this button here is only for strictly emergency purposes because we added a bunch of sensors, computers and motors to the vehicle and then in case something totally unexpected happens like you start smelling smoke, you have no idea what is going on, you just push the button and it mechanically, electronically stops the car at that point in time. Knock on wood, we havent had to engage that when driving yet. Peter whats the cost of all time. The different systems that youre added to this car . Raj many of the sensors, cameras we use are oneoff units, they tend to be expensive. So, it is an expensive vehicle because of that. The real idea to be thinking about is that when the volumes go up, the costs will go down significantly. I guess our thinking is that when these vehicles are mass produced, it would add about 5000 extra on top of the cost. But we think he will be very affordable to many people. Peter youre going to give us a driverless ride. Is it truly driverless . Is that fair to say . Raj it is an automated vehicle. It can drive itself under many conditions but not all conditions yet. Not yet. So lets start driving. I guess ill take you and ill explain a few more things. Peter and youre driving manually. Raj im driving manually. It says manual on the screen. Ill point out a few things on the screen. It tells us that the screen interfaces to the same thing. So i guess if i flip back and forth. This is a screen that we added. By flipping a switch you can go back and forth. Peter what are these images were seeing here . Raj this is the display which shows what the vehicle is doing at any point in time, so we as humans can be comfortable that the vehicle is indeed doing the right thing. So what we see here is some icons here, and the icons for example say that i can actually launch, i can stop. I can tell the vehicle to go to the airport, go home or go to work. And this basically lets you zoom in or zoom out. And what you see on the screen is you see two blue lines. They represent the lane the vehicle can drive in. Peter there are no lines on this parking lot were in. Raj we have programmed the map for this parking lot that allows to us drive along these lanes and we go to the main public roads. That represents the map that your g. P. S. Has. The blue line represents the map. Then you see a green line there. All the people who basically ride around this park next to where we are. Your g. P. S. Device calculates. The g. P. S. Has a builtin map database. You as a user punch in your destination. You also see a very short red line up there. That is basically the car knowing the green line route, knowing the blue line maps. It basically uses its sensory data from the radar cameras and basically says for the next 15, 20 meters, this is how, where, im going to drive. Peter dr. Rajkumar, did you have to Program Every route ahead of time . Or it orchid and go on a street it has never been on before . Raj it needs to have a map of the roads. Peter a g. P. S. Map. Would that count . Raj you can think of this as a g. P. S. Map, g. P. S. Navigation device and then you tell the system where you want to go. It uses your navigation and your calculation for the route. Just like a garmin device or google maps. And the red line is what the vehicle is deciding in real time. The red line. And so now youll see if i zoom back a little bit. You see a bunch of dots on the screen. Those dots are the laser point from the laser sensors updating in real time. So what you see here is basically a bunch of yellow dots. Peter is that that white car . Raj thats the white car. These Dark Navy Blue colors is the dumpster there. And you basically see a peter and all these white lines are the trees. Ok. Basically the car consents ther can sense radars and cameras and lasers act as the eyes and ears of the vehicle. Peter how far can it see . Raj 775 meters. It has builtin Wireless Communications radios that can go as far as 600 meters. Peter all right. Raj so lets engage the vehicle in autonomous mode. Im going to do the following. The vehicle is in parking mode. Im going to engage autonomous mode. Peter while its in park . Raj while its in park. Autonomous driving. Peter it started driving. Does it ever make you nervous . Raj i guess the normal reaction for anybody new at this is to feel anxiety and anxious and angst. Its a very normal reaction but lets see how the vehicle does. Peter ok. Dallas, you doing ok back there . Dallas i think so. Peter and it turns on its turn signals. Because you told it where you wanted to go already, correct . Raj yes. Peter ok. All right. So you hit the brakes . Raj i did not. Breaks . H, it hit the and it knows the speed limit. Raj even though the legal speed limit is 25, pretty much everybody drives at 35 or 40. But this is a stickler to the rules. So basically right now we have a vehicle behind us and of course the driver doesnt like driving so slow. Peter uhhuh. It seems to do a little meandering in the lane. Is that a Fair Assessment . Raj it could be better, yes. Peter but its reading constantly. Raj yes. This is a curvy, winding road. Peter uhhuh. Raj so im not controlling the steering or the brake pedal or the gas pedal and it was able to shift transmission by itself. Peter i see that. This is a curvy, winding road. Now theres a biker . It sensed that . Raj yes. Peter all right. How far have you come in 30 years . Raj we have come a very long way, but still some ways to go to basically completely remove the human from driving a car. Peter is this vehicle constantly learn something . Raj this vehicle is not constantly learning but its collecting data and using the data to teach the software about new features and functions. So it is not learning how to actually drive. Its learning after the fact. Peter how did it know theres a stop sign there . Raj the map basically has indications about where the stop line is. It did it all by itself. Peter that was the car that did that . Raj the car did it all by itself, yes. Peter it wasnt sure of its speed . Raj basically saw those parked cars. In the interest of time i do need to get back so i will take over manually. Ill just push this down. The vehicle has gone back to manual mode. Autonomous ready. Raj it says autonomous ready. Peter and can you do that on the fly . Raj yes. You can switch back and forth seamlessly. Peter and its seeing all of these things . Raj yes. This crosswalk is not on the map. It understands to wait for this lady. Peter oh, ok. Raj these lanes have been changed recently. Peter ok. So its not quite ready to be sent out on a road its never been on before. Raj we have done that on highways. Highways, weve never been on before. In open areas you have pedestrians, cyclists, more things. We do that on the highway, not in open corridors. Peter ok. A bike. Raj ill zoom and you can see it better. So we are back on that curvy, winding road. Peter can it read signs . Raj it can read some signs, yes, but not all signs. There are thousands of distinct signs. It does not understand all of them. So now we see that red light well. The green is the path it wants you to take. The red line is exactly on top of that. Peter how far have you driven in this car autonomously . Raj we have driven a total of about 20,000 miles autonomously. Peter whats the longest trip youve ever taken . Raj weve done a couple hundred miles on highways. To technology has been used basically drive from San Francisco to new york city, about a 3500mile journey and the vehicle drove itself on highways about 98. 6 of the time. The technology had been used so highways are not a problem. Autonomous ready. Raj so, you have taken your first ride in an Autonomous Car. Peter when will we do this regularly as consumers . Raj simple question, basic question, and ill give you a long and complex answer. You can already buy vehicles, for example, tesla with an auto pilot feature. The vehicle can drive itself, but the human must be paying attention. General motors next year will introduce a similar feature they call super cruise, where the vehicle can steer itself and apply the brakes and the gas pedal as well and that will be in a cadillac sometime next year. And many highend vehicles can already drive themselves today. So, some of these features are already available on the market. Then three to five years from now, we expect that the vehicles will be able to drive themselves , but in wellspecified, welldefined geographically constrained regions. That is called geofencing. Basically, for example, where bicyclists are not allowed and there is no heavy rain or heavy snow. So come in california. So some of these technologies were deployed earlier but when when you asked about when the human not drive at all . That basically implies that the technology should be able to drive the vehicle itself from any point a to any point b that you and i and other experienced drivers can drive in the u. S. That capability is going to take at least 10 years. We have come a pretty long way over the past couple of decades or so but still a ways to go before the human can take themselves out of the drivers seat and go to the back seat and take a nap. Peter have you allowed your kids and your wife to ride with you in the Autonomous Car . Raj sure, weve allowed family members to go a long. Many of the people from the project, yes. Peter i was a little surprised that we didnt have to sign a release before we got in. Raj i guess if you were with a corporation, but just because we are researchers. Peter why are we talking to you about Autonomous Cars in pittsburgh rather than detroit or Silicon Valley . Raj thats a great question. Carnegie mellon is globally well known, has a strong reputation for computer science, engineering as well as robotics. We have a Robotics Institute on campus. There internationally recognized and they have more than 100 researchers in it, excluding students, if you will, and theyre all extremely knowledgeable about robotics and ages in the field have been built at c. M. U. Since the early 1980s. In fact, we at Carnegie Mellon believe that we are the birthplace of Autonomous Technology dating back to about 1983 or so. A couple of years back in 2014, we literally celebrated the 30th birthday of this technology on campus. Peter you said before we started this interview that computers are simultaneously very intelligent and very stupid. Raj yes, so computers are simultaneously very intelligent. They can do things that amaze us, right . They can actually react very quickly and they can make decision that a normal person finds extremely smart. How does it know theyre driving at this speed and so on. They are really intelligent because they process a 360degree view of the vehicle with the multiple sensory data streams from lasers, radars and cameras. Very intelligent, but at the same time they are stupid, if you will, because they dont really have common sense. For example, we know that when we fall down or when we basically touch fire, it hurts, and the next time you wont do it, but computers cannot make that inference. Hey, i crashed into somebody last time around, next time dont do that. It will do the exact same thing unless it is programmed to do something else, specifically by a human being. Peter whats the difference between the vehicles here in the lab. The jeep, that seems to have a lot of equipment on it, and the cadillac that we road we rode . Raj we see two vehicles. One is a red jeep. The other is the cadillac that we were able to demonstrate the vehicle driving itself. The vehicle on the left, the red jeep, is called nav lab 11, meaning this was created by a Res