MIT Self-Driving Cars: State of the Art (2019)

- Today I would like to talk about the

state

of the art of autonomous vehicles, how I see the landscape, how others see the landscape, what excites us all, ways to solve the problem and what to expect in the future.

2019

, as we will also be able to hear from the different perspectives and the various leaders of the industry and autonomous vehicles in the coming, the coming weeks and the coming days. So the problem, the mission, the dream, what we are trying to solve for many may have to do with the entrepreneurial possibilities of making money, etc. But in reality it is about improving access to mobility, moving people around the world who do not have that ability, either due to age or simply because of access to the place where they live.

We want to increase efficiency in the way people move. The ability to be productive in the time we spend in traffic and transportation. One of the most hated things in terms of stress and emotion, the thing in our lives that if we could eliminate with the snap of a finger is traffic. So the ability to turn that into efficiency, into a productive aspect, into a positive aspect of life and really the most important thing, at least for me and for many of us who work in space, is saving lives, preventing accidents that lead to injuries, prevent accidents that will cause deaths.

Here is a counter. Every 23 seconds someone in the world dies in a car accident. He should be sobering, he is for me, something I think about every day. You go to bed, you wake up, you work on all the levels of deep learning, all the different papers get published, everything we're trying to push is really saving lives at the beginning and at the end, that's the main goal. . So with that groundwork, with that idea, with that foundation, the mission that we are all working towards from different ideas and different perspectives, I would like to review what happened in 2018.

First, Waymo has done an incredible job in implementing . and testing his vehicles in various domains and having reached the 10 million mile mark in October, autonomously, which is an incredible achievement. It's really a big step for fully autonomous vehicles in terms of implementation and obviously it's growing and growing day by day. And we'll have Drago from Waymo here to talk about his work there. Then at L2 on the semi-autonomous side, that's the torque, that's the mirror side of this equation. The other incredible number, perhaps less talked about, is the billion-mile mark achieved by Tesla in semi-autonomous Autopilot

driving

.

Now the autopilot is a system that is capable of controlling its position in the lane, focusing on the lane, it is capable of controlling longitudinal movement so as not to follow a vehicle when there is a vehicle in front, etc. But the degree of his ability to do it is the critical thing here, it's the ability to do it for many minutes at a time, even hours at a time, especially in highway

driving

. That's the critical thing. And the fact that they reached 1 billion with B miles is an incredible achievement. All of that from a machine learning perspective is data.

Those are data. And all autopilot models work with the main sensor as a camera, that's computer vision. Now, how does computer vision work today? Especially with the second version of the autopilot hardware, there is a neural network. There is a set of neural networks behind it. That's super exciting. This is probably the largest deployment of neural networks in the world that has a direct impact on human life, that is capable of deciding, that is capable of making life-critical decisions many times a second, over and over again. That is incredible. You go past the image classification step in ImageNet and you sit there with a tensor flow and you're very happy there.

You were able to achieve an accuracy of 99.3 with a

state

-of-the-art algorithm. From there, you step into a human life, your parents drive, your grandparents drive this, your children drive the system and there is a neural network that makes the decision whether they will live or not. So that billion mark is an incredible achievement. And on the sobering side and from various perspectives, the deaths, there were two deaths that occurred in March of 2018. One on the fully autonomous side of things with Uber in Tempe, Arizona, which hit a pedestrian and caused the death of a pedestrian. And on the semi-autonomous side with Tesla Autopilot, the third death caused by Tesla Autopilot and the one in 2018 is in Mountain View, California, when Tesla crashed into a median and killed its driver.

Now, the two aspects here that are sobering and really important to think about as we talk about the progression of autonomous vehicles, the proliferation in our world is our response as an audience, from the general public to the engineers, the media , etc. how we think about these deaths. And obviously disproportionate attention is paid to these deaths. And that's something engineers need to think about too: the bar is much higher at all levels in terms of performance. So to be successful, as I will argue, to design successful autonomous vehicles, those vehicles are going to have to take risks.

And when the risks don't pan out, the public, if the public doesn't understand, the overall problem that we're addressing, the objective of the mission, those risks when they don't, the risks that are taken can have a significant detrimental effect on the progress in this autonomous vehicle space. So that's something we really have to think about. That's our role as engineers and so on. Ask if. So the question was: Do we know the rate of deaths per mile of vehicle driven, which is the closest level of how people think about safety? So there are about 80, 90, 100 million miles driven in manually controlled

cars

in each death.

So one death per person, depending on the numbers we look at, is 80 to 100 million miles. In the Tesla vehicle, for example, mortality is good: we could take the billion and divide it by three. Now, this is comparing apples and oranges and that's something we're working on to make sure we compare correctly. Compare the aspects of manual miles that are directly comparable to autopilot miles. Therefore, the autopilot is a modern and much safer vehicle. Tesla is a modern vehicle that is much safer than the general population of manual driving vehicles. Autopilot is driven primarily on a particular type of road on the highway, most of the miles.

The type of people who drive Autopilot and all of these types of factors need to be taken into account when comparing the two. But if you just look at the numbers, Tesla's Autopilot is three times safer than manually driven vehicles. But that's not the right way to look at it. And for anyone who has ever taken a statistics class, three deaths is not, is not a large number to draw meaningful conclusions from. However, that doesn't stop the media, the New York Times and everyone from responding to a single death, to which the PR and marketing aspects of these different companies are very sensitive, which of course is disturbing. and worrying for an engineer who wants to save lives.

But it's something we have to think about. Well, the 2018 review continued. There have been many announcements or rather actual launches of public tests of autonomous taxi services. So, the companies that on public roads have been delivering real people from one place to another. Now there are many warnings. In many of these cases it is on a very small scale, just a few vehicles, in most cases it is at very low speed, in a restricted environment, in a restricted community and almost always, really always, with a safe driver. There are some exceptions for demonstration purposes, but there is always a real driver in the seat.

Some of the brilliant people who represent these companies will be speaking in this course: Voyage does it in an isolated community, the incredible work they are doing in towns in Florida, Optimus Ride here in Boston and the community in Union Point, Drive.ai in Texas, May Mobility expands beyond Detroit, but actually most operations are in Detroit, Waymo has launched its service. Waymo, one that has received some publicity in Phoenix, Arizona. Let Nuro make zero-occupancy grocery deliveries autonomously. So we didn't say it had to deliver humans, but autonomously deliver groceries. Uber has quietly, or not so quietly, resumed its

self

-driving taxi service tests in Pittsburgh in a very careful and limited way.

Aptiv, after acquiring Carl Iagnemma and nuTonomy, has been conducting extensive testing of large-scale taxi services everywhere from Las Vegas to Boston, Pittsburgh and, of course, Singapore. Aurora, who spoke here last time, the head of Tesla Autopilot launched Aurora and Chris Urmson behind this young startup is testing in San Francisco and Pittsburgh and then Cruise, Kyle will be here to speak from GM, is testing in San Francisco. , Arizona and Michigan. So when we talk about predictions, I'll talk about some people who predict when we will have autonomous vehicles and when you your

self

think about what that means, when will they be here?

When will autonomous vehicles emerge so that that Uber you call is autonomous and not populated with a driver? So what we have to think about is what we think about what, how we define autonomy, what that experience is like. And the most important thing in these discussions is that we have to think about scale. So here at MIT, our MIT Human-Centered Autonomous Vehicle group, we have a fully autonomous vehicle that people can get into if they want and it will take them to a certain location. But that is a vehicle, it is not a service and it only works on certain roads.

It is extremely limited. In some ways, it's not much different from most of the companies we talk about today. Now scale here, there's a magic number, I'm not sure what it is, but for this, the purpose of this conversation let's say it's 10,000, where there is significant implementation, when you actually go beyond prototype demo mode to where that's all. under control, to the point where it is actually affecting the general population in fundamental ways. Scale is everything here and it starts at, say, 10,000. Just for reference, there are 46,000 active Uber drivers in New York City. That's what it feels like for some 10,000, you know, 25 or 30 percent of Uber drivers in New York City to suddenly become passengers.

So predictions, I'm not a marketing PR person, so I don't understand what everyone has to do to make a prediction, but everyone seems to understand it. Although major automakers have made a prediction of when they will have a deployment, when they will be able to deploy autonomous vehicles. Tesla made a prediction in early 2017 that it would have autonomous vehicles in 2018. In 2018 they have now adjusted the prediction to

2019

. Nissan, Honda and Toyota have made predictions for 2020 under certain limitations on urban roads. Hyundai and Volvo have done it in 2021. BMW and Ford, Ford says at scale, so a full-scale rollout in 2021.

And Chrysler in 2021 and Daimler says in the early 20s. Then there are the predictions that are extremely optimistic and perhaps driven by the company's instinct to declare that it is at the forefront of innovation. And then there are many of the leading engineers behind these teams, including MIT's Carl Iagnemma and Gill Pratt, who inject some caution and informed thinking about how difficult it is to take the human out of the automation loop. So Carl says that basically teleoperation gives this analogy of an elevator and the elevators are completely autonomous, but there is still a button to call for help if something happens.

And that's how he thinks about autonomous vehicles. Even with an increasing degree of automation, there will still need to be a human in the loop, they will still be a way to contact a human for help. And Gill Pratt and Toyota are making some announcements at CES, basically saying that the human in the loop is the fundamental aspect that we need to address this problem and removing the human from our consideration is a long, long way away. And Gill, who historically and currently is one of the world's great roboticists who defined many of DARPA's challenges and much of our progress historically speaking up to this point.

So they're really the full spectrum, we can think of it as Elon Rodney's spectrum of optimism versus pessimism. Elon Musk, who is extremely bold and optimistic about his predictions. I often connect with this type of thinking because sometimes you have to believe that the impossible is possible for it to happen. And then there's Rodney, also one of the great roboticists, the former director of CSAIL, the artificial intelligence laboratory here, he's a bit of a pessimist. So for Elon, now fully autonomous vehicles will be here in 2019. For Rodney, thevehicles are really, completely autonomous, they will be beyond 2050.

But there, he believes, that in the 30s there will be a major and important city that will be able to allocate an important region of that city where manual driving is totally prohibited. Here's how he thinks those vehicles could proliferate: Autonomous vehicles really proliferate when manually driven vehicles are banned in certain parts. And then in the 40s, in 2045 or beyond, most American cities will ban manually driven vehicles. Of course, Elon Musk's quote in 2017 is that, I guess, probably 10 years from now it will be very unusual for

cars

to be built that are not completely autonomous. So we also have to think about the long tail of the fact that many people drive cars that are 10 or 20 years old.

So even when all cars are built as fully autonomous, it will still take time for vehicle dissipation to occur. And so my own vision beyond the predictions, to pause briefly in the ridiculousness and fun of explaining the vision. Yes, that's me playing guitar in our self-driving vehicle. Now the point of this ridiculous and embarrassing video, I should have never played it. Yeah, okay, I think it will end soon. Now, for those of you who were born in the '90s, that's classic rock. (Audience laughs) So what I'm trying to point out beyond the predictions is that humans will not adopt autonomous vehicles anytime soon, in the next 10 to 15 years, because they are safer.

Safety won't do it, they may be safer, but they won't be so much safer that that's the reason they adopt. It won't be because they get you to the place faster. All we see with autonomy is that they will be slower until the majority of the fleet is autonomous. They are cautious and therefore slower and therefore more disruptive in the way we think about how we navigate this world. We take risks, we drive assertively with speed above the speed limit all the time. This is not how autonomous vehicles work today. So they're not going to get us there any faster and for every promise, every hope that they're cheaper, there's actually still a significant investment in them and there's no good economics in the short term on how to make them, obviously significantly . more economical.

What I believe is that Uber and Lyft have taken over the taxi service because of the human experience. In the same way, autonomy will only take over if it does not take over but is adopted by human beings if it creates a better human experience. If there is something in the experience that you really enjoy. This video and many others we are publishing show that in natural language communication, interaction with the car, the ability of the car to sense everything you are doing, from the activity of the driver to the attention and being of the driver. able to transfer control back and forth in a playful but really serious way personalized to you.

That's really the human experience, the efficiency of the human experience, the richness of the human experience, that's what we need to figure out as well. That's something to think about because many of the people who will be speaking in this class and many of the people who are working on this problem are not focused on the human experience. It's kind of an afterthought that once we solve the autonomous vehicle problem it will be a lot of fun to be in that car. I think we first have to make it fun to be in the car and then solve the autonomous vehicle problem together.

So in the language we're talking about here there are various levels of autonomy that are defined from level zero to level four. Level zero without automation, four and five, level three, four and five increasing automation. So level two is when the driver is still responsible, level three, four, five is when there is less and less responsibility. But really in three, four, five, there are parts of driving where the responsibility falls on the car. So as far as I'm concerned, there are really only two levels: autonomy of the human center and total autonomy. Human-centered means that human being is responsible.

Full autonomy means that the car is responsible both from a legal point of view and from an experience and algorithm point of view. That means that full autonomy does not allow teleoperation. Therefore, it does not allow the human to intervene and remotely control the vehicle because that means the human is still aware. It doesn't allow for the 10 second rule that it will be completely autonomous, but once it starts warning you, you have 10 seconds to take control. No, it is not fully autonomous if it cannot guarantee safety in any situation. You have to be able to do it, if the driver doesn't respond within 10 seconds you have to be able to find a safe harbor.

You have to be able to pull over to the side of the road without hurting anyone else to be safe. That is the totally autonomous challenge. So how do we envision these two levels of automation proliferating across society and being implemented on a massive scale? The 10,000, 10 million beyond. On the fully autonomous side, the way to think about it with the predictions we're talking about here is that there are several different possibilities for how to deploy these vehicles. One is last-mile delivery of goods and services such as groceries. These are zero occupancy vehicles that deliver food or deliver human beings in the last kilometer.

What the last mile means is a slow transportation to the destination where most of the complicated driving along the way is done manually and then the last mile delivery in the city in the urban environment is done by autonomous vehicles. zero occupancy. Road transport, possibly in platoons, where a sequence of trucks follow one another. So what people think of as a pretty well-defined problem of highway driving with well-marked lanes and well-mapped routes throughout the United States and around the world, highway driving is automatable. Specific urban routes are kind of what many of these companies are working on, defining this personalized taxi and public transportation service.

There are certain pickup locations you can go to, there are certain drop-off locations and that's it. It's like taking the train here, but unlike getting on the train with 100 other people or the bus, you get in the car, when you're alone or with another person. Gated communities, something that Voyage's Oliver Cameron and Optimus Ride are working on, define a particular community that you now have a monopoly on, you define the limitations, you define the customer base and then you simply deliver the vehicles. You map the entire road, you have slow transit that takes people from A to B to anywhere in that community.

And then there's the world of zero-occupancy rideshare delivery. So the Uber that comes to you, instead of you driving it yourself, it arrives autonomously with no one there and then you get in and drive it. So imagine a world where we have empty vehicles driving around and delivering themselves to you. The semi-autonomous side is thinking about a world where teleoperation plays a really crucial role, where it is completely autonomous under certain restrictions on the road, but a human can always intervene. High autonomy on the road, something like what Tesla is looking to achieve more recently. It is an entrance ramp to an exit ramp.

Now the driver is still responsible, in terms of responsibility and in terms of just observing the vehicle and algorithmically speaking, but the autonomy is at a fairly high level to the point where much of the driving on the road could be done fully autonomously. . And unrestricted low range travel as an advanced driver assistance system, which means cars are like the Tesla, the Volvo S90 or the Super Cruise and the Cadillacs, all these types of L2 systems that can keep you in the lane , you already know. 10 to 30% of the miles you drive and a fraction of the time take some of the stress out of driving.

And then there are some ideas, right? The idea of ​​connected vehicles, vehicle-to-vehicle communication, and vehicle-to-infrastructure communication allows us to navigate, for example, an intersection efficiently without stopping, eliminating all traffic lights. So here shown at the bottom is our conventional approach that there is a queuing system that is formed due to traffic lights turning red, green, yellow and without traffic lights and with communication to the infrastructure between the vehicles, it is actually you can optimize that to increase significantly. the traffic load through a city. Of course, there is the boring solution of tunnels under cities, layers of tunnels under cities.

Tunnels to the end. Autonomous vehicles, basically by the design of the tunnel, limit the problem to such an extent that, I mean, the idea of ​​autonomy just completely transforms. Basically, a car is capable of transforming into a mini train, a mini public transport entity, for a certain period of time. So you go into that tunnel, you drive 200 miles per hour and or you don't necessarily drive, you get driven 200 miles per hour and then you come out of the tunnel. Of course, there are flying cars, custom flying vehicles. I won't, I mean, Rodney, as I mentioned before, believes that we will have them in 2050.

There are a lot of people who are thinking seriously about this problem: obviously there is a level of autonomy that is required here for a person. I don't know anyone who doesn't have, for example, a pilot's license to be able to take off and land. Making that experience accessible to everyday people means there will be a significant amount of autonomy involved. One of the people really, one of the companies that is really working seriously on this, is Uber with Uber Elevate, I think it's called Uber Air and the idea is that you find your vehicle not on the street but on a roof, you take the elevator, you meet them on the roof of a building.

This video is from Uber. They are seriously addressing this issue. At some point they have laughed at many of the great solutions to the world's problems. So we don't laugh too much at these possibilities. In my days we drove down the street. Well, huh, 10,000 vehicles, if that's the bar. I asked out of curiosity and did a small public survey. 3,000 people responded. He was asked who will be the first to deploy 10,000 fully autonomous cars that drive on public roads without a safety driver. And several options were leaked: Tesla got 57% of the vote and Waymo got 21% of the vote, 14% from someone else and 8% from the curmudgeons and engineers who said no one in the next 50 years would. will do.

And again in 1998, when Google came out, the leaders in the space were Ask Jeeves, Infoseek and Excite, all services that I've used and probably some people in this room have used, Lycos, Yahoo, obviously they were the leaders in the space. and Google completely disrupted that space. This survey shows the current leaders, but it is very open to ideas and that is why there are many autonomous vehicle companies. Some companies are taking advantage of the hype and the fact that there is a lot of investment in the space, but some companies, like some of the speakers who visited this course, are really trying to solve this problem.

They want to be the next Google, the next billion-dollar, multi-million dollar, trillion-dollar company solving the problem. So it's completely open. But currently Tesla with a human, with the semi-autonomous vehicle approach working to try to become fully autonomous. And Waymo, starting with fully autonomous and working toward scale in fully autonomous, are the leaders in the space. Given that, classified in 2019, let's quickly rewind to 2005 with the DARPA challenge when the story began. The Race to the Desert, when Stanford's Stanley won a race through the desert that really captured people's imaginations about what is possible. And a lot of people have said that the autonomous vehicle problem will be solved in 2005.

They actually said the idea was especially because in 2004 no one finished that race, in 2005 four cars finished the race, it was like we solved it. That's all. And then you know that some critics said that city driving is actually nothing comparable to desert driving, the desert is very simple, there are no obstacles, etc. It's really a mechanical engineering problem, not a software problem. It's not fundamentally, it's not really an autonomous driving issue as it would be delivered to consumers and of course in 2007, DARPA hosted the Urban Grand Challenge and several people finished it with the head of CMU winning.

And then the thought was: that's it, we're done. As physicist Ernest Rutherford said, physics is the only real science, the rest is stamp collecting, all biology and chemistry. Certainly, wow, I wouldn't want to know what he thinks about computing. Are all thesestupid and silly details. Physics is the fundamental thing. And that was the idea with the DARPA Grand Challenge and by solving it we solved the fundamental problem of autonomy. And the rest is just for the industry to figure out some of the details of how to build an app and turn it into a business. So that could be true.

And the underlying belief is that driving is an easy task, which has a solution. What we do as human beings is quite formalizable and is quite easy to solve autonomously; The other idea is that humans are bad drivers. This is a common belief. Not me, not you, but everyone else, no one in this room except everyone else is a terrible driver. The kind of intuition we have about our experience of traffic leads us to believe that humans are really bad at driving. And from the point of view of human factors, psychology, there has been over 70 years of research that shows that humans are not capable of monitoring, maintaining surveillance, monitoring a system.

So when you put a human in a room with a robot and say, look at that robot, they start texting within like 15 seconds. That is fundamental psychology. There are thousands of articles about this. People tune out, trust the system too much, misunderstand it, and lose vigilance. Those are the three underlying beliefs. It may very well be true, but what if it isn't? So we have to consider that it is not. The task of driving is easy because if you believe that the task of driving is easy, formalizable, and solvable by autonomous vehicles, you must solve this problem.

The subtle non-verbal vehicle-to-vehicle and vehicle-to-pedestrian communication that happens here in a dramatic sense but actually happens in a subtle sense millions of times every day in Boston. Subtle non-verbal communication between vehicles, you go, no, you go. You have to solve all the crazy road conditions where in a split second you have to make a decision, so in snow, ice, rain or limited visibility conditions, you have 100 or 200 milliseconds to make a decision. Your perception-based algorithm has to make a control decision. And then you have to deal with non-verbal communication with pedestrians, these irrational, irrational creatures that we humans are.

You not only have to understand what the intention of the movement that is anticipated is. So, anticipating the path of the pedestrian, you also have to assert yourself in a form of game theory, as crazy as it may seem, you have to threaten yourself, you have to take a risk. You have to take the risk that if I don't slow down like the ambulance didn't, the pedestrian will slow down. Algorithmically we are afraid to do that. The idea that a pedestrian who is moving, we anticipate his trajectory based on the simple physics of the current velocity of the impulse, will continue forward with some probability.

The fact that by accelerating we can make that pedestrian stop is something that we have to incorporate into the algorithms and we do not do it today. And we don't really know how to do it. So, if driving is easy, we have to solve that too. And of course what I showed yesterday with the coastal corridors and the boat going around and all the ethical dilemmas, from the moral machine to the more serious engineering aspects, from the unintended consequences that arise from having to formalize the objective function under which planning is carried out. the algorithm operates.

If there is some learning that, as I showed yesterday, a boat on the left led by a human wants to finish the race, the boat on the right realizes that it doesn't have to finish the race, it can take turbos along the way and you get much more reward. So if the objective function is to maximize the reward, you can hit the wall over and over again and that's actually the way to optimize the reward. And those are the unintended consequences of an algorithm that must be formalized into the objective function without a human being involved.

Humans are bad at driving. As I showed yesterday, if humans are bad at anything, it's having a good intuition about what is difficult and what is easy. The fact that we have 540 million years of data in our visual perception system means that we don't understand how impressive it is to be able to perceive and understand the scene in a fraction of a second, maintain the context, maintain the understanding of how to do all the things . visual localization tasks about anticipating the physics of the scene, etc. And then there is the control side. We humans don't give ourselves enough credit.

We are incredible, the next-generation soccer player on the left (audience laughs) and the next-generation robot on the right. I think he scored four or five times (audience laughs). And this is all the movement and stuff involved with that, of course, here's the human robot, it's some really amazing work that was done for the DARPA Robotics Challenge with the humanoid robots on the right and some amazing work done by human people doing the same. type of tasks, much more impressive task, I would say. So that's where we find ourselves. And those on the right are actually not completely autonomous, there is still some human being in the loop.

There is simply noisy and broken communication. So humans are incredible in terms of our ability to understand the world and in terms of our ability to act in that world. And the fact that humans, the idea, the view, the popular psychology-based view that humans and automations don't mix well, because of trust, misunderstanding, loss of vigilance, command, etc. , it is not an obvious fact. It happens a lot in the lab. Most experiments are done in the laboratory. This is the difference. Many of you put an undergraduate or graduate student in a lab and say here, look at this screen and wait for the dot to appear.

They'll tune out immediately, but when it's your life and you're on the road, it's just you in the car, it's a different experience. It's not completely obvious that vigilance will be lost and it's not completely, when it's just you and the robot, it's not completely obvious what the psychology is, what the attention mechanism is, with the vigilance it seems. So one of the things we did was we instrumented 22 Teslas here and observed people over a two-year period about what they actually do when they're driving autopilot, driving these systems. Manually controlled vehicles are shown in red and the vehicle control autopilot is shown in cyan.

Now there's a lot of detail here and we have a lot of presentations on this, but really, the bottom line is that they drive 34%, a large percentage of the miles on autopilot and in 26,000 transfer of control moments they are always attentive. There is not a single moment in this data set where they respond too late to a critical situation, to a challenging role situation. Now the data set, 22 vehicles, is 0.1% or less than Tesla's entire fleet that has Autopilot. But it's still an indication. It is not obvious that it is not possible to build a system that works together with a human being and that system essentially looks like this.

Some percentage, 90%, maybe less, maybe more, when they can solve the problem of autonomous driving they solve it and when they need human help they ask for help. That is the compensation, that is the balance. On the completely self-contained side, right you have to resolve here with citations and there are always references at the bottom. All problems must be solved exceptionally and perfectly, from mapping localization to scene perception, control, planning, the possibility of finding a safe harbor at any time and also the possibility of establishing external HMI communication with others. pedestrians and vehicles on the road. scene and then there is teleoperation, vehicle to vehicle, vehicle to AI.

You have to solve them perfectly if you want to solve the problem of full autonomy, like I said, including all the crazy things that happen when driving. And if you approach the shared autonomy side, the semi-autonomous side where you are only responsible for a large percentage but not 100% of the driving, then you have to solve the human side, the human interaction, the feeling of what the driver is doing. driver. , the collaborative communication with the driver and the personalization aspect that learns with the driver. Like I said, you can go online, we have a lot of demos of these types of ideas.

But I think natural language, communication, is fundamental for all of us, since we tweet like we all do. (people chatting) So it's as simple as, this is just a demonstration of how Eco takes control when attention over time, that the driver is, okay, we've got it, thanks. Well, basically smartphone usage has increased year after year and we are doing a lot of analysis on it. In reality, what people do in the car is use their phone, whether in manual, autonomous or semi-autonomous driving. . So, being able to manage that, communicating with the driver when he needs to pay attention, which may not always be the case.

You're sort of balancing when it's a critical time to paying attention when it's not and communicating effectively, learning with the driver, that problem is a fundamental machine learning problem. There is a lot of visible data, all about the driver and it is a psychological problem. We have data, we have complicated humans, and it is a human-robot interaction problem that deserves a solution. But as you'll hear beyond the human side looking out into the world, people who are trying to solve the fully autonomous vehicle is really a two-pronged consideration. One approach is vision, cameras, and deep learning, right?

Collect a large amount of data. So the cameras have this aspect that they are the highest resolution of information available. It's rich information about textures and there's a lot of it, which is exactly what you know the networks love. So to be able to cover all the edge cases, the vision data, the camera data, the visible light data, is exactly the kind of data that you need to collect a lot of, to be able to generalize over all the countless edge cases. that happen. It is also feasible, all major data sets, all, in terms of cost, interest, scale, all major data sets of visible light cameras.

That's another advantage and they're cheap and the world as it happens, whoever designed the simulation that we all live in, made our world, our roads and our world, designed for human eyes. Eyes are how we perceive the world and therefore lane marking is also visual, most of the road textures that you use to navigate, to drive are visible and made for human eyes. The disadvantages are that without a lot of data and we don't know how much, they are not accurate. You make mistakes because ultimately driving is 99.99999% accurate and that's what I mean by not accurate.

It's really difficult to get to that level. And then the second approach is LIDAR, which takes a very particular restricted set of roads, maps all of them, fully understands them in different weather conditions, etc., and then uses the most accurate sensors available. A suite of sensors but really LIDAR at the forefront. Be able to locate yourself effectively. The advantages are that it is consistent, especially when machine learning is not involved, it is consistent, reliable and explainable. If it fails, you can understand why, you can account for those situations. This is not as true for machine learning methods.

It is not so explainable why it failed in a particular situation. The precision is greater as we will talk about. The disadvantages of LIDAR are that it is expensive and most approaches to perceiving the world using LIDAR primarily do not rely on deep learning and therefore do not learn over time. And if they were based on deep learning, there's a reason they're not, it's because you need a lot of car, you're going to need a lot of lidar data. And obviously only a small percentage of cars in the world are equipped with LIDAR to collect that data.

So fast running through the sensors. The radar is something like the offensive line in football. In reality they are the ones who do all the work and never get the credit. So that's what radar is. It's always behind to catch, to really do the detection in terms of obstacles, the most safety-critical obstacle avoidance. It's cheap, works extremely well, and performs well in extreme weather conditions, but it has low resolution, so it can't achieve any kind of high degree of autonomy on its own. Now on the LIDAR side it's expensive, it's extremely accurate depth information, 3D cloud, point cloud information.

Its resolution is much higher than radar, but still lower than visible light, and depending on the sensor, there is 360-degree visibility built in. So there is a difference in resolution here, displayed LIDAR on the right, radar on the left. The resolution is much higher and improving and the costis going down, etc. Now on the camera side, it's cheap, everyone has one, the resolution is extremely high in terms of the amount of information transferred per frame and you all really know the scale of the number of vehicles that have it equipped is enormous. Therefore, it is ripe for the application of deep learning.

And the challenge is that it's noisy, it's not good at estimating depth, and it's not good in extreme weather conditions. So if we use this graph to look, compare these sensors, compare these different approaches. So LIDAR works in the dark, in variable lighting conditions, it has pretty good resolution, pretty good range, but it's expensive, it's huge and it doesn't provide rich texture contrast information and it's also sensitive to foggy conditions and rain. Now ultrasonic sensors detect many of these problems. They are better at detecting proximity, they have high resolution on objects that are close, so they are often used for parking, but they can also be integrated into the sensor fusion package for an autonomous vehicle.

They really detect many of the problems that radar has. They complement each other well and the radar, cheap, small, detects speed and has a fairly good range but terrible resolution. Very little information is provided. And then the cameras are very rich in information, they are cheap, their small range is excellent, actually the best range of all the sensors and it works in bright conditions but it doesn't work in the dark, in extreme conditions and it is just susceptible. to all these kinds of problems and it doesn't detect speed unless you make some complicated structure out of movement.

So this is where the sense of fusion comes in and everyone works together to build a complete image. That's how this plot works. You can stack them on top of each other. So if you look at a set that, for example, Tesla is using, which is an ultrasonic radar and a camera, and you compare it with just LIDAR and see how these paths compare, actually the camera, radar and ultrasound set are comparable to LIDAR. So those are the two comparisons we have. It has the expensive LIDAR way without machine learning and it has the cheap way, but it needs a lot of data and is not explainable or reliable in the short-term vision based approach.

And those are the two competing approaches. Now, of course, Eggs will talk about how they're trying to use both, but ultimately the question is who gets the catch, who's safe? In the semi-autonomous way, when there is a camera-based method, the human being is the one that is fail-safe. When you say, oh shit, I don't know what to do, the human gets you. In fully autonomous mode, which Waymo and others are working on, the security system is LIDAR, the security system is maps that cannot be trusted by humans. But you know this wayWell, if the camera gets scared, if there are any of the sensors that you can scare, you have such good maps, you have such good and accurate sensors, that the fundamental problem of avoiding obstacles, which is what security is all about , can be resolved.

The question is what kind of experience is created. Meanwhile, as people debate, try to make money, start businesses, there's tons of data. The Ford F-150 remains the most popular car in the United States. Manually driven cars still exist. So there's a lot of data happening. Semi-autonomous cars, all companies are launching more and more semi-autonomous technology. That's all the data. And that comes down to the two paths they're walking down: vision versus LIDAR, L2 versus L4, semi-autonomous versus fully autonomous. Tesla on the semi-autonomous front has reached 1 billion miles. Waymo, the leader on the autonomous front, has reached 10 million miles.

The pros and cons as I have described them. One, division one, which I'm obviously very excited about and which excites machine learning researchers, which is fundamentally based on big data and deep learning. The neural networks that run inside the Tesla and with its new features, are more or less the same type of path that Google was taking from the GPU to the TPU, the one that Tesla took from the Nvidia Drive PX2 system, a kind of more generally GPU-based. to creating your own ASIC and having a bunch of awesome neural networks running in your car. It's really interesting to think about that kind of path, which others are starting to take, for machine learning engineers.

And then people who are maybe more grounded and really want, are really, value, safety, reliability and something like that from the automotive world, think we need machine learning, it can't be explained, it's hard to work with, it's not reliable, etc In that sense, we have to have a set of sensors that are extremely reliable. Those are the two paths. If he asks. The question is that there are all kinds of things that you need to perceive, stop signs and traffic lights, pedestrians, etc. Some of them, if you hit them it's a problem, some of them are a bag flying through the air and they all have different visual characteristics, they all have different characteristics for different sensors.

So LIDAR can detect solid body objects, the camera is better at detecting, as Sasha Arnu talked about last year, I think fog or smoke. These are interesting things. They may appear to be an object to certain sensors and not to others, but fortunately the traffic light detection problem with cameras is pretty much resolved at this point. Fortunately, that's the easy part. The difficult thing is when you have a green light and there is a drunk, drugged, drowsy or distracted, the four D's that run over a pedestrian in line trying to cross what to do. That's the hard part.

So the path ahead of us as engineers, the science, is what I'm really excited about, the possibility of artificial intelligence having a big impact, is taking the step of having them, even if they're big, toy data sets. , toy problems, The ImageNet ranking toy benchmarks in cocoa, all the exciting and deep RL stuff we'll be talking about in the next few weeks, really toy examples, the game of go and chess, etc. . But taking those algorithms and putting them in cars where they can save people's lives and actually directly touch and impact our entire civilization, that's actually the defining problem for artificial intelligence in the 21st century: AI affecting people of all kinds. a real way and I think cars. , autonomous vehicles, is one of the big ways that happens.

We deal with the aspects of psychology, philosophy and sociology, how we associate it, how we think about it, with the problem of robotics, with the problem of perception. It's a fascinating space to explore and we have a lot of guest speakers exploring in different ways and it's really exciting to see how these people are trying to change the world. That being said, I would like to thank you very much. Go to deeplearning.mit.edu and the code will always be available online. (people applauding)