New AI Tool Explains Why Self-Driving Cars Make the Decisions They Do

Self-driving cars are often controlled by deep learning models that sometimes fail in unexpected situations — for instance, a car might inexplicably brake and block the path of an oncoming emergency vehicle. Researchers from MIT and autonomous vehicle company Motional have developed a method called the Concept-Wrapper Network, or CW-Net, that explains such decisions using concepts a person can understand, without altering the vehicle's driving performance. The research, led by Eoin Kenny, a former MIT postdoc now at J.P. Morgan Chase, and MIT professor Julie Shah, appears in the journal Nature.

CW-Net is a "concept classifier" plugged into the middle of an autonomous vehicle's existing machine-learning planner. It translates the model's internal reasoning into concepts like "approaching stopped vehicle" or "close to cyclist," then forces the final stage of the planning model to use those concepts when deciding what the car should do next. "Especially in high-stakes settings like self-driving cars, it's important that the explanations are not potentially misleading," Kenny said. "Because CW-Net is causally faithful in how it makes decisions, that provides certain guarantees around the explanations."

The researchers trained CW-Net to recognize concepts using a dataset of 130 million labeled scenes from self-driving cars, and designed it to mimic the decisions of existing planners so it would not hurt vehicle performance. The module outputs its explanations in real time alongside the car's planned trajectory. "Instead of just wondering why the car stopped, having real-time data provides feedback that lets you test the system during deployment," Kenny said.

In road tests on a private track with a Motional robotaxi, CW-Net helped a safety driver more accurately predict the vehicle's behavior, and a larger simulation study with nonexpert users found similar results. In one case, the vehicle consistently stopped when approaching a cyclist, and the safety driver assumed it had detected the cyclist. CW-Net's explanations revealed instead that the model was not properly configured to detect the cyclist and had chosen a trajectory that would have caused a collision; the car stopped only because its emergency braking procedure kicked in when it got too close.

"This work shows how explanations are supportive to the human's mental model and understanding of the behavior of a system, and how it could be used in engineering and development to improve the technology," said Shah, director of the Interactive Robotics Group at MIT's Computer Science and Artificial Intelligence Laboratory. "Unless we are building these technologies in a way that we can rely on and predict their behavior, then it is a shaky and unsafe foundation for their use."