Autonomous car makers hand over data on glitches and failures to California DMV

Google, Delphi, Tesla, and others submit their first disengagement reports.

Megan Geuss – Jan 13, 2016 4:45 AM | 61

Delphi's autonomous vehicle. Credit: Delphi

If you want to build a self-driving car and test it on public roads in California, the state’s Department of Motor Vehicles says that every year you have to submit a disengagement report—basically a list of every time the human driver had to take over for the car. This year, Bosch, Delphi, Google, Nissan, Mercedes-Benz, Tesla, and Volkswagen Group were required to submit disengagement reports, and the results are largely what you’d expect from a novel and complicated technology.

Google, as the company that has driven the most miles on public roads in California, said it experienced 341 significant disengagement events over 424,000 miles of driving (PDF). Similarly, Nissan reported that it drove 1,485 miles on public roads in California, and it experienced 106 disengagements. Delphi’s two autonomous vehicles drove 16,662 miles, and the company reported 405 disengagements. Tesla, for its part, reported no disengagements (PDF) from fully autonomous mode from the time it was issued a permit to test self-driving cars in California.

While it’s tempting to use those numbers as a comparison point as to how good a company’s autonomous vehicles are, there are many variables that could obscure an otherwise accurate comparison. The numbers only reflect miles driven on California roads and disengagements that happen in that state. If a company primarily tests its public road driving in another state, those numbers won’t be reflected in these reports.

Also, as Google notes, the technology is still in development, so engineers don’t necessarily want to keep disengagements from happening because every mistake is a chance to build more fail-proof software. "Our objective is not to minimize disengagements; rather, it is to gather, while operating safely, as much data as possible to enable us to improve our self-driving system,” Google writes.

Google also offers the most detail of all the automakers as to why the disengagements occurred. 272 disengagements happened when the car detected some sort of anomaly that it could not reconcile—these include things like a broken wire, anomalies that come from accelerometer or GPS readings, or "anomalies in the monitoring of key functions like steering and braking.”

69 of Google’s disengagements happened when the driver was forced to take control of the car—in these cases, Google determined that without intervention from the driver, the car would have crashed with another car or an object. The Guardian points out that Google actually says that its drivers took over “many thousands of times,” but “the company is reporting only 69 incidents because Google thinks California’s regulations require it to only report disengagements where drivers were justified in taking over, and not those where the car would have coped on its own."

The other automakers did not include as much analysis in their reports. Mercedes-Benz lists (PDF) all of its disengagement events with a short comment on the cause for the disengagement: “technology evaluation management” and “driver was uncomfortable” seem to be the two most-used reasons for stopping the autonomous drive. Volkswagen Group similarly uses vague language (PDF) to describe its autonomous drive failures. Bosch reported that (PDF) it only ever disengaged autonomous mode for a “planned test of technologies.”

Nissan and Delphi both had more robust reports of what happened and why the disengagement was needed. “A route could not be generated due to a localization error,” Nissan wrote in one instance (PDF). “Autonomous control shut off and the driver resumed manual control of the AV [Autonomous Vehicle].”

”AV did not recognize stopped vehicle in front of it,” another entry noted. “The driver overrode the system with manual brake input, causing the control to disengage.”

Delphi even broke down the causes (PDF) for disengagement in a table. Among the reasons that the test drivers disengaged the company's two autonomous vehicles were to make way for “emergency vehicles,” as well as “poor lane markings,” “precautionary intervention to give extra space for a cyclist,” and “traffic light detection” caused by “poor sun conditions.”

The Guardian noted that when the California DMV was drafting rules to allow testing of autonomous vehicles on its roads, Google lobbied hard against the requirement to report disengagement numbers. Separately, companies that test autonomous driving on public roads are required to report any accidents that occur to the DMV. In June of this year, Google reported that it had been in 13 minor accidents with its self-driving cars, none for which the self-driving car was at fault.

nproxy.org