Insurers: Beware self-driving cars' AI misjudging road risks

• Key insight: Insurers should rethink AI risks in autonomous vehicles
• Forward look: Insurers will change how they cover AVs

Autonomous vehicles' use of AI changes how insurers evaluate the risks, especially when the decision process is not properly documented, according to experts in underwriting and insurance risk.

An AI piloting AVs makes decisions in dangerous situations differently than human operators, said Nick Gibbs, class leader at Apollo, a Skyward Group insurance platform operated through Lloyd's of London.

"They have developed AI to perform these autonomous vehicle tasks," he said. "If there's an accident, it may be that an AV has to make a decision as to whether it's going to hit one person or hit another, or injure the passenger."

John Shetron

AI models for AVs that are trained on typical driving scenarios "can potentially misjudge unusual situations or conditions," said Patrick Schmid, chief insurance officer of the Insurance Information Institute. "When that vehicle is going to make a bad call, it's hard to know why the AI made that choice because the decision-making construct associated with it is not as clear. It's not as binary."

Figuring out what an AV's AI has done in a risky situation can be difficult to determine, Schmid said. 

"Insurers don't necessarily have visibility into the AI system's decisions and its logs of data," he said. "Without that data, in a claims investigation, what did AI perceive and why did it act in a certain way? Pricing on that becomes a little bit harder than with conventional driving or even telematics."

When AVs get into accidents, insurers have to figure out whether it is an auto-claim loss, a product=liability issue or even a cybersecurity loss if it involved a disruption in over-the-air communication with the AV, Gibbs said. (Some AV services use remote operators to guide or assist the vehicles.)

Insurers must also consider whether an AV is using AI for any of three functions, Schmid said: 

• Perception. Reading information collected from cameras or radar on the AV. 
• Prediction. Anticipation of nearby vehicles' behaviors, whether human-operated or automated.
• Planning. Choosing and carrying out the AV's next maneuver or action.

"Each of these different layers is a distinct AI model with its own failure modes," said Schmid. "Perception could have a misclassification of what it sees. Prediction could potentially misjudge another road user's movements or intent. Planning to select a safe maneuver can be based on a flawed read of what's occurring on the road."

AI in AVs uses probability and data from sensors to make judgments, Schmid said. "That sets the stage for these edge-case failures, limited explainability, fleet-wide risk and cybersecurity exposure," he said.

In some potential accident scenarios, since developers of an AV's AI may have programmed the vehicle to weigh risks of different actions, insurers should act accordingly, Gibbs said. "Therefore, we need to refine the standard policy terms to make sure that they're fit for purpose," he said.

Expect significant change in how insurance policies for AVs are written, Schmid said. 

"Staying abreast of AI policy angles, regulatory angles related to AI, what the NAIC [National Association of Insurance Commissioners] is doing, and trying to understand how it's going to impact risk is pivotal," he said. 

The NAIC has Model AI Bulletin guidance and an AI evaluation initiative.