Re: [SystemSafety] Static Analysis

From: Patrick Graydon < >
Date: Mon, 3 Mar 2014 08:02:45 +0100

Hmm. While my (possibly ill-informed) opinion is that the non-safety world over-uses a try-it-and-see approach, I wonder if we can categorically say that try-it-and-see is /never/ appropriate in safety.

Here’s my thought experiment. Suppose that you work for a car company and you come up with some brilliant idea for an active safety system that will help to save drivers from themselves. You can (and, I think we agree, should) engineer your implementation such that the risks associated with predictable system and component failures are well-managed*.

But the actual risk that drivers will experience in the field depends a lot on how the dodgy, unpredictable meat component reacts to the addition of this new system. Automation fatigue might set in. Drivers might drive faster, brake later, etc., trusting the new technology to save them. Etc.

So you set up some simulators, recruit a few dozen test subjects to drive in simulated environments, and try to see if there is a difference between an unmodified simulator and one modified to reflect your new gizmo. Your technology seems to substantially reduce aggregate risk, but a simulator is a simulator and the participants know it**. So you move on to a prototype on a closed test track with extra people around to monitor what’s going on, a system to remotely brake the car if needed, medics around to provide medical treatment if something goes horribly wrong, etc. Again, your study shows good news, but again this is not the real world and so the study is not perfectly convincing.

Every method of assessing how drivers will react to the technology has a key weakness***: the drivers know that they are in a study, not the real world, and so might react differently. Certainly they are less likely to discipline their children in the back seat or ring someone on their mobile while your camera crews are watching. But every sincere effort to make an unbiased assessment of the effect of the system shows that introducing it would reduce aggregate risk to the motoring public****.

Morally, ethically, should your company:
(a) Not release your new technology until it can accurately***** assess total risk, including contributions from how drivers will use it in practice
(b) Release your new technology on a few selected vehicles, monitor real-world use as closely as privacy regulations allow, and (i) recall those cars, (ii) phase out the technology, or (iii) roll the technology out more broadly as the real-world results become clear
(c) Release the new technology on every model you can (monitoring as closely as practicable, as per [b]) because the best information in hand suggests that this will save lives

If it is /never/ appropriate to try it and see, the answer must be (a). But, speaking for myself only as an occasional driver, I’d rather go with (b) or (c) as these seem paths to quicker overall risk reduction.

Disagreement welcome, of course.

— Patrick

Dr Patrick John Graydon
Postdoctoral Research Fellow
School of Innovation, Design, and Engineering (IDT) Mälardalens Högskola (MDH), Västerås, Sweden

Received on Mon Mar 03 2014 - 08:03:02 CET

This archive was generated by hypermail 2.3.0 : Sat Feb 16 2019 - 07:17:06 CET