Re: [SystemSafety] NYTimes: The Next Accident Awaits

From: Matthew Squair < >
Date: Tue, 4 Feb 2014 14:06:20 +1100

>On Mon, Feb 3, 2014 at 7:07 PM Patrick Graydon <patrick.graydon_at_xxxxxx wrote

> I don't see how those experiments (either the original or the follow-up
work) are particularly relevant.

Apologies for the late response, I blame the time difference and pressures of work.

Before I start, my definition of a safety case is 'a documented body of evidence that provides a convincing and valid argument that a system is adequately safe for a given application in a given environment'. Usually (but not always) it includes graphical representations.

The original Fischoff, Slovic and Lichtenstein 1978 study (on DTIC) was funded by DARPA and looked at fault trees because of the WASH 1400 reactor safety study. As the authors pointed out fault trees were the primary methodological tools for that study, hence their interest in whether the technique was sound. As graphical techniques such as fault trees, event trees, master logic diagrams and goal structuring notation etc etc are the primary tools we use to analyse and express the 'safety' of systems I think it's still relevant.

The out of sight, out of mind affect they found is not I suggest a small issue that can be easily resolved, it's actual a persistent and significant problem even for experts, as Fischoff et al found in the original study when they asked experts to review the provided tree. To quote "The most dramatic result of these studies was subjects' inability to appreciate how much had been omitted from the pruned fault trees". So the effect is evident in experts review of evidence presented in such a fashion and is still pertinent to independent assessment by any expert.

Greenwell, Knight, Holloway and Pease (2006) have reviewed a number of safety cases for logical fallacies, they found omission of evidence in all three of the safety cases studied, the Opalinus Clay repository, EUR Whole Airspace (preliminary study) and EUR RVSM (pre implementation). As a specific example of what they found in, in the case of the EUR Whole airspace argument they found it failed to consider possible interactions between geographical areas, while for the Opalinus clay study they found that the selection of uncertainty scenarios was based on expert judgement with no evidence as to which were rejected. Peter Ladkin has also separately published an analysis of the RVSM safety case, which looked at how that case failed to address the safety of the system as it was actually going to be operated. These are not to my mind nickel and dime type issues.

All of the cases examined had been published, with I presume the review and blessing of the associated regulator, yet these omissions escaped?

Maybe we need a meta analysis of all the work done.

>...whether a safety case regime systematically accepts more shoddy systems
after regulator/ISA review than a so-called 'prescriptive' system would...

Rather what is the likelihood that we don't know that we have a problem in either regime, and why?

>As to arguments that a system is unsafe, could you explain how that would

Well like Popper said, science advances on the basis of disconfirming evidence. So at the organisational level apply the approach used in security where a red team is constituted to specifically find a weakness in the system, which would offset the,"all that we've covered is all that there is" syndrome. Or at the methodology level the use of a technique such as TRIZ during hazard identification to reverses how we look at a system. Or during testing to emphasise tests that will break the system, regardless of whether the test is reasonable, then looking at what that tells us about predicted behaviour.


On Mon, Feb 3, 2014 at 7:07 PM, Patrick Graydon <patrick.graydon_at_xxxxxx

> On 3 Feb 2014, at 02:36, Matthew Squair <mattsquair_at_xxxxxx >
> > There is for example experimental evidence going back to Slovic and
> Fischoffs work in the 70s and Silveras follow up work in the 00s on how the
> structuring of fault trees can lead to an effect known as omission neglect,
> see here ( for further discussion of the effect. I
> see no reason why such graphical techniques as GSN should be immune to the
> same problem, or safety cases in the broader sense.
> I don't see how those experiments (either the original or the follow-up
> work) are particularly relevant. In all of them, the subjects were given
> the fault trees and told to use them as an aid to a subsequent task; the
> experimenters were measuring how presentation in them biased their
> performance in that task. But in none of them was anyone explicitly tasked
> with checking the given fault trees, as an ISA or a regulator would a
> safety case. Because no-one took on the role of a skeptical critic, I
> don't see the experimental context as particularly analogous to safety-case
> regulatory regimes.
> Moreover, if this was really to weigh in on the question of whether a
> safety case regime systematically accepts more shoddy systems after
> regulator/ISA review than a so-called 'prescriptive' system would, the
> experimental context would have to clearly be more analogous to the context
> of one of those than the other. But in *both* we have people presenting
> information (that might be framed one way or another) to
> regulators/assessors.
> Don't get me wrong, I am not claiming to have the answer here. But I find
> the evidence that has been offered to date so weak as to be useless. I
> second Drew's call for serious, systematic study of this.
> As to arguments that a system is unsafe, could you explain how that would
> work? Trying to discover all of the ways that a system is dangerous is a
> good way to find them, as trying to discover all of the ways that an
> argument is flawed is how we find flaws in arguments (safety and
> otherwise). But what are the criteria on which we decide whether something
> is good enough?
> This approach seems to be a case of demonstrating a negative. In an
> inductive argument, you do this by showing how many possibilities you have
> examined and discarded. E.g., if I wanted to claim that there are no
> Ferraris in my bedroom, I could back that up by claiming that I have looked
> into every space in that room big enough to hold one in such a way that I
> would expect to see one if it was there and that my search revealed
> nothing. In the case of safety, wouldn't you have to argue over how you'd
> gone about looking for hazards (and dealt with all you'd found), how you'd
> gone about looking for causes to those (and dealt with all of those), how
> you'd gone about verifying that your system as deployed did what your
> analysis (and the resulting safety requirements) required, etc. This
> sounds an awful lot to me like the standard guidance for safety case
> structure. Or do you have something else in mind?
> -- Patrick

*Matthew Squair*

Mob: +61 488770655
Email: MattSquair_at_xxxxxx
Website: <>

_______________________________________________ The System Safety Mailing List systemsafety_at_xxxxxx
Received on Tue Feb 04 2014 - 04:06:31 CET

This archive was generated by hypermail 2.3.0 : Mon Feb 18 2019 - 10:17:06 CET