- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ]

From: Steve Tockey < >

Date: Sat, 2 Apr 2016 05:37:53 +0000

The System Safety Mailing List

systemsafety_at_xxxxxx Received on Sat Apr 02 2016 - 07:38:06 CEST

Date: Sat, 2 Apr 2016 05:37:53 +0000

Bev wrote:

"This is indeed interesting. But what was the the definition of “efficiency of fault detection” here? Finding the most faults for a given outlay of effort? Maximising the chance of finding all faults? I can imagine other definitions."

Generally speaking, efficiency is looking at investment per unit of work done. In other words, increasing efficiency means that the same work is done at lower investment—the same (number of?) defects are found for a lower investment. Maximizing the chance of finding defects would be an issue of effectiveness. Effectiveness is looking at the rate at which the job is done correctly (I.e., a defect is found, not missed). One needs to look at both efficiency and effectiveness of processes to make a fair comparison.

- steve

Date: Wednesday, March 30, 2016 6:48 AM

To: David MENTRE <dmentre_at_xxxxxx
Subject: Re: [SystemSafety] Data on Proof effectiveness from real projects

Hi David

On 30 Mar 2016, at 07:39, David MENTRE <dmentre_at_xxxxxx

And there is also King et al. paper "Is proof More Cost Effective than Testing?" on SHOLIS project.

Interestingly, for SHOLIS the efficiency of fault detection was, in decreasing order, "Z proof" (i.e. spec proof), "System Validation" (i.e. System tests), "Integration Test", Code proof and "Acceptance" (client tests?) and Unit test. This illustrates well that the best approach is a mix of test (especially for integration and validation) and proof (especially at spec level, very efficient, but code proof is also more efficient that unit test). Of course, as Matthew said, I should not generalize too much from some samples.

This is indeed interesting. But what was the the definition of “efficiency of fault detection” here? Finding the most faults for a given outlay of effort? Maximising the chance of finding all faults? I can imagine other definitions.

Some of these definitions of efficiency might not correspond with what a potential user of the system would want. If, for example, they (quite reasonably, I think) wanted their system to be reliable, they might define efficiency, ideally, in terms of how much the reliability improved for a given outlay of fault-detection effort.

Maximising the number of faults found (for a given outlay) is, of course, not the same as maximising the reduction in unreliability. There is data from Ed Adams in IBM, many years ago (early 80s), which showed that, for the systems he studied, faults differed enormously in their sizes (i.e. in the contributions they made to the system’s unreliability). The most famous observation from this work was that 30% of the faults found during operation were 5000-year faults (i.e. such a fault would occur in operational use - over thousands of sites - no more than once every 5000 years of exposure).

Clearly, finding lots of 5000-year faults will not improve reliability a great deal. You’d prefer to find the larger faults. So efficiency defined just in terms of numbers of faults found for a given outlay of effort could be misleading unless we know something about the “seriousness” of the faults - i.e. if your interest is in reliability, what their contributions to unreliability are. It might be nice if the “more serious” faults were always found with higher probability than the “less serious” ones.

The only known procedure that has this property - that the chances of discovery are exactly ranked with their seriousness (in terms of contributions to unreliability) - is operational testing. However, it may not be very efficient, even at reducing unreliability, compared with other approaches… (and that may be an understatement!)

I’m not, of course, suggesting that “testing it in” is a good means of achieving reliability (I’d better say that explicitly before readers of this list jump on me). But it seems there are some subtleties here that may not be captured in some simple notions of “efficiency of fault detection".

Cheers

Bev

Bev Littlewood

Professor of Software Engineering

Centre for Software Reliability

City University London EC1V 0HB

Phone: +44 (0)20 7040 8420 Fax: +44 (0)20 7040 8585

Email: b.littlewood_at_xxxxxx

The System Safety Mailing List

systemsafety_at_xxxxxx Received on Sat Apr 02 2016 - 07:38:06 CEST

*
This archive was generated by hypermail 2.3.0
: Mon Apr 22 2019 - 20:17:07 CEST
*