Re: [SystemSafety] Data on Proof effectiveness from real projects

From: Littlewood, Bev < >
Date: Wed, 30 Mar 2016 13:48:02 +0000

Hi David

On 30 Mar 2016, at 07:39, David MENTRE <dmentre_at_xxxxxx

And there is also King et al. paper "Is proof More Cost Effective than Testing?" on SHOLIS project.

Interestingly, for SHOLIS the efficiency of fault detection was, in decreasing order, "Z proof" (i.e. spec proof), "System Validation" (i.e. System tests), "Integration Test", Code proof and "Acceptance" (client tests?) and Unit test. This illustrates well that the best approach is a mix of test (especially for integration and validation) and proof (especially at spec level, very efficient, but code proof is also more efficient that unit test). Of course, as Matthew said, I should not generalize too much from some samples.

This is indeed interesting. But what was the the definition of “efficiency of fault detection” here? Finding the most faults for a given outlay of effort? Maximising the chance of finding all faults? I can imagine other definitions.

Some of these definitions of efficiency might not correspond with what a potential user of the system would want. If, for example, they (quite reasonably, I think) wanted their system to be reliable, they might define efficiency, ideally, in terms of how much the reliability improved for a given outlay of fault-detection effort.

Maximising the number of faults found (for a given outlay) is, of course, not the same as maximising the reduction in unreliability. There is data from Ed Adams in IBM, many years ago (early 80s), which showed that, for the systems he studied, faults differed enormously in their sizes (i.e. in the contributions they made to the system’s unreliability). The most famous observation from this work was that 30% of the faults found during operation were 5000-year faults (i.e. such a fault would occur in operational use - over thousands of sites - no more than once every 5000 years of exposure).

Clearly, finding lots of 5000-year faults will not improve reliability a great deal. You’d prefer to find the larger faults. So efficiency defined just in terms of numbers of faults found for a given outlay of effort could be misleading unless we know something about the “seriousness” of the faults - i.e. if your interest is in reliability, what their contributions to unreliability are. It might be nice if the “more serious” faults were always found with higher probability than the “less serious” ones.

The only known procedure that has this property - that the chances of discovery are exactly ranked with their seriousness (in terms of contributions to unreliability) - is operational testing. However, it may not be very efficient, even at reducing unreliability, compared with other approaches… (and that may be an understatement!)

I’m not, of course, suggesting that “testing it in” is a good means of achieving reliability (I’d better say that explicitly before readers of this list jump on me). But it seems there are some subtleties here that may not be captured in some simple notions of “efficiency of fault detection".



Bev Littlewood
Professor of Software Engineering
Centre for Software Reliability
City University London EC1V 0HB

Phone: +44 (0)20 7040 8420 Fax: +44 (0)20 7040 8585

Email: b.littlewood_at_xxxxxx

The System Safety Mailing List
systemsafety_at_xxxxxx Received on Wed Mar 30 2016 - 15:48:17 CEST

This archive was generated by hypermail 2.3.0 : Tue Jun 04 2019 - 21:17:08 CEST