Re: [SystemSafety] Paper on Software Reliability and the Urn Model

From: Peter Bernard Ladkin < >
Date: Wed, 25 Feb 2015 12:20:26 +0100

On 2015-02-25 11:37 , jean-louis Boulanger wrote:
> For the software, no evaluation of reliability are acceptable or representative.

I think that is clearly wrong. I think my paper shows it is clearly wrong.

Software for protection systems in UK nuclear power plants has regularly used statistical reliability assessment techniques for decades, and some of the best people in the field have worked on them.

> Software contain bug (no idea of the number)
> the change process are not monitored (not the same team, not the same method, ...)

If the SW is being changed, then any statistical assessment has to start anew, for well-known reasons. Statistical assessment is not at all practical for SW that is constantly changing.

> We don't have no operational history of software ....

Well, you obviously can't do statistical assessment of reliability if there is no adequate operational history. Actually, it's pretty hard to do *any* assessment of reliability if there is no adequate operational history. And - please let's be clear about this - you really do need to perform reliability assessment on critical software, be it statistical or whatever you can do.

> Lots of things come up. People don't understand what the urn model has to do with software
> evaluation. I have recently experienced reliability experts making incorrect claims, and non-experts
> finding it difficult to adjudicate those claims.
> experienced reliability expert making incorrect claims because software reliability assessment is
> not a subject

I'll let you argue that with colleagues who have thirty or forty years doing it and getting their work published in the premium forums on dependability in software.

> software are not reliable ...

If that were generally so, then it shouldn't be running our planes, trains and cars, let alone our safety-critical process plants. You've got a lot of campaigning ahead of you .........

But in fact lots of it is very reliable indeed.

The space shuttle control system software turned out to have been completely reliable.

In 27 years of fly-by-wire commercial aircraft, with thousands flying the skies every day, the reliability of the control software, in the sense in which we mean it when using the Bernoulli modelling, has been first-rate. There has only ever been one accident, when a few people were injured. And that was due to a transient hardware fault, which was not filtered out by the control SW even though the phenomenon was known, so that was a design decision, not a Bernoulli-type-reliability issue.

The one incident which might have fallen into range was the 2005 Boeing 777 incident out of Perth, which turns out to have been a configuration mistake. Such things wouldn't have been picked up by a Bernoulli-type reliability assessment either.

The kinds of things which do keep some of us awake at night are generally not the kinds of phenomena which could be identified through Bernoulli-type statistical assessment. They are the kinds of things which fall through the gaps in specification.

> I am not an experienced mathematician but I understand that is not a good idea to apply the basic
> mathematics to a complexe product

On the contrary, if the product is complex, then basic math is what you want. Applying complex math, there are so many ways to get the assumptions wrong without noticing.

> For some software, I am the assessor from 10 years, and i confirm that the number of known bug
> increase after each version ....

Well, *we* occasionally do better than that :-)

PBL Prof. Peter Bernard Ladkin, Faculty of Technology, University of Bielefeld, 33594 Bielefeld, Germany Je suis Charlie
Tel+msg +49 (0)521 880 7319

The System Safety Mailing List
systemsafety_at_xxxxxx Received on Wed Feb 25 2015 - 12:20:40 CET

This archive was generated by hypermail 2.3.0 : Tue Jun 04 2019 - 21:17:07 CEST