Re: [SystemSafety] Data on Proof effectiveness from real projects

From: Les Chambers < >
Date: Sun, 3 Apr 2016 17:58:21 +1000


Steve

RE: " Inspections efficiency is 0.8 to 1.2 labor hours per defect found"

Impressive: the spread here is .4, not .45 or .39 but .4. But what does this mean to the average Joe in the development shop in Outer Mongolia?

I should point out that while it is laudable for an organisation to collect these stats for the purposes of process improvement, they are rarely transferable across organisations (sometimes the're useless between projects in the same organisation) as they are wholly a function of the inspection processes used and the skills of the inspectors. As a rookie instrument engineer I was once on a team inspecting instrument installations in a chemical plant. Our team was led by a guy with 30 years experience. I was in awe of this guy. Just about every installation had problems. The phone rang in the control room and he excused himself to take the call instructing us to carry on without him. Like magic we stopped finding problems. After 10 minutes we smelt a rat, stopped for a convenient cup of coffee and awaited his return.

Organisations that do not have a "stable" development environment - that is, people come and go and much use is made of contract labour, not to mention the usual churn in development technology with the resulting "learning experiences" - really struggle to make sense of these numbers because they vary wildly.  

I'm not saying we shouldn't try though. The answer must lie in more effective automation of inspection tasks. In my experience human beings are just so bad at it. Meanwhile I guess we will just have to put up with gross measures.  

As a matter of interest, one gross measure that fascinates me is how many lines of the 10,000,000 + line of code body in my car has actually been reviewed by someone other than the author. Right now we have a debate in Australia over the definition of free range eggs. We are moving towards one square metre per chook (Australian for chicken) , surely VW et al , owe us an eyeball count for the lines of code in our vehicles (I don't even want to think about our aircraft).  

Cheers

Les    

From: systemsafety
[mailto:systemsafety-bounces_at_xxxxxx Steve Tockey
Sent: Sunday, April 3, 2016 5:11 AM
To: martyn_at_xxxxxx
systemsafety_at_xxxxxx Subject: Re: [SystemSafety] Data on Proof effectiveness from real projects    

Martyn wrote:  

"Is anyone actually interested in the number of defects found? Surely that's not a measure of anything useful about the software."  

Maybe it doesn't tell me much about the software, but it speaks volumes about the software process being used. I care about how many requirements defects were found when requirements work was done. I also care about how many requirements defects were found after that requirements work was claimed to have been done. Dividing the first by the sum of the first plus the second gives me an effectiveness measure for my requirements QA/QC work. As well, dividing labor hours (or, money) invested in requirements QA/QC work by the number of requirements defects found gives me a measure of efficiency.  

Published data comparing Testing to Inspections is striking:  

Testing effectiveness is typically 60-70%

Testing efficiency is typically 6-8 labor hours per defect found

Effort to repair defects found in Testing averages 3.4 to 27.2 hours per  

Inspections effectiveness is reported to be 60-90% although I've had inspections run at 95% efficiency

Inspections efficiency is 0.8 to 1.2 labor hours per defect found

Effort to repair defects found in Inspections averages 1.1 to 2.7 hours per  

I think it's very useful to know that Inspections can be far more effective than Testing. I also think it's very useful to know that Inspections are between 5x and 10x more efficient than Testing. But I could never know either of those if I didn't count the number of defects found.    

If I have a base historical of data for estimating total defects, I can subtract defects already found from estimated total to get an approximation for defects remaining. Of course, it is only an approximation. But even being only an approximation, it can be useful to know.    

From: systemsafety <systemsafety-bounces_at_xxxxxx behalf of Martyn Thomas <martyn_at_xxxxxx Reply-To: "martyn_at_xxxxxx Date: Saturday, April 2, 2016 8:02 AM
To: "systemsafety_at_xxxxxx <systemsafety_at_xxxxxx Subject: Re: [SystemSafety] Data on Proof effectiveness from real projects  

On 02/04/2016 06:37, Steve Tockey wrote:

Generally speaking, efficiency is looking at investment per unit of work done. In other words, increasing efficiency means that the same work is done at lower investment-the same (number of?) defects are found for a lower investment. Maximizing the chance of finding defects would be an issue of effectiveness. Effectiveness is looking at the rate at which the job is done correctly (I.e., a defect is found, not missed). One needs to look at both efficiency and effectiveness of processes to make a fair comparison.    

Is anyone actually interested in the number of defects found? Surely that's not a measure of anything useful about the software.

An important benefit of verification is that it can tell you (for some classes of defect) that the number of defects remaining is zero.

Martyn



The System Safety Mailing List
systemsafety_at_xxxxxx Received on Sun Apr 03 2016 - 09:58:46 CEST

This archive was generated by hypermail 2.3.0 : Wed Apr 24 2019 - 18:17:08 CEST