Re: [SystemSafety] OpenSSL Bug

From: Steve Tockey < >
Date: Wed, 16 Apr 2014 15:36:21 +0000

David,

"Probably because the whole discussion started from a bug (in OpenSSL) that precisely lies in those 7%. :-)"

Actually I'm going to respectfully disagree with that. I'm going to claim I can trace it back to incomplete requirements & resulting inadequate design. A parameter on a function call was missing a range check. Why? Because there wasn't an explicit requirement that the value represented in that parameter be within a specific range. Had there been an explicit requirement, and had the designer/programmer paid attention to that requirement then they would have needed a range checking 'if' statement to satisfy that requirement.

And even if there were no explicitly stated requirement, had the developer had any brains and used Design-by-Contract then they would surely have noticed that the value range of that parameter's data type was much bigger than what the function was supposed to be able to handle. The need for a range check would have been bloody obvious.

It wasn't at all a 7% defect, it was an 83% defect.

"More seriously, I would make the distinction between two kinds of approaches: ..."

Again, slight disagreement. I don't know the exact source of the defect injection data (I.e., what specific kinds of projects the data came from). However, I do know (of) the person who published the data: James Martin. Knowing (of) Mr. Martin, I'm hard pressed to say that the data came from any safety critical projects.

Besides that, my own experience with both safety-critical and nondevelopment  projects doesn't show any appreciable difference in the defect *injection* statistics. The big difference, IMHO, is that the safety-critical projects make much better use of reviews and the defect *detection* statistics are heavily skewed towards being found and fixed earlier.

"Thus I have a practical question: suppose I have a company that lacks such good practices (at least for the review or safe coding rules one). I'm not a project manager, just a low level software engineer working on a given project. How can I help improve the development process to reach such level of good practices?"

Contact me directly, I'll send you a paper called "How Healthy is your Software Process?". By instrumenting the project's defect tracking system with very simple data we can easily show that the majority of defects are requirements & design injected and that more than half of the organization's capacity to do development work is being wasted in finding and fixing mistakes that were made much earlier. Managers have a tendency to discount what the techies are telling them, but it's hard to ignore the data when it's staring them in the face.

If anyone wants the paper, I'm happy to share it.

-----Original Message-----
From: David MENTRE <dmentre_at_xxxxxx Date: Wednesday, April 16, 2014 6:32 AM
To: "systemsafety_at_xxxxxx <systemsafety_at_xxxxxx Subject: Re: [SystemSafety] OpenSSL Bug

Hello,

Le 16/04/2014 13:23, Steve Tockey a écrit :
> No offense inferred, but thanks for pointing it out.

The same for me. I should have kept the classical "if (a == SUCCESS)" example. Another example: C language does not forces you to check the return value of a function (that might flag an error case). Most of other languages I know would force you to check it (or at least issue a warning).

> Besides, echoing this and Derek M. Jones' last post, I think this whole
> discussion is focusing on relatively minor issues and completely missing
> the big picture.

[...]
> the simple fact is that 83% of
> all defects exist before a single line of code was ever written. Why
> aren't we attacking the 83%, not the 7%???

Probably because the whole discussion started from a bug (in OpenSSL) that precisely lies in those 7%. :-)

More seriously, I would make the distinction between two kinds of approaches:

  1. People that produce software following the good practices you describe below, probably to satisfy DO-178C or EN 50128 standards. In such case, their error figures are probably in the above range (83% requirement + design, 7% code).
  2. People that produce software *without* following proper practices, probably like OpenSSL developers and most companies developing software in non-safety critical domain. In that case the coding errors are much more prevalent. And we are several people on this list to think that using better tool could help catch more of these errors. I do agree that following good practices could have similar effect, but the simple fact is that they are not doing it, probably due to lack of knowledge, time, money or because this is not a paid-for job. You might counter-argue that if they are not using good practices, they won't use static analysis tool neither. You might be right. ;-)

> A
> team with good requirements, design, and review practices can write good,
> safe code in any language, including C.

I agree.

Thus I have a practical question: suppose I have a company that lacks such good practices (at least for the review or safe coding rules one). I'm not a project manager, just a low level software engineer working on a given project. How can I help improve the development process to reach such level of good practices?

Sincerely yours,
david



The System Safety Mailing List
systemsafety_at_xxxxxx

The System Safety Mailing List
systemsafety_at_xxxxxx Received on Wed Apr 16 2014 - 17:36:36 CEST

This archive was generated by hypermail 2.3.0 : Thu Apr 25 2019 - 12:17:06 CEST