Re: [SystemSafety] Qualifying SW as "proven in use" [Measuring Software]

From: Jean-Louis Letouzey < >
Date: Thu, 20 Jun 2013 09:34:56 +0000


Hi All,

And many thanks to Thierry for all its comments.

I will just add something:
Organization targeting "high quality code" insert into their SQALE requirements list an additional requirement related to code coverage. Something like: "All files have at least a 80% unit/integration test coverage."  This requirement is attached to the "reliability" characteristic.

In such case, poorly tested code will have a high SQALE index (i.e. Technical Debt) related to Reliability (in addition to other TD related to other requirements). As an exemple, such coverage measure is automatically collected and reported within the SQALE Sonar plugin. It is easy then to establish correlation between SQALE Quality index (technical debt) and "perceived" quality.

Best Regards

Jean-Louis Letouzey | Consultant expert
inspearit
Le Visium | 22 avenue Aristide Briand | 94742 Arcueil Cedex | France TÚl : +33 1 49 08 58 00 | Mob : + 33 6 31 29 06 48 | Fax : +33 1 49 08 95 88 www.inspearit.fr
www.sqale.org

-----Original Message-----

Sent: jeudi 20 juin 2013 11:19
Cc: denis.chalon_at_xxxxxx Subject: RE: [SystemSafety] Qualifying SW as "proven in use" [Measuring Software]

Hi Peter,

To answer your questions:
1░) Yes, there is some objective evidence that there is a correlation between a low SQALE index and quality code. For example ITRIS has conducted a study where the "good quality" code is statistically linked to a lower SQALE index, for industrial software actually used in operations.

No, there is not enough evidence, we wish there would be more people working on getting the evidence.

But more than that, SQALE IS a quality model: it DEFINES what is "good quality" code. It defines objective, measurable REQUIREMENTS that can be achieved, and that can be objectively measured between vendors and owners. If a given check point is later found to be not relevant, there is no problem to discard the point.

2░) For high-critical software. Yes, SQALE has been used on high-critical software, where it basically automates the (pre-)defined quality rules. It is a useful tool to help what you call "experienced critical-system programmers". There is a published paper by Jean-Pierre Rosen and myself on ADA open-source code where you can find more information: we've used open source because most applications of SQALE in industry are confidential.

In addition, we're always looking for clear, unambiguous (automatable), without exceptions, checkpoints. For example, one of our current checks points is a higher limit of cyclomatic complexity V(G) of 15 and 60 for one operation (method/procedure/function/whatever you call it). The V(G) can be measured by many tools today, or approximated to an extent. The usual limit in the literature is around 10 (for example, here: http://www.sei.cmu.edu/reports/97hb001.pdf). The thresholds in SQALE are high in order to reduce discussions about real-life implementations. Our measurements show that most operations fall within this limit But there are challenges to this check point. For example: a) V(G) actually measures size and not complexity. This fact is used in SQALE as it uses the aggregate V(G) to compute densities (very interesting, btw). SQALE does not have any checkpoint on aggregate V(G), only on individual operations, b) some operations are complex and breaking them down is "artificial". SQALE's approach is to require breakdown as a DESIGN decision. It's possible in some implementations of SQALE either to require a rewrite, or to have a review and flag the particular operation as having an "accepted non-compliance". The last approach is to accept high V(G) in an operation if it follows a particular pattern: the whole V(G) originates from one "CASE ... OF" or "SWITCH" construct. However a mix of "CASE OF", "IF" "WHEN" and other constructs would result in a non-compliance.

Another example is copy/paste. SQALE detects a non-compliance for identical code above 100 tokens. This is about 5-15 lines of code, depending on the language, IDENTICAL in 2 or several areas. What we have observed is that there are clear differences between projects. Some projects tend to have large copy/paste ratios, while other projects (and the two we analysed in our ADA paper) have really low copy/paste. We do know which has the better quality. Again we get challenges on the usefulness of copy/paste, although most quality plans identify this as a non-compliant practice: a) copy/paste does not in fact increase the number of defects. SQALE's point of view is that copy/paste in fact increases the test effort and reduces testability, so it is a non-compliance. Copy/paste may increase defects but the literature is not clear. It also reduces maintainability since a defect/change in one paste instance must be corrected in every instance, but SQALE's model puts testability before maintainability, so it counts copy/paste only once. b) copy/paste on "generated code" has no sense. SQALE's point of view is that there is no reason people writing code generators should abstain from applying the quality rules applicable to hand-crafted code. In fact we would expect them to do a better job at writing code.

Basically, SQALE is a framework to harmonize and hierarchize quality check points. Each of those check points (and their thresholds) may be challenged. But when performing assessments, the 10 worst pieces of code identified by their high aggregate SQALE index are much poorer quality, compared to the remainder of the software, as assessed by humans. You can try it yourself. Deliberately, SQALE has been made freely available so additional and independent results can be obtained. Several tool vendors have implemented SQALE to complement their measurement points, and the SQALE authors have no commercial interests with tool vendors.

Peter, any comments or suggestions for improvement coming from you would be really helpful. Or from this mail list ;-).

Thierry Coq
PS. All opinions are my own, not necessarily those of my employer.

-----Original Message-----

Sent: mercredi 19 juin 2013 17:07
To: systemsafety_at_xxxxxx Subject: Re: [SystemSafety] Qualifying SW as "proven in use" [Measuring Software]

Thierry,

is there any objective evidence that using SQALE improves the quality of the code?

If yes, is there any objective evidence that using SQALE improves the quality of high-quality code written by experienced critical-system programmers?

If yes, where would that be found in the engineering literature?

PBL Prof. Peter Bernard Ladkin, Faculty of Technology, University of Bielefeld, 33594 Bielefeld, Germany Tel+msg +49 (0)521 880 7319 www.rvs.uni-bielefeld.de



The System Safety Mailing List
systemsafety_at_xxxxxx

The contents of this e-mail message and any attachments are confidential and are intended solely for the addressee. If you have received this transmission in error, please immediately notify the sender by return e-mail and delete this message and its attachments. Any unauthorized use, copying or dissemination of this transmission is prohibited. Neither the confidentiality nor the integrity of this message can be vouched for following transmission on the Internet.


The System Safety Mailing List
systemsafety_at_xxxxxx Received on Fri Jul 12 2013 - 15:13:33 CEST

This archive was generated by hypermail 2.3.0 : Thu Apr 18 2019 - 23:17:05 CEST