Re: [SystemSafety] Modelling and coding guidelines: "Unambiguous Graphical Representation"

From: Les Chambers < >
Date: Sun, 6 Mar 2016 12:12:05 +1000

I'd like to endorse Steve's comments on the effectiveness of model-based development. My experience is identical to his in terms of highly reliable, safe software delivered at low cost in predictable timeframes and, I might add, delivered by people who were not professional programmers. But not on 19 projects. It was on hundreds of projects and by now, in that environment, it would be thousands of projects. The model of course was the state engine. It was used in every traditional phase of software engineering from requirements all the way through to training operators. The hardware we used was optimised for executing state engine logic, the language we used was specifically designed for state engine implementation and the user interface exposed state engine concepts that were very easy for operators to understand.

The application was chemical process control. We were one of the first groups to control reaction kinetics with software. In that project environment you are a small cog in a giant machine. All the projects I worked on were substantial plant upgrades or new plant developments where the cost of late delivery was so huge you career would not withstand it. Unpredictable software development schedules were just NOT an option. Your career would also not withstand destruction of equipment or loss of life because of your software. Maintenance was also a huge issue. In the chemical marketplace product technologies are constantly changing and plant operations were evolving. The plant engineer responsible for the program was likely to move on every 2 to 3 years requiring someone new to come in, understand the operation of a very complex machine and take responsibility for ongoing maintenance and upgrade. The model made this possible with nonprofessional programmers. We found it much more effective do have a process expert writing, and taking daily responsibility for the consequences of, "the program".

I find it depressing that we continue to have discussions over the worth of these models. For example the state engine model is more than 60 years old and was first used when the only control tool was a relay rack. It's become progressively more useful since then.

A thing I have noticed is that state engine evangelists like Steve and myself are incredibly passionate about the subject. I have only recently figured out why. And the answer is: that particular model played a large part in one of the most productive periods of my engineering career.

I was there when almost perfect start-ups occurred on highly complex multiunit operation chemical plants.
I was there when chemical engineers with no particular love of software learnt to embrace this new technology as an indispensable tool in getting their jobs done.
I was there when the light came on in the brains of plant operators who had never seen or heard of a computer until a month before and went on to actually contribute to code that made their plant run smarter. Some of these guys didn't have a high school education. I experienced, over and over again, the transcendence of getting something right the first time.
I experienced the satisfaction of driving out of the plant of an afternoon and going home to my wife and children confident that nothing I had done that day would destroy equipment or hurt another human being - together with the attendant long hours of peaceful sleep.

And so, if you have any doubts about the utility of these models let me say this to you. "We believe more than we can prove." Neither Steve, I or anybody else will be able to produce numbers to support our case. These numbers are buried in proprietary operations and will never see the light of day. All that we can tell you is that decades of experience when the torch was applied to the belly of a method/model/programming paradigm it came through with flying colours. This is the source of our belief and it is worthy of notice AND ACTION ON YOUR PART.

Many of the tools we use today, languages for example, are toys conceived by people who have never had to manage the massive size, complexity and criticality of real-world applications - on a budget - in predictable timeframes - using a broad church of developers. Models such as the state engine have passed these tests with flying colours for decades. Unfortunately the true believers in profoundly effective real-world methods will probably remain cloistered in these large development shops constrained by proprietary restrictions, an exclusive race of Jedi looking out at the great unwashed and pitying them.

And furthermore, if you are not embracing the proven models of our profession such as the state engine, consider yourself to be an unwashed, unevolved, uneducated, lazy and fundamentally incompetent actor - not worthy to be in charge of or associated with anything that could hurt a human being. An accident waiting to happen.

If you find yourself in this category I suggest you start reading here: Modelling Software With Finite State Machines, A Practical Approach, Wagner et al. It's cool aid worth consuming.

  end of rant



-----Original Message-----
From: systemsafety
[mailto:systemsafety-bounces_at_xxxxxx Steve Tockey
Sent: Sunday, March 6, 2016 6:54 AM
To: Derek M Jones; systemsafety_at_xxxxxx Subject: Re: [SystemSafety] Modelling and coding guidelines: "Unambiguous Graphical Representation"


"You seem to be willfully refusing to deal with a reality that does not conform to your ideal view of how the world should be."

This is the crux of where we differ, I guess. Yes, you are entirely concerned with how software *is* practiced today. And I respect that. Where I'm coming from is real, on-project experience involving at least 19 projects where the results are what I stated:

Development cost and schedule cut in half from typical Delivered defects < 0.1 of typical
Long term maintenance < 0.25 normal

I'm not dreaming of some pie-in-the-sky, never-been-done-before ideal. We
*have* done it. More than once. And yet the people who weren't on those
projects refuse to acknowledge those successes.

"Quality as seen by the paying customer, which can be very different from code quality."

The key word here is "can". As I said before, software could have perfect internal structure and completely meet the stated requirements, but the stated requirements could be totally wrong. So yes, high quality code internally, panned by customers externally. It happens. But I'm talking about how crappy internal code quality affects the paying customer's perception of quality. A word processor that goes into an infinite loop on a paste is crap quality regardless of how well the software might claim to meet the "functional" requirements. I can give you a laundry list of other defects that are clearly caused by crap code quality, not by incorrect requirements. A small sampling:

*) Re-opening a document after an infinite loop crash has about a 10%
chance of "There is insufficient memory". I have to quit and restart a second time before it will read the exact same document it just complained about being too big

*) I try to save a document and it will complain "Not enough space on
disk" despite having gigabytes of free space on the disk. Only one file is 7 meg, the rest are well under 1 meg, most are about 400k

*) Lines of text in the document disappear at page boundaries. The lines
are really there, page rendering defects make them disappear. Adding a blank line makes them come back. That's offset by opposite cases where the same line appears both before and after the page boundary

*) Columns in tables can't be resized if that table crosses a page boundary

*) Even when a table is contained in the same page, column margins can't
be moved by more than 1/8 inch at a time. It requires many click-drag, click-drag, click-drag events to move column margins any appreciable amount

*) If a word in a section of text was hyphenated by line-wrapping, when
that text is copied and pasted the hyphen is still in the pasted text even though line-wrapping isn't happening at that word anymore

*) If a table is pasted into a document and the right edge of the table is
outside the right hand page margin, that edge of the table can't be selected (for resize) until page layout is set to landscape so the table margin is back inside the right side page margin

*) Switching page orientation to landscape to re-size a table has a 50%
chance of resulting in "Document is to big for <schlock product> to handle" error

*) "Paste and match style" frequently doesn't match the style of what that
text was pasted into

*) Sometimes changing type size in a footnote causes the display content
to move by as much as 7 pages

*) Editing text within a footnote sometimes ends up as a footnote to the

I could go on. These are not mis-matched requirements issues, they are simply amateur programmers at work. The same holds for other products from this same company, I can give you a laundry list of crap-code induced problems in them as well. If I wanted to, I could start lists for products from other companies (next on my list would be Adobe) but seriously, I don't have the time.

"Ok. Can I have a copy of your data please. I would be happy to analyze and write about your success."

Unfortunately I don't have detailed to-the-person-hour data for these projects. Besides, that data would almost certainly be considered proprietary by the organization in question. That said, however, here's data that I can give you:

*) A discrete event simulation system was built to allow traffic
throughput calculations for commercial airports. The legacy code was 44K SLOC of Fortran and only handled a single runway. The company needed to support parallel runway simulations. The two full-time maintainers said the code could not handle parallel runways. The system was rebuilt from the ground up. One developer was coached on doing model-based requirements and design and replaced the legacy code in about 9 months with 8K SLOC of Ada that also handled parallel runways and needed less than 1/4 FTE to maintain from then on.

*) 767 Engine Simulator ATE: the most credible estimate for the project
was 18 months with 4 developers. The business need was 9 months. The 4 developers delivered in 7 months. Industry data for delivered defects in real-time systems (Boehm) were 15-17 per K SLOC. We delivered 13K SLOC. We claimed we would be no worse than the data for IT systems, 5-7 defects per K SLOC. Commercial products were supposed to deliver 1-3 defects per K SLOC. We based our estimate on 6 per K SLOC, this would estimate 78 defects delivered. Only 4 defects were reported by users in the entire first year. That's a delivered defect rate of 0.3 defects per K SLOC and that's for a real-time system. When we wlooked at defects found in model reviews (for both requirements and design), we counted pretty close to 72 defects found and fixed before coding. We made mistakes, we just found and fixed the majority of them before code was ever written.

*) 777 ATE: The business need was 30 months, the project was staffed to
get it done in that time. The original team wasted the first half of that time writing research papers, surfing the web, playing games, etc.--not doing any ATE requirements, design, or code work. When management found the team had done nothing in 1.25 years, that team got shuffled out. A new team was brought in. Using model-based development we were able to deliver to the factory floor 3 days ahead of the original schedule. Similar delivered defect densities as 767 ATE. The 777 Factory Manager insisted on 24x7 pager coverage for the first 6 weeks of use. If 777 ATE so much as hiccuped, he wanted someone there to fix it NOW. Not a single call ever happened. Interviews with shop floor mechanics (users) showed that they were astonished by how such complex functionality could work so flawlessly. They even set up a betting pool: $20 buy in, the first to be able to reputably crash 777 ATE would win a $600 pool. As long as I was with 777 ATE, nobody ever won the pool because nobody could crash it. Even with a $600 incentive.

*) ATE Switch Matrix Driver re-write: Part of general ATE software in
Everett was the driver for a "switch matrix". This device allows ATE instrumentation on one axis of wires to be arbitrarily switched into and out of airplane test points on the other axis of wires. There are rules about what switching was legal, like "No two output instruments can be switched onto the same airplane point at the same time". The existing code was about 4000 lines of spaghetti crap that was constantly failing. We re-developed it in roughly a month and ended up with about 800 lines of clean code that never broke from then on. When I say "never", I mean as long as I was on the 777 ATE project which was about another 8 months. I can't say it never failed after that because I lost contact with them for about 15 years. But, if the switch matrix code was going to fail then it should have failed at least once in that time.

*) 787 ATE: Can you say, deja vu? Again, the original team wasted the
first half of the schedule. Again, I got called in to bail out the project with a replacement team. We delivered earlier than was required by the original airplane development schedule. Remember that the whole airplane program slipped by about 2 years. We would have been ready even if there were no slips at all anywhere else in the airplane program. Similar quality performance as 777 ATE and 767 ATE.

*) Oso and Altair: A pair of projects at Los Alamos National Lab that
prepare input data (2-D and 3-D models) for computational fluid dynamics code. The need date for both projects was about 12 months, both were delivered in about 6. Similar quality performance as ATE.

*) 777 ATE ACS Re-write: FAA rules are that one can't test a commercial
airplane with obsolete instrumentation. One part of 777 ATE was "ACS", a driver to hardware that allows ATE to talk on the ARINC 629 bus. CDS, the maker of the hardware, notified Boeing that the version they were using would go obsolete in 12 months. ATE management estimated it would take at least 12 months with 4 people to rewrite the driver code. I was called in to help the team. We started with the original ACS requirements models and modified them to account for hardware changes in the new device. Those same model mods were forwarded into the design models and then into the code. They were back on the air with the new ARINC 629 hardware in 3 months.

*) P-8 Mission Systems: The P-8 is the replacement for the Lockheed P-3
Submarine Chaser ( All of the fly-the-airplane avionics was essentially stock 737 equipment--out of scope for us. Our project was the mission systems in the back of the plane: mission planning, mission execution, secure plane-to-plane communication, target acquisition and tracking, targeting, . . . There were about 250 developers on the project, working for about 7 years. Unfortunately, the project went black (need-to-know security clearance, I mean) before they finished and they're understandably tight-lipped about it now. My best guess is that mission systems code is in the 5-7 million SLOC range. I can't say that the software was done in half the time, because they won't tell me. On the other hand, it's normal for projects of this size to be substantially late, on the order of 100%. All I can say is that there have been no reports of schedule delay or software quality problems on the P-8. It's be so out of the news that most people don't even know it exists. OTOH, if it were anything like a typical military project of that size, it would have almost certainly been in the news for some software-induced delay or another.

*) Well Placement Engineering: This was for an oilfield services company.
Most people think that oil wells are drilled straight vertical. Not so. To take advantage of softer layers, wells can go many times the distance horizontally as vertically. The drill bit is literally steered by an operator to follow a pre-determined path. This project was for part of the software that gives the drill operator control over the drill. Again, model based requirements and design led to the project being delivered early (sorry, I don't have exact how-early figures). I also don't have delivered defect data from the field. However, the testers told me personally that they were amazed by the quality of that software, they literally could not find any defects in it at all. It passed almost every test they threw at it.

There have been other projects, too. Unfortunately, I have less data about those projects than these (above). And of course, anyone can throw out the "But this is all anecdotal data" defense. And there's nothing I can do to counter that. However, when you look at the typical results of software projects in terms of over-cost, over-schedule, under-scoped, defects-delivered, the pattern on these projects is decidedly different. None of these were over schedule (the pattern, as I said, is to be early), none were over cost, all delivered everything they were supposed to, and customers were delighted that the code did what they wanted it to without crashing or giving wrong answers.

Is this enough data? Or, do you need more?

-----Original Message-----
From: systemsafety <systemsafety-bounces_at_xxxxxx on behalf of Derek M Jones <derek_at_xxxxxx Organization: Knowledge Software, Ltd
Date: Friday, March 4, 2016 2:23 PM
To: "systemsafety_at_xxxxxx <systemsafety_at_xxxxxx Subject: Re: [SystemSafety] Modelling and coding guidelines: "Unambiguous Graphical Representation"


>> A very important question.  I'm not sure I have the answer, but I
>> data data showing it happening at multiple levels."
> That's interesting data, yes. But remember, that data is limited by how
> software development is being practiced today (by largely "highly paid

I want data about today's practices (the data is actually from 2005-2014, the dates used by the papers that originally worked on it). Why would you want data from further back (not that I think things have changed much)?

> amateurs"), and not how software development SHOULD be practiced.

You seem to be willfully refusing to deal with a reality that does not conform to your ideal view of how the world should be.

>> "The benefits of any upfront investment to make savings maintaining
>> what survives to be maintained has to be compared against the
>> costs for all the code that got written, not just what remains."
> Some of the software that gets built is intended to be single-use or is
> intentionally short lifetime. My experience is that's a very small

The data is from large, long lived (10-20 years) projects.

> I argue that what determines survivability of the vast majority of > software is, essentially, quality. High quality (I.e., well-structured,

Quality as seen by the paying customer, which can be very different from code quality.

> clean, etc.) code that reliably automates a solution is far, far more
> likely to survive than code that's crap (e.g., high technical debt) and

Unsubstantiated, biased point of view.

Cost of maintenance is a drop in the pond compared to marketing expenses.

What is important is ability to add new features quickly so the next version can be sold to generate revenue. If a product is selling management will pay what it takes to get something out the door.

> Using your terminology from

> I'm saying that I can make d1 and m1 such that:
> d1 = 0.5 * d
> m1 <= 0.75 * m
> Therefore, clearly, for any y >= 0
> d1 + y * m1 << d + y * m

Ok. Can I have a copy of your data please. I would be happy to analyze and write about your success.

Derek M. Jones           Software analysis
tel: +44 (0)1252 520667
The System Safety Mailing List

The System Safety Mailing List

The System Safety Mailing List
Received on Sun Mar 06 2016 - 03:12:32 CET

This archive was generated by hypermail 2.3.0 : Tue Jun 04 2019 - 21:17:08 CEST