Re: [SystemSafety] Another unbelievable failure (file system overflow)

From: Steve Tockey < >
Date: Tue, 2 Jun 2015 17:35:57 +0000

I can't speak for IEC 61508, but I do agree that in general the weaknesses you point out are at least borderline ethical issues.

From: Martyn Thomas <martyn_at_xxxxxx Date: Monday, June 1, 2015 1:34 AM
Subject: Re: [SystemSafety] Another unbelievable failure (file system overflow)


Thanks for this. There's little discussion of professional ethics in any forum that I read.

Do you think there's any hope that we might be able to make a small advance in a focused area, such as IEC 61508? The standard isn't fit for purpose, in that it largely ignores cybersecurity issues and does not provide a sound basis for assessing whether safety-critical systems are safe enough for their proposed application. It's also too long, inconsistent, too expensive, and can't be copied/republished for use in teaching, research or professional debate. I see these weaknesses, in the central international standard for the safety of computer-based systems, as an ethical issue. Do you agree?



On 31/05/2015 05:14, Les Chambers wrote: Steve
Thanks for referencing the code of ethics. It should be brought up more often. Unfortunately, for me, it makes depressing reading. Especially when you come upon paragraphs such as:

3.12. Work to develop software and related documents that respect the privacy of those who will be affected by that software.

Although he has probably never read it, there is a man, who will probably never see his homeland again because he took these sentiments to heart and attempted his own corrective action. And what of the thousands of scientists, engineers and technologists who contributed to the construction of the software, the existence of which, he exposed to the world?

My point is that non-compliance with this code of ethics is massive and almost universal. In fact, any engineer maintaining strict compliance with every paragraph of this code would be unemployable in our modern world.

Reading these paragraphs through the lens of experience I am blown away by their flippancy. From personal experience I can tell you that screwing up the courage to implement even one of these items can be a massive life changing event. This sentence would be lost on a graduate. They're all perfectly reasonable statements of how one should behave. Much like, "Thou shall not kill, thou shall not commit adultery ...". The issue lies in the moral courage to implement.

There is no quick fix to this problem as we are a decentralised, unorganised and generally fragmented lot. We don't have the luxury of the medical profession that deals with a single organism. We can't simply state and righteously comply with the notion of, "Do no harm." In fact, for us, the opposite is true, many of us work in industries where the primary purpose is to kill other human beings, and with high efficiency (fewer soldiers kill more enemy).

One thing we can do is deal with the problem at its root:

We are graduating incomplete human beings from science and engineering courses. There is insufficient focus on the moral issues surrounding the impact of our machines on humanity. For example, a study of applied philosophy, including ethics, should be a nonnegotiable component of all engineering courses. Not just a final year subject, but a subject for every year with a weekly reflection on the content. Much like the weekly safety meetings I was forced to attend in the chemical processing industry.

I'm sure there will be howls of laughter at this, but, let me tell you it's the only thing that caused me to back a senior manager about five levels above my pay grade into a corner - he could physically not escape me short of punching me out and stepping over my body - and berate him until he promised to properly train his operators in the emergency procedures for a safety critical system.

Popping a few paragraphs up on the web would never have done the trick.

That experience was trivia compared to where we are headed. The massive computing power now available means that our software is beginning to take higher level decisions away from human beings. Some of these decisions are moral ones (refer my previous post on lethal autonomous weapons systems). "Shall I kill all humans associated with this structure, or no?"

At a recent engineering alumni meeting I asked the head of my old engineering Department how much philosophy is taught to undergraduate engineers. He chuckled. "It is available as an elective but less than one percent participate," he said.

I plan to speak to him again soon.


The System Safety Mailing List
systemsafety_at_xxxxxx Received on Tue Jun 02 2015 - 19:36:12 CEST

This archive was generated by hypermail 2.3.0 : Tue Jun 04 2019 - 21:17:07 CEST