Re: [SystemSafety] Agile methods

From: Steve Tockey < >
Date: Mon, 2 Sep 2013 17:50:53 +0000

While we're on the topic...

Wallace & Kuhn wrote a paper titled, ³Converting System Failure Histories into Future Win Conditions²
(http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.28.2701&rep=rep1& type=pdf)

One interesting tidbit out of the paper is (page 11), "It is significant however, that of the 109 reports that are detailed, 98 % showed that the problem could have been detected by testing the device with all pairs of parameter settings." So, while it appears that all-pairs testing with respect to device parameters is pretty effective in revealing problems, it doesn't appear that the FDA has made any move on requiring that kind of testing before the candidate device is approved...

-----Original Message-----
From: Derek M Jones <derek_at_xxxxxx Organization: Knowledge Software, Ltd
Date: Monday, September 2, 2013 10:36 AM To: "systemsafety_at_xxxxxx <systemsafety_at_xxxxxx Subject: Re: [SystemSafety] Agile methods

> Given the sad state of medical device software, it does not seem that
> simply using Agile methods proves anything. Homa Alenzadeh, a grad
>student
> at UIUC, has looked at medical device recalls. The following is from an
> abstract she wrote for a talk she gave: "Medical device incidents are one

Paper + spreadsheet listing 1,211 recall events: http://users.crhc.illinois.edu/alemzad1/pub.html

> of the leading causes of serious injury and death in the United States.
> Between years 2006 and 2011, 5,294 recalls and approximately 1.2 million
> adverse events of medical devices were reported to the US Food and Drug
> Administration (FDA). Almost 23 percent of these recalls were due to
> computer-related failures, of which approximately 94 percent presented
> medium to high risk of severe health consequences (such as serious injury
> or death) to patients. '
>
> If my arithmetic is correct (1,200,000*.23 and then divide by 5 then
> multiply by .94), it appears from Homa's data that there are 51,888
> reported potentially serious incidents to the FDA regarding software each
> year. That is 144 every day. And many incidents are not reported (the
> reporting rules are pretty sketchy) and that is only in the U.S.
>
> The simple fact that mdedical device manufacturers have used Agile
>methods
> does not prove anything about their efficacy for safety-critical systems
> given the sad state of safety in that industry.
>
> Nncy
>
>
> On Mon, Sep 2, 2013 at 12:40 PM, Steve Tockey
><Steve.Tockey_at_xxxxxx >
>> All,
>> I have personal experience using Agile, and personal experience doing
>> safety-critical development (DO-178B), but not personal experience doing
>> agile development for a safety critical system. Instead of being
>>polarizing
>> and claiming "it can't be done", I'll simply say that I believe that it
>>is
>> possible to do it‹it could just end up being very costly to do so. The
>> worst case would be developing the code for the system in an truly agile
>> way, and then looping back and re-developing it with the necessary
>>safety
>> analysis, etc. in place using more traditional processes.
>>
>> All that said, I have a report titled, "AAMI
>> TIR45: 2012 Technical Information Report Guidance on the use of
>>AGILE
>> practices in the development of medical device software". AAMI is the
>> "Association for the Advancment of Medical Instrumentation" and they're
>> based at
>> 4301 N. Fairfax Drive, Suite 301
>> Arlington, VA USA 22203-1633
>> www.aami.org
>>
>> The report abstract states, "Over the past several years, AGILE
>>software
>> development has become an accepted method for developing software
>>products.
>> There have been questions from both manufacturers and regulators as to
>> whether (or which) AGILE practices are appropriate for developing
>>medical
>> device software. Enough medical device manufacturers have implemented
>>AGILE
>> practices in their software development so that answers to these
>>questions
>> can be documented. Having clear guidance of which practices have been
>>found
>> to be appropriate will be very useful for all developers of medical
>>device
>> software. This TIR will provide recommendations for complying with
>> international standards and U.S. Food and Drug Administration (FDA)
>> guidance documents when using AGILE practices to develop medical device
>> software."
>>
>> A preview copy is at:
>>
>>
>>http://marketplace.aami.org/eseries/scriptcontent/docs/Preview%20Files/TI
>>R45_1208_preview.pdf
>>
>> I have a licensed copy, but it's copyrighted material and can't be
>> shared. If you're really interested in the content, you'll have to get a
>> copy on your own (sorry).
>>
>> Having said that, three comments about all this:
>>
>> 1) I don't know what sort of status AAMI has with respect to FDA
>>policy
>> on medical device software development. In other words, I don't know if
>>the
>> FDA has actually agreed to this, or it's simply AAMI's suggestion on how
>> one might approach it.
>>
>> 2) The AAMI approach talks about agile development in a way that's
>>very
>> consistent with how most competent people would characterize agile
>> development.
>>
>> 3) I took a look at the Bruce Douglass report that Geoffrey Biggs
>> mentioned (Thanks, Geoffrey)
>>
>>
>>http://www.ibm.com/developerworks/rational/library/agile-analysis-practic
>>es-safety-critical-development/
>> I've worked with Bruce before, and I have a lot of respect for his
>>work.
>> However, I'd be willing to bet that most people who really understand
>>what
>> agile development is could easily argue that the Harmony method
>>discussed
>> by Bruce would not be considered agile. It's clearly iterative, but
>> iteration alone does not constitute true agility. *I* wouldn't consider
>> the Harmony method described by Bruce to be truly agile.
>>
>>
>> -- steve
>>
>>
>>
>> From: René Senden <rene.senden_at_xxxxxx >> Date: Monday, September 2, 2013 8:32 AM
>> To: 'M Mencke' <menckem_at_xxxxxx >> Cc: "systemsafety_at_xxxxxx >> systemsafety_at_xxxxxx >>
>> Subject: Re: [SystemSafety] Agile methods
>>
>> Myriam,****
>>
>> ** **
>>
>> You are right about my initial question being somewhat polarizing,
>>perhaps
>> because I tacitly assumed that anyone who¹d answer this question with
>> ³Yes², would also include some****
>>
>> of the corresponding experiences Š Regarding experience with
>> reconciliation of agile environments with typical/common/Š (software)
>> safety standards, I am very interested in the ****
>>
>> development processes and corresponding results/evidence/outputs, all
>>this
>> should be auditable. Is it (at all) possible to harmonize these very
>> different worlds, or would any such ****
>>
>> attempt result in compromising either? Transparency, auditability,
>> documentation, and safety specific processes such as analyses are
>>examples
>> that, to me, seem to be particularly ****
>>
>> difficult to address in an agile environment, let¹s take ³scrum² as an
>> example of a popular agile approach.****
>>
>> ** **
>>
>> Rene****
>>
>> ** **
>>
>> ** **
>>
>> *From:* systemsafety-bounces_at_xxxxxx >>
>>mailto:systemsafety-bounces_at_xxxxxx >>ounces_at_xxxxxx >> *On Behalf Of *M Mencke
>> *Sent:* maandag 2 september 2013 14:39
>> *To:* Jon Davies
>> *Cc:* systemsafety_at_xxxxxx >> *Subject:* Re: [SystemSafety] Agile methods****
>>
>> ** **
>>
>> René,****
>>
>> I have come across a situation in the railway field where an
>> organization was required by the customer to comply with a safety
>>standard
>> which has specific requirements on the software development process, in
>> this case, EN 50128, where this was previously not required. ****
>>
>> It seems to me that this type of situation typically arises in
>> industries/products where the SIL concept has traditionally not been
>> applied, or is being introduced. The customer creates a ³SIL
>>requirement²
>> as a type of marketing argument, where the SIL is understood by the
>> customer as applying to equipment rather than to a function (in order
>>to be
>> able to state ³This equipment is SIL X² in commercial scenarios).****
>>
>> Regarding your question, ³Have you encountered a situation, in
>>industrial
>> practice, in which an organization developing software following an
>>agile
>> methodology has to comply with a safety standard which has specific
>> requirements on the software development process?² ****
>>
>> This is a polar question, which can only be answered with a yes/no. ****
>>
>> If you are interested in practical experience, it might be useful to add
>> to this question which aspect(s) of such a situation you are interested
>>in.
>> For example, if the answer is ³yes², was the project actually a success?
>> Which measures/procedures were implemented in the company to approach
>>this
>> problem and reconcile the development process with such requirements?,
>>etc.
>> ****
>>
>> Depending on the type of information you are seeking, it may not be
>> possible to provide particular details regarding such a state of
>>affairs,
>> as this information may be company-specific.****
>>
>> Regards,****
>>
>> Myriam.****
>>
>> ** **
>>
>> 2013/9/2 Jon Davies <jdavies_at_xxxxxx >>
>> On 30 August 2013 18:02, René Senden <rene.senden_at_xxxxxx >>> Dear all,
>>> ****
>>
>>> Do any of you have practical experience with reconciling established
>> agile
>>> software development with software safety requirements (e.g. IEC-61508
>>>or
>>> DO-178..) ?****
>>
>> Yes, and we usually end up throwing away the software developed using
>> "agile" methods, and starting again properly.
>>
>> I'm taking "agile software development" as meaning the development of
>> software using processes consistent with the agile manifesto:
>> http://agilemanifesto.org/ - to quote the relevant part:
>> "...we value... working software over comprehensive documentation"
>>
>> this is fundamentally in conflict with many of the things we know
>> about building high integrity software, and so "agile" methods are
>> fundamentally in conflict with developing software for safety critical
>> systems.
>>
>> There's plenty to learn from agile development methods that might be
>> useful in high integrity software development, but that's a whole
>> different discussion. Every time we discuss agile development here,
>> we end up back at the need to use a development process that builds in
>> correctness - we can't test exhaustively, so we need a process that
>> builds integrity in. Agile methods don't do this.
>>
>> Jon****
>>
>> _______________________________________________
>> The System Safety Mailing List
>> systemsafety_at_xxxxxx >>
>> ** **
>>
>> _______________________________________________
>> The System Safety Mailing List
>> systemsafety_at_xxxxxx >>
>>
>
>
>
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety_at_xxxxxx >

-- 
Derek M. Jones                  tel: +44 (0) 1252 520 667
Knowledge Software Ltd          blog:shape-of-code.coding-guidelines.com
Software analysis               http://www.knosof.co.uk
_______________________________________________
The System Safety Mailing List
systemsafety_at_xxxxxx

_______________________________________________
The System Safety Mailing List
systemsafety_at_xxxxxx
Received on Tue Sep 03 2013 - 09:34:31 CEST

This archive was generated by hypermail 2.3.0 : Thu Apr 25 2019 - 04:17:05 CEST