[SystemSafety] Software Glitch Caused 911 Outage For 11 Million People

From: Taylor Johnson < >
Date: Wed, 22 Oct 2014 10:57:08 -0500

Software Glitch Caused 911 Outage For 11 Million People http://it-beta.slashdot.org/story/14/10/22/1428214/software-glitch-caused-911-outage-for-11-million-people

How a dumb software glitch kept thousands from reaching 911 http://www.washingtonpost.com/blogs/the-switch/wp/2014/10/20/how-a-dumb-software-glitch-kept-6600-calls-from-getting-to-911/

"...But on April 9, the software responsible for assigning the codes
maxed out at a pre-set limit; the counter literally stopped counting at 40 million calls. As a result, the routing system stopped accepting new calls, leading to a bottleneck and a series of cascading failures elsewhere in the 911 infrastructure..."

FCC Report of Incident:

"The outage was caused by a software coding error in the Colorado
facility, and resulted in a loss of 911 service for more than 11 million people for up to six hours. Over 6,600 calls to 911 never reached a PSAP. Although, fortunately, it appears that no one died as a result, the incident - and the flaws it revealed - is simply unacceptable. Americans rely on 911 as a reliable way to communicate in an emergency, and lapses like this cannot be permitted."

"At 11:54 p.m. PDT20 on April 9, 2014, the PSAP Trunk Member's (PTM)
counter at Intrado's
Englewood, Colorado, ECMC exceeded its threshold and could send no more 911 calls to PSAPs
using CAMA trunks. Under normal operations, the PTM assigns a unique identifier for each call
that terminates using CAMA trunks. This is how Intrado has implemented the ATIS protocol
commonly used to complete 911 calls over CAMA trunks, which (unlike SS7) require additional
features to carry the signaling along the TDM path."

"In this case, the trunk assignment counter reached a pre-set capacity
limit to assign trunks, which meant that no additional database entries to reserve a PSAP CAMA trunk could be created, no trunk assignments for call delivery could be made for PSAPs with CAMA trunks and, therefore, no 911 calls could be completed to these PSAPs or any backup PSAP through the Englewood ECMC."


Taylor Johnson
Assistant Professor
Computer Science and Engineering
University of Texas at Arlington
817-272-3610 (office)
Engineering Research Building 559 (office)
Engineering Research Building 105B (lab)
The System Safety Mailing List
Received on Fri Oct 24 2014 - 15:09:35 CEST

This archive was generated by hypermail 2.3.0 : Tue Jun 04 2019 - 21:17:06 CEST