Lumen: Redundancy Couldn't Stop 'Fluke' Neb. 911 Outage
“A fluke event outside of Lumen's control” resulted in a nearly statewide 911 outage in Nebraska four months ago, the telecom company’s attorney Katherine McNamara said during a Nebraska Public Service Commission hearing Thursday. During the livestreamed session, Lumen and 911 officials said the outage resulted from contractors' accidental fiber cuts: The first occurred in Minneapolis Aug. 30 and the second in Omaha Aug. 31. The 911 outage lasted from the evening of Aug. 31 until early morning the next day (see 2309010021).
Sign up for a free preview to unlock the rest of this article
Export Compliance Daily combines U.S. export control news, foreign border import regulation and policy developments into a single daily information service that reliably informs its trade professional readers about important current issues affecting their operations.
Nebraska PSC Chair Dan Watermeier urged Lumen to “think outside the box” and treat such incidents like a “black sky event,” that is, a catastrophe caused by a cyberattack or other event. Be proactive, not just reactive, he said.
The outage touched 41 of 68 public safety answering points (PSAPs) across the state, including legacy and next-generation 911 networks, said Nebraska 911 Director Dave Sankey. The NG-911 emergency services IP network (ESInet) wasn’t affected, he said. Some PSAPs received calls intermittently, some could receive calls only to their administrative lines, and others could get no calls, he said. Sankey said the RapidSOS system worked, so 911 centers could still map calls with callback information. That let some PSAPs return calls through their administrative lines or personal cellphones, he added.
Local and long-distance lines were down in addition to 911, said Buffalo County Sheriff Neil Miller. "It was pretty clear we had a major event going on." Calls couldn't get into the ESInet, he said. Buffalo County didn’t receive notification from Lumen for more than an hour after the outage was discovered, said Miller.
Lumen had appropriate redundancy, attorney McNamara of Fraser Striker argued. Third parties caused two fiber cuts in separate locations, and there was a “perfect storm of factors including delays by third-party contractors in Minnesota in repairing fiber ... and a contractor's failure to locate Lumen’s fiber in Omaha." Since the cuts, “Lumen has taken steps to improve its end-to-end diversity.”
Yet having two diverse routes doesn’t eliminate the possibility of a pair of unrelated events knocking out both, as occurred here, said Drew Groff, Lumen Network Operations Center director-public safety services. An influx of federal broadband funding for more projects requiring digs could increase the risk of cable cuts in the future, said Groff.
Notifications are automated and usually go out immediately, but the Minneapolis cut prevented notification reaching counties, including Buffalo County, Groff said. The company is taking steps to diversify alarms that generate the notifications, he said. In Minneapolis, the contractor who cut the line had performed a locate request to check for fiber in the digging area but still managed to hit the fiber, said Groff. In Omaha, the contractor didn’t perform a locate request for the area where fiber was cut, he said.
The Nebraska 911 director is working with PSAP officials to craft a better way to quickly gather information in such an emergency and communicate more effectively, Sankey said. They are also working on providing tools that will improve public messaging, he said. “As a result of this emergency ... we’ve all learned some lessons."
Add teeth to the state’s requirement that contractors call 811 before they dig, suggested Miller. Also, Nebraska should find how to more quickly share the 911 network’s health, the sheriff said. PSAPs should consider buying phone lines from multiple companies, not just Lumen, he suggested. "A little bit of diversity can help get calls delivered through the system." Completing the state’s migration to NG-911 could help reduce future problems, he added.