Optical Transport Network
January 28, 2026 11:36AM MST
RESOLVED
Root cause was a software bug affecting the south path optical control loop (which provides telemetry data for the composite interfaces for amp transmit levels) essentially stalled. This resulted in the ROADM's inability to signal each other for a desired transmit power resulting in the composite line going down. The troubleshooting steps that were taken after the bugs initial surfacing on 1/14 through 1/16 compounded the issue as the subsystem would restore prior state (even when physically power cycling the equipment).
The vendor provided guidance on 1/16 that resolution consisted of updating the software to latest as it was a known bug but difficult to identify. The software update that was completed on 1/20 did not appear to initially resolve the issue due the effects of the bug being persistent as the subsystem had not yet reset as there was no interruption to the composite link. At approximately 2:40am MST until 3:30am MST on 1/21, there was a small dark fiber event on the south path that caused a loss of light resulting in the optical control loops to hard reset with the new code and everything began functioning as expected.
The south path has been monitored for stability since and has remained stable restoring confidence.January 23, 2026 3:07PM MST
MONITORING
Vendor has reviewed majority of logs and chain of events. They are still working through a few final checks before clearing equipment for return to service. Will be monitoring equipment into next week as well for stability.January 21, 2026 8:15AM MST
MONITORING
An event internal to the equipment occurred at approximately 3AM MST this morning resulting in the resolution of the issue. Currently working with the vendor to understand the behavior and ensure will not surface again.January 20, 2026 11:24PM MST
INVESTIGATING
Completed software update to OTN which did not resolve degradation issues with south path. Working with vendor to continue to troubleshoot.January 16, 2026 12:43PM MST
IDENTIFIED
After consultation with the hardware vendor, it was determined the underlying issue to be a known bug that surfaced on the current software version. A change will be scheduled to update to the latest software version shortly.January 15, 2026 6:40PM MST
INVESTIGATING
The DCX OTN south path is currently experiencing full degradation. Currently investigating root cause. The DCX OTN North path is fully operational.
