Active Incident

Updated a few seconds ago

DCXGYR1 Intermittent Internet Transit ConnectivityPartial Service Disruption

Incident Status

Partial Service Disruption

Components

Internet Transit

Locations

DCX-GYR1, DCX-GYR1-DS1, DCX-GYR1-DS2


August 22, 2025 7:36AM MST
[Identified] A DCX edge router was found to be suffering from hardware failure coupled with incorrect configuration on a redundant edge router resulted in the service disruption. Configuration on the redundant edge router has been corrected and service has been restored although redundancy is in a reduced state. Will continue to troubleshoot the problematic edge router.

August 22, 2025 5:53AM MST
[Investigating] Alerts have been received indicating that the DCX internet fabric is seeing intermittent connectivity resulting in service disruption.

Critical Power




Operational

Critical Cooling




Operational

Internet Transit




Partial Service Disruption

Dashboard




Operational

Access Control




Operational

Facility




Operational

Optical Transport Network




Operational

3

Upcoming Maintenances

4

Incidents Last 30 Days

4

Maintenances Last 30 Days

External Services

Cloudflare

Salesforce

Status.io

Locations

Scheduled Maintenance

DCXGYR1 DS102 Troubleshoot ECS ComponentsPlanned Maintenance

Schedule

September 4, 2025 9:00AM - 5:00PM MST

Components

Critical Power, Critical Cooling

Locations

DCX-GYR1-DS1, DCX-GYR1-DS2

Description

The redundant power supply for the controls system for DS102 indicates failure. Troubleshooting will require access to main power distribution cabinets. No impact to customer load expected.
DCX OTN Dark Fiber Provider MaintenancePlanned Maintenance

Schedule

September 16, 2025 1:00AM - 7:00AM MST

Components

Optical Transport Network

Locations

DCX-GYR1, DCX-PHX2

Description

The dark fiber provider for the DCX OTN has a planned maintenance that will be impactful to the DCX OTN south route. They have indicated that during the change window, it is expected that services will be be impacted for up to 2 hours. If you have services that currently utilize the DCX OTN south path, it is advised to fail traffic to another non-impacted route during the change window.
DCX OTN Dark Fiber Provider MaintenancePlanned Maintenance

Schedule

September 17, 2025 1:00AM - 7:00AM MST

Components

Optical Transport Network

Locations

DCX-GYR1, DCX-PHX2

Description

The dark fiber provider for the DCX OTN has a planned maintenance that will be impactful to the DCX OTN south route. They have indicated that during the change window, it is expected that services will be be impacted for up to 2 hours. If you have services that currently utilize the DCX OTN south path, it is advised to fail traffic to another non-impacted route during the change window.

History (Last 7 days)

DCX OTN OutageService Disruption

Incident Status

Service Disruption


Components

Optical Transport Network


Locations

DCX-GYR1, DCX-PHX2



September 1, 2025 4:59PM MST
[Resolved] The dark fiber provider has indicated that as of 4:50pm MST that they are now hands off. Services remained stable as they worked to completion of the root issue. The issue is now resolved.

September 1, 2025 12:58PM MST
[Monitoring] The dark fiber provider has completed installation of a replacement cable and splicing of the impacted fiber for DCX. They are still working to splice other fibers. Connectivity is showing restored. Will continue to monitor until the technicians are hands off of the fiber.

September 1, 2025 5:15AM MST
[Identified] A break has been identified in the south path dark fiber. The dark fiber provider identified the event immediately and began triage. They have communicated that the break is near the same location as the last event and is again due to vandalism (a 432 count fiber cable was cut). Construction teams will begin repairs at 5:15am MST. Priority has been placed on the fibers supporting the DCX OTN.

September 1, 2025 5:06AM MST
[Investigating] DCX Support ticket is SUP-27

September 1, 2025 5:03AM MST
[Investigating] Received alerts are approximately 12:53am MST that the DCX OTN south path composite line has loss of light.
DCX OTN OutageService Disruption

Incident Status

Service Disruption


Components

Optical Transport Network


Locations

DCX-GYR1, DCX-PHX2



August 29, 2025 7:19AM MST
[Resolved] The dark fiber provider has indicated that as of 6:30am MST that they are now hands off of the splice case. Services remained stable through the night as they worked to completion of the root issue. Issue is now resolved.

August 28, 2025 7:21PM MST
[Monitoring] The dark fiber provider has completed splicing of the impacted fiber for the DCX OTN. They are still working to splice other fibers and place into a new case. Connectivity is restored. Will continue to monitor until the technicians are hands off of the fiber.

August 28, 2025 6:08PM MST
[Identified] The dark fiber provider indicated that a large fiber bundle was vandalized and cut (432 fibers) of which the south path fibers were impacted. They provided an ETA of 4-6 hours to complete the repairs starting at approximately 4:30pm MST. As it is now 6:00pm MST, repairs should be completed in the next 2.5-4.5 hours.

August 28, 2025 3:51PM MST
[Identified] Dark fiber provider has indicated that they are still continuing to work the issue. They have indicated that impact is not just limited to DCX and have other customers who are experiencing service disruption.

August 28, 2025 12:25PM MST
[Identified] Dark fiber service provider has identified a fiber break and is currently dispatching to have repaired.

August 28, 2025 12:20PM MST
[Investigating] DCX Support ticket is SUP-24

August 28, 2025 12:14PM MST
[Investigating] The DCX OTN south path composite line is currently down.
EMERGENCY - DCX OTN Dark Fiber Provider MaintenancePlanned Maintenance

Description

The dark fiber provider for the OTN has indicated that the south path dark fiber will be undergoing emergency maintenance during which connectivity may be interrupted. It is advised to utilize a different path if possible during the change window.


Components

Optical Transport Network


Locations

DCX-GYR1, DCX-PHX2


Schedule

August 27, 2025 2:00AM - August 27, 2025 4:00AM MST


August 27, 2025 4:00AM MST
[Update] Scheduled maintenance is complete.

August 27, 2025 2:00AM MST
[Update] Scheduled maintenance is starting.
EMERGENCY - DCX OTN Dark Fiber Provider MaintenancePlanned Maintenance

Description

The dark fiber provider for the OTN has indicated that the south path dark fiber will be undergoing emergency maintenance during which connectivity may be interrupted. It is advised to utilize a different path if possible during the change window.


Components

Optical Transport Network


Locations

DCX-GYR1, DCX-PHX2


Schedule

August 26, 2025 2:00AM - August 26, 2025 4:00AM MST


August 26, 2025 4:00AM MST
[Update] Scheduled maintenance is complete.

August 26, 2025 2:00AM MST
[Update] Scheduled maintenance is starting.