Services Status

New Cloud Platform Issues

In October 2017, we brought a new enterprise cloud platform online in test mode. This enterprise platform was to add new features such as fast snapshots, tiered storage, and fail-over redundancy. This platform was tested for 5 months until February 2018, when it went live. As minor issues surfaced, we worked closely with our enterprise vendors to resolve them in a timely manner. However, starting in mid-September 2018, we have experienced several severe performance issues with the platform that ultimately has led us to decide against sustaining the platform long-term. As such, we are now working with our vendors to move the 260TB worth of Virtual Machine (VM) data to a new environment.

This new environment is a previous dedicated environment that we are positive will regain the stability and reliability for our company and customers alike. This transition requires a full re-sync of the 260TB of VM data, which is a lengthy and cumbersome process. The challenge has been to keep the existing platform within performance specifications while we move our customers to the new stable environment. While this re-sync is in process, customer VM's will not perform within specification. Our engineers will continue to work 24/7 to make this move, while still keeping the cloud platform operational during the transition.

Century Link Outage – November 29 2018

Root Cause: A configuration was applied in error in Raleigh, NC, which impacted IP services.  
Fix Action: The necessary configuration was added to the device to restore services.  

Summary: On November 28, 2018 at 17:44 GMT, CenturyLink identified a service impact in Raleigh, NC. The IP NOC was working to replace the redundant Routing Engine within a device to clear a Loss of Redundancy state.  During the installation of the redundant Routing Engine a configuration was removed incorrectly. When that action was taken the configuration was inadvertently removed from both the primary and redundant Routing Engines causing the outage observed. The IP NOC engaged both Field Operations on site as well as the equipment vendor to assist with restoral. The necessary configuration was obtained from a backup configuration and applied the configurations back to both the primary and backup Routing Engines which lost their configuration as a result of the configuration error. Once the Routing Engines were recovered it was confirmed that all services had restored.  

Corrective Actions: This incident has been reviewed by CenturyLink leadership, the equipment vendor, and the individuals involved in this outage. The configuration method used on the redundant router resulted in the primary routing engine configurations to be lost. Additional training has been implemented with those involved to prevent incorrect configuration applications of this nature and subsequent outages.