Type of Announcement: Unplanned Downtime
Description:
On Saturday 28th at 09.17 Node4’s network management system reported alarms relating to multiple devices and services in our Data Centres. Initial investigations showed that connectivity between our NMS and multiple switches within the DC was intermittent, at this point the impact of this issue was not known and no common cause could be found.
Engineers investigated the fault and traced the problem to DC1. DC1 was isolated and full service to DC2 was restored at 09.50.
The cause of the issue at this point was unknown however the result was that some of our core switches had been operating at 100% cpu and therefore connectivity to some hosting customers had been severely disrupted.
Engineers suspected that a network loop had occurred in the DC1 shared hosting VLAN, this had resulted in trunk links being saturated and the spanning tree topology becoming unstable. Due to the nature of the fault standard troubleshooting analysis was ineffective and engineers therefore manually shutdown ports isolating areas of the network until the source of the problem was found.
The problem was traced back to a redundant/secondary connection in DC1, this port was immediately disconnected. Investigations are continuing as to the exact cause of the fault on this equipment, however the result was that a Spanning Tree loop was created in the network. All services were restored by 12.40, although many customers will have seen their service restored prior to this time.
Node4 follow best practice design and configuration within our infrastructure, however due to the specific nature of this fault the affect was wide spread. Node4 engineers worked quickly and effectively to identify the source of the problem and restore service. Node4 will be undertaking a review of our configurations in order to identify if anything further can be done to mitigate these types of faults and reduce their impact should something similar happen again.
Service(s) affected: Internet Access for all users