Unannounced maintenance Atrato
|Network Operations Center:||Type: Issue||State: Solved||Start Time: 22-3-2014 18:45:47||End Time: 22-3-2014 22:26:28|
Dear WorldStream Customer,
Currently one of our transit parties (Atrato / Hibernia networks) have unannounced maintenance in their network.
As this maintenance was unannounced our routers had to reroute the traffic. Unfortunately you might have experienced
high latency/packetloss up for several minutes.
Our apologies for the inconvenience.
Update from Atrato - Hibernia networks
18.16 - First 100G linecard flaps occured
19.33 - First customer reports packet loss and extensive problems on transport to AMS-IX
19.36 - On call engineer reloads first switch fabric.
19.37 - On call engineer reloads second switch fabric.
19.39 - On call engineer reloads third switch fabric.
At this point on duty engineer found the flaps cannot be resolved and linecard requires a reload.
19.40 - On call engineer power-cycles 100G line card.
19.43 - On call engineer notices 100G linecard did not come back up after power-cycle.
19.44 - 100G line card is stuck in interactive mode due to fail in loading firmware.
This was caused by a previous upgrade that was done for the maintenance number 581
19.50 - On call engineer initiated a downgrade of the firmware running on the 100G line card.
After numerous attempts to fix the line card, on call engineer found out there is no quick fix for the problem.
Therefor on call engineer called a manager to confirm it is best to reload the line card.
21.10 - On call engineer initiates reload of the router at Interxion AMS5.
21.25 - Services full restored.
Problem with transit party solved.