GB/LON02 Routing Instability (SN-20200505-1)

Created on Tuesday 5 May at 10:42 UTC
Resolved on Thursday 14 May at 15:27 UTC.

Issue Summary

Please be aware that we are currently investigating a Routing Instability within LON02 (Telehouse North).

Our engineers are investigating the issue and we will provide an update here shortly.

Need Help?

If you believe that your services have been affected by this incident, please raise a support ticket online at https://secure.vorboss.net

If the matter is urgent you may also contact the Vorboss Network Operations team on per the knowledge base page KB15052001

Issue Updates

Please note that these updates are in reverse order (most recent updates first).

Wednesday 6 May at 08:11 UTC

The works as part of the Emergency Maintenance Window last night were successful and services have been stable throughout the night. Follow-up works are still underway and will be referenced in the RFO document.

 

Please contact Vorboss directly via the Support Portal to request the RFO.

Tuesday 5 May at 19:45 UTC

Works to increase the speed of our responses to similar attacks are well underway. We will continue to provide updates here as the works progress and when the extended mitigation is in place.

Tuesday 5 May at 17:11 UTC

While services are currently stable, we are currently in the process of improving our ability to mitigate the attacks faster, so that the impact to our customer isn't noticeable, should a similar attack re-occur.

We will continue to provide updates here this evening as these works progress.

Tuesday 5 May at 17:36 UTC

Please be aware that while services are currently stable, we are leaving this case open while we continue to work on a solution to prevent this happening.

We will continue to provide updates here throughout the evening as this progresses.

Tuesday 5 May at 16:53 UTC

The most recent incident occurred between 15:15 UTC and 15:22 UTC. Our engineers are continuing to work on the issue and we will provide updates here as the situation progresses.

Tuesday 5 May at 15:56 UTC

Please be aware that we are currently seeing a further period of instability. Our engineers are still investigating the situation and we will provide a further shortly.

Tuesday 5 May at 15:47 UTC

Please be aware that we are seeing further network instability. Our engineers are currently investigating the situation and we will provide an update here shortly.

Tuesday 5 May at 11:21 UTC

Please note the address on the previous update was incorrect.

For details on the full RFO, please contact Vorboss on the Support Portal.

Tuesday 5 May at 11:13 UTC

Please be aware that the majority of services are now stable.

This issue was caused by two large targeted denial of service attacks. Full details will come out in the RFO.

The time of these two attacks were 10:02 UTC and 10:26 UTC. We have since taken steps to mitigate these and will continue to monitor the situation closely.

Please contact support@ai.co.uk to request the full RFO when available.

Tuesday 5 May at 10:08 UTC

Please be aware that this issue is still ongoing, we have been able to stabilize the majority of affected services, however, we cannot yet rule out any further disruption at this time.

We will provide further updates as the situation progresses.

Tuesday 5 May at 10:21 UTC

We are seeing subsequent issues and brief instabilities ongoing.

Our engineers are still working on resolving the issue fully. As soon as services are stable we will provide an update here.

Tuesday 5 May at 10:18 UTC

We have been able to stabilize the device in question and most services should now be restored.

However, we are still seeing an issue with our LINX interconnect which we are still working on resolving.