Datacenter Fiber Line Failure (Resolved)

[Resolved 16:56 EDT] Our Datacenter’s ISP has informed us that the repair of the fiber line has been made, that services will start to see the difference here over the next hour or so as traffic in the datacenter flows back into proper channels.

[Update 14:35 EDT] We have received the following update from our datacenter’s ISP “Estimated time of return 6.00pm (EDT)”

[Update 13:00 EDT] We are somewhat happy to announce we are back up and running our our Datacenter’s backup lines, service will not be as fast till everything is completely restored. Do expect some packet loss, but our datacenter is doing everything they can to assist in the packet loss. Our datacenter is pushing all their server through one ISP when it is normally distributed between 3 ISPs. The one ISP can take the hit, why they have 3 and not just one. However pushing so much data through one pipe causes major overhead slowing down data for everyone. The good news is the other ISPs are resolving their issue right now and everything should be back to normal automatically very soon. We also want to thank Google for providing us with a Public DNS resolving service, which is why we are able to continue providing you with service during this time.


[Update 12:06 EDT] We are successfully rerouting traffic, please be aware that we are restoring service using our Datacenter’s backbone network, this network is designed in case of an incident of this nature to keep data flow operational. Thanks to Google’s free Public DNS system we are bypassing our datacenter’s overloaded system for resolving hostnames. The backbone network may have high ping times, however is operational. The high ping times are due to all traffic is having to be rerouted through these backup lines, where normally traffic is split among 3 ISP(Internet Service Providers). Please bare with us as we continue to restore service over the backbone network.


[Update 11:52 EDT] Our Datacenter has informed us that their backbone network is still operational but is taking a huge hit. Their name resolving servers are overloaded due to massive overload on their backbone network. We are attempting to bypass our datacenters name service, and using Google’s public main service and restore service on a slightly degraded level until their ISP repairs the ripped line. This way service can still continue. Please stay tuned for more updates!


[Update 11:39 EDT] Rip was find in fiber line at “Boulevard Monseigneur bridge in QB, Canada. A team from the ISP that serves this line has already been dispatched and is in process of repair”


[first]We have just called and spoken with our datacenter, it turns out our sluggish issue is not result of our system. Our data center located in Canada is suffering from a Fiber line outage, they are running on backup fiber lines as we speak till the issue is resolved. That being said, several packets are failing and are causing a major service interruption for all of our clients at the moment as well as all of our data center’s clients. Our servers are up and running, they are working really hard with the Fiber company to restore service. Speaking with a rep from the datacenter they hope to have the issue fully resolved shortly.

For more news and updates on this please follow our blog at http://blog.zetamex.com/ 

Again, we are very sorry but there is nothing we can do during this outage other than prepare services for restart as soon as the fiber line is restored.

Thank You all for understanding.

Leave a Reply

Your email address will not be published. Required fields are marked *