As most of you are aware by now the entire RunUO network was offline for the better part of 12 hours yesterday. This post is an attempt to explain the issues we had, what it means for you as users and how we're planning to mitigate it from happening in the future.
At approximately 13:00 EST the entire network destabilized to the point it was virtually unusable, followed by a total and complete collapse around 15:00 EST.
The cause of this outage is now known, the service provider who runs our core infrastructure in Chicago had a complete meltdown of their Cisco 6509 Core switching platform. This issue required Cisco to come on site and provide a fix for a very serious software issue. This took nearly 12 hours to complete and at this point the network is again stable for the time being.
While this outage was completely out of our hands losing the entire network is completely unacceptable and cannot happen again in the future.
To keep this from happening again we are going to be separating the game server services and the web infrastructure while completely getting rid of the current provider. The original thought was to source another provider in Chicago to handle the game servers but our search was fruitless for a cost effective provider to meet our high demands/requirements. We've elected to move to the game servers to Dallas Texas.
While I recognize some of you may not like the choice of moving the game servers to Dallas we're choosing a world class facility run by Internap:
http://www.internap.com/colocation-provider-facility-overview/dallas-data-center/
This facility houses some of the biggest companies in the world and will provide us with a much more stable platform for UOGamers going forward. We have yet to determine where we will be moving the web sites to but will update you as soon as we have that information. For the time being they will stay in Chicago providing us redundancy in the event of a failure in the future.
Another interesting thing is that the facility (being Internap) utilizes the Internap Flow Control platform. This is widely regarded as the end-all-be-all of internet bandwidth optimization and can dynamically re-route or re-allocate routes to provide all of you with the best routing and latency possible worldwide. I'm providing the following test IP's for you to utilize but keep in mind the full effect will only be realized when we're online and you're an active IP in their network and the routing to you is fully optimized.
Test IPv4: 199.231.226.12
I recognize how painful these last 24 hours have been and I appreciate the amount of patience the vast majority of you have shown. This was not an easy time and the holidays always make for a very tight time for these larger un-budgeted items like an emergency move.
Thank you again for your support of RunUO and UOGamers and we look forward to many more years with you!
Update 1: Demise move is complete. World load time dropped by about 80% and save times are about 40% better. Please enjoy.
Update 2: Hybrid move is complete. World load time dropped by about 25% and save times are down about 30%! This is an amazing improvement.
Thank you for your support!
At approximately 13:00 EST the entire network destabilized to the point it was virtually unusable, followed by a total and complete collapse around 15:00 EST.
The cause of this outage is now known, the service provider who runs our core infrastructure in Chicago had a complete meltdown of their Cisco 6509 Core switching platform. This issue required Cisco to come on site and provide a fix for a very serious software issue. This took nearly 12 hours to complete and at this point the network is again stable for the time being.
While this outage was completely out of our hands losing the entire network is completely unacceptable and cannot happen again in the future.
To keep this from happening again we are going to be separating the game server services and the web infrastructure while completely getting rid of the current provider. The original thought was to source another provider in Chicago to handle the game servers but our search was fruitless for a cost effective provider to meet our high demands/requirements. We've elected to move to the game servers to Dallas Texas.
While I recognize some of you may not like the choice of moving the game servers to Dallas we're choosing a world class facility run by Internap:
http://www.internap.com/colocation-provider-facility-overview/dallas-data-center/
This facility houses some of the biggest companies in the world and will provide us with a much more stable platform for UOGamers going forward. We have yet to determine where we will be moving the web sites to but will update you as soon as we have that information. For the time being they will stay in Chicago providing us redundancy in the event of a failure in the future.
Another interesting thing is that the facility (being Internap) utilizes the Internap Flow Control platform. This is widely regarded as the end-all-be-all of internet bandwidth optimization and can dynamically re-route or re-allocate routes to provide all of you with the best routing and latency possible worldwide. I'm providing the following test IP's for you to utilize but keep in mind the full effect will only be realized when we're online and you're an active IP in their network and the routing to you is fully optimized.
Test IPv4: 199.231.226.12
I recognize how painful these last 24 hours have been and I appreciate the amount of patience the vast majority of you have shown. This was not an easy time and the holidays always make for a very tight time for these larger un-budgeted items like an emergency move.
Thank you again for your support of RunUO and UOGamers and we look forward to many more years with you!
Update 1: Demise move is complete. World load time dropped by about 80% and save times are about 40% better. Please enjoy.
Update 2: Hybrid move is complete. World load time dropped by about 25% and save times are down about 30%! This is an amazing improvement.
Thank you for your support!