Many comments and complaints have been heard regarding the latency and disconnections that have been occuring recently. We have had several seperate incidents that have accumulated together creating an overall bad game play experience in the past 7 weeks.
I want to talk about what has happened in the past week. We have been making numerous optimizations over the past months to find sources of lag and eliminate it. Things were going as well as expected with such a long term project, when our normally stable ISP began having its own problems. We had several downed routers throughout the internet (not all of them from our ISP) when that happens the network traffic redirects to less optimal routes that creates lag for the users. This is normal, it sucks when it happens but it is not an odd occurance in the internet world.
And then things got real.. a constuction project severed the lines from our Data Center completely cutting WarpPortal off from the world for almost a full day. This was repaired, but things were never quite the same. We began analyzing all packets coming in with a monitoring tool that was on loan, and it was illuminating for where bottlenecks were on our back end network (game servers, routers and DBs). Many optimizations were made over the weeks and regular improvements were noticed, which is expected given the size of such a project. However, at that time those pesky routers were not behaving and many connections between ISPs were very poor routings which causes more latency than would otherwise be there. We could see these problems as well, and reported them to the ISP which then negotiated and worked with the responsible people for the routers.
Things were fairly well sorted out by Tuesday of last week, there was 1 bad router that was being looked into on NTT side, and then the DDOS started on Thursday which made our services unavailable. This DDOS continues through now. Early this morning we hoped to perform a repair and check session at our IDC, this requires help from the Data Center and ISP staff. It was cancelled due to a work scheduling problem. We are doing the appropriate steps to mitigate the DDOS, which is extra annoying because the attack vector is fluctuating, and not persistent. We have taken appropriate prevention and repair procedures to lessen the damage, but due to the way internet traffic must work it is not possible to "turtle up" and still get legit traffic to get through. Our ISP and IDC is working with us but a final solution that doesn't kill normal traffic only comes with time or when the DDOS person stops (or is stopped).
I do expect the the latency and DCs to continue at some level for the next day or so. I do know we will be providing some treats as an apology and to help get you where you should have been. We will weather the storm and come out ahead in the end.
Merging all lag stuff here.
TL;DR
SERVERS ARE BEING HACKED BY A DISGRUNTLED PLAYER D:
for those of you who are still confused: wikipedia
Edited by CHiSEEN, 30 November 2010 - 08:01 PM.
