12/2 Latency, Disconnections and what is being done about it
#1
Posted 30 November 2010 - 05:30 PM
I want to talk about what has happened in the past week. We have been making numerous optimizations over the past months to find sources of lag and eliminate it. Things were going as well as expected with such a long term project, when our normally stable ISP began having its own problems. We had several downed routers throughout the internet (not all of them from our ISP) when that happens the network traffic redirects to less optimal routes that creates lag for the users. This is normal, it sucks when it happens but it is not an odd occurance in the internet world.
And then things got real.. a constuction project severed the lines from our Data Center completely cutting WarpPortal off from the world for almost a full day. This was repaired, but things were never quite the same. We began analyzing all packets coming in with a monitoring tool that was on loan, and it was illuminating for where bottlenecks were on our back end network (game servers, routers and DBs). Many optimizations were made over the weeks and regular improvements were noticed, which is expected given the size of such a project. However, at that time those pesky routers were not behaving and many connections between ISPs were very poor routings which causes more latency than would otherwise be there. We could see these problems as well, and reported them to the ISP which then negotiated and worked with the responsible people for the routers.
Things were fairly well sorted out by Tuesday of last week, there was 1 bad router that was being looked into on NTT side, and then the DDOS started on Thursday which made our services unavailable. This DDOS continues through now. Early this morning we hoped to perform a repair and check session at our IDC, this requires help from the Data Center and ISP staff. It was cancelled due to a work scheduling problem. We are doing the appropriate steps to mitigate the DDOS, which is extra annoying because the attack vector is fluctuating, and not persistent. We have taken appropriate prevention and repair procedures to lessen the damage, but due to the way internet traffic must work it is not possible to "turtle up" and still get legit traffic to get through. Our ISP and IDC is working with us but a final solution that doesn't kill normal traffic only comes with time or when the DDOS person stops (or is stopped).
I do expect the the latency and DCs to continue at some level for the next day or so. I do know we will be providing some treats as an apology and to help get you where you should have been. We will weather the storm and come out ahead in the end.
************************
Update December 2,
So we have finally gotten a good read on the metrics data from the attack... And yes it is a ZDOS this means infected computers around the internet are a part of the coordinated attacks. While this likely isn't surprising, the data from the history is suggesting this DDOS may have been slowly building since October and is a contributer to the overall lag...
Matching those IPs that are hitting us now matches many IPs of normal RO players.. so with that in mind I request
VIRUS SCAN YOUR COMPUTERS!!!!1!
#2
Posted 30 November 2010 - 05:34 PM
My gameplay experience is never good at the best of times due to my location but this week has been extra horrible and today especially i have been D/Cd 4 or 5 times and killed due to lag 10 maybe 12 times, which having bought some HEBMs from the kafra shop for the first time to try and make an effort in levelling hasn't left me thrilled to bits to say the least. I have used 5 so far and i havn't gained anything at all.
And to top things off i wait ages to get into geffenia only to be D/Cd and find myself back at the fountain again, minus EXP and a HEBM still ticking away.... not good Heim, not good.
#3
Posted 30 November 2010 - 05:35 PM
#4
Posted 30 November 2010 - 05:36 PM
#5
Posted 30 November 2010 - 05:36 PM
It may piss a few people off, but in the long run it should prove to be better for the player base if you hold off on any kind of compensation until players are actually able to play.
Edited by Kahlev, 30 November 2010 - 05:37 PM.
#6
Posted 30 November 2010 - 05:40 PM
I hope we get that one quest proposal that was made during the OCA fiasco, just spread it out for everything and keep the cute hat reward. Kind of like a "Yeah, things used to be , but now let's laugh about it!" kind of thing.~ (And since it takes 1-6 months for an item approval, maybe a voucher that can substitute and have a turn-in NPC later on.)
#7
Posted 30 November 2010 - 05:40 PM
#8
Posted 30 November 2010 - 05:43 PM
#9
Posted 30 November 2010 - 05:47 PM
#10
Posted 30 November 2010 - 05:47 PM
#11
Posted 30 November 2010 - 05:49 PM
#12
Posted 30 November 2010 - 05:56 PM
I agree with Kahlev... There's "please don't hate us" compensation, and then there's "Hell yes! Everything is fine again" compensation. One has a better feel to it than the other, and not to mention has a better chance of being used to it's fullest.
I hope we get that one quest proposal that was made during the OCA fiasco, just spread it out for everything and keep the cute hat reward. Kind of like a "Yeah, things used to be , but now let's laugh about it!" kind of thing.~ (And since it takes 1-6 months for an item approval, maybe a voucher that can substitute and have a turn-in NPC later on.)
Yes, +1 to the quest I made.
#13
Posted 30 November 2010 - 06:07 PM
#14
Posted 30 November 2010 - 06:08 PM
As much as i like treats, please don't give away anything until the problem appears to have been solved.
Seconded. Or Thirded by now. Who knows.
#15
Posted 30 November 2010 - 06:12 PM
Server went from 300 people to like 170 after I got d/cd. Ping was above 500. This is just sort of ridiculous.
#16
Posted 30 November 2010 - 06:12 PM
#17
Posted 30 November 2010 - 06:16 PM
#18
Posted 30 November 2010 - 06:24 PM
#19
Posted 30 November 2010 - 06:25 PM
#20
Posted 30 November 2010 - 06:27 PM
blame Insur! punch them and put it on youtube
nah i duno...hope u get it fixed! x
#21
Posted 30 November 2010 - 06:27 PM
Does Gravity know if the DDOS attacks can be linked to a certain ip address or something? I noticed on DS last week it was fine to play until yesterday.
#22
Posted 30 November 2010 - 06:36 PM
I blame Insur for the DDOS.
i heard they all went to ygg. or back to WoW, which is preferred
#23
Posted 30 November 2010 - 06:40 PM
#24
Posted 30 November 2010 - 06:40 PM
Agreed.As much as i like treats, please don't give away anything until the problem appears to have been solved.
#25
Posted 30 November 2010 - 06:51 PM
Its just unfortunate that the community has to suffer the past 2 days and not being able to enjoy the 150% Experience Event.
I hope the "treats" will be given after these latency/disconnect issues have been fixed.
0 user(s) are reading this topic
0 members, 0 guests, 0 anonymous users