[Resolved?] iRO server lag investigation and community input

All three of iRO's servers are laggy. Hopefully this post will convince you that it's not your imagination or your internet connection.

Have you noticed rubberbanding in-game or a brief stop when spamming skills/items and dismissed it as a you problem? You're not alone. I've gone around and asked multiple members of the community and without exception, every single one of them answered "yes." They were also asked to perform a simple test: Sprint from one end of a map to the other 5 times. I've asked some of them to share their results here if they wish to.

First of all, here is how that's supposed to look [video recorded on euRO Prime]. Notice that the character always keeps moving and never comes to a stop.

Now compare it to a video from iRO

At 00:28 and 02:18 the character comes to a complete stop before continuing on. That's 2 lag spikes within the span of 2 minutes. I encourage everyone (except those playing on a wi-fi connection) reading this thread to perform this test and share their results in this thread, preferably as a video. I can guarantee you that you will observe this brief stop/stutter.

It could be an issue with your connection to the server

Yeah it could be. But it's highly unlikely that every single person--multiple of whom live in close proximity to the servers--had a bad connection to the server.

Then maybe it's just an issue with Sprint?

No, there are further issues observed. Please take the time to watch this video from kRO first

Take note that the monster is never able to ignore and move through cells of Firewall in this clip. Compare this to iRO Chaos:

That's weird, the Novus completely ignored those Firewall cells and walked straight through. As anyone who has played a Mage can tell you, this is not normal. There would be lots dead Mages if it were. Again, I encourage you to perform the test and share results.

But maybe that's just a coincidence and is not indicative of lag

Huh, that monster walking through Firewall coincided with the massive lag spike.

The above test was performed around 20 times and this weird behavior coincided with the lag spike every single time. This leads me to believe that it is a server issue and not a network issue, as Firewall continues to behave normally even if the player is experiencing extreme lag. Refer to the kRO clip, that could be described as extremely laggy. Again, please test and share your results. This test may be a little difficult, so feel free to ignore it if you don't feel confident in your ability to pull it off.

How often do these lag spikes occur?

Frequent enough to be noticeable and irritated by it. They occur at random but you'll observe one every other minute on average. It took under 2 minutes from logging in to finish recording the Novus clip on iRO.

Does it only occur on certain maps?

It is not isolated to any map based on my observations.

Does it only occur at certain times of the day?

Nope, at all times of the day, every day.

Could it be due to server load?

I'm going to guess no, as the issue occurs on Freya and there are barely 100 players logged in.

Could it be due to EAC?

We've had EAC since 2020 and it wasn't an issue until sometime in 2023.

When did the issue start?

It's hard to say exactly when due to the server being on fire every other week during the later half of 2023.

It would be great to have this fixed. Sadly, going through the ticket system has proved to be an exercise in futility. And so I'm posting here in hopes that strength in numbers works.

Special thanks to: phaicm, Sigma, Dia, Wizzzzard, squirrel, and everyone else who responded to my questions about the lag

Edited by Zayaan, 10 June 2024 - 04:37 AM.

Posting my thoughts as requested by a player; succinctly "that is weird".

What is particularly interesting is that the character appears to actually stop movement, i.e. after the "lag" ends the character resumes from where they left off.  This is starkly different than what happens during client:server packet loss, as the character would "snap" to the server's position on the client.  Consequently, I want to assume that this "weirdness" originates from the zone server(s).

Working with that assumption, we know that the zone server "tries to evaluate" state every 20 ms.  This lends to TKM heat inflicting damage 50 times/second (1000 / 20 = 50), as that is the server "refresh" rate.  Fire Wall has similar behavior vs. undead enemies which also consume Fire Wall stacks at a rate of 50 / s, which is why "fast" enemies can "pass through" a Fire Wall cell without consuming it (i.e. they move faster than 1 cell / [20 * 14] ms).  Expanding on the latter observation, should the zone be under "load", the refresh rate drops and refresh rate dependent skills behave differently (i.e. TKM heat dps drops, decay enemies pass through Fire Wall more readily, etc.).  This looks like a state refresh drop (TK position did not update).

As to why that happens, it could be many things.  It could be supplemental load on the zone server (e.g. new NPCs/monsters that consume/exhaust evaluation intervals), a memory leak, or something else.  Without context, it is difficult to conclude anything with confidence (e.g. it could be another process on the machine/VM that consumes CPU/memory resources resulting in the disruption).

If I were to attempt to debug, I'd start by attempting to find if there were a "pattern" (e.g. does this happen every 2 minutes on the dot?).  Once a pattern has been identified, back trace to recent modifications to see if new behavior aligned with the observed schedule (e.g. if a new NPC//process had a sleep interval of 2 minutes, the same as the disruption frequency, that'd merit further investigation).

I can attest to what Zyn explained above regarding lag spikes.

In my 3min sprint video, I come to a complete stop at 0:29, 1:06, 2:19, 2:45, and 2:56.

With each lap/pass, there's visible signs of stuttering and rubber banding.

I can also confirm occasional rubber banding when snapping with my Rebel which is very annoying.

Here's my testing that I did with autoattacking. Notice there's lag spikes twice in the video.

One at 0:36, other at 1:13

Edited by FishDeity, 11 May 2024 - 01:46 AM.

what's the purpose of making the video of lag spike?

what's the purpose of making the video of lag spike?

It's not actually "lag" in the traditional sense. The RO server itself is failing to process everything on it for roughly half a second. This means that the entire game is essentially frozen (for everyone) for the duration of the "lag" spike. If someone else were to be observing you, they would see the same "lag" or freeze at the exact same time you did.

I conducted my own testing and tried to do the same thing that was posted by Zayaan and then I tried what FishDeity did in terms of autoattacking the training dummy

In the following video I used Sprint in alberta on a Star Emperor and I observed stuttering (brief noticeable pauses, a split second long) as well as complete pauses at the following time stamps:

Edited by CaptainSunbear, 12 May 2024 - 05:42 AM.

Just adding my test results. First video is a soul linker sprinting in Alberta. Second video is auto-attacking at 193 ASPD.

Edited by Rheinheart, 12 May 2024 - 04:03 PM.

The issue its persistent and doesnt arise from specific game actions like firewall, icewall, auto attack, etc. To conduct the following test i standed still doing nothing for 30 minutes and sending CZ_PING packets to measure the latency each 1 second in a empty ZoneServer in this case pay_gld (128.241.92.43:4507). This rules out the possibility of high latency due to overloaded ZoneServer like in Prontera.

The results are telling: latency spikes are a regular occurrence, happening approximately every 1.2 minutes and at times the latency can spike up x3 times the stable rate. Clearly this is resource contention problem or will be seeing the same issues in other official servers using Aegis. Of course identifying the root of the issue falls within the responsability of the development team to profile the ZoneServer, backend resources and review the game scripts to single out any potential cause.

The community has done a lot in this thread to demonstrate the issue from a game perspective, so please address this issue.

Edited by SpookyFenix, 12 May 2024 - 11:59 PM.

I will go on record to state that I too have had the issues with the lag.  It really makes you not want to play the game...

Good day,

Understood, I will follow up this issue to GV HQ to check,

Thank you for sharing detail information

This may seem dumb... Nagle algorithm is the first thing that jumps to mind.

The latency graphs provided seem suspicious to me especially since last I knew the game servers don't accept ICMP traffic. If the servers are virtualized (Would cover the web server answering and the rest of the servers) then I would look at CPU Power saving in the hardware on rails (The base OS and physical hardware, not the virualized OS or hardware) as I had a similar issue with virtualized hardware caused by powersaving kicking in once a minute.

If there isn't a virtiual environment here then the Firewall would be my next target. Set up a test host on the ISP connection but with a different IP (Plug into the ISP modem if possible) and see if the loss/spikes happen there.

To me though its very slight as I'm on the west coast and I barely stop once in a 2 minute window.

The frequency of this has reduced significantly since last maintenance. While the Ice Wall test still fails roughly every 10 full durations, it's infrequent enough to not be bothersome anymore.

And I'm not sure if this is in any way related to whatever measures were taken to mitigate the issue, but monsters inside of instances have been exhibiting some strange behavior at melee range after maintenance.

See:
https://forums.warpp...ance/?p=3048303
https://forums.warpp...ance/?p=3048272
https://forums.warpp...ance/?p=3048253