Dedicated Server: Massive performance drop during Bloodmoon since switching to A21

[theMickey_]

I've been running a 7D2D server on my 2nd PC with decent hardware on A20 for a couple of months, and a group of 3-5 people (including me ) were playing on it without any issues. No performance issues whatsoever. And although during Bloodmoon the (client side) FPS dropped a bit, it still was playable. We did use a custom 10k map with vanilla POIs only back then.

A few weeks ago we started playing A21. A new custom 10k map (vanilla POIs only), and I setup the server with pretty much the same settings: yes, we've changed the difficulty and such, but AFAIK nothing that should impact any performance. But during Bloodmoon all of our (client side) FPS drop to 10-15 FPS, which is horrible and almost unplayable -- and I don't know why.

Looking at my server logs, there's one massive difference though: the "server FPS" on A21 is never going past 20, it seems to be "hard locked" at 20. This wasn't the case in A20.

Here's a log entry from our (perfectly fine working) A20 Bloodmoon (3 player, 40 zombies, 69 entities) at 50 FPS:

2023-08-26T20:54:59 10040.209 INF Time: 167.14m FPS: 50.71 Heap: 4412.1MB Max: 4687.2MB Chunks: 447 CGO: 41 Ply: 3 Zom: 40 Ent: 69 (77) Items: 7 CO: 13 RSS: 5778.5MB




And this is a log entry from our last Bloodmoon on A21 (4 player, 47 zombies, 81 entities) at 20 FPS:

2023-12-31T01:14:50 10012.125 INF Time: 166.47m FPS: 19.77 Heap: 3271.0MB Max: 3442.8MB Chunks: 859 CGO: 50 Ply: 4 Zom: 47 Ent: 81 (240) Items: 13 CO: 15 RSS: 4872.3MB




There's no way I messed up the server config for A21 so bad that we get those massive performance drops. Again, even outside of Bloodmoon the server side FPS never gets higher than 20.0, which is different from A20.

What am I missing? Is there a new setting to "lock" the server side FPS which I now have to change?

Any help would be much appreciated! Thank you guys a lot in advance!

 

[theMickey_]

Another reason why I think the server side FPS in A21 is kinda "locked" or "capped": if I start our old A20 server and login, the server side FPS immediately increases (it was 20 when the server was idling):

2023-12-31T12:45:35 276.059 INF GMSG: Player 'theMickey' joined the game
2023-12-31T12:45:35 276.063 INF PlayerSpawnedInWorld (reason: JoinMultiplayer, position: 1154, 61, -107): EntityID=171, PltfmId='Steam_xxx', CrossId='EOS_xxx', OwnerID='Steam_xxx', PlayerName='theMickey'
2023-12-31T12:45:52 292.336 INF Time: 4.62m FPS: 36.48 Heap: 4294.4MB Max: 4294.4MB Chunks: 298 CGO: 18 Ply: 1 Zom: 0 Ent: 2 (2) Items: 0 CO: 1 RSS: 5506.4MB
2023-12-31T12:46:22 322.358 INF Time: 5.12m FPS: 30.69 Heap: 4273.3MB Max: 4294.4MB Chunks: 298 CGO: 18 Ply: 1 Zom: 0 Ent: 2 (2) Items: 0 CO: 1 RSS: 5506.2MB
2023-12-31T12:46:52 352.398 INF Time: 5.63m FPS: 31.42 Heap: 4318.3MB Max: 4318.3MB Chunks: 298 CGO: 18 Ply: 1 Zom: 0 Ent: 2 (2) Items: 0 CO: 1 RSS: 5506.2MB
2023-12-31T12:47:22 382.417 INF Time: 6.13m FPS: 30.58 Heap: 4373.2MB Max: 4373.2MB Chunks: 298 CGO: 18 Ply: 1 Zom: 0 Ent: 2 (2) Items: 0 CO: 1 RSS: 5506.6MB




If I do the exact same thing on A21 (starting the server, login in), the server side FPS stays at exactly 20:

2023-12-31T12:38:48 449.882 INF GMSG: Player 'theMickey' joined the game
2023-12-31T12:38:48 449.897 INF PlayerSpawnedInWorld (reason: JoinMultiplayer, position: 1305, 45, -943): EntityID=172, PltfmId='Steam_xxx', CrossId='EOS_xxx', OwnerID='Steam_xxx', PlayerName='theMickey', ClientNumber='1'
2023-12-31T12:38:51 453.436 INF Time: 7.09m FPS: 20.00 Heap: 2125.4MB Max: 2125.4MB Chunks: 298 CGO: 27 Ply: 1 Zom: 1 Ent: 3 (3) Items: 0 CO: 1 RSS: 3035.6MB
2023-12-31T12:39:21 483.473 INF Time: 7.59m FPS: 20.00 Heap: 2162.1MB Max: 2162.1MB Chunks: 298 CGO: 27 Ply: 1 Zom: 1 Ent: 3 (3) Items: 0 CO: 1 RSS: 3072.6MB
2023-12-31T12:39:51 513.488 INF Time: 8.09m FPS: 20.00 Heap: 2146.7MB Max: 2162.1MB Chunks: 298 CGO: 27 Ply: 1 Zom: 1 Ent: 3 (3) Items: 0 CO: 1 RSS: 3089.8MB
2023-12-31T12:40:21 543.518 INF Time: 8.59m FPS: 20.01 Heap: 2096.2MB Max: 2162.1MB Chunks: 298 CGO: 27 Ply: 1 Zom: 1 Ent: 3 (3) Items: 0 CO: 1 RSS: 3105.9MB

 

[SylenThunder]

Yes, server FPS being capped is intentional.  Getting 30FPS on a20 is a bit mid, as the upper end was in the 50's then. In a19 it was 40, and prior to that it was 30. Capping the FPS is done intentionally to optimize performance.

Please post the complete log file. Instructions are Pinned, and should have been read before posting.

 

[theMickey_]

Please post the complete log file. Instructions are Pinned, and should have been read before posting.

Thanks! In fact I did read this before I posted. I also used the search to find a solution, but to no avail. Then I assumed that posting the complete 800kb log file wouldn't be useful at all, that's why I only posted the parts I am worried about. But here you go -- this is the full log file, but again, I doubt it being very useful to answer my question: https://justpaste.it/cnt4r

Yes, server FPS being capped is intentional.  Getting 30FPS on a20 is a bit mid, as the upper end was in the 50's then. In a19 it was 40, and prior to that it was 30. Capping the FPS is done intentionally to optimize performance.

In A20 we usually had like 35-50 (server side) FPS, although it seemed to be capped at 20! Only during Bloodmoon it dropped to like 30-35 FPS. As said before I did launch our A20 server again and used "settargetfps" without any parameters and it did tell me that the FPS is capped to 20 -- which is the same return I get when using the same command on A21. But in A20 the FPS increases to 35-50 FPS (server side!) as soon as someone logs in, in A21 it stays at 20 at most.

 

[SylenThunder]

Ok, so you may not be happy with what I'm going to put out here, but here goes.

1. That CPU is a bit dated. I would put it near the bottom end for hosting the current dedi with the kind of mod setup you have. It has core speed going for it, but it is pathetically light on the number of cores, so it's basically bare minimum. And you're running it on Windows which saps even more performance.

2. MaxSpawnedZombies is set to high for your CPU. If you were on maybe a 3900X that might be ok. It for sure is just fine if you're sporting something like a 5950X. For your CPU though, that should not be higher than 80. It doesn't matter that you "compensated" by making meat hard to find with max animals set to 15. 

3. Your logs shows the name as "A21-Big-Cities", and larger cities is going to tax the CPU, RAM, and Disk bandwidth for that hardware. Especially since you are most likely limited to SSD's with that setup. (IIRC, the m.2 slot is not NVMe compliant for that generation of hardware.) Larger cities means a lot more blocks to calculate structural integrity on, and to manage AI pathing. It means loading and unloading a lot of data from disk storage into RAM. 

Throughout the run of the server, and even during bloodmoon, the server FPS is just fine. So by increasing your hardware to meet the demand of your settings, or adjusting your settings to match your limited hardware, you should be able to find a balance.  The server client itself is not struggling to perform in your situation.

Switching to Linux and making some changes in your settings may fix it temporarily, but you are still going to have some hard limits.

I myself started noticing this with our similarly aging hardware in a20, and is why I built the new server that we are using now.

 

[theMickey_]

Thank you for your input.

But the main question remains: why did this "outdated" hardware run a 10k map with massive cities and a similar set of mods perfectly fine on A20 and is struggling so hard on A21? Although I'm familiar with Linux, I'm not switching to Linux (neither as main OS nor as dual-boot) to run an A21 server just to get a bit more performance, eventually. The server's CPU might be outdated, but it's barely doing anything during Bloodmoon -- I've checked. Not a single core is above 30% utilization, and there are no I/O waits. We've just been able to run a Tier 6 Infestation in the Higashi Tower in the center of a huge town on A21 which hasn't been an issue at all, although this might be one of the POIs with the highest demand when it comes to CPU performance. Our base for Bloodmoon is outside any major city, pretty much as we built our base in A20 outside towns.

I still don't get it. A20 was perfectly fine (same amount of MaxSpawnedZombies and animal spawns left at the default of "50" -- the change of animal spawns in A21 was just a to figure out whether or not it might improve the performance; it doesn't though, so I will revert it back to 50).

Again: Same hardware, same OS, same size and of style of a 10k map, same server settings, same kind and amount of mods -- A20 is fine, A21 is literally unplayable during Bloodmoon.

 

[SylenThunder]

a20 is completely different from a21 in terms of what the server is doing, what the demand is, how the data is executed, ect. 

a21 has had huge changes in how the server runs, and that is a large reason why it performs poorly on older hardware. If you have older AMD hardware from a similar generation, it almost won't run at all.

And do note, that even though you aren't seeing your CPU spiking or long I/O waits, does not mean that you still do not have a bottleneck. If the client is waiting on your system to process data, it isn't going to be spiking the CPU. 

 

[theMickey_]

Thanks again, but I'm really confused now: you're saying the server is having bottlenecks because of the outdated CPU (and I don't doubt that)-- but I'm not able to see them? So no CPU spikes, not a single core, no I/O waits, no memory issues etc. ...?! That never happened to me before. Usually, if an app/program/game is having issues, I should be able to identify at least some kind of "spikes" somewhere, right?

Anyways: I'm not going to "upgrade" my old PC anymore, so I guess I'll just have to rent a 7D2D server then. But what specs should I be looking for? "8 vCPUs" without knowing what hardware they're running on and how many shared servers are running on the same physical machine doesn't give me any clue on whether the server will run smoothly or not. And how many GB of memory do we need for a 10k map with a couple of big cities? Again: if I can't see how much resources the A21 server is actually demanding on my own hardware, how will I be able to select a decent server?

Thanks again for your input, and maybe you can add some more details on what I should look for when trying to find a server...

 

[SylenThunder]

yeah it isn't going to peg CPU cycles when it's basically in a wait state for the CPU to do something.  The 7th-gen series is one of the crappy gen's from Intel. Basically it isn't a new and better chipset than the 6th-gen, it's just a 6th-gen chipset overclocked a bit and given a new label. Add to that the limited number of cores and it just becomes a slow ride all around. I have a 7700 and a i7-3930k. If I overclock the 7700 to 4.2GHz, the 3930k runs circles around it once it also hits 4.2GHz OC. And the 3930K overclocks higher, so the two extra cores add up to a lot and even though the 3930k was generations behind, it still ran the server better than the 7700.  (The i7-3930k OC'ed to 5.1GHz actually ran a wee bit faster than an i7-8700k.)

Sadly I have a LOT of older hardware laying around here to test with, so there is a good chance I have been able to throw the client or server onto it for testing. I try to do a lot of this with the internal builds before they are released to the public so I have a good idea of how a system will perform when someone has an issue like yours, or has lagging FPS in the game client.

Most hosting companies are using 12th-gen Intel chipsets or Zen 3/4 Ryzen. Those CPU's will be adequate. For RAM your will want 16GB with a 10k map. More if you plan to run any overhauls. Disk is preferred to be NVMe, but SSD's in RAID will work fine.

 

[Beelzybub]

Anyways: I'm not going to "upgrade" my old PC anymore, so I guess I'll just have to rent a 7D2D server then.

I think something else is going on that is causing your issues. Could you answer some more questions?

1. Why is this line in your log "2023-12-30T22:27:58 0.047 INF Last played version: Alpha 18.4"
 

A new custom 10k map (vanilla POIs only

2. How did you generate this "custom" map?

 

[Beelzybub]

Also, you have a bunch of zombie mods that, just as a test, you could disable those and see if things improve.

If I was you I would do some testing to see just how much your server can handle in A21 by making a new vanilla

10k world using a new seed and default settings. Get your 4 friends to help you test and see how many zombies it

can handle before things get weird. If it runs fine, then add in your mods and see if it now does not run fine. Then

you've found your problem.

 

[theMickey_]

1. Why is this line in your log "2023-12-30T22:27:58 0.047 INF Last played version: Alpha 18.4"
2. How did you generate this "custom" map?

1. I don't know why it shows 18.4 as "Last played version" -- as I said before, I've been running (multiple) A20 server on the same machine before.
   
2. I've used version 0.48.2 of Teragon to create this map.
   

Also, you have a bunch of zombie mods that, just as a test, you could disable those and see if things improve.

The only "Zombie Mod" we use is a mod that increases wild zombie spawns and and the size of the Wandering Horde by 4 (this does not affect the Bloodmoon Horde, which is the only time the performance of the game becomes unplayable). The other mods on the server are for a slightly different UI, for changing the melee ranges for zombies, for higher stack sizes of selected items and for additional quests types. We've used pretty much the exact same mods on A20 -- but I will deactivate those to test it, but I doubt it will improve a lot during Bloodmoon.

 

[Beelzybub]

I don't know why it shows 18.4 as "Last played version"

Generally that happens when things were not cleaned up between alphas.

Like Sylen already said (paraphrasing) it looks like you're trying to do too much with your server.

You may disagree, but you can empirically find the limits of what you have with some effort.

I run a vanilla server on worse hardware than you and don't have these issues. So

somewhere between Vanilla and what you have is the problem. (imo). Also, your confidence in

mods behaving themselves in unwarranted.