Server desynchronization

K_3_F_1_R

Refugee
Version
EXP 2.2b3
Platform
Windows
Hello, I would like to address the community and the technical aspects of the game. I don’t understand what the problem is and whether it’s a game bug or a technical limitation of the server itself.

The issue is as follows: Players are calmly playing on the server, completing quests, and building their base day after day. They reach tiers 5-6 of quests when the trader gives quests for the following locations:

  • bunker_00 (Area 7)
  • mine_01 (Hybrid Mining)
  • quarry_02 (Correa Rockworx)
When a player goes to complete one of these quests—clicks the marker [Activates the quest]—everything freezes. The server is running, there are no errors, everything seems fine, but each player on their PC can move around, shoot, etc. However, everything around them freezes, and after 5 to 15 seconds, synchronization occurs, and everything starts moving again. Some players get teleported to where they had moved during the freeze.

If two players are near each other and one activates the quest, one player sees the other standing still on their screen, while that player has actually moved far away on their own PC.

What could be the cause—is it a technical issue on my end, or is it poor optimization of these locations?

Dedicated server specifications:

 
Reproduction Steps
This error occurs only when you activate these locations
Link to Logs
https://drive.google.com/file/d/141tlmH3NeZOqT007tohDRjU5QbdkxKha/view?usp=sharing
Link to Screenshot/Video
https://www.youtube.com/
And here is a video of what it looks like with the default settings and vanilla version.
There are no players on the server.
The zombies are moving, but the POI (Points of Interest) are not updating and not reacting to the location change in any way.
The locations used for testing were Tier 5 "Fetch and Clear" quests:

  • mine_01 (Hybrid Mining)
  • quarry_02 (Correa Rockworx)
The bunker_00 (Area 7) location also doesn't work.
In all other locations, all POIs available in the game work fine.

UPD: When my friend joined, we repeated these actions and the zombies did not move.

 

Attachments

Managed to reproduce around 10 seconds freeze and investigated it under sampling profiler.

gRdfbQb.png


Relevant code is this thing from Chunk.SetBlock:
Code:
        bool flag = !blockValue.isair && _blockValue.isair;
        if (flag)
        {
            this.waterSimHandle.WakeNeighbours(x, y, z);
            if (blockValue.Block.StabilitySupport)
            {
                MultiBlockManager.Instance.SetOversizedStabilityDirty(vector3i);
            }
        }

and this from MultiBlockManager.SetOversizedStabilityDirty:
Code:
                foreach (KeyValuePair<Vector3i, MultiBlockManager.TrackedBlockData> keyValuePair in this.trackedDataMap.OversizedBlocks)

My guess is that problematic POIs contain a ton of blocks that are placed into the world as transition from non-air to air. Maybe underground air blocks?
And that transition is problematic because it iterates over all oversized blocks (simple mod reports 960 oversized blocks for my map after server start, before any player connections) for each POI (underground air?) block.
Iteration itself is non-trivial, SubsetAccessor.get_Current looks up blocks via hash lookup, not trivial index access.

Reproduction steps:
1) Install and configure dedicated server
2) Connect to it
3) Do something to increase the oversized blocks count that server is aware of. For example fly around a city, or do a visitmap console command, not sure what is most effective.
4) Teleport to a problematic POI for example quarry_02, and run "givequest tier6_clear_infested" console command/delete quest until you get a relevant quest
5) Kill some blocks to get an indication of quest activation completion.
6) Start the quest
 
Last edited:
Maybe the game client could also be coded to ignore the fake E-Cores in modern Intel CPU's. Other game engines are doing this where the cores will cause a problem with performance on CPU-heavy tasks.
More than half of the CPU cores in the OP's chipset are completely useless for the game, but Windows will try to put the client on them anyway.
 
When a chunk gets reset from a quest the server has to do some heavy lifting in some cases, and that can affect performance in a few ways:
  1. Unload Old Data
    • The server clears all block/entity data for that chunk from memory from bedrock to sky.
    • Any player-built blocks, loot containers, or entities in that chunk from bedrock to sky are deleted.
  2. Rebuild from Prefab/World Gen
    • The chunk is regenerated using its original prefab or random-gen template.
    • The server runs world-gen code to re-populate terrain, POIs, loot containers, and decorations.
  3. Save to Disk
    • The new chunk state is written back into the save files.
    • This means extra I/O load, especially on HDDs vs SSDs.
  • CPU Spikes: The quest reset involves recalculating terrain and regenerating structures, which can briefly spike CPU usage.
  • Disk I/O: Writing/rewriting chunk data can cause latency, particularly on slower drives.
  • Network Sync: If players are nearby, the server must re-send the updated chunk data to them, adding extra network load.
  • Temporary Lag: Nearby players might see a stutter, hitch, or rubberband as the server finalizes the reset and pushes the new chunk.
  • How many chunks reset at once: mass resets (e.g., players doing quest) can cause major lag effecting the whole server and players.
👉 In short: Resetting a quest prefab chunk causes temporary CPU + disk I/O spike, and if players are in the area, it can result in lag or rubberbanding. On good hardware mass quest resets can hammer server performance.
 
Maybe the game client could also be coded to ignore the fake E-Cores in modern Intel CPU's. Other game engines are doing this where the cores will cause a problem with performance on CPU-heavy tasks.
More than half of the CPU cores in the OP's chipset are completely useless for the game, but Windows will try to put the client on them anyway.
I switched the processor to a 9950X and everything became good. The servers themselves started working better. Once again I was convinced that Intel's processors are not very good.
 
When a chunk gets reset from a quest the server has to do some heavy lifting in some cases, and that can affect performance in a few ways:
  1. Unload Old Data
    • The server clears all block/entity data for that chunk from memory from bedrock to sky.
    • Any player-built blocks, loot containers, or entities in that chunk from bedrock to sky are deleted.
  2. Rebuild from Prefab/World Gen
    • The chunk is regenerated using its original prefab or random-gen template.
    • The server runs world-gen code to re-populate terrain, POIs, loot containers, and decorations.
  3. Save to Disk
    • The new chunk state is written back into the save files.
    • This means extra I/O load, especially on HDDs vs SSDs.
  • CPU Spikes: The quest reset involves recalculating terrain and regenerating structures, which can briefly spike CPU usage.
  • Disk I/O: Writing/rewriting chunk data can cause latency, particularly on slower drives.
  • Network Sync: If players are nearby, the server must re-send the updated chunk data to them, adding extra network load.
  • Temporary Lag: Nearby players might see a stutter, hitch, or rubberband as the server finalizes the reset and pushes the new chunk.
  • How many chunks reset at once: mass resets (e.g., players doing quest) can cause major lag effecting the whole server and players.
👉 In short: Resetting a quest prefab chunk causes temporary CPU + disk I/O spike, and if players are in the area, it can result in lag or rubberbanding. On good hardware mass quest resets can hammer server performance.
The server has become faster to process information, but if you activate the quest, there are no errors on the server side, but there are errors on the Client side in F1

Failed setting triangles. Some indices are referencing out of bounds vertices. IndexCount: 198, VertexCount: 37
Failed setting triangles. Some indices are referencing out of bounds vertices. IndexCount: 198, VertexCount: 37
"" mesh must have at least one non-degenerate triangle to be a valid collision mesh.
"" mesh must have at least one non-degenerate triangle to be a valid collision mesh.
 
The server has become faster to process information, but if you activate the quest, there are no errors on the server side, but there are errors on the Client side in F1

Failed setting triangles. Some indices are referencing out of bounds vertices. IndexCount: 198, VertexCount: 37
Failed setting triangles. Some indices are referencing out of bounds vertices. IndexCount: 198, VertexCount: 37
"" mesh must have at least one non-degenerate triangle to be a valid collision mesh.
"" mesh must have at least one non-degenerate triangle to be a valid collision mesh.
the game supports 8 players
with 8 players running around questing...
there is no denying it causes many performance issues.
 
Back
Top