Stopping lag.. I could use some help.

Leaving the vehicles on the ground isn't inherently bad. However many of the servers exceeding known stable values have had issues with them. A few have a job that wipes the vehicles.dat at every restart and just tell their players not to leave them laying about.


Not inherently bad, no, but the corrupted vehicle.dat file causing the vehicle issues isn't related to big servers, can repro it with one person in a SP game too. Just conjecture, gotta catch up with the rest.

 
Not inherently bad, no, but the corrupted vehicle.dat file causing the vehicle issues isn't related to big servers, can repro it with one person in a SP game too. Just conjecture, gotta catch up with the rest.
I see.  

File is locked. You can use justpaste.it or open it up.  

Work is especially busy tonight, so may take me a bit to get through it.

 
So I run a server with a healthy population.  After a few hours the server starts to lag. I use all the tools available, such as RAT.

Now, here is my stats, I run my server on a 2016 VM on a Dell host. THe stats are:

1.  PowerEdge R720 with Duel  E5-2697 v2 @ 2.70GHz

2.  256GB of ram, of which 16GB is dedicated to the VM.

3.  8 CPU's are dedicated to the VM.

4.  The server is running on a 10 RAID of 10K drives.

The server lagging after awhile makes no sense, this is with about 25 people.

It gets annoying to the players that I have to restart every few hours to fix this.  What is causing this lag after awhile.  Everyone's ping is below 100, some as long as 10.

What is causing this, the rubber banding gets worse and worse etc.  Zombies don't move.

Can I get some help on this please.
I had/have alot of issues with EAC and lag sometimes, depending on the 7d2d patch, steam patch and EAC updates and windows .netframework probz.
My suggestion would be to make sure windows has allllll its updates, disable EAC (At least to test and see if it reduces lag, if it does, probably EAC bottlekneck, nothing you can do cept disable EAC) and double check steams bad self for updates.
If youve already done that, ignore me and best of luck :x (Im assuming windows? Again, ignore me if youve already checked this stuff.)

 
Last edited by a moderator:
If your vehicles.dat file has been corrupted I have seen servers be brought to their knees due to invisible dupes. Known issue but deleting it will cure it. Just have everyone pick their vehicles up first, and yeah ones left out does allow the issue to happen.
Does the vehicles have to be on the player or can they be on storage?  I got a marker for a minibike that disappeared on me and couldn’t find it with god mode enable.

I been having issues with my bow firing as soon as I go to pull back on the string snd wondering if this is tied to it.

 
OK, I always leave my vehicles laying about and have never had issues to my knowledge. What triggers the corruption?


That is a gigantic, recursive can of worms, to put it lightly. Every cause is sporadic; I had to put a BOLO for a game save with a vehicle dupe in it and I spent many hours (besides the many hours investigating outside of it) debugging/breaking/investigating/reproing various issues in that save. It's nested deep in the backend, although enough information should be there to get it fixed. Pretty much the way the vehicle manager is handled sometimes causes it to mix up the stored data, sometimes you'll see a vehicle reappear where you left it before, a parked one reappearing in a different nearby spot, etc. Related to unload/reloading chunks/regions. Once it breaks, it turns into a waterfall; you'll spot many vehicles saved in the manager when you only have one or two or whatever and have invisible ones everywhere.

I will look at this log later on tomorrow while working myself. 

Does the vehicles have to be on the player or can they be on storage?  I got a marker for a minibike that disappeared on me and couldn’t find it with god mode enable.

I been having issues with my bow firing as soon as I go to pull back on the string snd wondering if this is tied to it.


No only vehicles handled in the world. Maybe. I do believe they are temporarily stored as idle (the x/z saved is x active z inactive) for a moment when you put them away but I'm pretty confident they aren't active as far as the vehicle manager is concerned. But, you can test this by going to where your minibike vanished and do a "killall all" in the console; if it's there invisible, it will blow up. Deleting the vehicle.dat file will only wipe vehicles placed in the world (or, invisible duped ones), so pick up the ones you want and delete it. 

There is also an inventory glitch, that's another issue with the manager. Both have been worked out for reproduction, though.

 
Ok right at the start, the server FPS is 20. Then it gets up to the 30's.

Then after you've had a couple of people connect, you get this error...

Code:
2021-07-11T08:29:43 625.636 ERR Could not save file 'E:\steamcmd\7_days_to_die_server\Saves/MerlinsBeard/MerlinsCape\Player/76561199149499235.map': Sharing violation on path E:\steamcmd\7_days_to_die_server\Saves\MerlinsBeard\MerlinsCape\Player\76561199149499235.map
2021-07-11T08:29:43 625.637 EXC Sharing violation on path E:\steamcmd\7_days_to_die_server\Saves\MerlinsBeard\MerlinsCape\Player\76561199149499235.map
IOException: Sharing violation on path E:\steamcmd\7_days_to_die_server\Saves\MerlinsBeard\MerlinsCape\Player\76561199149499235.map
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x0019e] in <567df3e0919241ba98db88bec4c6696f>:0 
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode) [0x00000] in <567df3e0919241ba98db88bec4c6696f>:0 
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode)
  at DatabaseWithFixedDS`2[KEY,DATA].Save (System.String _dir, System.String _filename) [0x00027] in <858b342e2ec84f598a70e52b667e9627>:0 
UnityEngine.DebugLogHandler:Internal_LogException(Exception, Object)
UnityEngine.DebugLogHandler:LogException(Exception, Object)
UnityEngine.Logger:LogException(Exception, Object)
UnityEngine.Debug:LogException(Exception)
Logger:masterLogException(Exception)
Logger:Exception(Exception)
Log:Exception(Exception)
DatabaseWithFixedDS`2:Save(String, String)
MapChunkDatabase:SaveAsync(TaskInfo)
ThreadManager:myQueuedTaskInvoke(Object)
System.Threading.QueueUserWorkItemCallback:System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
System.Threading.ThreadPoolWorkQueue:Dispatch()
System.Threading._ThreadPoolWaitCallback:PerformWaitCallback()
The server FPS for a while varies from 29-32. Serviceable, but not a lot of room for overhead. It does occasionally pick up to 35 or 36.

I'm seeing a number of errors for corrupted map tiles, and the above error appears a couple more times.

Sharing violation is most commonly a result of not excluding the client from security software. It could also be a result of some other process accessing the files, the drive not performing well enough to handle requests in a timely fashion, or a simple permission issue.

 
Ok right at the start, the server FPS is 20. Then it gets up to the 30's.

Then after you've had a couple of people connect, you get this error...

Code:
2021-07-11T08:29:43 625.636 ERR Could not save file 'E:\steamcmd\7_days_to_die_server\Saves/MerlinsBeard/MerlinsCape\Player/76561199149499235.map': Sharing violation on path E:\steamcmd\7_days_to_die_server\Saves\MerlinsBeard\MerlinsCape\Player\76561199149499235.map
2021-07-11T08:29:43 625.637 EXC Sharing violation on path E:\steamcmd\7_days_to_die_server\Saves\MerlinsBeard\MerlinsCape\Player\76561199149499235.map
IOException: Sharing violation on path E:\steamcmd\7_days_to_die_server\Saves\MerlinsBeard\MerlinsCape\Player\76561199149499235.map
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x0019e] in <567df3e0919241ba98db88bec4c6696f>:0 
  at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode) [0x00000] in <567df3e0919241ba98db88bec4c6696f>:0 
  at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode)
  at DatabaseWithFixedDS`2[KEY,DATA].Save (System.String _dir, System.String _filename) [0x00027] in <858b342e2ec84f598a70e52b667e9627>:0 
UnityEngine.DebugLogHandler:Internal_LogException(Exception, Object)
UnityEngine.DebugLogHandler:LogException(Exception, Object)
UnityEngine.Logger:LogException(Exception, Object)
UnityEngine.Debug:LogException(Exception)
Logger:masterLogException(Exception)
Logger:Exception(Exception)
Log:Exception(Exception)
DatabaseWithFixedDS`2:Save(String, String)
MapChunkDatabase:SaveAsync(TaskInfo)
ThreadManager:myQueuedTaskInvoke(Object)
System.Threading.QueueUserWorkItemCallback:System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
System.Threading.ThreadPoolWorkQueue:Dispatch()
System.Threading._ThreadPoolWaitCallback:PerformWaitCallback()
The server FPS for a while varies from 29-32. Serviceable, but not a lot of room for overhead. It does occasionally pick up to 35 or 36.

I'm seeing a number of errors for corrupted map tiles, and the above error appears a couple more times.

Sharing violation is most commonly a result of not excluding the client from security software. It could also be a result of some other process accessing the files, the drive not performing well enough to handle requests in a timely fashion, or a simple permission issue.


Okay so I am going to give the EVERYONE permission to the 7 Days Server directory. Start that way.

I have no security software, except  for Windows Defender, and the Firewall is off as well.

What else should I look at?

 
If I were to move this to an SSD RAID 0, or RAID 1, what SSD would you recommend?
Honestly, don't use anything but Samsung Evo's.

1. They are supported by your RAID controller.

2. They will last for literal years.

3. They are faster than most others.

We have multiple R710's with all the drive slots filled. Here is what we've run on them.

  • Our Evo 850's lasted almost ten years. One of which is still in use.
  • I tried Kingston drives. After about a year they were at 80% wear, and speed was reduced by 40%.
  • I tried Crucial drives. They didn't last 6 months. Critical drive failures across the board. Got them RMA'ed, next set did the exact same thing.
  • Tried Inland Professional. Same end result as the Kingston's, except we started seeing the drops in performance at about 6-9 months. I don't recall what the health was on these when I pulled them offhand. I think it was around 90%. I do know that when we pulled them, they were almost as slow as a freaking platter drive.
So now our array is almost fully Evo 860's with long term media storage being on Seagate Ironwolf drives. (Note that WD NAS drives are not worth the metal they are printed on.)

Our current drive layout.

On Miata the Linux OS is on a pair of 15k SAS drives in RAID-0 (650GB). The servers run on a pair of 860's in RAID-0 (500GB). The game save data is on 4 860's in RAID-0 (1TB). And the backups are stored on a pair of Seagate Firecuda SSHD's in RAID-0 (2TB), and mirrored to Challenger.

On Challenger the Windows OS is on an Evo 850 (120GB, health is at 93%, and estimated life remaining is about two years. It's only ever been a Windows drive). The servers and save data run on three 860's in RAID-0 (750GB). Media storage and all backups are on a Seagate IronWolF (8TB). One slot is currently reserved for another IronWolf drive that we are saving up for, but prices have been pretty steep for a year now.

The next system is planned to have NVMe's, and I am considering testing Sabrient or another brand I cannot for the life of me recall the name of now.

 
I know the EVo's are good, but I got better performance on a SiliconPower NVME compared tot he same size Evo at half the price. Been using it for a few years now, no issues. My SP chip has the same NAND that the Samsung did, so...

 
Sadly my board only has a single NVME slot, so no RAID. I do run a BTRFS RAID10 array of WD Black, single-platter disks for normal games (ie: Not Ark or Cyberpunk), but those are SATA disks.

You do realize that NVME is right on the PCI-E bus, right? NVME is faster than RAID on SATA even with a single disk. Mine averages around 3,500~3,600MB/s reads and around 3,300~3,500MB/s writes. If I was on Linux and could use F2FS instead of NTFS it would be even higher.

 
Last edited by a moderator:
You do realize that NVME is right on the PCI-E bus, right? NVME is faster than RAID on SATA even with a single disk. Mine averages around 3,500~3,600MB/s reads and around 3,300~3,500MB/s writes. If I was on Linux and could use F2FS instead of NTFS it would be even higher.
Absolutely correct. Huge difference between NVMe and the SSD's on our servers. And additionally between NTFS and F2FS. I can hit 5802 MB/sec with our fastest array, but that's pretty much the upper limit. (It can hit 6659 MB/sec in performance tests, but the limit for the RAID controller is 6.5GB/s)  You could do that with just a pair of NVMe's in RAID-0.

A standard NVMe array could easily hit 26GB/s with a full array of 6-8 drives. 

The AI server I used to get the 30-player limit on was running a full deck of Intel Optane drives at 45GB/s.

 
If you setup F2FS correctly, it can get far better speeds. The key is the correct block size (-w parameter) and using a good compression (mount option compress_algorithm). Between those two it DRASTICALLY decreases writes, prolonging life of your SSD/NVME and increasing throughput. It also tends to cache a lot of stuff in RAM so it isn't reading and writing as much. It's by far the best flash-based file-system I have used. Gentoo is the only OS that seems to do it right. I tried it on PCLOS and it cannot boot from F2FS but the root partition CAN live on F2FS. I love BTRFS, but for single-SSD setups or where you run RAID and only a logical volume is exposed, F2FS is the heat.

 
So if im reading this right, The server backend is very drive intensive. For now, should i run this off my ISCSi, RADZFS2 array with 10 gbps throughput or keep it on the 10k RAID10 raid until i get my ssds?

Because I honestly can't tell which has better I/O.

 
Last edited by a moderator:
So if im reading this right, The server backend is very drive intensive. For noe, should i run this off my ISCSi array with 10 gbps throughput or keep it on the 10k 10 raid until i get my ssds.
Yes. Especially if you have players spread about. It is actively loading and writing to region files that could literally be gigabytes in size. It's doing it live as changes happen, and as data needs to be loaded when people move around.  It's a large reason for the current limitations on vehicle speeds.

And you need not just fast storage, but a CPU and RAM that can keep up, and the bandwidth on the motherboard to keep up.

Our main Linux array is a little bit overkill, but I set this system up to run several 7 Days servers and a full Ark cluster.  For your particular case, I would advise at least 3-4GB/s throughput on the drives. 

 
Back
Top