Linux dedicated server RAM use increasing over time.

[AcidTDP]

Hi folks, I run a dedicated server for myself and some friends on a server I own (not a VPS) that is running CentOS 7.

Things have been going great for a few weeks, no real problems. However, 2 days ago somebody mentioned the server was a bit laggy. The physical server lives in the same rack as my work-related servers on a 1Gbps dedicated connection, in a datacentre in Germany. So I had to take a look to see if the problem was with the datacentre, and therefore affecting my production servers as well.

What I found, was that the 7dtd server executable was using 22GB of RAM on a box with only 16GB  RAM, meaning that virtual RAM was being used partially. For weeks it had been hovering between 9GB and 12GB, depending how many players were online. We run a 16k map, with the number of zombies and animals set quite high (several hundred). I closed the server down and restarted it, and it went back to around 10GB of RAM being used. But 12 hours later when I checked, it was back over 20GB of RAM being used. 

I have reduced the draw distance (8 down from 12) and halved the max number of zombies and animals, to see if this improved things, even though the server had been running fine previously with the original settings. But yet again, a few hours after restarting the server the RAM usage was approaching 20GB again. 

The only issues we have had prior to this, are several players getting the annoying console errors that are caused by electrical fences, something that has been documented elsewhere, and I can't imagine this has caused the server to start using almost twice as much RAM as before. I should also state that this server was setup after the latest update, so this is not a problem with a save from a previous version causing a problem.

EDIT: Forgot to add that this happens even when nobody has logged in to play. A few hours after being started the server has increased RAM usage of about 100%.

As you can see from the attached server log, there seems to be nothing that could indicate what this problem is being caused by. Does anyone have any ideas?

Thanks.

View attachment output_log__2020-10-05__04-21-39.txt

 

[meganoth]

Since it seems that your problem started only after playing the same world for a few weeks I would say it is highly likely the memory leak only shows up if some condition is true for your game world. That condition is likely a bug the game can't handle correctly.

And seriously console errors are the truest form of seeing a bug in action. So my bet is squarley on your electric fences being the culprits. Removing them and rebooting the server might have a chance to fix the memory leak.

 

[AcidTDP]

And seriously console errors are the truest form of seeing a bug in action. So my bet is squarley on your electric fences being the culprits. Removing them and rebooting the server might have a chance to fix the memory leak.

If it wasn't for the fact that the electric fence problem was happening a couple of weeks before that RAM issue started I would agree with you. However, I believe there is a file named power.xml or similar that I can delete and it will remove all wiring. I'll give that a try if my players agree to having to redo all their electrical connections. 

 

[meganoth]

Ok, sounds much less likely now with weeks inbetween. Another thing to try would be checking if the RAM usage goes up even if no players log in. 

If no (which is very likely), create a new pristine character preferably with a new PC or account so he needs to download the world and has clean settings again, log in. Does the RAM usage go up?

If yes, sending in a bug report or starting a new world are options, not much else I can see, the damage must be in the world files.

If no, you can either check if it is one specific player triggering it or simply have everyone delete their local caches first and check it that is already enough to fix the issue.

 

[AcidTDP]

I already said that it goes up even if nobody is logged in.

My guess is something is causing a recursive error, grabbing a few bytes of memory each time and it is mounting up. CPU usage has not changed. I did have a problem I had forgotten about. Around a week ago my own player data got corrupted somehow when my connection dropped. I remade my character but maybe something is left over causing the problem. 

I think we are keen on starting Darkness Falls soon, so the current map won't be played for a while. So I guess its just a case of restarting it every few hours to keep the RAM usage down until we start a new game.

 

[meganoth]

I already said that it goes up even if nobody is logged in.

Ah, right, sorry. So I assume this really means not even after starting server log in for a moment, log out, then leaving it running. Really pristine. That means the world files on the server are having the problem.

My guess is something is causing a recursive error, grabbing a few bytes of memory each time and it is mounting up. CPU usage has not changed. I did have a problem I had forgotten about. Around a week ago my own player data got corrupted somehow when my connection dropped. I remade my character but maybe something is left over causing the problem. 

I think we are keen on starting Darkness Falls soon, so the current map won't be played for a while. So I guess its just a case of restarting it every few hours to keep the RAM usage down until we start a new game.

 

[AcidTDP]

Yep, completely fresh, just start it up and not logging in at all, not even to check it. Doing that, the RAM usage goes up by about 1GB an hour. 

Actually, I have just seen that I was wrong, and that  CPU usage has gone up significantly. CPU usage is around 20 times higher with this save, than when running a fresh map. I had not noticed before until I tried this test, as I have plenty of spare CPU resources on this box. 

I think I will just have to chalk this one up to early access blues, and start from scratch. This time I will setup my server backup to save to a different file every day, rather than overwrite the previous days save. Then if something similar happens I can step back in time to before it started. 

 

[SylenThunder]

A 16k map isn't feasible at this time. 12k is the highest you can go and have it remain stable. This has been the case since the changes to RWG in a17/a18.

 

[AcidTDP]

A 16k map isn't feasible at this time. 12k is the highest you can go and have it remain stable. This has been the case since the changes to RWG in a17/a18.

I should probably have explained that I am using a Nitrogen map.  I will say that for our small group of about 6 people, a 16k map has ben overkill.

 

[SylenThunder]

I should probably have explained that I am using a Nitrogen map.  I will say that for our small group of about 6 people, a 16k map has ben overkill.

Doesn't matter if it's Nitrogen or not. Over 12k will experience sync issues and memory leaks. It's pretty much guaranteed.

 

[AcidTDP]

Doesn't matter if it's Nitrogen or not. Over 12k will experience sync issues and memory leaks. It's pretty much guaranteed.

Logically, those same issues must exist with smaller map sizes, but the effect is less pronounced. I am going to switch over to a smaller map, and I will keep data on the amount of RAM used over time. 

 

[Beelzybub]

Logically, those same issues must exist with smaller map sizes, but the effect is less pronounced.

The memory leak doesn't exist on the smaller maps, or on the larger map. You've already reported it working fine

for a while.

The issue with the larger map is that it is very likely to get corrupted over time. Once it's corrupted, the memory leak

starts. If you corrupted a smaller map the same way, it too would display the same symptoms.