Dedicated Server Random Crashes

Status
Not open for further replies.
Smugger

Smugger

Refugee

Dedicated Server Random Crashes

Unity Mono Error / 64bit Linux Guest VM

-- UNSOLVED -- NO FIX --

--- BUT POSSIBLE WORK AROUND IN POSTS BELOW ---

Link to Workround

​



I move my Dedicated Server over from a Windows PC to a Linux Server.

It worked great for a while, but then a little while ago, I started randomly crashing, there has been no method to the madness of the crashes.

I tried to collect as much information as I could, like what people were doing when it happened, if they got any errors on their end.

It did not seem to be attached to any particular action or reasoning, in fact, it has even just crashed randomly when no one was on it and no one was on it for a while, sometimes it will go a day with no issues, many times it will be many hours, but sometimes it will crash over and over again back to back like every 15-20 min. I can not figure this our at all.

It have even gotten a couple of crashes when I started it up and it ran for a few hours with no one logging in at all, and then it will just crash.

My current workaround is a script I wrote to check if the server is still up and running, if not it will start it back up again.

Most of the time the process will just completely crash and will not be running after the error, but sometimes when it crashes the process will still be running and it has to be killed.

Google searches have been of no help!

This is on a 64bit linux system, I have all the 32bit libraries

PS: Do anyone know how to get any additional debugging information!?!?

I already know about the loglevel command and how to use it, that provides no additional info.

Here is what the errors tend to look like:

Code: 
Receiving unhandled NULL exception
#0  0x007f682dd38140 in funlockfile
#1  0x007f682dd386d0 in mono_array_element_size
#2  0x007f682dd386e0 in mono_array_new_specific
#3  0x007f682dd38710 in (Unknown)
#4  0x007f682dd38720 in (Unknown)
#5  0x007f682dd38730 in (Unknown)
#6  0x007f682dd38740 in (Unknown)
#7  0x007f682dd38750 in mono_set_defaults
#8  0x007f682dd38c10 in mono_runtime_invoke
#9  0x007f682dd38c40 in mono_runtime_invoke_array
#10 0x007f682dd38d00 in mono_store_remote_field_new
#11 0x007f682dd38d60 in mono_thread_create
#12 0x007f682dd38da0 in mono_thread_create
#13 0x007f682dd38de0 in mono_thread_create
#14 0x007f682dd38e30 in mono_pthread_key_for_tls
#15 0x007f682dd38e50 in GC_end_blocking
#16 0x007f682dd38f00 in start_thread
#17 0x007f682dd38fc0 in clone


Here is the stacktrace info:
Stacktrace:

 at (wrapper managed-to-native) object.__icall_wrapper_mono_array_new_specific (intptr,int) <0x00064>
 at (wrapper managed-to-native) object.__icall_wrapper_mono_array_new_specific (intptr,int) <0x00064>
 at MapChunkDatabase.allocateDataStorage () <0x0001a>
 at DatabaseWithFixedDS`2<int, uint16[]>.read (System.IO.BinaryReader,uint) <0x001a8>
 at DatabaseWithFixedDS`2<int, uint16[]>.Load (string,string) <0x0024c>
 at MapChunkDatabase.LoadAsync (ThreadManager/TaskInfo) <0x0009f>
 at ThreadManager.myQueuedTaskInvoke (object) <0x00101>
 at (wrapper runtime-invoke) object.runtime_invoke_void__this___object (object,intptr,intptr,intptr) <0x00069>

Native stacktrace:

       /home/sdtd/engine/7DaysToDieServer_Data/Mono/x86_64/libmono.so(+0x98673) [0x7f6abc7c3673]
       /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f6ac0096890]
       /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f6abf18ae97]
       /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f6abf18c801]
       /home/sdtd/engine/7DaysToDieServer.x86_64() [0xb13b5c]
       /home/sdtd/engine/7DaysToDieServer_Data/Mono/x86_64/libmono.so(+0xd8f65) [0x7f6abc803f65]
       /home/sdtd/engine/7DaysToDieServer_Data/Mono/x86_64/libmono.so(+0x3721f) [0x7f6abc76221f]
       /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f6ac0096890]
       /home/sdtd/engine/7DaysToDieServer_Data/Mono/x86_64/libmono.so(mono_array_element_size+0x1) [0x7f6abc810dc5]
       /home/sdtd/engine/7DaysToDieServer_Data/Mono/x86_64/libmono.so(mono_array_new_specific+0x1f) [0x7f6abc869d54]
       [0x413457e4]

Debug info from gdb:


=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================
ALSO:

Hey Fun Pimps, why do you not have any Official Support? I should not have to depend on the community for help.

 
Last edited by a moderator:
Read and follow https://7daystodie.com/forums/showthread.php?37912-IMPORTANT-Please-Read-Before-Creating-New-Threads
Especially the stuff about providing the logfile.

As a shortcut, since you have a mono error: Should you see "Exception: Magic number is wrong: 542 " in your logfile, read https://7daystodie.com/forums/showthread.php?123206-Hosting-7DTD-Server-on-Debian , basically the solution is using "export TERM=xterm" before starting the server
Click to expand...
So I just did a search for just 542 in all the log files, and that Exception is no where.

Also I have read about posting a log file, but due to sensitive information (usernames, steam ids, passwords, etc, etc) I do not feel comfortable posting all that information. So I posted the information that is relevant, I am willing to post more information if someone feels that is relevant. But as I said, the crashes are completely random, there is no activity in the log files that points to what is going on.

I will try that export solution, but since you say it has something to do with that exception, and I do not see that exception anywhere in any of the files, I am not sure that will help.

- - - Updated - - -

When I have the time I will write a regex bash script to remove all sensitive info and post the files...

 
Really the only "sensitive info" would be your IP address, and that's not sensitive.

Unless you use your social security number as your username in Windows...

 
I am not about to get into an argument over what I feel is sensitive information or not, since open port information for console and telnet, system information I wish not to share, user names, steamids, users public ip addresses, directory structures, telnet failed login limits / blocktime, I would even consider the random blood moon days to be information I don't wish to share, so please respect that...

 
Last edited by a moderator:
Complete Log (Minus Sensitive Info)

So, I still don't get it. I deleted the old instance, verified all the Dedicated Server Files (So NO MODS). Then recreated the instance, and still getting crashes.

Here is a link to the Log File:

https://pastebin.com/TXbavp9X

I am thinking about just doing a complete server wipe and re-install everything!

Should I do this?

 
Another Link for another Log after removing the ServerIP Property from the config, which did not seem to do anything except remove GamePref.ServerIP from the startup allowing me to force the IP.

The Error that shows up about the GamePrefs.GetBool did not crash the server, it still ran for a long time after that error

Output Log:

https://pastebin.com/JAqErzBF

Stout Log:

https://pastebin.com/xWSDva6K

 
Working with someone on the 7DTD Discord Server, so far nothing is working yet, but he is trying to help. I will post any relevant info after we are finished...

And a solution if we find one...

 
I've run 7D2D servers under LInux for years, and it almost never crashes. I can think of a couple of things:

If it worked great for a while and no longer does, there is a very good chance your world is fubar. Have you tried removing the world *and the world template files, and tried a new world with a different seed?

Are you running on a VM or dedicated hardware? If VM, move it to dedicated hardware.

What distro are you running? Debian would be good, Ubuntu would not. Slackware would be excellent, that is what I use and like I said, it almost never crashes. Try another distro and see if that makes a different. I once had a couple of servers running Ubuntu - that was a disaster. I converted them to Slackware and a LOT of problems went away.

Are you running in a screen session? If so, try running it directly from the console, not a screen session, and see if that makes a difference.

How much memory does your computer have? How much is being used when the server is running? Most importantly, how much is not being used? 7D2D can be a memory pig, and lot of memory is a good thing.

Hardware problems? Try running it on another computer.

Random crashes when server is empty makes me lean towards hardware problems. Try memtst on your memory - burn it for many hours and see if anything pops up. All it takes is one bad bit in the wrong place to cause all of your problems.

 
Full Problem Description / Possible Workarounds

Here is some info, no "real" solution to the base problem, but here is what we do know:

OS: Ubuntu 18.04 64 Bit (x64)

Hardware: Virtual Box Virtual Machine (VM) with 16GB RAM provided and 6 cores of a Ryzen 2700x

Description of the problem:

Crashes after a random amount of time without any clear cause, there is no specific action that causes the problem

What we know:

We ended up getting down to the base of the problem, which has nothing to do with 7 Days to Die itself, it is a problem with Unity running on Linux within a Virtual Machine, there is no clear fix, we have tried a bunch of solutions and settings, different extended features and vitalization settings.

Nothing seems to work.

We are still unsure if this is specific to just 64 bit guest linux systems, have not tested 32 bit guest. We found some information that my suggest that could be an issue, it says that having AMD Secure Virtual Machine on in the BIOS/UEFI could cause the problem, but that setting needs to be on to allow 64 bit guests machines to even boot up and run. So that is not a real solution. (Here is the link)

So with that "maybe" a 32 bit host might work as long at the

System > Acceleration > Hardware Virtualization > Enable VT-x/AMD-V is unchecked

Since this us a problem with Unity itself, there is nothing this Development team can do to fix the problem as far as I am aware, I plan to get a hold of Unity Support to see what they know about the bug/issue

We did:

Every time it crashed, we did reset everything and deleted the instance, there was NO MODS running, everything was the base game, we tried many worlds of many sizes, crashes with every one. New Seeds / New RWG's, the works...

Moving to Dedicated Hardware is not a real solution, this does not solve the actual problem, this may not be an option for some people, the requirements to run this game are more than what some people can afford in Dedicated Hardware.

That being said, if you are reading this and you have the option to move to Dedicated Hardware, you can do that to avoid this error.

Another option is to run a Windows Guest VM system, if you are doing this on a paid VPS this will be more expensive than having a Linux VPS.

So until I either get a solution from the Unity team or they fix it, there may be nothing you can do to actually fix running Unity on a 64bit Linux Guest VM.

Here is the actual error you will get:

Code: 
Receiving unhandled NULL exception
#0  0x007f682dd38140 in funlockfile
#1  0x007f682dd386d0 in mono_array_element_size
#2  0x007f682dd386e0 in mono_array_new_specific
#3  0x007f682dd38710 in (Unknown)
#4  0x007f682dd38720 in (Unknown)
#5  0x007f682dd38730 in (Unknown)
#6  0x007f682dd38740 in (Unknown)
#7  0x007f682dd38750 in mono_set_defaults
#8  0x007f682dd38c10 in mono_runtime_invoke
#9  0x007f682dd38c40 in mono_runtime_invoke_array
#10 0x007f682dd38d00 in mono_store_remote_field_new
#11 0x007f682dd38d60 in mono_thread_create
#12 0x007f682dd38da0 in mono_thread_create
#13 0x007f682dd38de0 in mono_thread_create
#14 0x007f682dd38e30 in mono_pthread_key_for_tls
#15 0x007f682dd38e50 in GC_end_blocking
#16 0x007f682dd38f00 in start_thread
#17 0x007f682dd38fc0 in clone
It will always start with:

Code: 
Receiving unhandled NULL exception
#0  0x007f682dd38140 in funlockfile
And follow with some error in the "mono" file

I am sorry there is not an actual solution.

I did have to move this to dedicated hardware, but I am not going to stop to find an "actual" solution to this not just a workaround.

If you think you have any good helpful information, feel free to either post on the 7DTD Official Discord and tag me (Smugger | STAG) or post on here.

There is absolutely no settings in 7DTD you can do to solve this problem, as again, it is a problem with Unity on 64bit Linux Guest VM Systems.

 
Last edited by a moderator:
I have a virtual server rented with a 64 bit debian jessie (yes, fairly old but still getting patched). 7d2d works, there are no random crashes.

Did you try the game on a different and maybe older distribution? After all it could be just a bug in newer mono versions or spme other library.

 
I have a virtual server rented with a 64 bit debian jessie (yes, fairly old but still getting patched). 7d2d works, there are no random crashes.
Did you try the game on a different and maybe older distribution? After all it could be just a bug in newer mono versions or spme other library.
Click to expand...
To be honest, I could not narrow it down to figure out if this is a general Virtualization issue or specific to just Virtual Box, I was thinking about trying VMWare and also ESXi (Free VMWare OS) to see if it still happened, I don't have VMWare Hypervisor, so that is out of the question for me to test on, which is what most Large VPS Data Centers Use and the Companies you rent these from. So I can't tell you if something is different about those.

I will say though, most of the forums we have found in searches, it does not seem to be specific to any certain Virtualization Software. Who knows? :shrug:

I just wanted this out there in case anyone else runs across this error.

Do you have any specifics about your VPS service you used?

I have not looked into if Mono or Unity Version were different between versions of 7DTD. That is a good idea!

 
my vps is rented from netcup, a german VPS that is using kvm (!)

With "different and maybe older distribution" I was talking about the linux distribution, not the virtualization solution. Debian jessie is from 2015, quite a different beast than ubuntu 18.04.

 
my vps is rented from netcup, a german VPS that is using kvm (!)
With "different and maybe older distribution" I was talking about the linux distribution, not the virtualization solution. Debian jessie is from 2015, quite a different beast than ubuntu 18.04.
Click to expand...
Good point, I will try that some time also

Sorry, I must have read that wrong, I did not see you said "distro" Sorry Thanks

 
Status
Not open for further replies.
Back
Top