Garbage Collection causes indefinite loading screen.

TehBlizzy

Refugee
Version
1.4 Stable to 2.3 b9
Platform
Windows
I recently installed the game to try out the new changes and I attempted to start my own singleplayer world. Unfortunately, it reached a point where it was "initializing world..." and never finished. I even let it sit for 30 minutes with no changes.

I opened up the console and the last messages to be printed was "Started thread GenerateChunks" and the only thing I could do was close the game.

In my attempts to try and fix this issue, I have tried every branch available on Steam and the only one that had any difference was 1.4 where it would just crash instead of indefinitely loading.
I tried hosting a dedicated server, connecting to other servers, and tried random gen and the other presets and they all had the same problem.
I used the launcher tool to delete all of the data, I deleted the registry data, and I deleted the appdata folder for the game every time, and nothing changed.
I tried verifying the game on Steam multiple times, and nothing changed.
I even updated my Windows 11 from 10.0.22631 to 10.0.26100 with no changes.

Nothing was making any difference, so I decided to dig into the game code myself. Using the "GenerateChunks" string, I found the thread function ChunkProviderGenerateWorld.GenerateChunksThread and added logging to discover m_RegionFileManager was always null thus it never continued. I did some more logging, and tracked down that the Navezgane world uses ChunkProviderGenerateWorldFromRaw. I added some logging into the Init function and saw that the function calcWorldFileCrcs was never called but getFilenameDTM was. The issue was in GCUtils.UnloadAndCollectCo(). When I added logging into that function, the GCUtils.PostUnload task never completes. I actually *removed* that task from the function, and now my game loads perfectly fine.

I had to actually remove the Garbage Collection code from the Assembly-CSharp.dll just to play the game.

I've included a log file from before I made any changes to the game code, a log file from when I added logging to the game code, and then a log file after I had removed the section of code. The last two log files have quite a bit of extra logging from when I was tracking down the issue.

The 2nd log: https://pastebin.com/L6QRK9Wq
The 3rd log: https://pastebin.com/duTTwc4c


The section of code I modified in GCUtils:
C#:
public static IEnumerator UnloadAndCollectCo()
    {
        Log.Out("UnloadAndCollectCo START");
        Interlocked.Increment(ref GCUtils.m_working);
        try
        {
            Log.Out("UnloadAndCollectCo calling PreUnload");
            Task preUnload = Task.Run(new Action(GCUtils.PreUnload));
            yield return Resources.UnloadUnusedAssets();
            while (!preUnload.IsCompleted)
            {
                yield return null;
            }
            Log.Out("UnloadAndCollectCo PreUnload completed.");
            Log.Out("UnloadAndCollectCo calling postUnload");
            // v---- I DELETED THIS SECTION OF CODE ------v
            Task postUnload = Task.Run(new Action(GCUtils.PostUnload));
            while (!postUnload.IsCompleted)
            {
                yield return null;
            }
            // ^--- I DELETED THIS SECTION OF CODE -------^
            Log.Out("UnloadAndCollectCo postUnload completed.");
            // v--- I ALSO DELETED REFERENCED TO postUnload
            preUnload = null;
            postUnload = null;
            preUnload = null;
            postUnload = null;
            preUnload = null;
            postUnload = null;
            preUnload = null;
            postUnload = null;
            preUnload = null;
            postUnload = null;
        }
        finally
        {
            Log.Out("UnloadAndCollectCo FINISH");
            Interlocked.Decrement(ref GCUtils.m_working);
        }
        yield break;
        yield break;
    }

Because this issue generates no errors at all, I don't actually know what object in memory is causing the GC to indefinitely hang.
 
Reproduction Steps
1. Install game
2. Launch game
3. Attempt to join a server, create a new or start an existing singleplayer save of any kind
4. Watch console until it prints "Started thread GenerateChunks"
5. Observe that the game *never* finishes loading
6. Close game
7. Open Assembly-CSharp.dll in DNSpy and find the method GCUtils.UnloadAndCollectCO
8. Modify the method and remove the PostUnload task, then save the modified dll.
9. Run the game, and try to connect to a server or load a singeplayer save
10. The game finishes loading and is playable
Link to Logs
https://pastebin.com/iUmQTSF8
Link to Screenshot/Video
https://i.imgur.com/9d4VOTw.png
Fun.
Because this issue generates no errors at all, I don't actually know what object in memory is causing the GC to indefinitely hang.

Looking at GCUtils.PostUnload,
Code:
    public static void PostUnload()
    {
        GC.WaitForPendingFinalizers();
        GCUtils.FullCollect();
    }

it can be some finalizer, not garbage collecting itself.

If you feel like it, you can download process explorer, configure MS public symbol servers, and examine game threads stacks.
There is a reasonable chance that some thread will be stuck inside some third party driver .dll

Another approach is to download PerfView, and toggle .NET Loader checkbox in advanced collection options. It will cause Mono to emit symbol information about jitted methods, and you can examine what game threads are doing with resolved method names in stack. Collect for a few seconds, then PerfViewData.etl.zip -> CPU Stacks -> 7days process -> CallTree tab
 
Last edited:
As far as I know Unity itself handles all of the garbage collection, not sure how you're the only one I've ever heard of that has this issue. What storage drive do you have? Any third party antiviruses, or "game boosters"?
 
Fun.


Looking at GCUtils.PostUnload,
Code:
    public static void PostUnload()
    {
        GC.WaitForPendingFinalizers();
        GCUtils.FullCollect();
    }

it can be some finalizer, not garbage collecting itself.

If you feel like it, you can download process explorer, configure MS public symbol servers, and examine game threads stacks.
There is a reasonable chance that some thread will be stuck inside some third party driver .dll

Another approach is to download PerfView, and toggle .NET Loader checkbox in advanced collection options. It will cause Mono to emit symbol information about jitted methods, and you can examine what game threads are doing with resolved method names in stack. Collect for a few seconds, then PerfViewData.etl.zip -> CPU Stacks -> 7days process -> CallTree tab
I downloaded those tools and gave it a shot at trying to track down the correct thread or even a thread that seemed connected to the postUnload task and it's kind of hard to navigate those programs without any prior experience using them. I tried for about an hour before I gave up :P


As far as I know Unity itself handles all of the garbage collection, not sure how you're the only one I've ever heard of that has this issue. What storage drive do you have? Any third party antiviruses, or "game boosters"?

I have an i5-13600k, 32GB DDR5 RAM, RTX 3060Ti. My Windows C:\ drive is a Lexar 128GB SSD, my game was running on a Samsung 970 EVO 500GB on D:\, and I have a 1TB HDD for other programs. I made sure my C:\ drive and D:\ drive had plenty of space and also have no hardware/software issues.
I have no third-party AV, and I especially don't use any "game boosting" software. I run a pretty minimal system as I dislike having programs that I'm not actively using even running.

It was my understanding that Unity and C# can manage the GC entirely on their own which made me question why it was necessary for the GCUtils functions to even exist. Maybe older systems need a helping hand with any amount of memory saving possible? I don't know. I just know that removing that functionality myself from the Assembly-CSharp.dll fixed it for my system on every branch available on Steam.
 
I am just wondering how you're the only one affected by this ;), something has to be different. Do you have exceptions in windows security for the install/save folders?
 
I downloaded those tools and gave it a shot at trying to track down the correct thread or even a thread that seemed connected to the postUnload task and it's kind of hard to navigate those programs without any prior experience using them. I tried for about an hour before I gave up :P




I have an i5-13600k, 32GB DDR5 RAM, RTX 3060Ti. My Windows C:\ drive is a Lexar 128GB SSD, my game was running on a Samsung 970 EVO 500GB on D:\, and I have a 1TB HDD for other programs. I made sure my C:\ drive and D:\ drive had plenty of space and also have no hardware/software issues.
I have no third-party AV, and I especially don't use any "game boosting" software. I run a pretty minimal system as I dislike having programs that I'm not actively using even running.

It was my understanding that Unity and C# can manage the GC entirely on their own which made me question why it was necessary for the GCUtils functions to even exist. Maybe older systems need a helping hand with any amount of memory saving possible? I don't know. I just know that removing that functionality myself from the Assembly-CSharp.dll fixed it for my system on every branch available on Steam.
You manually run the garbage collector to keep peak memory use down as it can creep up causing the heap to grow which never shrinks back down. Important for limited memory systems like consoles.
 
It's been a few months and I somehow managed to forget to check this thread for any replies. My bad. I've reinstalled the game today just to discover that this issue still persists for me.

Here is the log file file: https://pastebin.com/v9117BCB

I don't have any special permissions / security settings on my system for any of the install or save folders, but I did find this in the Event Viewer:
Code:
Fault bucket 2275711467653789772, type 5
Event Name: RADAR_PRE_LEAK_64
Response: Not available
Cab Id: 0

Problem signature:
P1: 7DaysToDie.exe
P2: 2022.3.62.28864
P3: 10.0.26100.2.0.0
P4: 
P5: 
P6: 
P7: 
P8: 
P9: 
P10: 

Attached files:
\\?\C:\Users\Blizzie\AppData\Local\Temp\RDRC22D.tmp\empty.txt
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.1459a0f3-2725-4f61-a303-43bcd33160d3.tmp.WERInternalMetadata.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.2e982d5f-372d-4374-ae6e-58487fa13580.tmp.csv
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.3adbeff0-1308-4d25-941a-d6b3fcbd2da1.tmp.txt
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER.7af6b779-2acb-4f05-9765-e9e6e46d91dc.tmp.xml

These files may be available here:
NULL

Analysis symbol: 
Rechecking for solution: 0
Report Id: 5c1e8886-0bc7-498a-9611-001f0c607555
Report Status: 268435456
Hashed bucket: 5014a2648f3f0becef94f385c8c9784c
Cab Guid: 0

I promise to pay attention to the thread this time so I can run any form of tests or respond to questions about my setup since it would be nice to casually install the game and play whenever without editing the Assembly dll :P
 
I don't have any special permissions / security settings on my system for any of the install or save folders
Although it may not be related, you should have the game and save folders excluded with your antivirus.

The thread is so old that your logs are gone, so I can't see if I see anything, but I'm sure the TFP people who responded looked and if there was anything there, they'd have noticed. Most (not all) issues related to loading the game are due to either a corrupted save or having mods installed that are causing problems. A clean install without loading any existing saves or worlds, and where you have completely emptied out the Mods folders in both AppData and in the game directory would be a good start, as would verifying files. Additionally, if you've ever used a mod loader, you may have mods installed to another location than those two and you'd need to find and remove those as well.
 
Back
Top