Garbage Collection causes indefinite loading screen.

[TehBlizzy]

Version

1.4 Stable to 2.3 b9

Platform

Windows

I recently installed the game to try out the new changes and I attempted to start my own singleplayer world. Unfortunately, it reached a point where it was "initializing world..." and never finished. I even let it sit for 30 minutes with no changes.

I opened up the console and the last messages to be printed was "Started thread GenerateChunks" and the only thing I could do was close the game.

In my attempts to try and fix this issue, I have tried every branch available on Steam and the only one that had any difference was 1.4 where it would just crash instead of indefinitely loading.
I tried hosting a dedicated server, connecting to other servers, and tried random gen and the other presets and they all had the same problem.
I used the launcher tool to delete all of the data, I deleted the registry data, and I deleted the appdata folder for the game every time, and nothing changed.
I tried verifying the game on Steam multiple times, and nothing changed.
I even updated my Windows 11 from 10.0.22631 to 10.0.26100 with no changes.

Nothing was making any difference, so I decided to dig into the game code myself. Using the "GenerateChunks" string, I found the thread function ChunkProviderGenerateWorld.GenerateChunksThread and added logging to discover m_RegionFileManager was always null thus it never continued. I did some more logging, and tracked down that the Navezgane world uses ChunkProviderGenerateWorldFromRaw. I added some logging into the Init function and saw that the function calcWorldFileCrcs was never called but getFilenameDTM was. The issue was in GCUtils.UnloadAndCollectCo(). When I added logging into that function, the GCUtils.PostUnload task never completes. I actually *removed* that task from the function, and now my game loads perfectly fine.

I had to actually remove the Garbage Collection code from the Assembly-CSharp.dll just to play the game.

I've included a log file from before I made any changes to the game code, a log file from when I added logging to the game code, and then a log file after I had removed the section of code. The last two log files have quite a bit of extra logging from when I was tracking down the issue.

The 2nd log: https://pastebin.com/L6QRK9Wq
The 3rd log: https://pastebin.com/duTTwc4c


The section of code I modified in GCUtils:

public static IEnumerator UnloadAndCollectCo()
    {
        Log.Out("UnloadAndCollectCo START");
        Interlocked.Increment(ref GCUtils.m_working);
        try
        {
            Log.Out("UnloadAndCollectCo calling PreUnload");
            Task preUnload = Task.Run(new Action(GCUtils.PreUnload));
            yield return Resources.UnloadUnusedAssets();
            while (!preUnload.IsCompleted)
            {
                yield return null;
            }
            Log.Out("UnloadAndCollectCo PreUnload completed.");
            Log.Out("UnloadAndCollectCo calling postUnload");
            // v---- I DELETED THIS SECTION OF CODE ------v
            Task postUnload = Task.Run(new Action(GCUtils.PostUnload));
            while (!postUnload.IsCompleted)
            {
                yield return null;
            }
            // ^--- I DELETED THIS SECTION OF CODE -------^
            Log.Out("UnloadAndCollectCo postUnload completed.");
            // v--- I ALSO DELETED REFERENCED TO postUnload
            preUnload = null;
            postUnload = null;
            preUnload = null;
            postUnload = null;
            preUnload = null;
            postUnload = null;
            preUnload = null;
            postUnload = null;
            preUnload = null;
            postUnload = null;
        }
        finally
        {
            Log.Out("UnloadAndCollectCo FINISH");
            Interlocked.Decrement(ref GCUtils.m_working);
        }
        yield break;
        yield break;
    }

Because this issue generates no errors at all, I don't actually know what object in memory is causing the GC to indefinitely hang.

 

[valeskas]

Fun.

Because this issue generates no errors at all, I don't actually know what object in memory is causing the GC to indefinitely hang.

Looking at GCUtils.PostUnload,

    public static void PostUnload()
    {
        GC.WaitForPendingFinalizers();
        GCUtils.FullCollect();
    }

it can be some finalizer, not garbage collecting itself.

If you feel like it, you can download process explorer, configure MS public symbol servers, and examine game threads stacks.
There is a reasonable chance that some thread will be stuck inside some third party driver .dll

Another approach is to download PerfView, and toggle .NET Loader checkbox in advanced collection options. It will cause Mono to emit symbol information about jitted methods, and you can examine what game threads are doing with resolved method names in stack. Collect for a few seconds, then PerfViewData.etl.zip -> CPU Stacks -> 7days process -> CallTree tab