They were on their way. If you would have stayed outside just a little longer they would have shown up...
Another way to test whether they are bee lining to YOU or just following your smell trail as Atomic pointed out is to throw a rock out a window and se if they abandon the poi to go chase the rock. If they do then you know they are not aware of you because if they were, they would ignore the rock.
I don’t see the problem with staggered zombie spawning outside while you are inside. The OP is complaining about something that only has a noticeable and annoying impact when the game is modded. It’s doubtful the devs are going to compensate for a mod.
When I first started playing around with this part of the xml I went crazy with 8x the normal zombie counts because why not? I noticed what the OP is talking about right away and it made it impossible to fully explore- especially POIs with basements. Those were 100% death traps.
Eventually, I settled on 2x with a shorter respawn rate because that gave more outdoor zombies without the constant visitors to POIs Inwas clearing. You still get some but it doesn’t bother me. Sometimes they get in and find their way to where I am but most of the time they are just banging on the walls in a nonaggro state.