Been having a similar experience.
Current base is in the wasteland, in Zillions Tower (a 0 skull POI). I removed the entire first floor of the tower so zombies can't get in. Still, screamers and wandering hoards would show up. Due to the lack of a path to my location, I suppose, they took to...