I start with digging out a hole to hide out initially underground. Put a floor/ceiling above level with the ground around the base area so that if any wandering hordes come by they walk right over. At the corners of the excavated area, I place my initial support pillars, and then determine...
My experience digging out veins of gravel is that they can make some interesting cavern complexes.
All they would need to do to create better caves is generate some of the gravel patches with some of the gravel removed.
I voted for water, because boats.