I get lots of xp for building my base in flagstone and plenty more upgrading it to bricks, concrete and finally r concrete. I build next door to trader, so I sell all my things right on the way home, grab another quest, etc etc. I don't have to "move" later game. While you are moving and setting up a fort I've already got my farm up and I'm power leveling at that point. There is no right or wrong, its just a preference. I do dig quests and all that clay I use for flagstone and eventually fill my forges with it because you will be crafting 1000's of bullets. So I build my base mostly at night, early game its too risky to loot at night, you need good armor and good weapons and ample ammo to loot at night.
Something about hearing the trader horn saying "Trader Joel's is now open" makes me feel less alone too