Performance is always an issue for any complicated piece of software. You will always work to improve performance with optimizations. And unless your software works on a 286 computer with 1 MB of RAM, you are going to have to consider performance on different levels of hardware. Anything made for a modern computer will most likely not run on a 20 year old computer because of the performance (among other things). It has nothing to do with bad design. There are games that are designed specifically to work on really powerful computers. They perform horribly on less powerful computers. That isn't bad design. That is hardware limitation.
Now, if you want to say that bad design can impact performance, that's true. But performance is not "only an issue if your game is badly designed." That is false.
You also cannot compare a fully destructible voxel game's performance to that of a game with a fully static world. A fully static world requires significantly fewer resources than this game does, especially for the CPU.
And you clearly don't represent the majority of players if you want zombies to be unable to break into your base and don't want to fight a lot of zombies on your own or want the game to be so easy that the zombies are nothing more than a minor annoyance that you can simply run away from and being so slow that they aren't a danger.