Ive done a few fort setups across the updates and found taking points into brawler in the first few levels really to set me back too much personally. I push 2 into fort and 2 into regen for the starter quest points cause lets be real, when you hide the first few nights, that free regen is baller. Brawler is rough with zombie rage, you dont have the range of something like sledge/spear/bow or even club to give you that extra second to react, and you cant take hits like you need to to apply the fast attack speed that makes brawler great.
Then 2 points into pain tolerance, and 2 points into heavy armor off exp from doing some initial trader quests and figure out the resources for scrap armor as I go (had issues with that in my current game till I found a bear

). Only after getting the 3 defensive perks at level 2 do I think brawler starts to shine. Take very little damage, basically never get stunned, regen what little damage you do take, and start knocking off those heavy armor penalties before you can find the armor mods, which you can add 2 of to the level 3 armor you can make when you do find them. Then you can just go in and unleash without worrying so much and use that fast attack speed to decapitate everything
This is the first run I haven't died in the first few days and its def because I avoided knuckles for the first 2 hours