We added attributes that govern your perk choices. If your attribute gets damaged you lose the perks if you no longer meet the requirements. This allowed for us to make more interesting debuffs and death penalties than only having health and stamina to deal with.
Long story short, we had skills sandwiched between attributes and perks and all it did was confuse everyone and add clutter to the UI. So Attributes governed skills which governed perks? It was so confusing Everyone hated it.
I had considered making attributes gain through actions but at the end of the day its not that much fun to grind skills in order to buy perks. When it reaches 100 you cannot gain any skill doing what you love, like in Skyrim it suffered that problem. Once your skill is maxed you are leveling skills you hate just to raise your level. They later added the ledgendary reset thing, but I'm not a fan of it so much. You lose all your perks and started over at 15 skill. So suddenly you were a horrible swordsman after being the best of the best.
With this system you can play how you want to play and buy what you want. No spam crafting junk you don't need, no jumping off of bridges to raise your athletics (Oblivion), no sneaking in the corner of a bar to level your sneak (Oblivion/skyrim)... that stuff is bs. I can argue that skill grinding is even more unrealistic than what we have now because nobody learns how to chop off heads by punching grass. Think of it as reading a book and learning a new skill to go try out.
In real life you fundamentally walk through a new action in your brain before your body does it. Think of the perk as deciding to walk down a path and then doing it.
At the end of the day I'm playing the game how I want to play it. Crafting exactly what I need, specializing in the perks I want. Why have to jump 13,588 times to reach agility level 5? Is it fun bunny hopping? Nope. Sorry but this is the best system IMO.