Since I don't quite know how skills and books intertwine I can not tell you an exact response so I will make a two parter:
Possibility 1:
you can buy perkbooktalents with skillpoints:
the books are not needed and 500xp is allright (still less than most zombies) since finding them is just a bonus.
Possibility 2:
skills and "book bonuses" are seperate and you can only unlock those perks (like 10% more damage with fireaxes against wood) by finding them.
Now the books are essential. While their bonus is not that great, it is a character improvement. Especially huge things like beeing able to craft lockpicks. If I am missing one book to finally be able to craft them, this is HUGE and my frustration of finding the other books over and over and only getting 50xp for it will be insane. "YES BOOK! Oh god just another waste... YES BO-... f***... FINALLY A BOO... FK THIS ♥♥♥♥! IMMA JUST CHEAT THAT STUPID PERK!!!!"
But if there is like a 2000xp reward for finding a book twice I can go: "well... its not the right one again, but at least it is worth the xp"
I am sorry that I do not know this but it has me confused for a while now, with perks like fireaxe doing more damage in the burnt biome beeing more or less useless and I would never buy it with perks, but others like beeing able to craft your own lockpick beeing such a hardrequirement for any stealthbuild, that it would be close to the first things I would buy.
If you could elaborate what you can buy with perkpoints, that would be great (or show that in your next video)