I like the idea. Animals eat crops all the time. This may put much more emphasis on fences and gates, whether its added vanilla or it becomes a mod. The zombies could care less about your garden, but the wildlife are as hungry as you are. In terms of meat, you could just balance it out so killing the invading wild boar is bittersweet at best.
With alot of the zombie lore, you always have your passive traps or chain link perimiter fence up for the zeds. Its expected that the zeds do block damage in the game, but I like how they just kinda pile up against the fences in the shows and just kinda lean and linger. I think this would be alot more intimidating in-game if they just piled up outside of your base and lingered without block damage. Just knowing theyre piling up can be just as scary as if small groups are doing mindless block damage constantly.
This would encourage the player to lure the wandering horde away from the base, rather than try and cut through them all and risk triggering them to bring the fence down. If you choose to leave them, when it gets dark, they go feral and tear the fence down, possibly attacking your base if they detect you.