You're losing more max stamina than before, since it's based on a percentage, yeah. But you don't have less max stamina than you did when you were at base agility and the same amount of food.
You might want to invest in the metabolism perk that decreases the rate at which you lose hunger and...