The limitation stems from XBOX Series S' hardware. I doubt it could handle hosting more than 2 players and even if it could, it might not be comfortably. I think it may already struggles hosting the 2 its allowed to.
XBOX Series X is able to host 4 players because its hardware is a little better and can handle the overhead.