Without mods, the game is a headache for PvP now. Howlune the on Sleepless PvP server is trying hard, give that a look. NAPVP is trash, because Grandpa Minion doesn't ban certain cheaters, so cheating has been rampant there. That keeps people away for sure.
But the number one thing, HANDS DOWN that is keeping people from PvP now is that the game has too many bugs and exploits to make it fair. You can't just play the game as intended because too many people know the exploits and hacks (duplicating items, traveling/shooting/looking through the mesh, using consumables to basically be invincible, etc. etc.), not to mention how unbalanced it is.
BUT it could be one of the better PvP games if they cared enough to fix it.