I'm having a problem with removing mods also, it appears you have to click at the bottom left of the mod icon to remove it. I also noticed a major lag issue when opening inventory up after first loading the game for about 10 minutes after I installed the ui, hud, and bigger bag modlets. There is also noticable lag when crafting large stacks of items like cobblestone even after playing for a few hours. It only happens in the inventory screen and when I close it everything goes back to normal. I did remove all the other mods I had installed and it still happens.
My system specs are a 6700k overclocked to 4.8GHz (EK custom loop water cooled), 16GB ram, EVGA 1080ti FTW3 and the install is on my NVMe 1TB Samsung EVO m.2 drive. So I don't think the lag could be from low system specs.