On a whitelisted server you probably do not need one, because you know your players. On any public PvP or survival server you do.
Paper
- Grim (free, open source): detects movement and combat hacks well, low false-positive rate. The first choice.
- Vulcan (paid, SpigotMC premium): very complete, good support.
- Matrix, Spartan: older, still used.
Install via the Plugins tab (Vulcan: upload the jar yourself), restart, then look at plugins/<AntiCheat>/config.yml.
Fabric / Forge
Few anti-cheats exist for modded servers because mods legitimately change movement. Paper's anti-cheats do not run there. Rely on whitelisting and logging, or use Vanilla client-side mod restrictions with a Fabric mod like AntiXray + Polymer checks.
Configure, do not just install
- Start in log-only mode for a week and read what it flags. Elytra, riptide tridents, slime blocks and laggy players trigger checks.
- Raise thresholds for
FlyandSpeedbefore enabling automatic kicks. - Never auto-ban on a single detection; kick after several, ban only after a human looked.
What no anti-cheat catches
Chat-based scams, account sharing, coordinated griefing. Rules, moderators and logging cover those.