Exiling Fairgrounds Patrol is part of the activation cost of its ability. Once you've activated it, no player will be able to stop you by removing it from your graveyard.