Only time I have trouble in banish with people attacking me is revenges from other realms usually after their realms boss is over.