If you are within 3 levels of his level then he can revenge after 24 hours.
You can attack UP any number of levels. Down from your level is limited to 3.
Better gear, runes and heroes on your antagonist explain your losses.

Gorgon