Are you upgrading to hope bringers, champions etc? Cos you lose 3 levels per upgrade.