Some of the better stakes are broadcast on World Chat. When this happens, especially with 100 or 1000 gems, people switch realms to come find you in your arena, take your stakes, and then go back to their own realm. That's the only explanation I can offer, otherwise, there's a serious glitch going on.