Then why was I charged gems to switch realms when I'd never done it before?
Did you remain in the realm that the game originally place you in? A lot of people switch when they first join to be in the same realm as friends that also play and then forget that they did it.