Are you buying gems? It resets the pirates every time you do and if your re currently holding a map, it disappears.