I guess you guys can set up a system where you contact the person you are taking gems from with both PM and world chat to let them know, so if they are online and not afk they can ask you to power down afterwards.