I was stuck on wooden palace while beating the last guy.
So I refreshed my browser and went straight to the stone palace without picking a reward and lost my buff.