Mostly that's because the 2 hour refresh doesn't start unless you open the Tavern menu.
Basically, per time you see 1:59:00 or before on the countdown, you only refresh once.
Doesn't mean you don't get spirit for the off chance that one refresh contains a spirit giving option.