[Baertram]

If everything is unequipped on your main/back bar you cannot change back to it. The bar switching is disabled then.
But you might be good using an addon to unequip/Re equip items via hot keys, like Alpha gear is an addon to do so.