I have taken it off a couple of times, normally a fairly quick exercise, and I do prefer the look without with the more exposed spare wheel. Trouble is you are more or less compelled to leave the side screens on as the cars with spares don't have a slot for the sidescreens.
My sidescreens will store flat on the rear shelf