My series 2 used to have a similar problem if left for a while.
Like you I never used the handbrake but it was always my conclusion that it was binding.
I never sorted out the cause, but thankfully the car never failed to stop.
It tends to be worse if the discs are hot when the car is stopped. (Reason why on a track day, never use the handbrake when the car is at rest after running on the track- use wheel chocks- and NEVER put the handbrake on)
G
