My series 2 used to have a similar problem if left for a while.

Like you I never used the handbrake but it was always my conclusion that it was binding.

I never sorted out the cause, but thankfully the car never failed to stop.

Slowly going green