I had the same problem Manxmog, in the end I replaced the mirrors (complete with arms) with round ones (from the factory) and that solved it. There may be cheaper fixes of course, but I planned on switching to round mirrors anyway.
The round mirrors fitted to our car by the factory last year have exactly the same problem. I have a set if anyone wants them. I replaced them with a set from Phoenix Design
http://www.morganhoods.com/id31.html .
There are other aftermarket round mirrors that do work. There was a recent thread on the subject
here