On my factory mirror was a small plastic ring attached in a groove at the end of the mirror stalk!
I had to pull a little bit less soft!

The ring is "cutted" so he can be placed in the groove on the stalk. This ring looks a little bit like a tiny piston ring, made of plastic.
This ring should eliminate the gap between the stalk and the mirror base/ holder.
This cut in the ring makes the ring also flexible, so the ring fixes sometimes the stalk at the mirror base! The problem is to get the stalk out of the base and not the base/ holder from the door....