I think that when I swapped my mirrors I took the plastic cap off to get the stem out. This needs doing carefully to avoid putting pressure on the mirror mount and door, but isn't too difficult.
When putting the new mirror stem back in ensure the cap refits so the cutaway is in the right place for the stem screw to go through.
Once I figured out what to do it took about 5 minutes per side.
