Spacer is used on ball bearings, but none is required on taper roller bearings.If your spacer is removed, the rotor will move inboard and the caliper will no longer be in the right place. Is one axle longer than the other? I assume that the spacer is inboard of the inner bearing?