The Panhard rod made a noticeable difference to the handling of my 4/4 Sport Competition when I had it. There were no downsides. The feeling of the body moving sideways relative to the rear axle disappeared and the car felt much more "planted" on the road, even in normal road use. Just do it - you won't be disappointed.
Don.
