Both, Both dear boy!
The additional weight is no more than half a tank of fuel and the anti roll bar weighs a hell of a lot more than audio, dynamat and a load of other stuff!
The heavier it is the slower it is, even though you will have a roll bar (a must) any other weight saving is good. You will find most will remove carpets, spare wheels and anything else that comes off with ease. If you have full bumpers, time to buy overriders. Also more weight means more kinetic energy to stop and get round corners. Hill climbers look for every kilo because in 40-60 second dash it makes a difference.