Just to confuse you I've got almost the opposite advice to give !
I've been a few times and combine it with a visit to my brother who lives about 30 minutes away.
The city centre is vibrant and interesting with a few stalls, various classic cars and a good atmosphere. The Racing is difficult to view given the nature of the circuit. The paddock is worth viewing if only for the numerous tatty Bugattis on show. Costs have escalated over the years I've been attending and it's not as informal as it was.
The Rally des Ramparts follows a different route each year and the lunchtime stop is usually worth attending. You'll see hundreds of classic cars parked up and have time to chat with the owners.
I didn't enter, but followed the Rally cars part of the way (in a classic car) and it was thoroughly enjoyable. Every village had spectators along the route and the atmosphere was enthusiastic and friendly with much waving and cheering.
I don't think the Rally is value for money, and you can follow it anyway with the flexibility to divert if you get bored or see somewhere you'd rather stop and explore.
