I usually go for two days, Friday to have a good look around and then Sunday to watch the race finals. I really enjoy the Friday because you get a great feel for the atmosphere, there used to be a lot fewer people but that doesn't seem to be the case any more.
