The Driving Standards Agency sums it up very well on the danger of driving without shoes.
Would not relish undertaking an emergency stop while driving without appropriate footware. It could be argued wearing said footware is among the most important safety prerequisites in driving.