Happens all the time for me. Got used to it and live guilt free. One time someone waved at me and they were old friends and I did not wave back. On that particular section of road one would be insane to take a hand off the wheel to wave. They were a bit miffed. I'm still alive. Enough said.
On a side note, just being a sight seer in that car has damn near killed me. She is a jealous mistress and brooks no attention from others...
I remain focused while driving.