My thoughts also. If you loose the phone or its stolen you are stuffed.
Not as stuffed as losing a physical key as it happens. Any mobile phone, with your Tesla password, will open the car. Tesla themselves can remotely unlock the car and let you drive it - and, of course, there is a key card.
If you lose a physical key, I would say you are in a more difficult and expensive situation.
But I do understand that this technology is not for everyone. However it is possible to have a foot in both camps - as indeed I do and intend to continue doing so.