If I had known the reaction of the fairer sex to a Morgan when I was younger I would have bought one years ago. Having said that I am happy I did not and so ended up married and happy now.
Life is nothing without the backbone to follow up on your desires and accept a little compromise as a result. I drove a Porsche 968 Club Sport as a company car for 3 years. Nuts - but so great.
Do it.
Worst case you find it is not practical but you will have had great fun doing it. Will you have a problem selling it on? Will you loose your shirt on it like you would almost every other car?