The French stance, which I believe is correct, is to ban the covering of the face (not the use of head scarfs or other hair coverings) . I am not allowed to go around with my face covered or, for that matter, to carry a traditional arm, say a sword, on the grounds of my religious or cultural beliefs so why, in the name of political correctness, should we accept it from other residents who are in effect often guests.
I confess I've never been able to see why folk shouldn't be able to wear what they like in public as long as it doesn't involve, as you say, carrying an offensive weapon.