A woman was in full niqab at my local Tube station today. That I respect her right to dress as she likes, is for most libertarians all that there is to say on that subject. In truth, of course, there is much more. A wise friend of mine said recently that libertarians are wrong to treat such issues as cut and dried. We give the impression that we are uncaring, cold and more unlike other people than we really are.

This post of mine was a good example. My friend rebuked me for saying that "I don't care" if people want to enter into polygamous/polyandrous marriages, when I would actually be very concerned for any family member or friend embarking on one. He has a point. As witness the conventional lives that most of us lead, libertarians generally have a similar range of ethical scruples to everyone else. In a sense, we just have an extra scruple about interfering in the lives of others.

I would never advocate interfering with that young lady's right to dress as she does. That doesn't mean I don't have any other response. In truth my reaction was the same I would have to seeing her paraded in public on a leash. However much she and they might deny it, I feel it's degrading that her menfolk claim the right for her to be seen only by them. I feel her garb is the sign and symbol of misogynistic subjection.

Other libertarians might have different responses. We are not an army of liberty-minded robots. We are diverse, mostly rather ordinary humans with a range of views.

Why then do I feel so uncomfortable in expressing such a personal view? I am not afraid of being accused of islamophobia. As used in public discourse in Britain, I regard it as a bogus concept designed to close down discussion. Rather like racism, sexism and homophobia, it is usually no more than an incantation; a magic spell to shut opponents up.

Nor do I recognise the lady's right not to be offended. Someone is offended by any point of view. I am very offended by those who advocate enslaving their fellow-men on a time share basis; making them work for the state for months before permitting them to earn for their families. Yet I don't claim the right to suppress their foul views. There is no free speech without offence - real, imagined or bogus. Sticks and stones may break our bones, but if we want to live in a free society we can't ever allow mere words to hurt us.

My wise friend is right. If we don't talk about the many concerns we share with non-libertarians, we make it harder to win them over. We sound like cold, hard people lacking concern for our fellow men. It's not enough to say the lady in the niqab is entitled to wear it. We also need to say that, like our fellow citizens, some of us at least feel sorry for her and disgusted by the misogyny her garb represents.

In modern Britain, libertarians inevitably spend most of our time arguing against the increasing intrusion of the state into private lives. We need also to make clear that we only do so as a matter of ethical principle. It's not because we approve of whatever "evil" the state pretends it is trying to cure. We would oppose a hijab ban à la française in England for example, but that doesn't mean any or all of us are happy for the women concerned. Just because we claim no right to interfere doesn't mean we lack a moral response.

Perhaps the confusion arises from the fact that, in a radically statist society like ours, where government accepts no boundaries on its right to interfere, moral criticism is almost always the precursor to an attack on liberty. We used to separate the immoral (to be avoided in oneself and discouraged in others on ethical grounds) and the illegal (to be suppressed for the protection of others from genuine harm). That distinction has somehow been lost.

The loss is no accident, in my view. To advance their cause, statists have - in a cynical agitprop exercise - sought out "oppressed" groups and offered them the state's protection. They have given the right to those favoured groups (selected for the sympathy they evoke in a population of generally decent people) not to be offended and not to have hatred expressed against them. In doing so they have chilled free expression so effectively that it's hard not to imagine that was their objective. And they have caused a clamour from other groups to be added to the list of the elect.

The British media demonised the Polish and Ukrainian peoples as racist bigots in advance of UEFA 2012, for example. I am familiar with both countries and don't believe racism is more prevalent there than here. I simply think we have suppressed its expression here and in doing so may even have increased its incidence. Does that really make anyone's life better? Does it increase the chance of different communities growing together; learning to understand each others' concerns and to build trust? I think not.

The lady on the platform today may, as most human communication is non-verbal, have detected my unease. She may speculate as to its causes but she will never know the truth. Unless it's possible to talk openly to each other, how can we progress? How can we explain to those who are taught to assume we are hostile by our racist, sexist, homophobic and islamophobic natures, that we stand by the old English principle of "handsome is as handsome does?" That we really just want people to stop calling for us to be controlled like dangerous dogs and for all of us - citizen familes old and new - to sign up to the standards of tolerance and mutual respect that we think should define our society?

The key question is, as always, cui bono? I don't think it's the young lady in the niqab, who might well enjoy having me as a neighbour if not taught to fear me. I don't think it's the black and brown football fans who missed out on two wonderful countries. The only beneficiaries of this moral panic agitprop are those who seek ever more control over our lives. Every time we edit our speech for fear of PC "offence" we are losing the battle for our freedom.