The libertarian justification for restricting public nudity and public sex is complex. There are, I think, two issues here (setting aside the First Amendment issue, which I think is generally not an obstacle for banning public nudity and public sex, whether or not the public nudity and public sex is engaged in for expressive reasons — more on that later, maybe).
1. The Yuck Factor in Public Government-Run Spaces: When it comes to government-owned spaces, may the government forbid certain behavior because the majority finds it offensive? May a library, for instance, eject patrons who are too smelly?
On private property, libertarians would generally turn to the property owner’s right to control. Some property owners might well set up nudist colonies; most, reflecting their own preferences and those of the majority of their patrons, would ban nudity. But when certain spaces are owned by the government — even in a libertarian paradise most such spaces would be privatized, in the world in which most libertarians live, many spaces are government-owned — the question is whether the government can set up rules that try to maximize the aggregate enjoyment of those spaces.
There might be some constitutional or broader moral constraints on that, for instance if the “yuck” stems from the content of the message that someone is expressing, or from a patron’s race. But it’s not clear to me that “yuck,” whether it comes to nasty smells or to public sex, is a categorically illibertarian at least when it comes to behavior in government-run spaces (which I suspect are at issue in most public nudity / public sex prosecutions, though I realize that public nudity and public sex bans also extend to some private property).
2. Is It More Than Just “Yuck”? More importantly, might there be more than just “yuck” here? I’m not positive, but here’s one answer that I’ve heard and that I think can’t be easily dismissed:
Viewing nudity and especially sex does more than just make people say “yuck.” Rather, it has the capacity to create, at least as to many viewers, a substantial amount of sexual arousal. Sometimes people will pay good money to get that sort of arousal. But in many places, people don’t want such arousal, and find this involuntary arousal to be intrusive and troubling — not because the behavior is yucky, but because it plays with their hormones in a way that’s outside their conscious control.
Now we naturally tolerate a good deal of such arousal, and many of us probably welcome some modest amount of that arousal. Moreover, social and market norms tend to take care of most of the unwelcome sorts of arousal; people generally don’t wear bikinis to work, and most people who go places where there are bikini-dressed people either seek to see a lot of skin, or at least expect to see it.
But public sex, and to some extent public nudity, have, I think, a much greater effect on most of us than just bared skin (or than statues of nudes, or even, in many instances, pictures of nudes). The question is whether the law can shield us from unwanted arousal by coercing others not to engage in such behavior; I tend to think the answer is yes, but in any event I don’t think “it’s just people saying ‘yuck’ is much of an answer.”
Incidentally, if you do want to draw a First Amendment analogy here, it should probably be between public nudity/sex and publicly visible video screens that display nudity/sex — since restrictions on both may involve the same interest in preventing involuntary sexual arousal — rather than between public nudity/sex and flagburning, which involves a very different (and in my view much less legitimate) justification for restriction.