Let's look at that notion deeper. Are you able to claim that consenting adults engaging in sexual acts together in an uncontrolled manner (i.e. no marriage contract) is not hurting others? What about consenting teens with each other? I think you'll see that widespread STD's, abortions, teen pregnancies, single parent (especially single mother) families, and the emotion baggage that comes with these easy sexual relationships disagrees with you.

Secondly, what or who is your authority that people doing what they want as long as it doesn't hurt anyone is a valid generalization to make for all actions?