Good question. I assume you mean that they are more damaging overall, to the universe as a whole, rather than just to the person imprisoned.
Two arguments are typically offered to justify punitive sentences: retribution and deterrence. Personally, I can see no moral justification for retribution. It just seems to be a product of a primitive eye-for-an-eye kind of gut reaction that humans could and should transcend. Forgiveness, with a view making the world a better place for everyone, seems to me a worthier ideal. Deterrence is a different kettle of fish. As matters now stand many humans will commit crimes if not deterred. Given the deterrent value of prison sentences, it is hard to judge whether, on balance, they are damaging or beneficial to society as a whole. In cases where the deterrent value is high, that can provide a very good and morally acceptable reason to imprison offenders on occasion, even if it does the offender far more harm than good – in my opinion, anyway. But if the deterrent value is very low or non-existent, and the damaging effects are high, justification is lacking.