Before I begin, let me issue a quick reminder: Not all college sports is big business. Some of it is, to be sure: Big-time college football, basketball, and the like. But college golf, tennis, swimming, and gymnastics don't generate much revenue, except perhaps at the most elite programs, and college sports don't generate much revenue at all at institutions like, say, MIT. So when I talk about college sports and "student athletes" below, I'm talking about only some college sports programs.
So, that said, I used to be a huge fan of college basketball. (I went to Duke. Go figure.) Now I hardly watch at all, and the reason you mention is perhaps the most significant. The rules governing (that is, prohibiting) the compensation of "student athletes" were put in place many years ago to protect the interests of such students. For example, there was concern that a student might decide to go to school X rather than school Y, not because school X would better serve that student's long-term interests---which probably have little to do with sports---but rather because school X is offering certain kinds of financial incentives. That made a certain amount of sense.
But that was a long time ago, when the term "student athlete" didn't need scare quotes. Now, as you say, colleges and universities make large sums of money from their sports programs, and their "student athletes" are essentially prohibited from receiving any compensation. Of course, the "student athletes" do receive scholarships and limited amounts of subsidy, but the real value of these forms of compensation are trivial compared to what the coaches, atheltic directors, and the like make. (Compare professional sports, where the players typically make more money than the coaches and front office staff.) That's all the more worrying when so few "student athletes" graduate, and it's even more worrying when the graduation rate for black "student athletes" is so far below that of white "student athletes" at most programs. (Boston Globe columnist Derrick Jackson publishes such statistics every year as regards football and basketball.) The simple reason for this is that "student athletes" are admitted to colleges and universities absolutely all the time who have, and are known to have, absolutely no chance whatsoever of receiving a degree. Some of them, indeed, cannot even read.
One might well reach the conclusion that "student athletes" are being exploited for four years and that the fruits of their labor are going to enrich the colleges and universities that claim to be protecting their interests.