Do people have intrinsic value?
Where the weightiness you experience comes from is a psychological, psychobiological, or anthropological question and therefore not a question that philosophers, as such, are competent to answer. Having said that, I'll speculate anyway! It wouldn't surprise me to learn that natural selection has favored a tendency in human beings to treat all, and only, human beings as belonging to a morally special category. But, of course, a tendency favored by natural selection might nevertheless be hard to defend with argument.
About the weightiness itself: Speaking as a philosopher, I'd urge us to distinguish between persons (i.e., people) and human beings. I regard person as a psychological category: members of the category possess distinctive (if broadly and vaguely defined) psychological traits and dispositions, such self-consciousness and rationality. I think those traits and dispositions make any person morally significant in a way in which any non-person isn't (even a non-person that's morally significant to some degree). I don't think this distinction gives persons the right to treat non-persons any which way they like -- I don't have the right to torture a cat just for fun even if I'm a person and the cat isn't -- but the distinction does carry some moral weight. By contrast, human being is a biological category -- a species -- and I can't see how species membership, all by itself, carries any moral weight. So I think it's possible to hold that people (i.e., persons) as such matter but that human beings as such don't. A member of our species might fail to be a person and therefore lack the moral significance that comes with personhood; a member of some non-human species might be a person and therefore have the moral significance that persons have.
For further reading, you might start with this SEP entry.