Let's start with "test taking is difficult." There's a difference between "test taking is difficult for me" and "test taking is difficult." If what I mean is just that I personally find test taking difficult, then simply saying "test taking is difficult" is a recipe for being misunderstood.
Now of course if I say something is hard, I'm usually not saying it's hard for literally everyone. I mean, roughly, that it's hard for most people. That's fine, but it can be true—whether or not the task is hard for me personally.
This gives us our first point: if a statement doesn't seem just to be about the speaker, then don't read it that way unless there's a good reason to.
Now let's turn to moral statements. When people say that punching a child in the face is wrong, they don't just mean something about themselves. In fact, most people, including most philosophers, would reject that reading out of hand. If I say "it's wrong to punch a child in the face" and you say "what you really mean is that you have negative feelings about punching kids," I'd say no; that's not what I mean. I mean that it's wrong, and it would still be wrong even if I happened to enjoy it.
Now maybe there's no real difference between right and wrong. But at this stage, we're talking about what statements mean. Roughly, that's a matter of what people who use the statements us them to mean. And I think we're on safe ground in saying that when most people say something is wrong, they don't just mean that they don't like it.
You might say that there's nothing else meaningful that they could mean. But the argument you give doesn't show this. Your argument is that there's a causal story that accounts for my saying that it's wrong to punch a child—some story about my upbringing and my influences and my neurons or whatnot. The problem is that even granting this, it doesn't show what you need to show. Compare: I say that 537 times 24 is 12,888. There's certainly a story about my upbringing, education, and the workings of my brain that explains why I say this. But the statement that 537 times 24 equals 12,888 isn't a statement about me, and in fact it's a true statement. My upbringing and the workings of my brain made me into the kind of person who's a reasonably reliable source of information about basic arithmetic.
This brings us to our second point: just because there's a causal story explaining why someone says, that doesn't make their statement subjective, and doesn't give us any reason to think it's not true. On the contrary, the causal story might help us understand why the person is reliable about this sort of thing.
Finally, back to moral claims. When people say that punching children is wrong, they don't just mean that they have a certain feeling about punching kids. If Mary says punching kids is wrong, there's certainly some complicated story about Mary's upbringing, influences, and brain that explains why she says this. But for any statement there's always some such story, and that doesn't usually give us a reason to doubt that what the person says is correct. (It could, of course; maybe the person is on drugs. But the fact that some causal stories give us reasons to doubt what someone says doesn't mean that all of them do.) Mary might be (is, I'd say!) right about punching kids. And because of her empathy, sensitivity, intelligence and wide experience, Mary might even be an unusually reliable source of moral advice.
This isn't to say that there aren't any reasons to doubt moral realism. But those reasons will have to be quite different from the ones you've proposed.