I'd suggest that we need to keep three things separate: 1) whether the word is offensive, 2) whether offense was intended, and 3) whether the hearer was offended. All eight possibilities are real. To take the most relevant, a word might be offensive, and yet the person using it might not have intended to offend and the hearer might not be offended.
For example: suppose someone who's not a native speaker uses a deeply racist term to refer to someone. The speaker is not at all a racist and would be deeply mortified if she knew how the word is normally used. She intended no offense. But that's because she didn't know that the word is an offensive word.
The person she was speaking to, meanwhile, is a racist. The speaker doesn't know that; she's just met him. He's not offended, but only because of is racism. On the contrary: he thinks he's met a kindred spirit.
There's no mystery here. The word is offensive because of its history, its usual meaning, and the way people typically respond to it. None of that changes if the speaker is unaware of this or the hearer, for whatever reason, doesn't have the usual reaction.
You're right, of course, that if I come to learn that a speaker didn't realize the full connotations of his words, it might be unreasonable to hold on to my offense. But if I tell the speaker "You might want to know: that word is actually a very offensive one," I could be right even if in light of the full situation I'm not offended.