Your view about the possibility that people perceive things radically differently is an old philosophical puzzle. The so-called "inverted spectrum" hypothesis is the most common variant. Assuming that color space is organized symmetrically, so that one can switch primary colors like red and green, or blue and yellow, while maintaining all the similarity relations among them, it seems someone could see red wherever others see green and yet there be no difference in their behavior. As you point out, they would learn to call red things "red", even though they looked green to them. Perhaps there could be inversions of other properties as well. So is this really possible?
One way of approaching the issue is to distinguish between what we might call the "representational content" of a perceptual experience and its "qualitative character". The representational content is what the perceptual experience is "saying" about the world around you. So if you see a red apple, your visual experience is "telling you" that there is a red, round object, an apple, at a certain location. The qualitative character is the subjective "feel" of the experience, what it's like for you to be seeing the apple. Given this distinction, we can ask whether the possibility of a behaviorally undetectable inversion - a difference in experiences that can't be detected by what the people say or how they act - applies to both the representational content and to the qualitative character, or just one (or, indeed, neither).
There are good reasons for thinking that an inversion with respect to what the experience is representing is not possible. The point is that on a very plausible theory of what determines what the experience is representing, a theory that grounds the representational content in the facts about what conditions in the world reliably produce the experience, the sort of inversion you are imagining won't occur. Since both people are experiencing what is reliably produced by the same wordly conditions, their experiences will automatically count as representing the same conditions. However, when it comes to the more elusive feature of qualitative character, it does seem possible that an inversion occur. Some philosophers try to argue that such inversions aren't possible, but I, for one, have never been convinced by their arguments.