There are a couple of ways we might go here. Thinking about the sky, it's a pretty good bet that most people with normal color vision (roughly, people who can pass a color-blindness test like this one ) will say that the sky is blue and will agree that a good many other things are blue, even if they've never seen them before. And so while there is a certain amount of arbitrariness and convention in just where we draw the lines between colors and while the names themselves are certainly matters of convention, there are facts about human physiology here as well. Thus, one way to think about colors is in terms of the responses of "normal" individuals in "normal" conditions (where both of those "normals" are not altogether easy to pin down.)
Another way is to think about the physical characteristics of the things we apply the color words to. For example: light with frequency around 450 nanometers is blue light. If the light reflected from a body has this frequency, we call the body blue. But things are a bit complicated here. We can't identify colors with frequencies in any simple way because non-monochromatic light (light that's a mixture of frequencies) will typically appear to be a definite color.
So color is not a simple matter. There are objective elements having to do with our visual systems, objective elements having to do with the properties of physical objects and the way they interact with light, and there are conventions about where certain boundaries get drawn. A really good account would be even more complicated. For example: context makes a difference to what a color looks like. Here's one example and here's another, related one. What we can say pretty confidently is that it's not just a matter of convention, though the full story calls for a good deal more expertise than I have.