'Impartiality' is in no way a simple moral concept. Yet one thing most moral philosophers would agree upon is that impartiality cannot be plausibly equated with treating everyone the same. Rather, impartiality seems to have both an exclusionary and and an inclusionary aspect. Here's what I have in mind. Being impartial means not allowing a certain fact or consideration about people to influence a choice or a policy. A judge who routinely convicts defendants with mustaches while routinely acquitting the clean shaven makes her decisions on the basis of a fact or consideration — the state of a person's facial hair — that ought not influence her decisions. Here she fails to be impartial because she does not exclude from her decision making a factor she ought to exclude. Conversely, suppose a judge issues her rulings without regard to whether the evidence provided indicates a defendant's guilt. Here she fails to be impartial because she does not include a factor she ought to include in order to be impartial.
Taken together, these two factors indicate that impartiality is linked to equality, not in the sense that impartiality requires everyone be treated exactly alike, but instead requires that everyone be treated equally on the basis of all and only the considerations that are morally relevant to a decision or policy.
The issues thus raised for your examples of special accommodations, etc., for senior citizens, the disabled, and pregnant women is whether the characteristics that distinguish these individuals from others are morally relevant to how we should distribute goods such as places in lines, etc. In other words, such accommodations would violate impartiality if the characteristics on the basis of which the accommodations are being made are not morally relevant to how the accommodations ought to be distributed.
Unfortunately, a full treatment of these issues would take more space than we have here. But I hope you can imagine how those defending such accommodations would try to defend them against the charge that they fail to be impartial. With regard to the disabled, senior citizens, and pregnant women, they might argue that impartiality requires that certain goods be distributed in ways that do not disproportionately burden anyone — and perhaps having to stand in long lines is simply harder for those groups than for others. They would, then, accept your claim that our actions or policies "should not favor one person over the other simply because they are two different individuals." Yet they would also claim that such accommodations don't rest merely on the fact that the individuals who benefit from there are different from others in some way. They instead rest on the fact that these individuals have some morally relevant characteristic that justifies treating them differently.
I don't offer this proposal with the expectation that this settles the issue for good. Rather, the point here is that impartiality does not mean treating everyone the same -- it means treating everyone the same with regard to certain facts about them but differently with regard to other facts about them -- and it is possible to offer arguments to the effect that treating people with different characteristics differently need not violate the demand that our choices and policies be impartial.