Perhaps examining the root meaning of "privilege" could help us unpack this question a bit. The term means "private-law" - or to put it another way - the laws that operate for most persons do not apply to some particular thing or person or group. For example, one might have "privileged information" which means it is not available to most others.
When I teach a college class of students from a wide variety of socio-economic and cultural backgrounds, I am keenly aware of the disparity of their life circumstances. But the very fact that they are in college is a form of "privilege" insofar as the vast majority of human beings (across cultures and time) have not had access to higher education - and advantages that follow from this privilege may continue to accrue throughout life, regardless of race, gender or class.
When we look at various and exceptional gains that appear to be generated by groups, we see real clusters of advantages in terms of race or whatever other category/social marker you choose. The term White privilege suggests that the question of color does not occur to white folks - precisely because they are the "normative" human in some societies; others are outliers identified with markers such as "race." "White privilege" - even if dirt poor and uneducated - exists because the social laws that apply to them differ than from the "others". Think how different is the mundane experience of shopping or driving and how one is followed or pulled over based on perceptions of race.
The prosperity of persons of European descent is not neutral data or statistical fact, but must be understood within a whole context of a dominant culture that skews advantage in a particular way, unevenly, to the group deemed to be the norm. White males have been at the top of the pyramid with white women reaping the rewards of privilege by attaching to an elite male (i.e. a "good catch!").
A great question - I hope others will respond to give a more fulsome reply.