Isn't racist to find the word "nigger" racist? As in when it's merely said

Read another response by Allen Stairs
Read another response about Language, Race
Isn't racist to find the word "nigger" racist? As in when it's merely said around you and not directed towards you. When someone calls another an "asshole," there isn't a normally a particular ethnicity that comes to mind -- yet The "N" Word is automatically associated with people of African descent. This all seems to fit into the ideology of race making racism possible.