It's not clear to me which of two questions you're asking: (a) Is it always racist to use the word "niggardly"? (b) Can it be racist to use the word "niggardly"? I'd answer "no" to (a). It's not racist, and it's accurate, to describe Ebenezer Scrooge (before his conversion) as a niggardly character. But suppose someone uses "niggardly," perhaps mistakenly thinking that it's related to the N-word, in order to express racial hatred. I think that counts as a racist use of "niggardly," so I'd answer "yes" to (b).
How about sharing some of the literature from the social psychology research on stereotypes? This way you won't be arguing back and forth about "intentions" (conscious or unconscious) but instead giving them some robust research on social cognition that shows how racial (and gender and other social group) categories can bias thinking even in well-intentioned individuals. Ziva Kunda's book _Social Cognition_ is a good place to start, as is Virginia Valian's book _Why So Slow?_.