Don't ask me, ask the US Judiciary.
It'd stand to reason though that if your government legally oppressed your grandfather through segregation, substantially diminishing the quality of his life, culture and community, which proceeds to affect his people's, and by relation, your status in society - then that same government decides some decades later that it was actually in error, you'd probably be a little "peeved" and expect something in return more than some words.
Racism doesn't end at a single person or with a single generation. I think you're missing the fact that a lot of the people looking for compensation have on some level or another, found themselves affected ("harmed") personally by it. In fact, I'd say that that's the reason they're looking for compensation. If they hadn't been harmed by it, they probably wouldn't be looking for anything in the first place.