I just don't think it's fair to assume people with them are racist. I think that's hypocritical. They shouldn't be flying at government buildings though. Everyone just needs to light one up and stop getting offended so easily.
It is a lesson in empathy. The origin and initial meaning of the flag don't really matter in this context. What matters is that to a significant portion of people the flag does indeed represent not just "racism" the word, but — and to some actual memories — of oppression, violence and murder. When the Little Rock Nine started going to school with the white kids the flag was there in the crowd as protestors hurled insults and curses. When George Wallace said "segregation now, segregation tomorrow, segregation forever," the flag was there in the hands of those cheering him on. To this day when klansmen walk down city streets, there you'll see the flag.
So when you tell me that it's really just about Southern Pride or small-government values, or whatever else it doesn't really carry water, because to a significant number of people, for a very real and legitimate reason, it means a hell of a lot more than that.
Putting yourself in someone else's shoes when you make a choice is important, especially when those shoes have been a lot less comfortable than your own. Once you do that it makes it difficult to see whatever it is you're debating over so trivially.