I don't think you know what the word "inherently" means.
The American flag itself was created under the same "particular circumstances" which were racist. The founding fathers owned slaves. There are mounds of laws, court decisions, etc, that go back to the very creation of the country, where they said "Freedom for everyone -- oh, but not for you black people, you stay in the under-class."
We, as a nation, have acknowledged that background, and are able to condemn those racist circumstances while also acknowledging that the founding fathers were way ahead of their time in terms of freedom, equality, and liberty. Never mind the fact that a flag is a piece of cloth and can't be "inherently" any kind of message, I'll go even further and say that just because the nation that founded the flag had slavery at the time, doesn't make the ideals it stands for racist.
People from the Southern states uphold the Confederate flag as a symbol of a lot of things besides racism. Some assholes use it to promote that, but if you think that the majority of southern people don't associate the flag with a multitude of ideals OTHER THAN racism, then you are the close-minded idiot, not them.
Hell, look at the British flag. The British Empire is almost single-handedly responsible for the importation of black slaves onto the American continent to begin with. If you really believed that a flag that was used to spread racism at one point in history must ALWAYS carry racist connotations, then you should be protesting the British flag like people protest the Swastika, as it was responsible for a faaaaar bigger cost of human life.
But people don't do that, because it's nonsense, and complaining about the Confederate flag is just a guise for complaining about Republican politics, so no one is fooled.
For the record, I could be convinced either way that the flag should stay or should go, because I don't know if I agree about flying the flags of other defeated former countries on government property. But it has nothing to do with the silly racism angle.