I can't see any judge buying that, ownership implies control, unless you are a minority share owner (meaning percentage, not race).
The league (and when I say "the league" I mean the other owners he signed the contracts with) is not his employer. Like the owner of a small business, that owner is free to make racist decisions that carry liability that bring down his business. A more accurate analogy is that Adam Silver is actually Donald Sterling's employee, not vice versa.
The only power the league has over him is pursuant to what amounts to a series of contracts that he signed with the rest of the owners.
The league wrote up those contracts and had him sign them. However, there are many reasons why any two entities could write up a "contract" and then have it fail to be enforceable.
Yes, the NBA has very smart lawyers who undoubtedly wrote those contracts so as to make them as enforceable as they could, but at the end of the day they are writing them to favor the NBA and they are just guessing based on the closest precedents they have in court decisions. Since there has never been a case exactly like this before, it's new ground.
I see the argument the league can make. I'm just not sure if it can overcome stripping him of ownership.