I believe that racism is an idea that is not limited by being black. It has been tried in different forms with different cultures including whites. But being black is a fundamental aspect of racism not because we are black but because it provides a truth from the perspective of the person that is trying to exercise it.
If they see a black person, they are able to focus their racism because it applies to all black people. If racism applies to white people, how do they know which one of the white people are deserving of racism and which ones are not. There is too much debate and that will cause conflict within the white community.
So the color of the skin is not what determines racism, its the economics of the effectiveness of applying racism to all blacks that makes it apply specifically to black people.
For the racial economist, its easier in their minds to add up the numbers to fundamentallly apply racism.