(Disclaimer - I have no opinions on either side in this issue).
If it matters I’m with the OP on this. I’m Romanian, I’m “white”, my mother is, for lack of a better word, “brown-ish” (between “black” and “white”, that is), my brother has the same skin “color” as my mum. My brother has been the victim of racist remarks over the years of which I was not, I suspect because of our slightly different skin “color”.
Nevertheless, I find the essentialist policy the US has when it comes to “races” quite baffling. For example I don’t know how me and my brother would potentially be “labeled” were we to move to the States. Like I said, technically we have different skin “colors” but, after all, we’re brothers.
Don’t get me wrong, I wish you were 100% right on this, i.e. that US racial things should only be discussed by those directly involved with the same happening in the Romanian case (from where I’m from), but unfortunately that is not the case.
The original sin of the US was the Atlantic slave trade. It so happens that almost of the people swept up into that slave trade were dark skinned. And, because that slave trade was so monumentally huge in terms of how many people it brought to the US, a very large fraction of dark skinned people in the US today and all throughout history either were slaves or are primarily descended from them.
In order to morally justify the enslavement of millions of people, white slave-owners manufactured a justification that people of certain races are physically, mentally, and morally different from whites. They created a (factually false) narrative that skin tone didn't just mean skin tone but also implied all sorts of other things about the person.
That narrative is profoundly embedded in all aspects of US politics, culture, and institutions. It's been there since before the three-fifths compromise which is directly in the US Constitution. It's everywhere.
So it's not just about skin tone. It's about generations of Americans growing up in a culture that says people with a certain skin tone are less human, and then building social institutions predicated on that as fact.
Culture and institutions change very slowly (especially when many participating in them actively benefit from their current state) and it takes a lot of work to root something like this out.