Because nobody will write software for 300 unique hardware variations of a platform that have inconsistent capabilities. Consistency is one of the reasons why x86-64 with extensions like like SSE, AVX2 etc is popular.
They didn't choose 8051 there for experience, but because it was a tiny core with a decent IPC they could license for a small part, then focus on the main cores. I wouldn't be surprised if they eventually switch to riscv there too.
Also, these 8051 cores tend to be extremely diverse. I don't think I've come across cores from different manufacturers that were actually compatible for real code. They all seem to want to handle accessing 16/32 bit memory differently, have different interrupt details, etc.
I really don't know anything about this space, but you just said that 8051 is dominant because it has one dominant architecture since the 70s. It has hundreds of manufacturers making identical parts.
As you say, software matters. If the Software can't run because of hundreds of extensions that can't be checked for, then you're going to pick a target that works, no? So in fact the ISA matters most: which ISA has the most software? Which ISA means my software runs on the most devices?
And yet you couldn't help yourself...
but you just said that 8051 is dominant
I said absolutely no such thing.
If the Software can't run because of hundreds of extensions
The software in the x86 world runs because there aren't hundreds of mutually incompatible extensions. I think the last time there was a major completely incompatible x86 ISA divergence was AMD "3DNow" vs other SIMD extensions. AFAIK the rest were "processor X got feature Y later than competitor Z".
Which ISA means my software runs on the most devices?
Easiest question evah: x86.
SSE has inconsistencies like SSE4.x vs SSE4a. AVX is an even more mixed bag. There are some 19 AVX-512 extensions and ZERO chips support all of them.
The situation is so bad that AMD and Intel got together to make AVX10 to unify everything. That seemed great, but Intel now has AVX 10.1 and 10.2 in addition to the base set, so there we go again...
x86 is a massive battleground with tons of competing extensions like FMA3 vs FMA4 (why did FMA3 win???) and in cases where one of the competing variants didn't win, we get something like virtualization extensions being completely different between Intel and AMD. There's also the rash of security extensions that have gone through various support and dropped support (not to mention using some of this stuff for market segmentation and further fragmenting the ecosystem).
x86 is anything but consistent if you look into its history (or even it's present).
Intel committed to not slicing and dicing the AVX instruction set going forward. AVX 10.(n+1) will not remove instructions that were in AVX 10.n. Feature testing is also easier: a single feature bit and a single version number.
I bet Intel's management will find some new counterproductive way of segmenting the market but AVX-10 (and therefore AVX-512) should be safe now.
Who said they have to? One can select a RISC-V configuration for a baseline for a particular purpose. Desktop? Choose the one that's most powerful.
ARM is more popular than x86 and is less consistent than it.