Right now if you're based in SV you can hire the best talent in the world, pay them a fortune and build something great. Investors are reluctant to touch anyone outside of that bubble.
With lack of housing (and no more construction virtually) alongside better tools for working remote, and more managers experienced in working remote, I can see remote teams leading the charge towards the end of this decade.
It takes time.
It is obviously difficult to have a pro-remote company if you hire executives or management that values "face time" above all else, and prefer phone to email/chat, this is why likely it's a lot simpler to start a company fully remote than to change an already existing one.
Given the advantages of cost of labor (you can be a lot more flexible in compensation terms if you don't have to compete with SV salaries), pool of talent and retention advantages (if you have a great remote environment I think it'd be a lot easier to have very low turnover) I am really surprised more companies are not being formed doing this.
But then again as you said a lot of VCs/investors won't touch companies not based in SV so that definitely can be a contributing factor in the low % of fully remote outfits.