- Costs money: the physical sensors (a dozen of them), wiring it up, assembling it, maintain inventory, code it, etc.
- Time spent on maintaining, improving software stack for the non-vision sensors as well as efforts needed to fuse the data with vision, takes away from focusing on vision alone. It also holds back vision in relevant areas.
- Existing non-vision sensors used by Tesla are orders of magnitude lower fidelity than vision. It has historically (as the case with radar) led to vision essentially having to overriding radar because vision just performed much better (see AI day 2021).
My take:
As with any new tech, it likely sucks at the start (think HDD and SSDs, and how a mechanical thing with lots of moving parts was way more reliable than SSDs at the start). However, by essentially moving past the local maxima, you get to innovate better, faster in the future.
In case of ultrasonic sensors, they are for low speed cases anyway and most people are fine without them. Majority of fatalities and injuries happen at higher speeds.