Also, if we had more types of cones then each type would have to be more spread out over the retina, so we would lose resolution.

In some sense the eyes and ears are at opposite ends of a spectrum. The eyes give you lots of information about the direction light is coming from, but barely any information about its spectrum (only three types of cone), whereas the ears give you lots of information about sound's frequency spectrum, but barely any information about its direction (only two ears).

Come to think of it, this could be phrased as a tradeoff between resolution in position and resolution in frequency. It's the uncertainty principle!