Indeed. I learned color theory from "The Measurement of Appearance" by Richards Hunter and Harold, and one of the strengths of that work is that goes well beyond simple color into things like gloss, opalescence, etc. Highly recommended.
Also Bruce MacEvoy's in-progress book "Color Experience"[1] looks like it will be a fairly definitive treatment of the topic. It is also rooted in physical colorants (the author is an extremely serious watercolorist).