OSI/FSF decided to use generic words as label to promote their specific ideas. The ambiguity of that unspecific wording choice is on them, not on the rest of the world.
Just because "gravy boat" has the word boat in it does not mean it is actually a real boat. "Whisky on the rocks" has ice in it, not actual rocks.
Free Software and Open Source Software have widely agreed upon meanings, and just because you think intuitively it would make more sense for "whisky on the rocks" to be served over actual rocks doesn't mean you're better at understanding english words than the rest of us.
Who is “we”?
My point is that I don’t think that your premise of “collective agreement” is true for “open source” or “free software”. I don’t agree with it, and I know a bunch of other people that don’t do either.
Language is cultural and context-specific. Not everyone has to agree, but if you talk to software people about "open source" and don't mean what everybody else means, you're just going to confuse and annoy people instead of communicating.
Language is not set in stone either, and the perception of what terms mean may change over time, even within one and the same cultural context. That’s why we are having debates and discussions. The world of computer people is no exception of this phenomenon – the etymology of the word “computer” is a literal example for that.