Who is “we”?
My point is that I don’t think that your premise of “collective agreement” is true for “open source” or “free software”. I don’t agree with it, and I know a bunch of other people that don’t do either.
Language is cultural and context-specific. Not everyone has to agree, but if you talk to software people about "open source" and don't mean what everybody else means, you're just going to confuse and annoy people instead of communicating.
Language is not set in stone either, and the perception of what terms mean may change over time, even within one and the same cultural context. That’s why we are having debates and discussions. The world of computer people is no exception of this phenomenon – the etymology of the word “computer” is a literal example for that.