EDIT: It's not a perfect analogy but if you have a system that can be publicly accessed, just seeing that it's publicly available isn't a grey area to me that's all.
Conversely, we don't need a pedestrian ethicist to conceptualize looking at doors by asking us to think about walking around and observing houses innocently as being somehow congruent to portscanning. Analogies offer similarities, not equations.
Isn't the inverse of that true as well? If you don't want someone using a port, close it.
These days I hope people realize that the equivalent of "lock the front door, but leave the back door open" isn't viable.
Also, the door analogy is (as other commenters have pointed out) flawed. Should an open port be translated as a door you can knock at or ring the bell, a door you can open, or just a door you can see. I personally agree that "An open port is a door on your house" but others see it as one you've just knocked on, and others see it as one you've willfully tried to break into.
The informedness of the port owner will define how they interpret the port scan.
For example on AWS you can do so but need permission first.