Typically you want to know who is connecting to what server via what service and log these connections. If something is off, an alert can be generated. If ssh isn't served on a standardized port, logging and alerting becomes more complicated - albeit not impossible.
There is more housekeeping to do. In case of a handoff, things like this need to be documented. If all services work on their default port, there is no need for documenting them.
In the case of compromise, it becomes very hard to identify how a machine got compromised.
Yes, a lot of people do not do a full port scan. But those are not the people exploiting risky vulnerabilities. Security by obscurity reduces your risk, but only to a certain extent. Having a proper patch management or firewall management in place reduces your risk a lot more.
A lot of owls do get killed by humans, despite their camouflage.