To me, from the perspective of submissions and comments, it appears that the ability to down-vote is relegated to those that the existing community who approves.
If one's world-view, sentiment, and such does not agree with the existing team - reflected in their posts/comments, that individual will unlikely to make it into the approved group. One has to be echoing the chamber's sentiments to reach the karma level required. It is a self reinforcing mechanism.
But, HN is a private entity that can decide whatever they want to; so there is that.
This is not unusual, and have been shown to work out even back in the BBS era.