Note that down votes are a key part of what drives a number of automatic scripts that clean up things.
* People who waste too much of the communities time (asking poor questions that get closed and down voted ) get question banned - driven strongly impart by down votes.
* Closed questions that have a negative score and no answers with a positive score get deleted after a bit over a week. (If people complain about how much crud there is, one should look at how much crud gets automatically deleted).
* A down voted question helps signal to people reading to not waste their time reading it because its not useful. When you've got 8000 questions/day, someone saying "don't bother with this" can save you a bit of time reading it.
For smaller communities where a person who wastes your time can get addressed through individual moderator attention. Where there are 8k questions per day asked by a similar number of people, you can't have every moderator address each poor question and time wasting troll. It is for this that simple down votes can help drive the automated system when there is enough signal that the person is wasting everyone's time without involving moderators.
Update: So, because of how the system was designed, it isn't something that one can easily go back to and change (and remove down votes). Its part of the software and the culture of the group and the two are inseparable - the technological and social part of the community is tightly interwoven. Further reading on this at http://www.shirky.com/writings/group_enemy.html