Downvotes serve to help signal to the person that there is something wrong - and it does it noticeably. Flags are things that go to various levels of moderation (close flags go to the close vote queue, moderator flags go to the elected moderators, low quality flags (only doable on negative score posts) go to the low quality review queue, and not an answer flags go to elected moderators).

Downvotes mean that its not helpful, or that its not correct, or that it insufficiently answers the question. Flags and downvotes are often orthogonal to each other - one doesn't mean the other is needed.

With a good question, viewpoints shouldn't be an issue at all - there's a question, there's an answer.

An individual who systematically down vote another individual will have the votes reversed by the system.