One more argument in addition to eldarerathis' answer: Downvotes are not intended to punish the poster, but to cause him to improve his post. If he does, you can take back your downvote (and get your rep back).
If it was a real bad post, and receives enough downvotes (or attention from our mods/high-reppers [not "high rappers" :)]), the post gets deleted. In such a case, all "down-voters" get their "lost rep" back as well – provided the deletion happens in a given time-frame. I don't remember the exact numbers here; but if the post only gets deleted a year later, the rep stays lost.
Remains the question: Does it make sense to down-vote a post of a user just having 1 rep? He won't see (or feel) that. You only can lose rep if you've got some (which as down-voter, you have). But see previous paragraph: Given enough down-votes, a special user called "Community" (which even has mod status) automatically deletes such a post – and once more, you regain your "lost rep". Thus it makes pretty much sense for e.g. spam and offending posts, to speed up their "demise".