Yes, your reputation decreases when you downvote an answer, but only by a small amount. Downvotes on questions, however, have no cost associated with them (though they did previously). Click on the arrow beside the What is reputation? topic in the site's FAQ for details on which actions give (and take away) reputation.
This is done to make people think seriously about downvoting anything.
Any positive action you do will get you at least five times as many reputation as you lost on that downvote, so it's really not worth worrying about too much.
See this StackExchange blog post The Value of Downvoting, or, How Hacker News Gets It Wrong for more info on why, especially this section:
The net effect is that you have to feel very strongly about
something to downvote it. Downvotes are serious business, and not to
be cast lightly. We designed our system around that maxim.
In circumstances where an answer is totally unrelated, or is spam, or really shouldn't be there for any reason you can click the "Flag" link to let a moderator know what you think. If a moderator agrees with you and deletes the post, then you may get your rep points refunded. More on flagging here Improved Flagging, including the side-effects of over-using the Flag option.