OP is not bothered by the downvotes, they are bothered that they are being used as nothing more than a "I personally disagree" or "I don't like your opinion" button and that the vast majority downvoters are too cowardly to put forward their own counter opinion.
Any comment that is making an honest contribution to the discussion shouldn't be getting downvotes. When comments are downvoted for no legitimate reason, the website becomes at risk of being an echo chamber.
A feature that is consistently misused is a useless feature. Either get used to them being used to express disagreement or admit that they're a failure and stop trying to make them work.