I don't have an issue with it. It's already an anonymous platform, so why do downvotes need to anonymized from your anonymous account? If someone is very concerned about that, they could just make a separate account just to use to down vote and never comment.
If anything maybe it's unfair that server admins can see the list of who downvoted and the users can't, so maybe it should be a feature for anyone to see who downvoted.