I think the need for downvotes will become more crucial the more Lemmy grows, but waiting to make this change later will annoy people more (since we all know how people usually hate change).

The inclusion of downvotes have always been a double-edged sword. Remove them and the bad gets mixed in with the good. Add them and you introduce brigading and hive-minds. However, I still think the addition of downvotes outweigh the negatives. Not having a downvote means the only way for "bad" things to "move down" is for other posts/comments to be voted up so that might just force people to vote-up mediocre posts just to subdue the bad stuff - it would feel like a sub-par workaround. And worse than that, Lemmy will be littered with some really "bad" stuff that will undoubtedly have some upvotes. All you need to do is look at YouTube (without the "return dislikes" extension) and you'll see how bad it can get.