It probably breaks mostly along line of if you played it at release. As only the 2nd game, it's not like they'd established a formula that felt quintessentially Zelda at that point. I remember it feeling like a decent game that was just doing something different from the 1st one.
It does depend what you consider "the whole series."
Does Tingle RPG count? Does Link's Crossbow Training?
Other questions that probably wouldn't matter for best or worst game in the series, but would for a complete ranking might be whether remakes/remasters count as separate games or not and if not, do we judge on the basis of the remaster or the original? (Either way, all the remasters are inferior to the originals. Maybe that's my most radical, non-left-right position. That and Breath of the Wild was better than Tears of the Kingdom.)