I find it androgynous and will use it for anyone, but if anyone told me they were uncomfortable with me referring to them that way, I would stop, as with anything. It’s simply decent to refer to people as they prefer.
I think sexual activity adds implied gender relationships that shift how 'dude' is interpreted. While 'dude' can feel androgynous in casual contexts, in a phrase like 'fucked a bunch of dudes,' the sexual framing amplifies its masculine connotations. This shows how cultural norms around sexuality and gender can override attempts at neutrality.