Me too - for me because things like my floor are meant to be durable, and seeing it being destroyed so easily just by some spores flying around discomfits me.
Fungus can consume nearly anything organic, but it has to be damp. Even "dry rot" is only dry when you see it, it was once wet for the mycelium to spread through it.
I grew up in a wet climate where we feared mold and fungus, now I live in a dry one where we run humidifiers. You won't ever see mildew or black mold here without a constant moisture source.