There is a difference between choosing a soy product for some protein replacement, and living of soy as if it's the only source of protein.
Adding to that. The phytoestrogen seems to compete and reduce the effect of estrogen in some situations. It is possible that excessive soy would have the opposite effect on a woman. So reducing the harshness of puberty.
And also. My grandmother was powerfully endowed too. On a balanced diet. So I don't find this anecdotal evidence enough.
That doesn't mean it's worth considering its impact. But I think the focus should still be on proper nutrition regardless.