Who the are these people talking in the stalls. I be worried someone would try and kick my ass. I've seen people go ape shit if you disturb them during a shit in public.
I don't know about being mad, but if a coworker tries to talk to me in the restroom regardless of specifics they immediately go on my "perhaps has body in basement" list. Usually it's the Cxx people...who are sociopaths. So the math works out.