My first language is English and some accents/dialects are very difficult. Certain Indian speakers, Scottish speakers, Newfies, and West Virginians (which has a lot of Scots and Irish roots) can take me some time to acclimate to.
I'm a native speaker and have absolutely no issue whatsoever with Australian and British accents, but people with a heavy Indian accent still sound like they're not speaking English to me.
It doesn't help that Indian English often still uses a lot of colonial terms, like Capsicum instead of bell pepper. That being said most Indians in the US will adjust to the local vocabulary pretty quickly.