Is Gentoo more-or-less pointless for the average Linux user?

More specifically, Portage. I know use flags and "optimization" are all the hype, but really, would the average user even see a benefit from customizing all their use flags? Especially a benefit that compensates for the constant compilation?

I installed it once to help grow my e-peen, but immediately switched back to Arch after watching my system compile.

Those who daily drive it, do compilation and use flags annoy you, and do you see any real benefit?