I don't tell other Linux users that I use linux because it almost always leads to a bunch of questions that I don't know the answer to. So let me assure you, that awkwardness is on both sides of that conversation you're describing.
even the linux-from-scratch snobs are usually pretty decent (really it's only when it's their first time out of the server room in too-long a time). I usually try to switch it up to something like "have you tried playing The Battle for Wesnoth. (it's a turn based fantasy strategy game that's... uhm. surprisingly good.)