It annoys me because answering truthfully often leads to judgment, unwanted advice, etc. None of which I appreciate. It seems to depend a lot on where you live though. Where I grew up, no one would blink an eye at hearing we homeschool (admitting to unschooling probably wouldn't have gone over well), and saying you're a stay-at-home mom when asked what you do for a living wouldn't have been an issue either. Where I live now, homeschooling is really viewed in a negative light and I get responses like "Well my kids are in *real* school so they'll learn, go to college and get a GOOD (high-paying) job!" Responses to being a stay-at-home parent were similarly negative when I lived in another area (I was a SAHM then, now I'm the sole wage earner and work from home...)
So it's not the question but the reaction to the answer that bugs me. It's surprising how ballsy people can be, especially when they're ignorant of the subject at hand. I really, really hate conflict (and even debate, when it's with strangers). I have it easy in a way because DD is gifted and above grade level in all subjects (we check her progress yearly, which she enjoys). If she were unschooled, "behind" in any subject and I admitted to it, I'd probably incite a riot.