I'm expecting baby #2, another girl, and she will be our last - partially because we feel 2 is right for us, and partially because of my age (I'm 41.)
I get so many comments about how it's too bad we're not having a boy, and of course we'll HAVE to try again for a boy, but I was quite happy to hear we were having another girl. My dh was too. I'm sure if we were having a boy we'd be happy with that too, but we were both immediately happy when we heard she was a girl. Dh gets more of it than I do - people say, "Don't you want a boy to work on projects with you" etc. And he'll say that having a boy doesn't guarantee that, and there's no reason to assume a girl wouldn't like to work with tools.
Anyway, yes it's annoying, and worse so that people are saying it in front of your dds. They don't need to hear that.